SpringerLink
Log in

Integration of Fast-Evolving Data Sources Using a Deep Learning Approach

  • Conference paper
  • First Online:
Software Foundations for Data Interoperability and Large Scale Graph Data Analytics (SFDI 2020, LSGDA 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1281))

  • 411 Accesses

Abstract

Data scientists spent 80–90% of their efforts in data integration and there is still no end-to-end automatic integration and wrangling pipeline working for a large number of data sources. This work proposes a data integration system that transforms fast-evolving raw data sources to user desired tables. Based on a set of pre-trained models, a user only needs to specify the schema of the outcome feature vector as well as a few examples of rows, the system will automatically generate the outcome table from the raw data sources. The training process is automatically injected with provisioned schema evolution so that the model is resistant to data source changes. Our experiments show that the proposed approach is particularly effective for the integration of data with fast evolving schemas.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Thailand)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 42.79
Price includes VAT (Thailand)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 49.99
Price excludes VAT (Thailand)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Caltech covid-19 modeling. https://github.com/quantummind/caltech_covid_19_modeling

  2. Cdc covid-19 death forecasting models. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/forecasting-us.html

  3. Community mobility reports. https://www.google.com/covid19/mobility/

  4. Covid-19 data repository by the center for systems science and engineering (CSSE) at johns hopkins university. https://github.com/CSSEGISandData/COVID-19

  5. Harvard covid-19 data: county\(\_\)age and sex\(\_\)with\(\_\)ann. https://dataverse.harvard.edu

  6. Kaiser health news data repository. https://khn.org/news/as-coronavirus-spreads-widely-millions-of-older-americans-live-in-counties-with-no-icu-beds/#lookup

  7. Abadi, D., et al.: The seattle report on database research. ACM SIGMOD Rec. 48(4), 44–53 (2020)

    Article  Google Scholar 

  8. Bernstein, P.A.: Applying model management to classical meta data problems. In: CIDR, vol. 2003, pp. 209–220. Citeseer (2003)

    Google Scholar 

  9. Bernstein, P.A., Rahm, E.: Data warehouse scenarios for model management. In: Laender, A.H.F., Liddle, S.W., Storey, V.C. (eds.) ER 2000. LNCS, vol. 1920, pp. 1–15. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45393-8_1

    Chapter  Google Scholar 

  10. Cappuzzo, R., Papotti, P., Thirumuruganathan, S.: Creating embeddings of heterogeneous relational datasets for data integration tasks. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1335–1349 (2020)

    Google Scholar 

  11. Christen, P.: Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer Science & Business Media, Berlin (2012)

    Book  Google Scholar 

  12. Curino, C.A., Moon, H.J., Zaniolo, C.: Graceful database schema evolution: the prism workbench. Proc. VLDB Endowment 1(1), 761–772 (2008)

    Article  Google Scholar 

  13. DiScala, M., Abadi, D.J.: Automatic generation of normalized relational schemas from nested key-value data. In: Proceedings of the 2016 International Conference on Management of Data, pp. 295–310 (2016)

    Google Scholar 

  14. Doan, A., Halevy, A.Y.: Semantic integration research in the database community: a brief survey. AI Mag. 26(1), 83–83 (2005)

    Google Scholar 

  15. Ebraheem, M., Thirumuruganathan, S., Joty, S., Ouzzani, M., Tang, N.: Deeper-deep entity resolution. ar**v preprint ar**v:1710.00597 (2017)

  16. Fernandez, R.C., Abedjan, Z., Koko, F., Yuan, G., Madden, S., Stonebraker, M.: Aurum: a data discovery system. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 1001–1012. IEEE (2018)

    Google Scholar 

  17. Fernandez, R.C., et al.: See** semantics: linking datasets using word embeddings for data discovery. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 989–1000. IEEE (2018)

    Google Scholar 

  18. Gionis, A., et al.: Similarity search in high dimensions via hashing. Vldb 99, 518–529 (1999)

    Google Scholar 

  19. Gottlob, G., Senellart, P.: Schema map** discovery from data instances. J. ACM (JACM) 57(2), 1–37 (2010)

    Article  MathSciNet  Google Scholar 

  20. Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing, pp. 604–613 (1998)

    Google Scholar 

  21. Kasai, J., Qian, K., Gurajada, S., Li, Y., Popa, L.: Low-resource deep entity resolution with transfer and active learning. ar** using diverse noisy evidence. IEEE Trans. Knowl. Data Eng. 31(8), 1426–1439 (2018)

    Article  Google Scholar 

  22. Klein, M.C., Fensel, D.: Ontology versioning on the semantic web. In: SWWS, pp. 75–91 (2001)

    Google Scholar 

  23. Konda, P., et al.: Magellan: toward building entity matching management systems. Proc. VLDB Endowment 9(12), 1197–1208 (2016)

    Article  Google Scholar 

  24. Meduri, V.V., Popa, L., Sen, P., Sarwat, M.: A comprehensive benchmark framework for active learning methods in entity matching. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1133–1147 (2020)

    Google Scholar 

  25. Miller, R.J.: Open data integration. Proc. VLDB Endowment 11(12), 2130–2139 (2018)

    Article  Google Scholar 

  26. Miller, R.J., Haas, L.M., Hernández, M.A.: Schema map** as query discovery. VLDB 2000, 77–88 (2000)

    Google Scholar 

  27. Mior, M.J., Salem, K., Aboulnaga, A., Liu, R.: Nose: schema design for nosql applications. IEEE Trans. Knowl. Data Eng. 29(10), 2275–2289 (2017)

    Article  Google Scholar 

  28. Moon, H.J., Curino, C.A., Deutsch, A., Hou, C.-Y., Zaniolo, C.: Managing and querying transaction-time databases under schema evolution. Proc. VLDB Endowment 1(1), 882–895 (2008)

    Article  Google Scholar 

  29. Mudgal, S., et al.: Deep learning for entity matching: a design space exploration. In: Proceedings of the 2018 International Conference on Management of Data, pp. 19–34 (2018)

    Google Scholar 

  30. Nargesian, F., Pu, K.Q., Zhu, E., Ghadiri Bashardoost, B., Miller, R.J.: Organizing data lakes for navigation. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1939–1950 (2020)

    Google Scholar 

  31. Nargesian, F., Zhu, E., Pu, K.Q., Miller, R.J.: Table union search on open data. Proc. VLDB Endowment 11(7), 813–825 (2018)

    Article  Google Scholar 

  32. Rahm, E., Bernstein, P.A.: An online bibliography on schema evolution. ACM Sigmod Rec. 35(4), 30–31 (2006)

    Article  Google Scholar 

  33. Sheng, Y.: Non-blocking Lazy Schema Changes in Multi-Version Database Management Systems. PhD thesis, Carnegie Mellon University Pittsburgh, PA (2019)

    Google Scholar 

  34. Stonebraker, M., Ilyas, I.F.: Data integration: the current status and the way forward. IEEE Data Eng. Bull. 41(2), 3–9 (2018)

    Google Scholar 

  35. Ten Cate, B., Kolaitis, P.G., Tan, W.-C.: Schema map**s and data examples. In: Proceedings of the 16th International Conference on Extending Database Technology, pp. 777–780 (2013)

    Google Scholar 

  36. Thirumuruganathan, S., Parambath, S.A.P., Ouzzani, M., Tang, N., Joty, S.: Reuse and adaptation for entity resolution through transfer learning. ar** consistency under schema changes. VLDB J. 13(3), 274–293 (2004)

    Article  Google Scholar 

  37. Wang, L., Zhang, S., Shi, J., Jiao, L., Hassanzadeh, O., Zou, J., Wangz, C.: Schema management for document stores. Proc. VLDB Endowment 8(9), 922–933 (2015)

    Article  Google Scholar 

  38. Yu, C., Popa, L.: Semantic adaptation of schema map**s when schemas evolve. In: Proceedings of the 31st International Conference on Very Large Data Bases, pp. 1006–1017. VLDB Endowment (2005)

    Google Scholar 

  39. Zhao, C., He, Y.: Auto-EM: end-to-end fuzzy entity-matching using pre-trained deep models and transfer learning. In: The World Wide Web Conference, pp. 2413–2424 (2019)

    Google Scholar 

  40. Zhu, E., Deng, D., Nargesian, F., Miller, R.J.: Josie: overlap set similarity search for finding joinable tables in data lakes. In: Proceedings of the 2019 International Conference on Management of Data, pp. 847–864 (2019)

    Google Scholar 

  41. Zhu, E., Nargesian, F., Pu, K.Q., Miller, R.J.: LSH ensemble: internet-scale domain search. ar**v preprint ar**v:1603.07410 (2016)

Download references

Author information

Authors and Affiliations

  1. Arizona State University, Tempe, USA

    Zijie Wang, Lixi Zhou & Jia Zou

Authors
  1. Zijie Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lixi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jia Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jia Zou .

Editor information

Editors and Affiliations

  1. University of Technology Sydney, Sydney, NSW, Australia

    Lu Qin

  2. The University of New South Wales, Sydney, NSW, Australia

    Wenjie Zhang

  3. University of Technology Sydney, Sydney, NSW, Australia

    Ying Zhang

  4. The University of New South Wales, Sydney, NSW, Australia

    You Peng

  5. National Institute of Informatics, Tokyo, Japan

    Hiroyuki Kato

  6. The University of New South Wales, Sydney, NSW, Australia

    Wei Wang

  7. Osaka University, Osaka, Japan

    Chuan **ao

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Zhou, L., Zou, J. (2020). Integration of Fast-Evolving Data Sources Using a Deep Learning Approach. In: Qin, L., et al. Software Foundations for Data Interoperability and Large Scale Graph Data Analytics. SFDI LSGDA 2020 2020. Communications in Computer and Information Science, vol 1281. Springer, Cham. https://doi.org/10.1007/978-3-030-61133-0_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61133-0_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61132-3

  • Online ISBN: 978-3-030-61133-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation