Data Pipeline Management in Practice: Challenges and Opportunities
2020 (English)In: Product-Focused Software Process Improvement: 21st International Conference, PROFES 2020, Turin, Italy, November 25–27, 2020, Proceedings / [ed] Maurizio Morisio; Marco Torchiano; Andreas Jedlitschka, Springer, 2020, p. 168-184Conference paper, Published paper (Refereed)
Abstract [en]
Data pipelines involve a complex chain of interconnected activities that starts with a data source and ends in a data sink. Data pipelines are important for data-driven organizations since a data pipeline can process data in multiple formats from distributed data sources with minimal human intervention, accelerate data life cycle activities, and enhance productivity in data-driven enterprises. However, there are challenges and opportunities in implementing data pipelines but practical industry experiences are seldom reported. The findings of this study are derived by conducting a qualitative multiple-case study and interviews with the representatives of three companies. The challenges include data quality issues, infrastructure maintenance problems, and organizational barriers. On the other hand, data pipelines are implemented to enable traceability, fault-tolerance, and reduce human errors through maximizing automation thereby producing high-quality data. Based on multiple-case study research with five use cases from three case companies, this paper identifies the key challenges and benefits associated with the implementation and use of data pipelines.
Place, publisher, year, edition, pages
Springer, 2020. p. 168-184
Series
Lecture Notes in Computer Science, ISSN 0302-9743, E-ISSN 1611-3349 ; 12562
National Category
Other Computer and Information Science
Identifiers
URN: urn:nbn:se:mau:diva-56800DOI: 10.1007/978-3-030-64148-1_11ISI: 000766320200011ISBN: 978-3-030-64147-4 (print)ISBN: 978-3-030-64148-1 (electronic)OAI: oai:DiVA.org:mau-56800DiVA, id: diva2:1720426
Conference
21st International Conference, PROFES 2020, Turin, Italy, November 25–27, 2020
2022-12-192022-12-192022-12-19Bibliographically approved