Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Data Pipeline Management in Practice: Challenges and Opportunities
Department of Computer Science and Engineering, Chalmers University of Technology, Hörselgången 11, 412 96, Gothenburg, Sweden.
Department of Computer Science and Engineering, Chalmers University of Technology, Hörselgången 11, 412 96, Gothenburg, Sweden.
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).
2020 (English)In: Product-Focused Software Process Improvement: 21st International Conference, PROFES 2020, Turin, Italy, November 25–27, 2020, Proceedings / [ed] Maurizio Morisio; Marco Torchiano; Andreas Jedlitschka, Springer, 2020, p. 168-184Conference paper, Published paper (Refereed)
Abstract [en]

Data pipelines involve a complex chain of interconnected activities that starts with a data source and ends in a data sink. Data pipelines are important for data-driven organizations since a data pipeline can process data in multiple formats from distributed data sources with minimal human intervention, accelerate data life cycle activities, and enhance productivity in data-driven enterprises. However, there are challenges and opportunities in implementing data pipelines but practical industry experiences are seldom reported. The findings of this study are derived by conducting a qualitative multiple-case study and interviews with the representatives of three companies. The challenges include data quality issues, infrastructure maintenance problems, and organizational barriers. On the other hand, data pipelines are implemented to enable traceability, fault-tolerance, and reduce human errors through maximizing automation thereby producing high-quality data. Based on multiple-case study research with five use cases from three case companies, this paper identifies the key challenges and benefits associated with the implementation and use of data pipelines.

Place, publisher, year, edition, pages
Springer, 2020. p. 168-184
Series
Lecture Notes in Computer Science, ISSN 0302-9743, E-ISSN 1611-3349 ; 12562
National Category
Other Computer and Information Science
Identifiers
URN: urn:nbn:se:mau:diva-56800DOI: 10.1007/978-3-030-64148-1_11ISI: 000766320200011ISBN: 978-3-030-64147-4 (print)ISBN: 978-3-030-64148-1 (electronic)OAI: oai:DiVA.org:mau-56800DiVA, id: diva2:1720426
Conference
21st International Conference, PROFES 2020, Turin, Italy, November 25–27, 2020
Available from: 2022-12-19 Created: 2022-12-19 Last updated: 2022-12-19Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textFulltext

Authority records

Olsson, Helena Homström

Search in DiVA

By author/editor
Olsson, Helena Homström
By organisation
Department of Computer Science and Media Technology (DVMT)
Other Computer and Information Science

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 38 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf