Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Towards Automated Detection of Data Pipeline Faults
Chalmers.
Chalmers.
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0002-7700-1816
Ericsson, Gothenburg, Sweden..
2020 (English)In: 2020 27TH ASIA-PACIFIC SOFTWARE ENGINEERING CONFERENCE (APSEC 2020), IEEE, 2020, p. 346-355Conference paper, Published paper (Refereed)
Abstract [en]

Data pipelines play an important role throughout the data management process. It automates the steps ranging from data generation to data reception thereby reducing the human intervention. A failure or fault in a single step of a data pipeline has cascading effects that might result in hours of manual intervention and clean-up. Data pipeline failure due to faults at different stages of data pipelines is a common challenge that eventually leads to significant performance degradation of data-intensive systems. To ensure early detection of these faults and to increase the quality of the data products, continuous monitoring and fault detection mechanism should be included in the data pipeline. In this study, we have explored the need for incorporating automated fault detection mechanisms and mitigation strategies at different stages of the data pipeline. Further, we identified faults at different stages of the data pipeline and possible mitigation strategies that can be adopted for reducing the impact of data pipeline faults thereby improving the quality of data products. The idea of incorporating fault detection and mitigation strategies is validated by realizing a small part of the data pipeline using action research in the analytics team at a large software-intensive organization within the telecommunication domain.

Place, publisher, year, edition, pages
IEEE, 2020. p. 346-355
Series
Asia-Pacific Software Engineering Conference, ISSN 1530-1362
Keywords [en]
component, data pipeline, fault detection, anomalies, mitigation, robustness, failure recovery, data quality, fault-tolerance
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:mau:diva-44960DOI: 10.1109/APSEC51365.2020.00043ISI: 000662668700036Scopus ID: 2-s2.0-85102388726ISBN: 978-1-7281-9553-7 (print)OAI: oai:DiVA.org:mau-44960DiVA, id: diva2:1585978
Conference
27th Asia-Pacific Software Engineering Conference (APSEC), DEC 01-04, 2020, Singapore, SINGAPORE
Available from: 2021-08-18 Created: 2021-08-18 Last updated: 2024-02-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Olsson, Helena Holmström

Search in DiVA

By author/editor
Olsson, Helena Holmström
By organisation
Department of Computer Science and Media Technology (DVMT)
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 62 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf