Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Modelling Data Pipelines
Chalmers Univ Technol, Gothenburg, Sweden..
Chalmers Univ Technol, Gothenburg, Sweden..
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0002-7700-1816
Ericsson, Gothenburg, Sweden..
2020 (English)In: 2020 46TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA 2020) / [ed] Martini, A Wimmer, M Skavhaug, A, IEEE, 2020, p. 13-20Conference paper, Published paper (Refereed)
Abstract [en]

Data is the new currency and key to success. However, collecting high-quality data from multiple distributed sources requires much effort. In addition, there are several other challenges involved while transporting data from its source to the destination. Data pipelines are implemented in order to increase the overall efficiency of data-flow from the source to the destination since it is automated and reduces the human involvement which is required otherwise. Despite existing research on ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform) pipelines, the research on this topic is limited. ETL/ELT pipelines are abstract representations of the end-to-end data pipelines. To utilize the full potential of the data pipeline, we should understand the activities in it and how they are connected in an end-to-end data pipeline. This study gives an overview of how to design a conceptual model of data pipeline which can be further used as a language of communication between different data teams. Furthermore, it can be used for automation of monitoring, fault detection, mitigation and alarming at different steps of data pipeline.

Place, publisher, year, edition, pages
IEEE, 2020. p. 13-20
Series
EUROMICRO Conference Proceedings, ISSN 1089-6503
Keywords [en]
Data pipelines, conceptual model, data work-flow, domain specific language, agile methodology
National Category
Other Computer and Information Science
Identifiers
URN: urn:nbn:se:mau:diva-46699DOI: 10.1109/SEAA51224.2020.00014ISI: 000702094100003Scopus ID: 2-s2.0-85096624237ISBN: 978-1-7281-9532-2 (electronic)OAI: oai:DiVA.org:mau-46699DiVA, id: diva2:1609477
Conference
46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), AUG 26-28, 2020, ELECTR NETWORK
Available from: 2021-11-08 Created: 2021-11-08 Last updated: 2024-02-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Olsson, Helena Holmström

Search in DiVA

By author/editor
Olsson, Helena Holmström
By organisation
Department of Computer Science and Media Technology (DVMT)
Other Computer and Information Science

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 59 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf