Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Data-Driven Software Development at Large Scale: from Ad-Hoc Data Collection to Trustworthy Experimentation
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0003-4908-2708
2018 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Accurately learning what customers value is critical for the success of every company. Despite the extensive research on identifying customer preferences, only a handful of software companies succeed in becoming truly data-driven at scale. Benefiting from novel approaches such as experimentation in addition to the traditional feedback collection is challenging, yet tremendously impactful when performed correctly. In this thesis, we explore how software companies evolve from data-collectors with ad-hoc benefits, to trustworthy data-driven decision makers at scale. We base our work on a 3.5-year longitudinal multiple-case study research with companies working in both embedded systems domain (e.g. engineering connected vehicles, surveillance systems, etc.) as well as in the online domain (e.g. developing search engines, mobile applications, etc.). The contribution of this thesis is three-fold. First, we present how software companies use data to learn from customers. Second, we show how to adopt and evolve controlled experimentation to become more accurate in learning what customers value. Finally, we provide detailed guidelines that can be used by companies to improve their experimentation capabilities. With our work, we aim to empower software companies to become truly data-driven at scale through trustworthy experimentation. Ultimately this should lead to better software products and services.

Place, publisher, year, edition, pages
Malmö university, Faculty of Technology and society , 2018. , p. 357
Series
Studies in Computer Science ; 6
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:mau:diva-7768DOI: 10.24834/2043/24873Local ID: 24873ISBN: 9789171049186 (print)ISBN: 9789171049193 (electronic)OAI: oai:DiVA.org:mau-7768DiVA, id: diva2:1404709
Public defence
2018-06-15, NI:B0E07, Nordenskiöldsgatan 1, 13:00 (English)
Opponent
Note

In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Malmö University's products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink.

Available from: 2020-02-28 Created: 2020-02-28 Last updated: 2024-04-04Bibliographically approved
List of papers
1. The Lack of Sharing of Customer Data in Large Software Organizations: Challenges and Implications
Open this publication in new window or tab >>The Lack of Sharing of Customer Data in Large Software Organizations: Challenges and Implications
2016 (English)In: Agile Processes, in Software Engineering, and Extreme Programming, Springer, 2016, p. 39-52Conference paper, Published paper (Refereed)
Abstract [en]

With agile teams becoming increasingly multi-disciplinary and including all functions, the role of customer feedback is gaining momentum. Today, companies collect feedback directly from customers, as well as indirectly from their products. As a result, companies face a situation in which the amount of data from which they can learn about their customers is larger than ever before. In previous studies, the collection of data is often identified as challenging. However, and as illustrated in our research, the challenge is not the collection of data but rather how to share this data among people in order to make effective use of it. In this paper, and based on case study research in three large software-intensive companies, we (1) provide empirical evidence that ‘lack of sharing’ is the primary reason for insufficient use of customer and product data, and (2) develop a model in which we identify what data is collected, by whom data is collected and in what development phases it is used. In particular, the model depicts critical hand-overs where certain types of data get lost, as well as the implications associated with this. We conclude that companies benefit from a very limited part of the data they collect, and that lack of sharing of data drives inaccurate assumptions of what constitutes customer value.

Place, publisher, year, edition, pages
Springer, 2016
Series
Lecture Notes in Business Information Processing, ISSN 1865-1348 ; 251
Keywords
Customer feedback, product data, qualitative and quantitative data, data sharing practices, data-driven development
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mau:diva-12608 (URN)10.1007/978-3-319-33515-5_4 (DOI)2-s2.0-84971574775 (Scopus ID)20862 (Local ID)20862 (Archive number)20862 (OAI)
Conference
XP 2016, Edinburgh, Scotland (2016/05/02 - 2016/05/05)
Available from: 2020-02-29 Created: 2020-02-29 Last updated: 2024-06-17Bibliographically approved
2. Differentiating Feature Realization in Software Product Development
Open this publication in new window or tab >>Differentiating Feature Realization in Software Product Development
2017 (English)In: Product-Focused Software Process Improvement: Product-Focused Software Process Improvement. PROFES 2017., Springer, 2017, p. 221-236Conference paper, Published paper (Refereed)
Abstract [en]

Software is no longer only supporting mechanical and electrical products. Today, it is becoming the main competitive advantage and an enabler of innovation. Not all software, however, has an equal impact on customers. Companies still struggle to differentiate between the features that are regularly used, there to be for sale, differentiating and that add value to customers, or which are regarded commodity. Goal: The aim of this paper is to (1) identify the different types of software features that we can find in software products today, and (2) recommend how to prioritize the development activities for each of them. Method: In this paper, we conduct a case study with five large-scale software intensive companies. Results: Our main result is a model in which we differentiate between four fundamentally different types of features (e.g. ‘Checkbox’, ‘Flow’, ‘Duty’ and ‘Wow’). Conclusions: Our model helps companies in (1) differentiating between the feature types, and (2) selecting an optimal methodology for their development (e.g. ‘Output-Driven’ vs. ‘Outcome-Driven’).

Place, publisher, year, edition, pages
Springer, 2017
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 10611
Keywords
Data, Feedback, Outcome-driven development, data-driven development, Goal-oriented development
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mau:diva-12579 (URN)10.1007/978-3-319-69926-4_16 (DOI)000439967400016 ()2-s2.0-85034596851 (Scopus ID)24152 (Local ID)24152 (Archive number)24152 (OAI)
Conference
Product-Focused Software Process Improvement (PROFES), Innsbruck, Austria (29 November - 01 December)
Available from: 2020-02-29 Created: 2020-02-29 Last updated: 2024-06-18Bibliographically approved
3. Time to Say 'Good Bye': Feature Lifecycle
Open this publication in new window or tab >>Time to Say 'Good Bye': Feature Lifecycle
2016 (English)In: Proceedings 42nd Euromicro Conference on Software Engineering and Advanced Applications SEAA 2016, IEEE, 2016, p. 9-16Conference paper, Published paper (Refereed)
Abstract [en]

With continuous deployment of software functionality, a constant flow of new features to products is enabled. Although new functionality has potential to deliver improvements and possibilities that were previously not available, it does not necessary generate business value. On the contrary, with fast and increasing system complexity that is associated with high operational costs, more waste than value risks to be created. Validating how much value a feature actually delivers, project how this value will change over time, and know when to remove the feature from the product are the challenges large software companies increasingly experience today. We propose and study the concept of a software feature lifecycle from a value point of view, i.e. how companies track feature value throughout the feature lifecycle. The contribution of this paper is a model that illustrates how to determine (1) when to add the feature to a product, (2) how to track and (3) project the value of the feature during the lifecycle, and how to (4) identify when a feature is obsolete and should be removed from the product.

Place, publisher, year, edition, pages
IEEE, 2016
Series
Proceedings of the Euromicro Conference, ISSN 2376-9505
Keywords
value modeling, customer feedback, feature lifecycle
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mau:diva-12477 (URN)10.1109/SEAA.2016.59 (DOI)000386649000002 ()2-s2.0-85020708719 (Scopus ID)21729 (Local ID)21729 (Archive number)21729 (OAI)
Conference
Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus (31st August-2nd September, 2016)
Available from: 2020-02-29 Created: 2020-02-29 Last updated: 2024-06-17Bibliographically approved
4. The Online Controlled Experiment Lifecycle
Open this publication in new window or tab >>The Online Controlled Experiment Lifecycle
2020 (English)In: IEEE Software, ISSN 0740-7459, E-ISSN 1937-4194, Vol. 37, no 2, p. 60-67Article in journal (Refereed) Published
Abstract [en]

Online Controlled Experiments (OCEs) enable an accurate understanding of customer value and generate millions of dollars of additional revenue at Microsoft. Unlike other techniques for learning from customers, OCEs establish an accurate and causal relationship between a change and the impact observed. Although previous research describes technical and statistical dimensions, the key phases of online experimentation are not widely known, their impact and importance are obscure, and how to establish OCEs in an organization is underexplored. In this paper, using a longitudinal in-depth case study, we address this gap by (1) presenting the Experiment Lifecycle, and (2) demonstrating with four example experiments their profound impact. We show that OECs help optimize infrastructure needs and aid in project planning and measuring team efforts, in addition to their primary goal of accurately identifying what customers value. We conclude that product development should fully integrate the Experiment Lifecycle to benefit from the OCEs.

Place, publisher, year, edition, pages
IEEE, 2020
Keywords
Measurement, Companies, Software, Computer science, Product development, Media, Planning, Data-driven development, A/B tests, Online Controlled Experiments, experiment lifecycle
National Category
Software Engineering
Identifiers
urn:nbn:se:mau:diva-2309 (URN)10.1109/MS.2018.2875842 (DOI)000520152900011 ()2-s2.0-85055195833 (Scopus ID)28040 (Local ID)28040 (Archive number)28040 (OAI)
Available from: 2020-02-27 Created: 2020-02-27 Last updated: 2024-06-17Bibliographically approved
5. The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale
Open this publication in new window or tab >>The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale
2017 (English)In: International Conference on Software Engineering. Proceedings, IEEE, 2017, p. 770-780Conference paper, Published paper (Refereed)
Abstract [en]

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.

Place, publisher, year, edition, pages
IEEE, 2017
Keywords
A/B testing, continuous experimentation, data science, customer feedback, continuous product innovation, Experimentation Evolution Model, Experiment Owner
National Category
Engineering and Technology
Identifiers
urn:nbn:se:mau:diva-12651 (URN)10.1109/ICSE.2017.76 (DOI)000427091300068 ()2-s2.0-85027682332 (Scopus ID)24149 (Local ID)24149 (Archive number)24149 (OAI)
Conference
International Conference on Software Engineering (ICSE), Buenos Aires, Argentina (20-28 May 2017)
Available from: 2020-02-29 Created: 2020-02-29 Last updated: 2024-06-18Bibliographically approved

Open Access in DiVA

fulltext(5369 kB)1192 downloads
File information
File name FULLTEXT01.pdfFile size 5369 kBChecksum SHA-512
e41ec702016cfbab24fa5550a738f6878540fe88aef45239453f8c581cc4d33f976e521a1383fe352ae7bb76b7452344a861de8ccb617c026ee895e637caf63e
Type fulltextMimetype application/pdf

Other links

Publisher's full text

Authority records

Fabijan, Aleksander

Search in DiVA

By author/editor
Fabijan, Aleksander
By organisation
Department of Computer Science and Media Technology (DVMT)
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar
Total: 1194 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 947 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf