Malmö University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Online Controlled Experimentation at Scale: An Empirical Survey on the Current State of A/B Testing
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0003-4908-2708
Microsoft, Analysis and Experimentation, Redmond, United States.
Malmö University, Faculty of Technology and Society (TS), Department of Computer Science and Media Technology (DVMT).ORCID iD: 0000-0002-7700-1816
Chalmers University of Tech., Dep. of Computer Science, Göteborg, Sweden.
2018 (English)In: Proceedings of the EUROMICRO Conference, IEEE, 2018, p. 68-72Conference paper, Published paper (Refereed)
Abstract [en]

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and Booking.com report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.

Place, publisher, year, edition, pages
IEEE, 2018. p. 68-72
Series
Proceedings of the Euromicro Conference, ISSN 1089-6503
Keywords [en]
controlled experimentation, A/B testing, empirical survey, Experimentation Growth Model
National Category
Engineering and Technology
Identifiers
URN: urn:nbn:se:mau:diva-12624DOI: 10.1109/SEAA.2018.00021ISI: 000450238900012Scopus ID: 2-s2.0-85057169416Local ID: 27273OAI: oai:DiVA.org:mau-12624DiVA, id: diva2:1409671
Conference
44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Prague, Czech Republic (29-31 Aug. 2018)
Available from: 2020-02-29 Created: 2020-02-29 Last updated: 2024-06-17Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopushttp://dsd-seaa2018.fit.cvut.cz/seaa/

Authority records

Fabijan, AleksanderOlsson, Helena Holmström

Search in DiVA

By author/editor
Fabijan, AleksanderOlsson, Helena Holmström
By organisation
Department of Computer Science and Media Technology (DVMT)
Engineering and Technology

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 50 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf