Publikationer från Malmö universitet
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
Malmö universitet, Fakulteten för teknik och samhälle (TS), Institutionen för datavetenskap och medieteknik (DVMT).ORCID-id: 0000-0003-4908-2708
Outreach.io, Seattle, WA, United States.
Malmö universitet, Fakulteten för teknik och samhälle (TS), Institutionen för datavetenskap och medieteknik (DVMT).ORCID-id: 0000-0002-7700-1816
Chalmers University of Technology, Goteborg, Sweden.
Visa övriga samt affilieringar
2019 (Engelska)Ingår i: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2019), IEEE, 2019, s. 1-10Konferensbidrag, Publicerat paper (Refereegranskat)
Abstract [en]

Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies into an experimental laboratory. Despite their great power in identifying what customers actually value, experimentation is very sensitive to data loss, skipped checks, wrong designs, and many other 'hiccups' in the analysis process. For this purpose, experiment analysis has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Depending solely on scarce experts, however, is neither scalable nor bulletproof. To democratize experimentation, analysis should be streamlined and meticulously performed by engineers, managers, or others responsible for the development of a product. In this paper, based on synthesized experience of companies that run thousands of OCEs per year, we examined how experts inspect online experiments. We reveal that most of the experiment analysis happens before OCEs are even started, and we summarize the key analysis steps in three checklists. The value of the checklists is threefold. First, they can increase the accuracy of experiment setup and decision-making process. Second, checklists can enable novice data scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy platforms and tools for OCE set-up and analysis.

Ort, förlag, år, upplaga, sidor
IEEE, 2019. s. 1-10
Nyckelord [en]
Online Controlled Experiments, A/B testing, Experiment Checklists
Nationell ämneskategori
Programvaruteknik
Identifikatorer
URN: urn:nbn:se:mau:diva-39375DOI: 10.1109/ICSE-SEIP.2019.00009ISI: 000503218300001Scopus ID: 2-s2.0-85072103557OAI: oai:DiVA.org:mau-39375DiVA, id: diva2:1519607
Konferens
IEEE/ACM 41st International Conference on Software Engineering, 25-31 May 2019, Montreal, QC, Canada, Canada
Tillgänglig från: 2021-01-19 Skapad: 2021-01-19 Senast uppdaterad: 2024-06-17Bibliografiskt granskad

Open Access i DiVA

Fulltext saknas i DiVA

Övriga länkar

Förlagets fulltextScopus

Person

Fabijan, AleksanderOlsson, Helena Holmström

Sök vidare i DiVA

Av författaren/redaktören
Fabijan, AleksanderOlsson, Helena Holmström
Av organisationen
Institutionen för datavetenskap och medieteknik (DVMT)
Programvaruteknik

Sök vidare utanför DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetricpoäng

doi
urn-nbn
Totalt: 23 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf