With agile teams becoming increasingly multi-disciplinary and including all functions, the role of customer feedback is gaining momentum. Today, companies collect feedback directly from customers, as well as indirectly from their products. As a result, companies face a situation in which the amount of data from which they can learn about their customers is larger than ever before. In previous studies, the collection of data is often identified as challenging. However, and as illustrated in our research, the challenge is not the collection of data but rather how to share this data among people in order to make effective use of it. In this paper, and based on case study research in three large software-intensive companies, we (1) provide empirical evidence that ‘lack of sharing’ is the primary reason for insufficient use of customer and product data, and (2) develop a model in which we identify what data is collected, by whom data is collected and in what development phases it is used. In particular, the model depicts critical hand-overs where certain types of data get lost, as well as the implications associated with this. We conclude that companies benefit from a very limited part of the data they collect, and that lack of sharing of data drives inaccurate assumptions of what constitutes customer value.

Software is no longer only supporting mechanical and electrical products. Today, it is becoming the main competitive advantage and an enabler of innovation. Not all software, however, has an equal impact on customers. Companies still struggle to differentiate between the features that are regularly used, there to be for sale, differentiating and that add value to customers, or which are regarded commodity. Goal: The aim of this paper is to (1) identify the different types of software features that we can find in software products today, and (2) recommend how to prioritize the development activities for each of them. Method: In this paper, we conduct a case study with five large-scale software intensive companies. Results: Our main result is a model in which we differentiate between four fundamentally different types of features (e.g. ‘Checkbox’, ‘Flow’, ‘Duty’ and ‘Wow’). Conclusions: Our model helps companies in (1) differentiating between the feature types, and (2) selecting an optimal methodology for their development (e.g. ‘Output-Driven’ vs. ‘Outcome-Driven’).

With continuous deployment of software functionality, a constant flow of new features to products is enabled. Although new functionality has potential to deliver improvements and possibilities that were previously not available, it does not necessary generate business value. On the contrary, with fast and increasing system complexity that is associated with high operational costs, more waste than value risks to be created. Validating how much value a feature actually delivers, project how this value will change over time, and know when to remove the feature from the product are the challenges large software companies increasingly experience today. We propose and study the concept of a software feature lifecycle from a value point of view, i.e. how companies track feature value throughout the feature lifecycle. The contribution of this paper is a model that illustrates how to determine (1) when to add the feature to a product, (2) how to track and (3) project the value of the feature during the lifecycle, and how to (4) identify when a feature is obsolete and should be removed from the product.

Online Controlled Experiments (OCEs) enable an accurate understanding of customer value and generate millions of dollars of additional revenue at Microsoft. Unlike other techniques for learning from customers, OCEs establish an accurate and causal relationship between a change and the impact observed. Although previous research describes technical and statistical dimensions, the key phases of online experimentation are not widely known, their impact and importance are obscure, and how to establish OCEs in an organization is underexplored. In this paper, using a longitudinal in-depth case study, we address this gap by (1) presenting the Experiment Lifecycle, and (2) demonstrating with four example experiments their profound impact. We show that OECs help optimize infrastructure needs and aid in project planning and measuring team efforts, in addition to their primary goal of accurately identifying what customers value. We conclude that product development should fully integrate the Experiment Lifecycle to benefit from the OCEs.

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.