Deep reinforcement learning has made significant development in recent years, and it is currently applied not only in simulators and games but also in embedded systems. However, when implemented in a real-world context, reinforcement learning is frequently shown to be unstable and incapable of adapting to realistic situations, particularly when directing a large number of agents. In this paper, we develop a decentralized architecture for reinforcement learning to allow multiple agents to learn optimal control policies on their own devices of the same kind but in varied environments. For such multiple agents, the traditional centralized learning algorithm usually requires a costly or time-consuming effort to develop the best-regulating policy and is incapable of scaling to a large-scale system. To address this issue, we propose a decentralized reinforcement learning algorithm (DecRL) and information exchange scheme for each individual device, in which each agent shares the individual learning experience and information with other agents based on local model training. We incorporate the algorithm into each agent in the proposed collaborative architecture and validate it in the telecommunication domain under emergency conditions, in which a macro base station (BS) is broken due to a natural disaster, and three unmanned aerial vehicles carrying BSs (UAV-BSs) are deployed to provide temporary coverage for missioncritical (MC) users in the disaster area. Based on the findings, we show that the proposed decentralized reinforcement learning algorithm can successfully support multi-agent learning, while the learning speed and service quality can be further enhanced.

Deep reinforcement learning has advanced signifi-cantly in recent years, and it is now used in embedded systems in addition to simulators and games. Reinforcement Learning (RL) algorithms are currently being used to enhance device operation so that they can learn on their own and offer clients better services. It has recently been studied in a variety of industrial applications. However, reinforcement learning, especially when controlling a large number of agents in an industrial environment, has been demonstrated to be unstable and unable to adapt to realistic situations when used in a real-world setting. To address this problem, the goal of this study is to enable multiple reinforcement learning agents to independently learn control policies on their own in dynamic industrial contexts. In order to solve the problem, we propose a dynamic multi-agent reinforcement learning (dynamic multi-RL) method along with adaptive exploration (AE) and vector-based action selection (VAS) techniques for accelerating model convergence and adapting to a complex industrial environment. The proposed algorithm is tested for validation in emergency situations within the telecommunications industry. In such circumstances, three unmanned aerial vehicles (UAV-BSs) are used to provide temporary coverage to mission-critical (MC) customers in disaster zones when the original serving base station (BS) is destroyed by natural disasters. The algorithm directs the participating agents automatically to enhance service quality. Our findings demonstrate that the proposed dynamic multi-RL algorithm can proficiently manage the learning of multiple agents and adjust to dynamic industrial environments. Additionally, it enhances learning speed and improves the quality of service.

Continuous experimentation (CE) refers to a set of practices used by software companies to rapidly assess the usage, value, and performance of deployed software using data collected from customers and systems in the field using an experimental methodology. However, despite its increasing popularity in developing web-facing applications, CE has not been studied in the development process of business-to-business (B2B) mission-critical systems. By observing the CE practices of different teams, with a case study methodology inside Ericsson, we were able to identify the different practices and techniques used in B2B mission-critical systems and a description and classification of the four possible types of experiments. We present and analyze each of the four types of experiments with examples in the context of the mission-critical long-term evolution (4G) product. These examples show the general experimentation process followed by the teams and the use of the different CE practices and techniques. Based on these examples and the empirical data, we derived the HURRIER process to deliver high-quality solutions that the customers value. Finally, we discuss the challenges, opportunities, and lessons learned from applying CE and the HURRIER process in B2B mission-critical systems. 

Continuous deployment has been practiced for many years by companies developing web- and cloud-based applications. To succeed with continuous deployment, these companies have a strong collaboration culture between the operations and development teams. In addition, these companies use AI, analytics, and big data to assist with time-consuming postdeployment activities such as continuous monitoring and fault identification. Thus, the term AIOps has evolved to highlight the importance and difficulty of maintaining highly available applications in a complex and dynamic environment. In contrast, software-intensive embedded systems often provide customer product-related services, such as maintenance, optimization, and support. These services are critical for these companies as they provide significant revenue and increase customer satisfaction. Therefore, the objective of our study is to gain an in-depth understanding of the impact of continuous deployment on product-related services provided by software-intensive embedded systems companies. In addition, we aim to understand how AIOps can support continuous deployment in the context of software-intensive embedded systems. To address this objective, we conducted a case study at a large and multinational telecommunications systems provider focusing on the radio access network (RAN) systems for 4G and 5G networks. The company provides RAN products and three complementing services: rollout, optimization, and customer support. The results from the case study show that the boundaries between product-related services become blurry with continuous deployment. In addition, product-related services, which were conducted in sequence by independent projects, converge with continuous deployment and become part of the same project. Further, AIOps platforms play an important role in reducing costs and increasing postdeployment activities' efficiency and speed. These results show that continuous deployment has a profound impact on the software-intensive system's provider service organization. The service organization becomes the connection between the R&D organization and the customer. In order to cope with the increased speed of releases, deployment and postdeployment activities need to be largely automated. AIOps platforms are seen as a critical enabler in managing the increasing complexity without increasing human involvement.

Artificial intelligence (AI) and the use of machine learning (ML) and deep learning (DL) technologies are becoming increasingly popular in companies. These technologies enable companies to leverage big quantities of data to improve system performance and accelerate business development. However, despite the appeal of ML/DL, there is a lack of systematic and structured methods and processes to help data scientists and other company roles and functions to develop, deploy and evolve models. In this paper, based on multi-case study research in six companies, we explore practices and challenges practitioners experience in developing ML/DL models as part of large software-intensive embedded systems. Based on our empirical findings, we derive a conceptual framework in which we identify three high-level activities that companies perform in parallel with the development, deployment and evolution of models. Within this framework, we outline activities, iterations and triggers that optimize model design as well as roles and company functions. In this way, we provide practitioners with a blueprint for effectively integrating ML/DL model development into the business to achieve better results than other (algorithmic) approaches. In addition, we show how this framework helps companies solve the challenges we have identified and discuss checkpoints for terminating the business case.

Artificial Intelligence (AI) engineering is an engineering discipline that is concerned with all aspects of development and evolution of AI systems (that is, systems that include AI components). AI engineering is primarily an extension of software engineering, but it also includes methods and technologies from data science and AI in general.

Fast and reliable connectivity is essential to enhance situational awareness and operational efficiency for public safety mission-critical (MC) users. In emergency or disaster circumstances, where existing cellular network coverage and capacity may not be available to meet MC communication demands, deployable-network-based solutions such as cells-on-wheels/wings can be utilized swiftly to ensure reliable connection for MC users. In this paper, we consider a scenario where a macro base station (BS) is destroyed due to a natural disaster and an unmanned aerial vehicle carrying BS (UAV-BS) is set up to provide temporary coverage for users in the disaster area. The UAV-BS is integrated into the mobile network using the 5G integrated access and backhaul (IAB) technology. We propose a framework and signalling procedure for applying machine learning to this use case. A deep reinforcement learning algorithm is designed to jointly optimize the access and backhaul antenna tilt as well as the three-dimensional location of the UAV-BS in order to best serve the on-ground MC users while maintaining a good backhaul connection. Our result shows that the proposed algorithm can autonomously navigate and configure the UAV-BS to improve the throughput and reduce the drop rate of MC users.

In recent years, the application of artificial intelligence (AI) has become an integral part of a wide range of areas, including software engineering. By analyzing various data sources generated in software engineering, it can provide valuable insights into customer behavior, product performance, bugs and errors, and many more. In practice, however, AI for software analytics and business intelligence often remains at a prototypical stage, and the results are rarely used to make decisions based on data. To understand the underlying causes of this phenomenon, we conduct an explanatory case study consisting of and interview study and a survey on the challenges of realizing and utilizing artificial intelligence in the context of software-intensive businesses. As a result, we identify a vicious circle that prevents practitioners from moving from prototypical AI-based analytics to continuous and productively usable software analytics and business intelligence solutions. In order to break the vicious circle in a targeted manner, we identify a set of solutions based on existing literature as well as the previously conducted interviews and survey. Finally, these solutions are validated by a focus group of experts. (C) 2021 Elsevier Inc. All rights reserved.

AI in the Engineering, Procurement and Construction (EPC) industry has not yet a proven track record in large-scale projects. Since AI solutions for industrial applications became available only recently, deployment experience and lessons learned are still to be built up. Several research papers exist describing the potential of AI, and many surveys and white papers have been published indicating the challenges of AI deployment in the EPC industry. However, there is a recognizable shortage of in-depth studies of deployment experience in academic literature, particularly those focusing on the experiences of EPC companies involved in large-scale project execution with high safety standards, such as the petrochemical or energy sector. The novelty of this research is that we explore in detail the challenges and obstacles faced in developing and deploying AI in a large-scale project in the EPC industry based on real-life use cases performed in an EPC company. Those identified challenges are not linked to specific technology or a company's know-how and, therefore, are universal. The findings in this paper aim to provide feedback to academia to reduce the gap between research and practice experience. They also help reveal the hidden stones when implementing AI solutions in the industry.

Continuous deployment has become a widely used practice in web-based software applications. Deploying a new software version to production is a seamless automated process executed thousands of times per day. Continuous deployment reduces the time between a code commit and that commit is active in production. While continuous deployment promises many advantages to software development organizations, the adoption of continuous deployment in the software-intensive embedded systems industry is limited. Several empirical studies have highlighted the challenges associated with software-intensive embedded systems. However, very few studies, if any at all, have attempted to provide a practical approach to realize continuous deployment to these systems. This paper proposes a Controlled Continuous Deployment (CCD) approach, which considers the constraints software-intensive embedded systems have, such as high reliability and availability requirements, limited possibility for rollback after deployment, and the high volume of in-service systems in the market. We derived the approach by conducting a case study at Ericsson AB, focusing on three Radio Access Networks (RAN) technologies embedded software used in 3G, 4G, and 5G mobile networks.