Deep reinforcement learning has made significant development in recent years, and it is currently applied not only in simulators and games but also in embedded systems. However, when implemented in a real-world context, reinforcement learning is frequently shown to be unstable and incapable of adapting to realistic situations, particularly when directing a large number of agents. In this paper, we develop a decentralized architecture for reinforcement learning to allow multiple agents to learn optimal control policies on their own devices of the same kind but in varied environments. For such multiple agents, the traditional centralized learning algorithm usually requires a costly or time-consuming effort to develop the best-regulating policy and is incapable of scaling to a large-scale system. To address this issue, we propose a decentralized reinforcement learning algorithm (DecRL) and information exchange scheme for each individual device, in which each agent shares the individual learning experience and information with other agents based on local model training. We incorporate the algorithm into each agent in the proposed collaborative architecture and validate it in the telecommunication domain under emergency conditions, in which a macro base station (BS) is broken due to a natural disaster, and three unmanned aerial vehicles carrying BSs (UAV-BSs) are deployed to provide temporary coverage for missioncritical (MC) users in the disaster area. Based on the findings, we show that the proposed decentralized reinforcement learning algorithm can successfully support multi-agent learning, while the learning speed and service quality can be further enhanced.