IJEEC | Scalable Decentralized Multi-Agent Federated Reinforcement Learning: Challenges and Advances

Indexing and Abstracting

Scalable Decentralized Multi-Agent Federated Reinforcement Learning: Challenges and Advances

Praveen Kumar Myakala , Srikanth Kamatala

International Journal of Electrical, Electronics and Computers (IJECC), Vol-8,Issue-6, November - December 2023, Pages 8-16, 10.22161/eec.86.2

Download | Downloads : 141 | Total View : 1351

Article Info: Received: 24 Oct 2023; Accepted: 30 Nov 2023; Date of Publication: 07 Dec 2023

Abstract:

The increasing prevalence of decentralized multiagent systems has spurred interest in Federated Reinforcement Learning (FRL) as a privacy-preserving framework for collaborative learning. However, scaling FRL to multi-agent settings introduces significant challenges, particularly in communication efficiency, decentralized aggregation, and handling nonstationary environments. This survey explores recent advancements in Scalable Decentralized Multi-Agent Federated Reinforcement Learning (MA-FRL), with a focus on communication efficient strategies and decentralized aggregation techniques. We review key approaches such as selective agent communication, local model updates, and gradient compression, analyzing their impact on scalability, convergence, and performance trade-offs. Additionally, we highlight comparative insights into different methods, their limitations, and real-world applicability in decentralized systems such as autonomous vehicles and smart grids. By identifying open challenges, including robustness against adversarial attacks and adaptive communication mechanisms, we outline promising directions for advancing decentralized MAFRL.

Keywords:

Federated Reinforcement Learning, Multi-Agent Systems, Decentralized Learning, Scalability, Communication Efficiency, Aggregation Techniques, Non-Stationarity

References:

[1] Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019). Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2), 1-19.
[2] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020.
[3] McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282). PMLR.
[4] Fu, H., Tang, H., Hao, J., Lei, Z., Chen, Y., & Fan, C. (2019). Deep multi-agent reinforcement learning with discrete-continuous hybrid action spaces. arXiv preprint arXiv:1903.04959.
[5] Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., & Gao, Y. (2021). A survey on federated learning. Knowledge-Based Systems, 216, 106775.
[6] Foerster, J., Assael, I. A., De Freitas, N., & Whiteson, S. (2016). Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29.
[7] Aji, A. F., & Heafield, K. (2017). Sparse communication for distributed gradient descent. arXiv preprint arXiv:1704.05021.
[8] Blot, M., Picard, D., Cord, M., & Thome, N. (2016). Gossip training for deep learning. arXiv preprint arXiv:1611.09726.
[9] Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint arXiv:1903.03934.
[10] Lalitha, A., Kilinc, O. C., Javidi, T., & Koushanfar, F. (2019). Peer-to-peer federated learning on graphs. arXiv preprint arXiv:1901.11173.
[11] Xu, G., Zhou, Z., Dong, J., Zhang, L., & Song, X. (2023). A blockchain-based federated learning scheme for data sharing in industrial internet of things. IEEE Internet of Things Journal.
[12] Briggs, C., Fan, Z., & Andras, P. (2020, July). Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In 2020 international joint conference on neural networks (IJCNN) (pp. 1-9). IEEE.
[13] Nedic, A., Olshevsky, A., Ozdaglar, A., & Tsitsiklis, J. N. (2008, December). Distributed subgradient methods and quantization effects. In 2008 47th IEEE conference