2000 character limit reached
Optimal Flow Admission Control in Edge Computing via Safe Reinforcement Learning
Published 8 Apr 2024 in cs.NI | (2404.05564v3)
Abstract: With the uptake of intelligent data-driven applications, edge computing infrastructures necessitate a new generation of admission control algorithms to maximize system performance under limited and highly heterogeneous resources. In this paper, we study how to optimally select information flows which belong to different classes and dispatch them to multiple edge servers where deployed applications perform flow analytic tasks. The optimal policy is obtained via constrained Markov decision process (CMDP) theory accounting for the demand of each edge application for specific classes of flows, the constraints on computing capacity of edge servers and of the access network. We develop DR-CPO, a specialized primal-dual Safe Reinforcement Learning (SRL) method which solves the resulting optimal admission control problem by reward decomposition. DR-CPO operates optimal decentralized control and mitigates effectively state-space explosion while preserving optimality. Compared to existing Deep Reinforcement Learning (DRL) solutions, extensive results show that DR-CPO achieves 15\% higher reward on a wide variety of environments, while requiring on average only 50\% of the amount of learning episodes to converge. Finally, we show how to match DR-CPO and load-balancing to dispatch optimally information streams to available edge servers and further improve system performance.
- J. Achiam et al. Constrained policy optimization. In Proc. of International Conference on Machine Learning (ICML), 2017.
- Reinforcement learning for admission control in wireless virtual network embedding. In Proc. of IEEE ANTS, 2021.
- E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
- E. Altman and A. Schwartz. Adaptive control of constrained Markov chains. IEEE Transactions on Automatic Control, 36(4), 1991.
- Real-time video analytics: The killer app for edge computing. IEEE Computer, 50(10), 2017.
- Optimal control of admission to a multiserver queue with two arrival streams. IEEE Trans. on Automatic Control, 37(6), 1992.
- Real-time packet-based intrusion detection on edge devices. In Proc. of Cyber-Physical Systems and Internet of Things Week. 2023.
- V. Borkar. An actor-critic algorithm for constrained Markov decision processes. Elsevier Systems & Control Letters, 54(3), 2005.
- Traffic refinery: Cost-aware data representation for machine learning on network traffic. Proc. of the ACM on Measurement and Analysis of Computing Systems, 5(3):1–24, 2021.
- Optimizing admission control while ensuring quality of service in multimedia networks via reinforcement learning. Proc. of NIPS, 11, 1998.
- Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Internet of Things Journal, 6(3), 2018.
- L. Cherkasova and P. Phaal. Session-based admission control: A mechanism for peak load management of commercial web sites. IEEE Trans. Comput., 51(6), 2002.
- A. Choudhary and S. Chaudhury. Video analytics revisited. IET Computer Vision, 10(4), 2016.
- X. Fan-Orzechowski and E. A. Feinberg. Optimality of randomized trunk reservation for a problem with a single constraint. Advances in Applied Probability, 38(1), 2006.
- Optimality of randomized trunk reservation. Probability in the Engineering and Informational Sciences, 8(4), 1994.
- M. Fu and S. Hill. Optimization of discrete event systems via simultaneous perturbation stochastic approximation. IIE Transactions, 29(3), 1997.
- Edge-based video analytics: A survey. arXiv preprint arXiv:2303.14329, 2023.
- Dynamic admission control and resource allocation for mobile edge computing enabled small cell network. IEEE Transactions on Vehicular Technology, 71(2), 2022.
- Videoedge: Processing camera streams using hierarchical clusters. In Proc. of the IEEE/ACM Symposium on Edge Computing (SEC), 2018.
- Chameleon: scalable adaptation of video analytics. In Proc. of ACM SICOMM, 2018.
- Explainable reinforcement learning via reward decomposition. In Proc. of IJCAI, 2019.
- Edge-computing-enabled smart cities: A comprehensive survey. IEEE Internet of Things Journal, 7:10200–10232, 2019.
- Elastic admission control for federated cloud services. IEEE Transactions on Cloud Computing, 2(3), 2014.
- H. J. Kushner and G. Yin. Stochastic approximation algorithms and applications. In Applied Mathematics. Springer, 1997.
- N. Lilith and K. Dogancay. Using reinforcement learning for call admission control in cellular environments featuring self-similar traffic. In Proc. of IEEE TENCON, 2005.
- S. A. Lippman. Applying a new device in the optimization of exponential queuing systems. Operations Research, 23(4), 1975.
- Ipo: Interior-point policy optimization under constraints. In Proc. of AAAI, 2019.
- Policy learning with constraints in model-free reinforcement learning: A survey. In Proc. of IJCAI, 2021.
- Resource scheduling in edge computing: A survey. IEEE Communications Surveys & Tutorials, 23(4), 2021.
- Resource management with deep reinforcement learning. In Proc. of ACM HotNets, New York, NY, USA, 2016.
- Optimal trunk-reservation by policy learning. In Proc. of IEEE INFOCOM, 2019.
- B. L. Miller. A queueing reward system with several customer classes. Management science, 16(3), 1969.
- Achieving predictable and low end-to-end latency for a network of smart services. In Proc. of IEEE GLOBECOM, 2018.
- Reinventing video streaming for distributed vision analytics. In Proc. of USENIX HotCloud, 2018.
- Constrained reinforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019.
- M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
- Reinforcement learning-based admission control in delay-sensitive service systems. In Proc. of IEEE GLOBECOM, 2020.
- S. J. Russell and A. Zimdars. Q-decomposition for reinforcement learning agents. In Proc. of ICML, 2003.
- Call admission control in cellular networks: a reinforcement learning solution. International journal of network management, 14(2), 2004.
- Marina: Realizing ml-driven real-time network traffic monitoring at terabit scale. IEEE Transactions on Network and Service Management, 2024.
- Dynamic preemption call admission control scheme based on Markov decision process in traffic groomed optical networks. Journal of Optical Communications and Networking, 3(4), 2011.
- Kerveros: Efficient and scalable cloud admission control. In Proc. of USENIX OSDI, 2023.
- Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018.
- Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proc. of AAMAS, 2011.
- C. Szepesvari. Algorithms for reinforcement learning. Number 9 in Synthesis lectures on artificial intelligence and machine learning. Morgan & Claypool, 2010.
- Reward constrained policy optimization. In Proc. of ICLR, 2019.
- Hybrid reward architecture for reinforcement learning. In Proc. of NIPS, 2017.
- Retina: analyzing 100gbe traffic on commodity hardware. In Proc. of the ACM SIGCOMM, pages 530–544, 2022.
- Joint configuration adaptation and bandwidth allocation for edge-based real-time video analytics. In Proc. of IEEE INFOCOM, 2020.
- Bandwidth-efficient live video analytics for drones via edge computing. In Proc of IEEE/ACM SEC, 2018.
- Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection. IEEE Access, 6:1792–1806, 2017.
- C. J. Watkins and P. Dayan. Q-learning. Machine Learning, 8(3/4), 1992.
- Live video analytics at scale with approximation and {{\{{Delay-Tolerance}}\}}. In Proc. of USENIX NSDI, 2017.
- Hetero-edge: Orchestration of real-time vision applications on heterogeneous edge clouds. In Proc. of IEEE INFOCOM, 2019.
- Decomposable intelligence on cloud-edge IoT framework for live video analytics. IEEE Internet of Things Journal, 7(9), 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.