2000 character limit reached
Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e
Published 28 Dec 2023 in cs.LG, cs.AI, and physics.acc-ph | (2312.17372v1)
Abstract: We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab). Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale. We treat the Mu2e accelerator system as a Markov Decision Process suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and enhance training stability. A key innovation in our approach is the integration of a neuralized Proportional-Integral-Derivative (PID) controller into the policy function, resulting in a significant improvement in the Spill Duty Factor (SDF) by 13.6%, surpassing the performance of the current PID controller baseline by an additional 1.6%. This paper presents the preliminary offline results based on a differentiable simulator of the Mu2e accelerator. It paves the groundwork for real-time implementations and applications, representing a crucial step towards automated proton beam intensity control for the Mu2e experiment.
- Edge ai for accelerator controls (reads): beam loss deblending. Technical report, Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States), 2023.
- Mu2e technical design report, 2015.
- Synchronous High-Frequency Distributed Readout for Edge Processing at the Fermilab Main Injector and Recycler. JACoW Publishing, Geneva, Switzerland, 2022. URL https://napac2022.vrws.de/papers/mopa15.pdf. presented at NAPAC’22, Albuquerque, New Mexico, USA, Aug. 2022, paper MOPA15, unpublished.
- Robert H Bernstein. The mu2e experiment. Frontiers in Physics, 7:1, 2019.
- Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.
- Department of Energy, Office of Science. Data, Artificial Intelligence, and Machine Learning at DOE Scientific User Facilities, 2020. URL https://science.osti.gov/-/media/grants/pdf/lab-announcements/2020/LAB_20-2261.pdf.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801.01290, 2018. URL http://arxiv.org/abs/1801.01290.
- K.J. Hazelwood. Disentangling beam losses in the fermilab main injector enclosure using real-time edge ai, 10 2023. presented at The 19th Biennial International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS) 2023, Cape Town, South Africa.
- K.J. Hazelwood et al. Real-Time Edge AI for Distributed Systems (READS): Progress on Beam Loss De-Blending for the Fermilab Main Injector and Recycler. In Proc. IPAC’21, number 12 in International Particle Accelerator Conference, pages 912–915. JACoW Publishing, Geneva, Switzerland, 08 2021a. ISBN 978-3-95450-214-1. doi: 10.18429/JACoW-IPAC2021-MOPAB288. URL https://jacow.org/ipac2021/papers/mopab288.pdf.
- Real-time edge ai for distributed systems (reads): Progress on beam loss de-blending for the fermilab main injector and recycler. Technical report, Fermi National Accelerator Lab.(FNAL), Batavia, IL (United States), 2021b.
- On sparse modern hopfield model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2309.12673.
- Preliminary design of mu2e spill regulation system (srs). Technical report, Fermi National Accelerator Lab.(FNAL), Batavia, IL (United States), 2019.
- Fpga architectures for distributed ml systems for real-time beam loss de-blending. Technical report, Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States), 2023.
- Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273–1286, 2021.
- Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112–2120, 2021.
- J. Mitrevski. Edge ai for accelerator controls (reads): Beam loss deblending, 09 2023. URL https://indico.cern.ch/event/1283970/contributions/5550643/attachments/2721973/4729145/READS%20FastML%20v3.pdf. presented at Fast Machine Learning For Science Workshop 2023, London UK.
- A. Narayanan et al. Optimizing Mu2e Spill Regulation System Algorithms. In Proc. IPAC’21, pages 4281–4284. JACoW Publishing, Geneva, Switzerland, 2021a. doi: 10.18429/JACoW-IPAC2021-THPAB243. URL https://jacow.org/ipac2021/papers/THPAB243.pdf.
- A. Narayanan et al. Machine Learning for Slow Spill Regulation in the Fermilab Delivery Ring for Mu2e. JACoW Publishing, Geneva, Switzerland, 2022. URL https://napac2022.vrws.de/papers/mopa75.pdf. presented at NAPAC’22, Albuquerque, New Mexico, USA, Aug. 2022, paper MOPA28, unpublished.
- Optimizing mu2e spill regulation system algorithms. Technical report, Fermi National Accelerator Lab.(FNAL), Batavia, IL (United States), 2021b.
- History compression via language models in reinforcement learning. In International Conference on Machine Learning, pages 17156–17185. PMLR, 2022.
- Stable-baselines3: Reliable reinforcement learning implementations. The Journal of Machine Learning Research, 22(1):12348–12355, 2021.
- Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020.
- Feature programming for multivariate time series prediction, 2023.
- Proximal policy optimization algorithms, 2017.
- Accelerator real-time edge ai for distributed systems (reads) proposal, 2021a.
- Accelerator real-time edge ai for distributed systems (reads) proposal, 2021b. URL https://arxiv.org/abs/2103.03928.
- Ml-based real-time control at the edge: An approach using hls4ml. arXiv preprint arXiv:2311.05716, 2023.
- M. Thieme. Machine learning for slow spill regulation in the fermilab delivery ring, 11 2022. URL https://indico.bnl.gov/event/16158/contributions/69563/attachments/44212/74590/ICFA%20SRS%20Presentation.pdf. presented at third ICFA Beam Dynamics Mini-Workshop on Machine Learning Applications for Particle Accelerators, Chicago, Illinois, USA, Nov. 2022.
- Semantic regression for disentangling beam losses in the fermilab main injector and recycler. Technical report, Fermi National Accelerator Lab.(FNAL), Batavia, IL (United States), 2022.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
- Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770–40803. PMLR, 2023.
- Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023.
- A new pid neural network controller design for nonlinear processes. Journal of Circuits, Systems and Computers, 27(04):1850065, 2018.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.