A pipeline for Megahertz X-ray Photon Correlation Spectroscopy on soft matter samples at the MID instrument of European XFEL

Abstract: In this paper we present the experimental protocol and data processing framework for Megahertz X-ray Photon Correlation Spectroscopy (MHz-XPCS) experiments on soft matter samples, implemented at the Materials Imaging and Dynamics (MID) instrument of the European X-ray Free-Electron Laser (EuXFEL). Due to the introduction of a standard configuration and the implementation of a highly automated data processing pipeline, MHz-XPCS measurements can now be conducted and analyzed with minimal user intervention. A key challenge lies in managing the extremely large data volumes generated by the Adaptive Gain Integrating Pixel Detector (AGIPD) - often reaching several petabytes within a single experiment. We describe the technical implementation, discuss the hardware requirements related to effective parallel data processing, and propose strategies to enhance data quality, in particular related to data reduction strategies and an improvement of the signal-to-noise ratio. Finally, we address strategies for making the processed data FAIR (Findable, Accessible, Interoperable, Reusable), in alignment with the goals of the DAPHNE4NFDI project.