The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data
Authors: Kuan-Chieh Hsu, Sairam Sri Vatsavai, Ozgur O. Kilic, Sankha Dutta, Yihui (Ray) Ren, and David Park (Brookhaven National Laboratory); Tania Korchuganova and Joseph Boudreau (University of Pittsburgh); Tasnuva Chowdhury (Brookhaven National Laboratory); Shengyu Feng (Carnegie Mellon University); Raees Ahmad Khan (University of Pittsburgh); Jaehyung Kim (Carnegie Mellon University); Norbert Podhorszki and Scott Klasky (Oak Ridge National Laboratory (ORNL)); Tadashi Maeno, Paul Nilsson, and Verena Ingrid Martinez Outschoorn (University of Pittsburgh); Fred Suter (Oak Ridge National Laboratory (ORNL)); Wei Yang (SLAC National Accelerator Laboratory); Yiming Yang (Carnegie Mellon University); and Shinjae Yoo, Alexei Klimentov, and Adolfy Hoisie (Brookhaven National Laboratory)
Abstract: Large-scale international collaborations such as ATLAS rely on globally distributed workflows and data management to process, move, and store vast volumes of data. ATLAS’s Production and Distributed Analysis (PanDA) workflow system and the Rucio data management system are each highly optimized for their respective design goals. However, operating them together at global scale exposes systemic inefficiencies, including underutilized resources, redundant or unnecessary transfers, and altered error distributions. Moreover, PanDA and Rucio currently lack shared performance awareness and coordinated, adaptive strategies.
This work charts a path toward co-optimizing the two systems by diagnosing data-management pitfalls and prioritizing end-to-end improvements. With the observation of spatially and temporally imbalanced transfer activities, we develop a metadata-matching algorithm that links PanDA jobs and Rucio datasets at the file level, yielding a complete, fine-grained view of data access and movement. Using this linkage, we identify anomalous transfer patterns that violate PanDA’s data-centric job-allocation principle.
Back to The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data Archive Listing Back to Full Workshop Archive Listing