The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: ISAV25: In Situ AI, Analysis, and Visualization
Authors: Caitlin Ross and Scott Wittenburg (Kitware Inc.), Greg Eisenhauer (Georgia Institute of Technology), and Corey Wetterer-Nelson (Kitware Inc.)
Abstract: Efficient parallel I/O is essential for large-scale scientific workflows, particularly in in situ visualization pipelines where output sizes and data distribution can vary significantly over time. We present a new data-size based aggregation strategy for the ADIOS2 BP5 engine, designed to improve parallel I/O performance under load imbalance. Whereas existing aggregation strategies for ADIOS group writers based on compute node assignment, our approach dynamically balances subfile sizes according to the amount of data each process will write. We evaluate this strategy using a synthetic workload under low and severe load imbalance. Results show that the data-size based aggregation matches or outperforms existing strategies. These findings highlight the potential of adaptive aggregation strategies to improve I/O performance for imbalanced scientific workloads.
Back to ISAV25: In Situ AI, Analysis, and Visualization Archive Listing Back to Full Workshop Archive Listing