The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: WORKS 2025: 20th Workshop on Workflows in Support of Large-Scale Science
Authors: Dante D. Sanchez-Gallegos (Universidad Carlos III de Madrid), J. L. Gonzalez-Compean (Cinvestav Tamaulipas), Jesus Carretero (Universidad Carlos III de Madrid), and Raffaele Montella (University of Naples Parthenope)
Abstract: The computing continuum has emerged as a promising paradigm for decentralized data processing. This approach brings computation closer to data sources, reducing latency and enabling faster insights. However, managing such distributed systems introduces new challenges, particularly in ensuring the availability and reliability of data across heterogeneous and failure-prone environments. In this paper, we focus on addressing these challenges by introducing DagOnStore as a novel component of the DAGonStar workflow engine, integrating it with the DynoStore wide-area storage system to provide resilient and location-transparent data access. DagOnStore implements reliability and availability schemes based on erasure codes and utilization-aware load-balancing to guarantee that input and output data remain accessible and consistent, even in the presence of storage node failures or disconnections. We validate our approach through different tests, demonstrating that DagOnStore enables scalable and fault-tolerant workflow execution across the computing continuum with minimal user intervention.
Back to WORKS 2025: 20th Workshop on Workflows in Support of Large-Scale Science Archive Listing Back to Full Workshop Archive Listing