Workshops Archive

State Machine Orchestration of an HPC Workflow in Cloud

Workshop: WORKS 2025: 20th Workshop on Workflows in Support of Large-Scale Science

Authors: Vanessa Sochat, Loïc Pottier, and Daniel Milroy (Lawrence Livermore National Laboratory (LLNL))

Abstract: The High Performance Computing (HPC) community is facing a period of change, where access to resources is uncertain and workflows must move between available environments. The economic and innovative power of cloud presents opportunity by offering state-of-the-art orchestration frameworks like Kubernetes. However, the existence of environments does not guarantee access to them, and porting HPC applications to cloud is non-trivial. In this work, we redesign the orchestration of an ensemble-based workflow – the Multiscale Machine-Learned Modeling Infrastructure (MuMMI) for Kubernetes. We perform experiments representing a progression from a traditional MuMMI run on HPC to a fully portable variant running in cloud to assess the relative contributions of cloud-native features to workflow performance improvement. Moving from a traditional design based on service and filesystem components to an event-driven design we demonstrate 62.24% and 40.29% faster workflow completion times for CPU and GPU setups, respectively, resulting in 45.0% and 38.3% lower costs.

Back to WORKS 2025: 20th Workshop on Workflows in Support of Large-Scale Science Archive Listing Back to Full Workshop Archive Listing