Workshops Archive

Seamless End-to-End Containerized HPC Environments

Workshop: 7th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)

Authors: Brandon Cook, Shane Canon, Adam Lavely, and Daniel Margala (Lawrence Berkeley National Laboratory (LBNL))

Abstract: High-performance computing environments face increasing challenges from diverse scientific workflows, imposing conflicting demands for stability, customization, and reproducibility that traditional monolithic software stacks cannot accommodate. We present a comprehensive approach to seamless end-to-end containerized HPC environments which decomposes the technical challenge into five manageable areas: specification and construction of environments, session provisioning, scheduler integration, system integration, and security. We develop and evaluate prototypes across these five technical areas, demonstrating practical feasibility through Spack-based environment construction with CI/CD pipelines, transparent session access via PAM and Kubernetes, and flexible job execution using Slurm's native container support. % Our system integration framework supports multiple MPI strategies from host library injection to fully containerized stacks, accommodating diverse performance and portability requirements. Through this work, we demonstrate that comprehensive containerization of HPC environments can be achieved using open standards, providing enhanced reproducibility and flexibility without sacrificing user experience.

Back to 7th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Archive Listing Back to Full Workshop Archive Listing