The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Bjoern Enders (Lawrence Berkeley National Laboratory (LBNL), National Energy Research Scientific Computing Center (NERSC)), John MacAuley (Energy Sciences Network (ESnet)), Paul Rich (Argonne National Laboratory (ANL)), Juan Pablo Dorsch (Swiss National Supercomputing Centre (CSCS)), Ilya Baldin (Jefferson Lab), Tiffany Connors (Lawrence Berkeley National Laboratory (LBNL), National Energy Research Scientific Computing Center (NERSC)), Frank Wuerthwein (University of California, San Diego (UCSD)), Eli Dart (Energy Sciences Network (ESnet))
Abstract: DOE has recently launched the Integrated Research Infrastructure (IRI) program, which is designed to enable new modes of integrated science across DOE user facilities. Common or unified interfaces are needed for these workflows to seamlessly orchestrate resources across high performance computing, data, and network providers.
The IRI Interfaces working group, composed of members of ASCR user facilities, has spent the last year designing and implementing APIs for compute facilities. Here we’d like to present our current development status, deployed prototypes, and future roadmap for discussion. We are seeking feedback from the user community to help guide these IRI efforts.
Long Description: The Integrated Research Infrastructure (IRI) program stands out with its innovative strategy to more effectively enable science across the DOE user facilities at scale. It is set to radically accelerate discovery and innovation within the DOE with a unique approach that empowers scientists to seamlessly combine DOE research tools, infrastructure, and user facilities into their orchestrated workflows. A key component of this approach is the introduction of collaborative interfaces for users and orchestration tools, enabling workflows that run seamlessly across multiple user facilities.
In recent years, user facilities like NERSC, CSCS, TACC, ESnet, and OLCF have introduced novel interfaces, often with the broader DOE user community in mind. However, these are locally developed, limited by facility-specific technologies (e.g., HPC hardware), and lack solutions for multi-facility job resiliency, failover, and monitoring. Today, despite the use of common toolchains and similar HPC providers, the landscape of novel HPC interfaces is still rather fragmented, with no common interface emerging that functions across all user facilities. This is a major hurdle for scientists aiming to build resilient workflows with multiple facilities in mind, locking them into one specific computing and data environment and exposing them to risks like major interruptions. Hence, the success of IRI hinges on the delivery of collaborative interfaces that scientists can use to build resilient and performant cross-facility workflows.
The IRI interfaces technical subcommittee is formed by key staff across ASCR with a charter to identify specific interface design, functionality, and compatibility requirements. For this endeavor to be successful, deep engagement with a broad spectrum of stakeholders is necessary to ensure that the developed interfaces cater to a wide range of use cases. The IRI interfaces technical subcommittee has been at work for a year now, designing interfaces and building prototypes. We are looking to engage with the community again to show our work and receive feedback.
This BoF builds on the success of last year's "IRI: What novel interfaces will HPC expose for cross-facility workflows?" BoF. The significant attendance (around 70) at the previous session clearly indicated strong community interest. In that BoF we gave a review of current API/workflow initiatives and engaged with the community as to what their priority is. We received quite a bit of questions in regard to authentication/authorization and job submission so this year we will give that topic ample time at the end of the BoF with specialists on the panel. This discussion will align perfectly with our roadmap which contains authenticated endpoints and job submission starting at the end of the year. We expect to gain valuable insights on how to design these interfaces for our user community and will factor these insights into our development process.
SC25 is an ideal venue for this BoF as it brings the major ASCR user facility providers together with scientific computing users. This provides an opportunity to discuss the existing interfaces of several international HPC centers (both within and outside the DOE), stimulate conversation on implementation and adoption, and steer future efforts.
Website: https://sites.google.com/lbl.gov/sc25-bof-iri-interfaces/home