Birds of a Feather Archive

Integrated HPC: Lessons Learned From Multi-Facility Integration Projects Around the World

Authors: Deborah Bard (National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory (LBNL)), Frank Würthwein (San Diego Supercomputer Center (SDSC); University of California, San Diego (UCSD)), Richard Gunn (UKRI), Seppo Sahrakorpi (Digital Research Alliance of Canada), Vangelis Floros (EuroHPC Joint Undertaking)

Abstract: Science and computing are increasingly integrated, as large-scale scientific and computing challenges take on a national, and even international, scale. The need for resources at multiple facilities may be driven by access to site-specific hardware, security policy, or to ensure resilient operations. Deeper integration between facilities can create efficiencies for scientists, funding agencies, and the facilities themselves, but also exposes site incompatibilities in both technology and culture.

In this BoF we bring together seasoned experts in integrating supercomputing resources across institutions to discuss the challenges and opportunities of creating and managing the frameworks (political and technical) needed to integrate HPC.

Long Description: Goals: Bring together people who have experience in building, operating and managing HPC infrastructure across institutions Identify commonalities of experience - what was hard? What was easy? Identify some best practices for how to set up cross-institutional HPC infrastructure frameworks Lessons learned: what do participants wish they’d known earlier? Form the basis of a Community of Practice around Integrated HPC

Topics: How do we make the case for integration? How do we persuade stakeholders and scientists of the need for more integrated HPC? What are the models for managing a digital research infrastructure? Centralized vs distributed vs… Which institutional boundaries are sacrosanct? Where does integration stop? What are the major policy challenges? How can they be successfully addressed? What surprises have we encountered along the way?

Scientists increasingly need access to large computing and storage resources across multiple experiment and HPC sites, connected by high-performance networks. As a result, HPC facilities are increasingly called upon to be part of large initiatives that cross institutional boundaries. Examples include DOE’s Integrated Research Infrastructure, the Open Science Grid, Canada’s Alliance (Digital Research Alliance of Canada), the UK’s Iris (digital research infrastructure to support STFC science) and Europe’s EuroHPC. There are many technical challenges associated with building a truly integrated multi-institutional digital ecosystem - but some of the most interesting and difficult challenges are not technical, but political and sociological. In this BOF we will discuss what it takes to build and manage integrated infrastructure for HPC – including computing, storage, and networking – across multiple sites. We will focus on the unique opportunities and challenges brought by integrating highly specialized and independent supercomputing centers.

This BOF has never been held before - and to our knowledge, there has never been a gathering of global experts with experience in cross-institutional HPC infrastructure. This discussion will be of interest to HPC facilities, funding agencies and other organisations that see the need to integrate their resources across institutional boundaries, and could profit from the lessons learned by others. It will also be an opportunity for HPC facility users to learn about the pressures and challenges - and opportunities - around cross-facility integration.

We have five confirmed participants for the panel discussion, representing some of the most exciting infrastructure projects around the world - both established and emerging: The DOE’s Integrated Research Infrastructure program (Debbie Bard) Europe’s EuroHPC consortium (Vangelis Floros) The UK’s Iris program of digital research infrastructure (Richard Gunn) Canada’s Digital Research Alliance of Canada (Seppo Sahrakorpi) The US Open Science Grid (Frank Wuerthwein)

Back to Birds of a Feather Archive Listing