The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Bjoern Enders (Lawrence Berkeley National Laboratory (LBNL), National Energy Research Scientific Computing Center (NERSC)), Rafael Ferreira da Silva (Oak Ridge National Laboratory (ORNL)), Sam Welborn (Lawrence Berkeley National Laboratory (LBNL), National Energy Research Scientific Computing Center (NERSC)), Alex Upton (Swiss National Supercomputing Centre (CSCS)), Adam Thompson (NVIDIA Corporation), Fred Suter (Oak Ridge National Laboratory (ORNL)), Matthieu Dorier (Argonne National Laboratory (ANL))
Abstract: Emerging scientific needs are driving a new class of workflows that require near real-time processing, urgent computing, and time-sensitive decision-making. These workflows must bypass traditional buffering and shared file systems, instead relying on direct data streaming over WAN into HPC compute nodes. Latency and variability must be minimized to enable timely responses and experiment steering. This BoF will explore strategies, technologies, and policies to support these streaming workflows at scale, with a focus on building shared infrastructure and community practices to routinely enable this next generation of high-impact, time-critical scientific computing.
Long Description: Today, scientific data typically arrives at HPC facilities via gateway nodes into shared file systems, a model supported by tools like Globus and suited for workflows that tolerate queue delays. However, as HPC, experimental, and observational facilities become more integrated, time-sensitive workflows increasingly require real-time feedback, where latency and variability from shared storage are unacceptable. Despite early examples of streaming workflows, most facilities remain unprepared to support them due to policy, scheduling, or hardware limitations. Addressing these gaps is critical to enabling seamless, compute-in-the-loop workflows that define the next era of scientific discovery. This BoF addresses this groundbreaking and growing field in supercomputing.
SC25 is an ideal venue for this BoF as it brings the major HPC user facility providers together with scientific computing users who already run a realtime, streaming computing workflow or expect that need to arise in the near future. This provides an excellent opportunity to screen for current solutions, and discuss limitations in HPC, data, and network providers.
This BoF is the third of its kind (and 2nd at SC) and has consistently drawn a significant crowd, 85 at SC`24 and 75 at ISC`25, despite being a new BoF. We also managed to form a community that we leveraged this time to rotate in a new list of speakers for workflows and software across the HPC community. We also noticed that due to the novelty and complexity of the topic, the audience appreciates if the presentations are a little longer than lightning talks in order to engage in a more meaningful discussion and notice parallels with their own priorities. This year we aim for a series of 8-10 minute presentations drawn from our community ensued by in-depth discussions where the audience can ask questions to presenters, or can bring their specific streaming workflow to the attention of the workflow community.
The final section will be a very interactive part that we call a "Workflow Challenge Workshop".
In this workshop, participants will be divided into small groups representing different stakeholders (facility operators, users, software developers). Each group will receive a real streaming workflow scenario and work together to identify technical, policy, and infrastructure requirements for implementation. Groups will document their solutions using a shared digital whiteboard, allowing cross-group visibility and iteration.
As we have up to five speakers and the workflow challenge, we aim for a 90 minute BoF to cover everything. The responses have been strong so far and we intend to give our community more space. But we are also able to scale down the BoF to 60 minutes if needed. As outcomes we want to (1) further grow our community for streaming-type workflows; and (2) organize a full day workshop about streaming hosted at a DOE user facility in early 2026.
Website: https://workflows.community/bof/sc25-streaming