Workshops Archive

StreamHub: High-performance Managed SciStream as a Service

Workshop: The 12th Annual International Workshop on Innovating the Network for Data-Intensive Science (INDIS)

Authors: Seena Vazifedunn (University of Chicago); Flavio Castro (Argonne National Laboratory (ANL)); Rajkumar Kettimuthu (Argonne National Laboratory (ANL), University of Chicago); and Ian Foster and Kyle Chard (University of Chicago, Argonne National Laboratory (ANL))

Abstract: Scientific data streaming enables the real-time transfer, processing, and analysis of high-throughput experimental data, providing low-latency insights that are essential for adaptive experiment control. We present SciStream-as-a-Service (StreamHub), a secure, high-performance, and scalable framework that integrates Globus Compute with SciStream to deliver continuous memory-to-memory data transfer, in-transit processing, and robust zero-trust security while orchestrating the entire streaming setup across distributed facilities with only approximately 4.3s overhead. We describe StreamHubs design, core components, and deployment model, and evaluate its performance under diverse configurations. Our results demonstrate that StreamHub can be deployed without privileged access, ensures end-to-end encryption and authentication across institutions, and achieves near line-rate throughput with under 2% overhead compared to unencrypted transfers, making it a practical solution for real-time scientific discovery and experiment steering.

Back to The 12th Annual International Workshop on Innovating the Network for Data-Intensive Science (INDIS) Archive Listing Back to Full Workshop Archive Listing