The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Scot Breitenfeld (The HDF Group), Elena Pourmal (Lifeboat, LLC), Suren Byna (The Ohio State University), Quincey Koziol (NVIDIA Corporation)
Abstract: HDF5 has been a vital HPC I/O library for over 25 years, continually evolving to integrate modern technologies and architectures. This session brings together HDF5 developers and community members to discuss best practices and showcase exciting new features for utilizing HDF5 on today's HPC systems. We will begin with a panel of community experts who will focus on forthcoming, new, and established HDF5 features that represent best practices. Following this, the audience will have the opportunity to share their experiences, insights, and questions, making them an integral part of this collaborative journey to advance HDF5.
Long Description: HDF5 is an essential tool designed to leverage the capabilities of data collections of any kind fully. This open-source, high-performance package integrates an abstract data model, a powerful library, and an efficient file format, all tailored to meet the complex data needs of scientific, engineering, and business environments.
The HDF5 community is a vibrant ecosystem that encompasses diverse fields, and hence, HDF5 is considered a core software component by all major HPC system vendors. With over 1,000 projects on GitHub utilizing HDF5, its versatility is evident. Widely used by scientific and industrial applications, HDF5 enhances I/O performance and improves data organization within HPC settings. Its self-describing data model effectively represents complex data objects, their relationships, and associated metadata, allowing data not only to be stored but also to carry its description—a vital feature for researchers, users, and developers alike. In today's fast-evolving technological landscape, machine learning (ML) is transforming computing, and HPC is no exception. The increasing share of HPC workloads dedicated to ML brings unique I/O patterns and challenges that traditional HPC codes cannot adequately address. This is precisely where HDF5 comes in, evolving to manage these new I/O demands and workflows effectively.
Organizations like The HDF Group, Lifeboat LLC, The Ohio State University, and NVIDIA are actively engaged in a community dedicated to ensuring HDF5 adapts to future I/O challenges. We invite the SC25 attendees to join this BoF, designed for HDF5 enthusiasts from all backgrounds—whether they're experienced developers working in the exascale domain, experimental scientists, or individuals exploring HDF5 for scientific HPC and ML applications. This session is a platform for everyone, and we value the diverse perspectives and experiences that attendees bring. Together, we will discuss the solutions and challenges users face in both classical computational HPC and AI/ML contexts, and their feedback will play an essential role in shaping the future of HDF5.
The session will begin with an overview of the HDF5 roadmap, setting the stage for an interactive panel featuring HDF5 community developers and users. They will present their work and share how HDF5 is utilized in research, for example, in complex workflows, along with new and innovative features such as GPU I/O, threadsafety, and security. Following the panel discussion, we invite the attendees to share their experiences, contributing to the conversation about HDF5's role in their workflows.
This session is a platform for users' voices to be heard and their experiences to be shared. As we gather insights and requirements, we want to emphasize the importance of users' feedback. We encourage them to share suggestions on how HDF5 can be improved to better suit their needs in upcoming releases. Their contributions are invaluable, helping to strengthen our interconnected community and ensuring HDF5 remains a robust tool for managing data in the evolving I/O landscape. Don't miss this opportunity to learn and engage in an insightful conversation that will help shape the future of data management.