The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Tim Osborne (Oak Ridge National Laboratory (ORNL)), Jeff Hanson (Hewlett Packard Enterprise (HPE)), Melissa Romanus (ESnet; Lawrence Berkeley National Laboratory (LBNL)), Michael Ott (Leibniz Supercomputing Centre (LRZ)), Woong Shin (Oak Ridge National Laboratory (ORNL)), Terry Jones (Oak Ridge National Laboratory (ORNL)), Wolfgang Frings (Forschungszentrum Jülich)
Abstract: Operational data analytics (ODA) provides unique opportunities to analyze, understand, and optimize operations of HPC systems. However, those opportunities are often missed because access to data is restricted, siloed, or misaligned with the people best positioned to act on it. System administrators, researchers, and HPC users could combine their skills to optimize efficiency of HPC systems if data and expertise were shared between all parties.
How can we bridge these gaps and make more operational data available to more stakeholders in an easily digestible way while still maintaining operational safety, privacy, and legal requirements? What data is relevant to whom?
Long Description: Most sites that operate HPC systems are engaged in Operational Data Analytics (ODA) one way or another. Some may only be monitoring their HPC system for faults or emergencies while others try to collect as much data as possible from their HPC operations, covering the whole data center with its supporting infrastructure, the system hardware and software, and the applications running on the system. The potential for using this data for optimizing system operations is tremendous, but many sites are struggling to fully leverage it because they lack the tools, capabilities or human resources to do so. At the same time, other interested stakeholders who could potentially help with this task are struggling to get access to this data: researchers in data science have the right skillset and also great interest in analyzing such data but many HPC operators are reluctant to make operational data available to them out of data security or privacy concerns. Users of HPC systems have great interest in the performance of the jobs they are submitting and could leverage their domain expertise to identify opportunities for optimization if they were given easy to use tools and operational data that goes beyond the wall clock time of their jobs.
Some HPC sites have successfully addressed data security and privacy requirements and made operational data available to the public or associated researchers. Others have developed web-based tools to provide their users with insights into the operational performance of their jobs and to highlight potential for optimization. This BoF will be a unique opportunity to share their experiences and lessons learned and also for the “underprivileged” consumers of operational data to share their expectations and wishes.
This BoF builds on previously held BoFs at SC and ISC and is being organized by members of the ODA team within the Energy Efficient HPC Working Group (EEHPCWG). The ODA team comprises practitioners in HPC operations who deal with monitoring and data analytics on a daily basis. This BoF covers several challenging topics in the frontier of ODA every year, starting from ODA tools, frameworks and building blocks, data analysis methods, approaches, AI/ML application, and data standardization. This BoF establishes a larger community that is backed by the EEHPCWG, fosters collaboration among different HPC sites, and drives further discussion towards best practices while advancing the field.