Workshops Archive

Weak Scaling of NVSHMEM Applied To Hashed Distributed Structured Data

Workshop: The 8th Annual Parallel Applications Workshop: Alternatives To MPI+X (PAW-ATM 2025)

Authors: Andrew Davis (University of Virginia), Hans Johansen (Lawrence Berkeley National Laboratory), and Xinfeng Gao and Stephen Guzik (University of Virginia)

Abstract: We present a parallel, hash-based software library, HashBrick, for sparse, block-structured applications on CPUs and NVIDIA GPUs. We use a brick-based layout, where data is aggregated into small regular bricks, and hashes hide the complexity of neighbor indexing. This exposes an extra level of flexibility for irregular data and avoids packing communication buffers for ghost zones. One-sided NVSHMEM is used for GPU-GPU communication of an irregular distribution of ghost bricks. Weak scaling experiments using a high-order CFD application and a Jacobi iteration benchmark were run on a NVIDIA GH200 cluster. For constant problem size per node, the computation time is constant, and communication scales well despite load imbalance. We find that the variation in the distribution of ghost bricks more or less correlates with scaling efficiency. Our results show that MPI and NVSHMEM have similar scaling, but MPI wall-clock times are 64-84% higher for these experiments.

Back to The 8th Annual Parallel Applications Workshop: Alternatives To MPI+X (PAW-ATM 2025) Archive Listing Back to Full Workshop Archive Listing