Research and ACM SRC Posters Archive

Productive Scalable Distributed Task Scheduling Using an MPI-based Backend for Dagger

Poster Type: ACM Student Research Competition, Undergraduate

Author: Yan Guimarães (University of Brasilia)

Supervisor: Felipe Tomé (Massachusetts Institute of Technology (MIT))

Abstract: Distributed computing frameworks are vital for managing complex workloads in high-performance computing and scientific research. Julia's Dagger.jl supports task-based parallelism using TCP communication, suitable for cloud and local environments. However, TCP limits performance on modern HPC systems with low-latency, high-bandwidth interconnects. We introduce MPIAcceleration, an MPI-based backend replacing TCP with MPI-aware task placement and data movement. We benchmarked MPIAcceleration against TCP using parallel Cholesky decomposition on the Aurora exascale supercomputer at Argonne National Laboratory. Results show MPI successfully enables Dagger on HPC interconnects, significantly outperforming TCP on Aurora by overcoming the latency limitations inherent in standard TCP.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF

Back to Poster Archive Listing