The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data
Authors: Skyler Ruiter (Indiana University), Jiannan Tian (Oakland University), and Fengguang Song (Indiana University)
Abstract: Modern scientific simulations and instruments generate data volumes that overwhelm memory and storage, throttling scalability. Lossy compression mitigates this by trading controlled error for reduced footprint and throughput gains, yet optimal pipelines are highly data and objective specific, demanding compression expertise. GPU compressors supply raw throughput but often hard‑code fused kernels that hinder rapid experimentation, and underperform in rate–distortion. We present FZModules, a heterogeneous framework for assembling error‑bounded custom compression pipelines from high‑performance modules through a concise extensible interface. We further utilize an asynchronous task-backed execution library that infers data dependencies, manages memory movement, and exposes branch and stage level concurrency for powerful asynchronous compression pipelines. Evaluating three pipelines built with FZModules on four representative scientific datasets, we show they can compare end‑to‑end speedup of fused‑kernel GPU compressors while achieving similar rate–distortion to higher fidelity CPU or hybrid compressors, enabling rapid, domain-tailored design.
Back to The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data Archive Listing Back to Full Workshop Archive Listing