Workshops Archive

Compression Error Sensitivity Analysis for Different Experts in MoE Model Inference

Workshop: The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data

Authors: Songkai Ma (Hong Kong Polytechnic University); Zhaorui Zhang (The Hong Kong Polytechnic University); Sheng Di (Argonne National Laboratory (ANL)); Benben Liu (The University of Hong Kong); Xiaodong Yu (Stevens Institute of Technology); Xiaoyi Lu (University of California, Merced); and Dan Wang (Hong Kong Polytechnic University)

Abstract: With the widespread application of Mixture of Experts (MoE) reasoning models in the field of LLM learning, efficiently serving MoE models under limited GPU memory constraints has emerged as a significant challenge. Offloading the non-activated experts to main memory has been identified as an efficient approach to address such a problem, while it brings the challenges of transferring the expert between the GPU memory and main memory. We need to explore an efficient approach to compress the expert and analyze how the compression error affects the inference performance.

To bridge this gap, we propose employing error-bounded lossy compression algorithms (such as SZ3 and CuSZp) to compress non-activated experts, thereby reducing data transfer overhead during MoE inference. We conduct extensive experiments across various benchmarks and present a comprehensive analysis of how compression-induced errors in different experts affect overall inference accuracy.

Back to The 11th International Workshop on Data Analysis and Reduction for Big Scientific Data Archive Listing Back to Full Workshop Archive Listing