Workshops Archive

Invited Talk 1: Memorization vs Reasoning in MoEs and Estimating Memory Consumption in Distributed Training

Workshop: ExHetAI: Extreme Heterogeneity and AI Convergence in HPC

Authors: Rio Yokota (Institute of Science Tokyo)

Abstract: Mixture of Experts (MoE) allows one to increase the model capacity with minimal training/inference cost. Recent LLMs such as Qwen3-235B-A22B, gpt-oss-120B, Kimi-K2, GLM-4.5, DeepSeek-R1 are very sparse MoEs, though there are some subtle differences in the details of the architecture. The first part of this talk will focus on our recent efforts to measure the effect of sparsity on memorization tasks and reasoning tasks. We initially find that increasing the total parameters without increasing the active parameters increases the performance on memorization tasks but shows an inverse scaling on reasoning tasks. However, when the dataset is carefully constructed we show that the inverse scaling on reasoning tasks disappears. The second part of this talk will describe a tool to estimate the memory consumption during distributed training, and its effectiveness when trying to maximize per GPU Flop/s on a given system.

Back to ExHetAI: Extreme Heterogeneity and AI Convergence in HPC Archive Listing Back to Full Workshop Archive Listing