Workshops Archive

Exploring Reduced Precision for Deep Learning Activation Functions

Workshop: 9th International Workshop on Software Correctness for HPC Applications (Correctness '25)

Authors: Epifanio Sarinana (The University of Texas at El Paso) and Christoph Lauter and Shirley Moore (University of Texas at El Paso)

Abstract: The increasing scale of deep neural networks has heightened the need to optimize training and inference efficiency. Reduced-precision computation has emerged as a promising approach to improve memory usage, energy efficiency, and computational throughput. While formats such as FP16 and FP8 are increasingly supported by modern hardware for tensor operations, ultra-low-precision formats like FP4 and FP2 remain largely unexplored for non-linear activation functions, which play a critical role in model convergence and stability. In this work, we introduce a PyTorch framework to emulate and train models where activation functions are computed using FP8, FP6, FP4, FP3, and FP2 representations throughout the entire training process. Through comprehensive experiments across multiple models and datasets, we evaluate the feasibility of training neural networks when activation functions operate at low precision and identify those that maintain accuracy despite the noise introduced by quantization.

Back to 9th International Workshop on Software Correctness for HPC Applications (Correctness '25) Archive Listing Back to Full Workshop Archive Listing