Workshops Archive

Efficient Embedding Initialization via Dominant Eigenvector Projections

Workshop: 16th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH'25)

Authors: Quentin Petit (Mines Paris - PSL University France); Chong Li (Huawei Technologies France); Nahid Emad (Maison de la Simulation, University of Paris-Saclay); and Jack Dongarra (University of Tennessee, Knoxville)

Abstract: The embedding layer is essential in deep learning, transforming high-dimensional data into compact representations. However, growing datasets and model sizes pose challenges in training time, memory, and generalization. We propose a scalable method for embedding initialization via spectral dimensionality reduction using dominant eigenvector projections.

The proposed approach leverages on MIRAMns, multiple implicitly restarted Arnoldi method with nested subspaces, to extract most informative directions from large and potentially sparse data representations. Unlike traditional embeddings or autoencoders, this proposed approach requires few tunable parameters and is inherently parallel. We apply MIRAMns to matrix representations such as covariance and co-occurrence matrices to compute low-dimensional embeddings that preserve data structure and variance. Experiments across diverse datasets show that the proposed method achieves comparable or better accuracy with significantly reduced dimensionality, enabling smaller, faster deep networks. Additionally, our parallel implementation scales efficiently on HPC platforms, making it well-suited for large-scale scientific and AI workloads.

Back to 16th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH'25) Archive Listing Back to Full Workshop Archive Listing