Modern neural networks not only achieve impressive accuracy but also reveal surprisingly simple geometric structures in their learned representations. One such phenomenon, called neural collapse, emerges once a network has effectively reached zero training error: the network’s internal representations align into elegant, low-dimensional patterns.
In this project, we will study how this phenomenon interacts with quantization, a widely used technique for compressing neural networks by representing weights and activations with low-bit precision. Quantization is not only practical for deploying resource-efficient models but also introduces a natural form of regularization that may reshape the geometry of learned representations. Through this project, you will experimentally investigate how quantized networks behave in the post–zero-loss regime, and you will use geometric and combinatorial tools to understand the structure of both neural collapse and quantization. The work offers a unique opportunity to explore a cutting-edge topic at the intersection of deep learning theory, geometry, and efficient AI models.
Hivatkozások
[1] V. Papyan, X.Y. Han, and D.L. Donoho: Prevalence of neural collapse during the terminal phase of deep learning training, Proc. Natl. Acad. Sci. U.S.A. 117 (40) 24652-24663, https://doi.org/10.1073/pnas.2015509117 (2020)
[2] Vignesh Kothapalli: Neural Collapse: A Review on Modelling Principles and Generalization, https://arxiv.org/abs/2206.04041 (2022)
[3] Ashkboos, Saleh, et al. : EFQAT: An Efficient Framework for Quantization-Aware Training (Sections 1 and 2) https://arxiv.org/abs/2411.11038 (2024)