Neural Collapse in Quantised Neural Networks

Témavezető:	Lukács András
	ELTE Matematikai Intézet, MI Kutatócsoport
email:	andras.lukacs@ttk.elte.hu

Projekt leírás

Modern neural networks not only achieve impressive accuracy but also reveal surprisingly simple geometric structures in their learned representations. One such phenomenon, called neural collapse, emerges once a network has effectively reached zero training error: the network’s internal representations align into elegant, low-dimensional patterns. In this project, we will study how this phenomenon interacts with quantization, a widely used technique for compressing neural networks by representing weights and activations with low-bit precision. Quantization is not only practical for deploying resource-efficient models but also introduces a natural form of regularization that may reshape the geometry of learned representations. Through this project, you will experimentally investigate how quantized networks behave in the post–zero-loss regime, and you will use geometric and combinatorial tools to better understand the structure of both neural collapse and quantization. The work offers a unique opportunity to explore a cutting-edge topic at the intersection of deep learning theory, geometry, and efficient AI models.
A joint supervision with Rainie Heck (ELTE Institute of Mathematics, AI Research Group).

Előfeltételek

Good programming knowledge in Python and some familiarity with PyTorch, or a strong motivation to learn these skills.
Ability to read mathematics and machine/deep learning papers in English.

Hivatkozások

[1] V. Papyan, X.Y. Han, and D.L. Donoho: Prevalence of neural collapse during the terminal phase of deep learning training, Proc. Natl. Acad. Sci. U.S.A. 117 (40) 24652-24663, https://doi.org/10.1073/pnas.2015509117 (2020)
[2] Vignesh Kothapalli: Neural Collapse: A Review on Modelling Principles and Generalization, https://arxiv.org/abs/2206.04041 (2022)
[3] Ashkboos, Saleh, et al. : EFQAT: An Efficient Framework for Quantization-Aware Training (Sections 1 and 2) https://arxiv.org/abs/2411.11038 (2024)