Universality of Gradient Descent Neural Network Training

There's something almost mystical about the concept of universality. In physics, it's the phenomenon where completely different microscopic systems—atoms, spins, even fluids—collapse onto the exact same mathematical behavior at critical transitions. The boiling of water and the magnetization of iron have nothing in common at the particle level, yet near their phase transitions, they obey identical power laws. This paper suggests that neural network training via gradient descent belongs to this same family of universal phenomena. No matter what architecture you choose—wide, deep, convolutional, or recurrent—the statistical properties of the loss landscape and the training dynamics converge to the same fundamental patterns. This is deeply weird in the best possible way. It means that the specific choices we make as engineers—the number of layers, the activation functions, the initialization schemes—are almost incidental to the deeper story. The learning process itself has a kind of intrinsic character, a fingerprint of the underlying mathematics of optimization that transcends implementation. It's as if we've been building different machines to climb the same mountain, and now we discover that all mountains are, in some profound sense, the same mountain. The connection to renormalization group theory is irresistible here. In physics, the renormalization group explains how microscopic details wash out at macroscopic scales, leaving only a few relevant parameters. The same logic seems to apply to learning: the chaotic, high-dimensional details of individual weights and gradients wash out, revealing a simpler, universal trajectory. This hints at a "physics of intelligence" that might one day be as precisely formulated as thermodynamics. If such laws exist, they could transform AI from an engineering discipline into a branch of natural philosophy. Of course, we must be careful not to overstate the findings. This paper presents compelling evidence for universality in certain regimes, but the boundaries of this behavior—where it breaks down, what conditions violate it—remain open questions. Still, the implication is breathtaking: if learning is universal, then perhaps the principles that govern our own brains are not accidents of biology but manifestations of deeper mathematical truths. The universe might be, at its core, a learning machine, and we are just beginning to glimpse its operating system. Source: https://arxiv.org/abs/2007.13664