The double descent phenomenon is one of the most counterintuitive findings in modern machine learning. Train a model with a small number of parameters and test error falls as you add more, following the classical bias-variance curve. But keep adding parameters past the point where the model can just barely fit the training data, and something strange happens: test error peaks sharply, then falls again. A model with 1,000 parameters fits a cubic function better than a model with exactly 4, even though the 4-parameter model is the true generating process. This behavior contradicts the classical understanding that more parameters mean more overfitting.
Exact treatments exist for simplified settings like linear regression and random features, and the interpolation threshold has been connected to jamming transitions and broken ergodicity in physical systems. Congzhou M Sha, at Penn Medicine Doylestown Hospital, offers a different explanation rooted in statistical mechanics. The argument is simple, physical, and surprisingly complete.
Training as a particle in an energy landscape
The core analogy: treat the training loss as an energy function over parameter space, and a stochastic gradient descent run as a particle wandering through that landscape at some effective temperature T. This is not a metaphor. The Boltzmann distribution, which assigns probability proportional to exp(-E/T) to each microstate of a physical system, emerges naturally from the fundamental postulate of statistical mechanics: if all you know about a system is its macroscopic observable (the training loss), then every parameter configuration consistent with that loss is equally likely.
The training loss L(theta) plays the role of energy. The set of all parameter vectors achieving a particular loss value E forms the macrostate. An ergodic training run, one that has visited a representative sample of configurations at that loss level, samples this macrostate uniformly. The probability of observing a particular parameter vector theta is then proportional to exp(-L_lambda(theta)/T), where L_lambda includes the loss plus a weight decay term (lambda/2)||theta||^2, and T is determined by the properties of the optimization algorithm.
The weight decay term is not arbitrary. The paper proves a theorem connecting finite training time to implicit regularization: a stochastic gradient run of length t behaves, in terms of its covariance structure, like a run with weight decay lambda proportional to 1/t. This is a known result in the optimization literature (Krogh and Hertz 1992, Ali et al. 2019), but the statistical mechanics framing makes it load-bearing. The weight decay term makes every parameter a quadratic degree of freedom, which is what allows the equipartition theorem to do its work.
Equipartition and the cooling of temperature
Near a local minimum of the loss, the landscape is quadratic. The Hessian H (or for least squares with weight decay, M = A^T A + lambda I) defines d principal curvatures. Under the Boltzmann weight, each of the d coordinates is an independent Gaussian with variance T/mu_i, where mu_i is the curvature along that direction. The equipartition theorem then says that each degree of freedom carries exactly T/2 of the energy above the minimum. The expected loss is E_min + T*d/2.
This is where the double descent mechanism becomes visible. For least squares, the data term (1/2)||A*theta - y||^2 changes only along the n directions spanned by the row space of A, which the paper calls "seen" directions. The remaining d-n directions, in the null space of A, are " unseen" by the data. The energy in the seen directions is (n/d)*Delta_E, where Delta_E is the total excess loss above the minimum. As d increases beyond n, this fraction falls as 1/d. At a fixed training loss, adding parameters cools the temperature along the directions that actually matter for the fit.
This cooling drives the Boltzmann distribution toward the stationary path, the minimum of the loss. And the paper proves a second theorem: the L^2 norm of the stationary path is nonincreasing in d. Each new basis function with nonzero overlap with the training data lowers the energy of the interpolant. The minimum-norm interpolant at d+1 parameters has norm less than or equal to the minimum-norm interpolant at d parameters, strictly decreasing unless the new basis function is orthogonal to the training data.
The path integral perspective
Read as a statement about curves rather than parameters, the Boltzmann distribution is a path integral. Each parameter vector maps to a function f(x; theta), and the sum over microstates is a sum over every curve the model can draw, each weighted by exp(-L_lambda/T). A prediction is the average over all curves. As T approaches zero, the noise vanishes and the parameters descend to the stationary point. As T approaches infinity, the dynamics become an Ornstein-Uhlenbeck process.
For models linear in their parameters, the path integral is a Gaussian process with mean equal to the ridge regression fit and covariance proportional to T times the Hessian inverse. The paper shows that at finite training time, the unseen directions have reached only a fraction of their stationary variance: the run behaves as if the effective stiffness were capped at 1/(2t). Late stopping removes this cap and leaves the ridge fit at the explicit weight decay lambda, whose quality depends on the conditioning of the data.
The polynomial fitting example makes this concrete. Fit 20 noisy samples of a cubic using Legendre polynomials with d parameters. For d less than or equal to 20, the minimum-norm solution is the least squares fit. For d greater than 20, every solution interpolates the data exactly. At d=20, the polynomial oscillates wildly. At d=1000, the fit tracks the true cubic nearly as well as the d=4 solution. The test error peaks at d=n=20 and then falls. The statistical mechanics explanation: at d=20, the seen and unseen dimensions are equal, and the temperature is at its highest relative to the data-dependent energy. Beyond d=20, the unseen dimensions absorb more energy, cooling the temperature along the seen directions and pushing the distribution toward the minimum-norm interpolant, which happens to be the right function.
Why this is more than a restatement of existing results
Exact treatments of double descent already exist for linear models (Hastie et al. 2022, Mei and Montanari 2022) and random features (Bartlett et al. 2020). The jamming transition (Spigler et al. 2019, Geiger et al. 2020) and broken ergodicity (Li and Goldenfeld 2026) provide physical explanations. What this paper adds is a unified statistical mechanics framework that connects these threads through the Boltzmann distribution, equipartition, and the path integral.
The key contribution is the cooling mechanism. The standard explanation for the second descent in linear models relies on the geometry of the minimum-norm interpolant: beyond the interpolation threshold, the minimum-norm solution becomes smoother because the null space of the design matrix grows. The statistical mechanics framing gives this a physical mechanism: equipartition distributes the fixed energy among more degrees of freedom, cooling the temperature along the data-relevant directions, and the nonincreasing norm of the stationary path ensures that the sampled solutions become smaller. It is not just that the minimum-norm solution happens to be smaller; it is that the Boltzmann distribution concentrates on smaller solutions as d increases, for thermodynamic reasons.
The paper also makes the connection to weight decay precise. Finite training time is equivalent to implicit weight decay (Theorem 1), and increasing d beyond the interpolation threshold is another form of weight regularization (the conclusion). The two mechanisms are distinct: one comes from the optimization algorithm's finite horizon, the other from the geometry of the solution space. But both are captured by the same statistical mechanics framework.
Limitations and scope
The paper focuses on pedagogy and intuition rather than technical novelty. The author explicitly states that the aim is to provide elementary physical intuition understandable to undergraduates. The proofs use the Legendre basis for polynomial fitting and the quadratic loss, which makes the calculations explicit but limits the direct applicability to non-quadratic losses and non-linear models.
The ergodicity assumption is stated as an assumption, not derived. Whether real SGD training runs are ergodic in the sense required by the fundamental postulate is an empirical question. The paper notes that training produces a trajectory of parameters that has "settled" at a training loss E, and assumes this trajectory samples the macrostate uniformly. For convex losses with stochastic gradient noise, this is plausible. For non-convex losses with deep networks, the assumption is stronger and the relationship to broken ergodicity (Li and Goldenfeld 2026) is worth exploring.
The temperature T is treated as a property of the optimization algorithm, not derived from first principles. The paper shows that T determines the covariance of the parameter distribution but does not provide a formula for T in terms of the learning rate, batch size, or number of steps. This is a gap: the framework explains what happens given a temperature but does not predict the temperature itself.
The path integral formulation is exact for linear-in-parameter models but approximate for non-linear models. The paper acknowledges this by focusing on the quadratic loss and the Legendre basis, where the calculations are tractable. Extending the framework to deep networks, where the loss landscape is non-quadratic and the parameter-function mapping is non-linear, remains an open problem.
What this means for understanding generalization
The practical message is that double descent is not mysterious. It is a consequence of three well-understood physical principles: the fundamental postulate (equilibrium sampling of the loss level set), equipartition (energy distributed equally among quadratic degrees of freedom), and the non-increasing norm of the stationary path (adding parameters can only reduce the minimum-norm solution). Together, these explain why test error peaks at the interpolation threshold and falls beyond it: the system cools along the data-relevant directions, concentrating on smaller, smoother solutions.
For practitioners, this framework suggests that the double descent curve is not a bug to be avoided but a feature to be understood. The peak at d=n is the point of maximum temperature along the seen directions. Beyond it, the system is cooling. The question is not whether double descent occurs, but how quickly the temperature drops, which depends on the geometry of the data and the model.
The connection to weight decay is also practical. If finite training time acts as implicit regularization, and adding parameters beyond interpolation is another form of regularization, then the choice of model size is a choice about the strength of implicit weight decay. The statistical mechanics framework makes this trade-off explicit and quantifiable.
For researchers, the paper provides a physical vocabulary for discussing generalization that complements the existing mathematical and statistical frameworks. The Boltzmann distribution, equipartition, and path integrals are tools with centuries of development in physics. Applying them to machine learning does not replace the existing theory, but it offers a different lens that may suggest new directions, particularly for understanding how temperature, model size, and data geometry interact in non-linear settings.
Read the paper on arXiv