Bounds on Extrapolation Across Phase Transitions with Generalized Regression

Jeffrey Wei, Manolis Zampetakis, and John Sous

Yale University

NeurIPS 2026

Also presented at the ICML 2026 AI4Physics Workshop

[Paper] [Code]

“One of the ways of stopping science would be only to do experiments in the region where you know the law.”

— Richard P. Feynman, The Character of Physical Law (1965)

A major goal of theoretical physics is to make predictions in regimes where information is limited or inaccessible. Continuous phase transitions are the canonical example, yet it is not known which models extrapolate across them or how large their error can be before the unobserved phase is measured. We show that fixed-feature regression trained on a single phase recovers the unobserved one while neural networks do not, and that its error obeys a sample-free bound that grows only as a power law in the distance to the transition.

T = 2.90, training region
Heisenberg free energy f(T, m) at the selected temperature (left) and the order parameter |m*(T)|, the minimizer of each fitted free energy (right). We compare polynomial regression, random Fourier features (RFF), a Gaussian process (GP), and a multilayer perceptron (MLP), a standard neural network, each fit only on the training region P above Tc. Below Tc the ground-truth free energy develops two symmetric minima. Polynomial, RFF, and GP recover both the free-energy landscape and the order parameter curve. The MLP stays close to a single well and underestimates the magnetization throughout the unobserved phase.

Can a model trained only on one phase predict the other?

Heat a magnet past its critical temperature Tc and its magnetization disappears. Cool it back down and the magnetization returns. This is a continuous phase transition, in which a smooth physical observable gives rise to nonanalytic behavior in derived quantities. In the ferromagnet below, the Landau free energy has a single minimum at m = 0 above Tc. Below Tc it splits into two minima at ±m*, and the magnet acquires a net magnetization.

T = 3.00, |m| = 0.00
Each tab is one transition. Left: Wolff-cluster Monte Carlo of the two-dimensional Ising model, with temperature rescaled so that its critical point matches that of the Landau free energy. Middle: the Landau free energy used in the paper, with the ball at its minimum. Right: the order parameter |m*(T)| traced by that minimum. The middle panel is shaded blue while the control parameter is in the training region P and gray while it is in the extrapolation region Q, as in the right panel.

We fit each model only on a training region P above Tc and test it on an extrapolation region Q that extends across Tc. In P the magnetization is zero everywhere, so the clearest sign of the coming transition is that the curvature of the free energy at m = 0 shrinks as T approaches Tc. To predict the double well, a model has to continue that trend across Tc.

Fixed-feature regression extrapolates across the transition

Extending the GP study of [Vargas-Hernández et al. 2018], we evaluate on five continuous transitions: two analytic free energies (Heisenberg, Flory–Huggins), two numerical simulations (SSH polaron, Rayleigh–Bénard convection), and one experiment (ARPES on a cuprate superconductor). For each, we compare the predicted observable with the ground truth on Q and extract the derived nonanalytic quantity from the predicted surface. Polynomial, RFF, and GP locate the critical point to within a few percent on every target with a sharp transition. The MLP fits the training phase to numerical precision but misplaces the transition on each of them.

Ground truth, the four fitted landscapes, and the derived quantity
Ground truth Polynomial RFF GP MLP train P extrap Q

Each tab is one transition. Left: the ground truth, the four fitted landscapes, and the derived quantity. Right: the relative error on Q and the error in the recovered critical point, from Table 1 of the paper.

Past the outermost kink of its activations, a ReLU network continues affinely, with a slope the training data do not determine. GELU networks become asymptotically affine, tanh networks saturate, and sinusoidal networks such as SIREN repeat frequencies learned on P, so none of them can produce the change in curvature that splits the single well into two below Tc. Polynomial, RFF, and GP features stay non-affine past the boundary of P and carry the curvature learned there into Q. The same neural network extrapolation failure is apparent across all other datasets.

Ground truth and five neural network fits of the Heisenberg free energy
The failure is not specific to MLPs. On Heisenberg, a Fourier-feature MLP, SIREN, a physics-informed network with a parity penalty, and a gray-box network with the Landau form f = a(T)m2 + b(T)m4 + c(T) all fail to produce the double well below Tc.

Fixed-feature Regression Empirically Extrapolates

Polynomial, RFF, and GP recover the unobserved phase on all five systems, while neural networks do not.

A sample-free bound on the extrapolation error

We measure extrapolation reliability with the transfer coefficient Γemp, the ratio of the mean squared error (MSE) on Q to the MSE on P. Polynomial and RFF models both predict with a weighted sum of fixed features φ, monomials for the first and cosines for the second. When the residual r = f* − f̂ is such a sum as well, its mean square on each region is set by the feature covariance there, B = EP[φφ⊤] on P and A = EQ[φφ⊤] on Q.

On a narrow training window the features are strongly correlated, so B is thin in some directions while A can be wide in them (Fig. 5). A residual along such a direction is nearly invisible on P and large on Q. The worst direction is the top generalized eigenvector of A and B, and its eigenvalue is the GEVP bound, ΓGEVP = λmax(A, B), which guarantees MSEQ ≤ ΓGEVP · MSEP. Both covariances use only the known measures on P and Q, so the bound requires no samples from the unobserved phase.

View

λmax =

λmax =

The two feature covariances for a straight-line polynomial (left) and for RFF with two features (right), with Q around the transition and a training window P of the same width a distance D away. Each input T gives a point whose coordinates are its two feature values. The blue ellipse is the covariance of these points over P and the grey ellipse over Q. Inputs in a narrow window give nearly the same point, so the training ellipse is thin. Rescaled stretches the plane until the training ellipse is a circle of radius 1. The extrapolation ellipse then reaches out to √λmax, the radius of the dashed circle, so no error can grow in mean square from P to Q by more than λmax. The Fourier frequencies ω and phases b are one random draw.
Γemp ΓGEVP Volume bound Remez bound
Empirical transfer coefficient Γemp and GEVP bound ΓGEVP against training-set size n, for the polynomial (left) and RFF (right) models on each target. ΓGEVP upper bounds Γemp at every n and does not change with n, since it depends only on the feature covariances on P and Q. The Volume and Remez bounds apply only to polynomials, depend on P and Q only through their volume ratios, and sit many orders of magnitude higher.

GEVP Tightly Bounds the Extrapolation Error

The GEVP bound stays just above the observed transfer coefficient at every training-set size, while the geometry-based bounds sit many orders of magnitude higher.

A scaling law for extrapolation

We move the training region away from the transition while holding Q fixed and measure both quantities against the distance D. Neither grows exponentially. On all five targets, the empirical transfer coefficient and the GEVP bound follow power laws in D (Fig. 7). For polynomials this scaling is implicit in the Volume bound [Kalavasis et al. 2024]. With the region volumes fixed, its factor ρQρP2d scales as |Rfull|2d+1. RFF have no notion of degree, yet their transfer coefficient obeys a power law as well (Theorem 4.1 of the paper):

Γ(𝒢m; P, Q) ≤ 2(2A2)2(k−1) (|I| / |P|)2(k−1)

Here k ≤ 2m counts the distinct frequencies among the m features, |I| = |P| + D + |Q| is the length spanned by both regions, and A > 1 is an absolute constant, so at fixed window widths the bound grows as a power of D. The proof uses the Turán–Nazarov inequality for exponential sums [Nazarov 1993] and is in Appendix C of the paper.

Γemp ΓGEVP
Γemp and ΓGEVP against the distance D at a fixed training-set size, for the polynomial (left) and RFF (right) models on each target. Lines are power-law fits, and β is the fitted exponent.

In the one-dimensional setting of Fig. 5, the bound grows as a power of D for polynomial and RFF models alike (Fig. 8). Because the growth is polynomial rather than exponential, the bound stays informative across large gaps, and it can be evaluated before any sample from the unobserved phase is collected. We view this as an important result for practitioners. The power law sets a ceiling on the extrapolation error as a function of the distance to the transition, so one can decide in advance how far to extrapolate with confidence.

Number of features
The GEVP bound λmax against the distance D for the polynomial (left) and RFF (right) models with the same number of features, in the one-dimensional setting of Fig. 5, on log–log axes. A polynomial of degree d has d + 1 features. Both grow as power laws, far below an exponential with the same start (dashed). The polynomial curves approach slope 2d. The RFF curves are medians over 32 random draws of frequencies and phases, consistent with Theorem 4.1.

Scaling Law of Extrapolation

The observed transfer coefficient and the GEVP bound both grow as a power law in the distance to the transition, and for polynomial and RFF models the bound provably grows no faster.

Citation

@inproceedings{wei2026extrapolation,
  title     = {Bounds on Extrapolation across Phase Transitions with Generalized Regression},
  author    = {Wei, Jeffrey and Zampetakis, Manolis and Sous, John},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
}

References