Out-of-Distribution and Uncertainty Tests

A surrogate needs a scientifically useful way to say “this request is unlike the cases on which I was validated.” Low interpolation error does not provide that signal automatically, and a neural ensemble, dropout variance, or Gaussian-process interval can remain confidently wrong under parameter shift. Out-of-distribution testing therefore pairs every uncertainty score with the true high-fidelity error while the shift is increased in a controlled, predeclared direction.

Required background. Probability and Inference for Gravity supplies coverage, proper scoring rules, calibration, and hypothesis testing.

Helpful background. Formulations of General Relativity supplies physical constraints that can reveal failed predictions; Differential Equations for Relativity supplies high-fidelity reference and residual checks; Computational Relativity supplies versioned domains, benchmarks, and reproducibility.

Distribution shift in a physical parameter space

Let $P_{\rm train}(\lambda,h)$ describe the finite training catalog and $P_{\rm use}$ the requested cases. “Out of distribution” is not one condition:

• interpolation gap: $\lambda$ is inside the declared domain but in a contiguous region omitted from training;
• boundary shift: $\lambda$ approaches a face, corner, extremal spin, high mass ratio, or other edge where sampling and model regularity change;
• covariate shift: the input distribution changes while the conditional solution map is intended to remain the same;
• concept or model shift: the physical map changes through new modes, eccentricity, matter, calibration, gauge, or waveform convention;
• representation shift: sampling, alignment, detector weighting, or preprocessing differs from training.

A Euclidean distance in normalized parameters can detect sparse coverage but not every physical change. Two nearby points may straddle a phase transition or mode switch; two distant points may be related by an exact symmetry. Define the domain and shift using physical coordinates and known structure, then retain a simple distance baseline as a check against a needlessly opaque detector.

For a waveform predictor $\widehat h(\lambda)$, pair its uncertainty score $s(\lambda)$ with true quantities such as detector-weighted mismatch, phase error, amplitude error, and any constraint residual. An uncertainty method succeeds at failure detection only if larger scores reliably accompany larger errors at the threshold relevant to the application.

Calibration and detection metrics

For a scalar prediction with interval $I_{1-\alpha}(\lambda)$, empirical coverage is

$$\widehat C_{1-\alpha} =\frac{1}{n}\sum_{i=1}^{n} \mathbf 1\!\left[y_i\in I_{1-\alpha}(\lambda_i)\right].$$

Report coverage separately for interpolation, boundary, hole, and each OOD severity; a global average can hide total failure in a small critical region. Also report interval width. Trivially wide intervals can cover without being useful.

For probabilistic regression, negative log predictive density penalizes both overconfidence and unnecessary width. For a binary failure event such as mismatch above $10^{-3}$, report reliability, Brier score, expected calibration error with disclosed bins, and the area under the ROC curve for the uncertainty score as a detector. Selective-risk curves answer a practical question: as the model abstains on its highest-score cases, how quickly does error among accepted cases fall?

Ovadia and collaborators compare ensembles, dropout, and other predictive-uncertainty methods under progressive dataset shift using likelihood, Brier, calibration, and entropy measures in Ovadia et al. 2019, §§ 3–4 and Figs. 1–5. Their image benchmarks are methodological negative evidence, not direct evidence about relativistic waveforms: rankings and calibration can degrade under shift even after strong in-distribution performance.

A warning score must be tested at an operating threshold

Suppose 1% of requests exceed the application’s waveform-error limit. A warning rule detects 90% of those failures and falsely warns on 5% of accurate requests. In 10,000 requests it flags 90 genuine failures and 495 accurate cases. Only

$$\frac{90}{90+495}\simeq15.4\%$$

of warnings identify failures, despite the high detection rate. Ten failures are still accepted. This elementary count shows why a ranking statistic such as AUROC does not specify the false-negative rate, warning burden, or accuracy among accepted requests.

Choose the warning threshold on validation data, then evaluate the frozen rule on untouched cases. Report the failure prevalence, true and false positives, true and false negatives, and accepted fraction. Deployment with a different prevalence can change the fraction of useful warnings even if sensitivity and specificity remain unchanged.

Coverage also has sampling uncertainty. For $n$ independent cases and true coverage $p$, the standard error of the observed fraction is $\sqrt{p(1-p)/n}$. At $p=0.95$ and $n=100$ it is about 0.022. A few percentage points of apparent undercoverage need an interval estimate; severe regional undercoverage cannot be repaired by a good global average. Correlated waveform families reduce the effective sample size.

Comparison of methods and results

The evidence below was checked through 2026-08-18.

Position	Evidence in its tested regime	Boundary of the claim
established baseline	held-out high-fidelity error, parameter distance, physical residuals, and explicit domain checks directly test applicability	these checks require reference cases and may be costly
useful uncertainty methods	deep ensembles and Gaussian processes often improve in-distribution calibration and can rank some shifted errors	no method in the cited comparisons remains uniformly calibrated under all shifts
waveform-specific positive evidence	GP variance has guided acquisition inside declared waveform parameter boxes	the score is conditional on kernel, preprocessing, simulator, and catalog
adverse evidence	progressive shift can produce confidently wrong predictions and degraded method rankings	IID coverage does not transfer to OOD coverage
not established	a universal numerical-relativity OOD detector or a nominal interval guarantee under arbitrary nonexchangeable parameter shift	each physical shift needs its own counterexample and validation data

Doctor and collaborators discuss kernel choice, parameter boundaries, and cost limitations for waveform emulation in Doctor et al. 2017, § V and Figs. 13–15. Ovadia et al. summarize the broader shift findings in Ovadia et al. 2019, § 5. The favorable position is that ensembles, GP variance, and calibrated scores can triage requests in a demonstrated domain. The competing position is that their confidence can fail silently under precisely the shifts for which warning is most needed.

Progressive compact-binary benchmark

Freeze a versioned high-fidelity compact-binary waveform family with declared modes, parameter coordinates, alignment, norm, data revisions, and numerical uncertainty. Train over $q\in[1,3]$ and aligned spin $\chi\in[-0.5,0.5]$. Before fitting, reserve:

1. scattered in-domain interpolation points;
2. all boundary faces and corners;
3. a contiguous interior hole;
4. outward shells such as $q\in(3,4]$ and $|\chi|\in(0.5,0.7]$; and
5. a model-shift set containing a mode or physical effect absent from training, evaluated only where a compatible reference exists.

Compare a neural ensemble, dropout model, Gaussian process, and simple nearest-training-distance score using the same reduced representation and splits. Repeat every learned method over declared seeds. For each shift shell, report median, 95th, and maximum mismatch; physical symmetry or constraint violations; failure-detection AUROC; and selective risk for every warning score. Report 68% and 95% interval coverage and negative log predictive density only for methods that supply predictive distributions. A distance score may enter those comparisons only after a calibration map has been learned without access to the OOD test cases. Report failure-event Brier score and calibration for methods that output a failure probability.

Increase training density, model capacity, optimization accuracy, ensemble size, and arithmetic precision separately. Sampling convergence for coverage and AUROC is assessed by confidence intervals or bootstrap resampling that respects any waveform-family dependence. The actual high-fidelity numerical error remains a separate component; do not score the surrogate more finely than its reference.

A representative deployment rule might require in-domain 95% coverage between 90% and 98%, maximum in-domain accepted mismatch below $10^{-3}$, and an OOD false-negative rate below 10% for cases whose true mismatch exceeds that threshold at the first shell. These numbers define one finite benchmark, not universal standards. Publish the actual discrepancy, abstention rate, and false-negative rate.

The decisive counterexample is the most confident wrong OOD case. Locate the point with smallest reported uncertainty among cases exceeding the mismatch threshold, reconstruct the high-fidelity waveform directly, verify the error with independent overlap code, and inspect phase, amplitude, modes, and physical residuals. A method with excellent mean calibration but a dangerous false negative fails the stated warning role.

Reproducing the calculation and testing its robustness

Pin data and high-fidelity revisions, split and shift manifests, preprocessing, reduced basis, architecture or kernel, priors, optimizer, seeds, precision, software and accelerator versions, operating system or container, inference code, and every uncertainty metric implementation. Record binning, interval construction, bootstrap scheme, thresholds, and whether hyperparameters saw boundary or OOD cases.

The clean sequence verifies these inputs; rebuilds every method and seed; sweeps coverage, capacity, optimization, ensemble size, and precision; computes error and uncertainty by shift shell; finds the most confident failure; reruns it through independent high-fidelity and overlap paths; and emits metrics, thresholds, costs, and false negatives in machine-readable form. Data leakage from OOD tests into model or threshold selection creates a new validation set requirement.

Failure modes include calibrated IID intervals with OOD undercoverage, a score that tracks parameter distance but misses a physical discontinuity, an ensemble whose members share the same bias, a detector tuned on the test shifts, and extrapolation accepted because the output is smooth. Conformal or other coverage claims require their exchangeability or shift assumptions to match the physical use case.

Review is triggered by a new simulation catalog or physical domain, a changed architecture or uncertainty method, a new shift guarantee that covers the nonexchangeable case at issue, an independent false-negative or coverage failure, or a benchmark that materially changes the relative method ranking.

Check your understanding

A surrogate has 95% empirical coverage on a random in-domain test set. On a predeclared contiguous high-spin holdout, only 40% of reference values lie inside its nominal 95% intervals, but the mean waveform mismatch remains small. Is its uncertainty calibrated for high-spin use?

Solution

No. Coverage must be assessed in the deployment regime, and 40% falsifies the nominal 95% claim there. The small mean mismatch does not repair undercoverage or rule out a dangerous worst case. Report interval width, worst mismatch, standardized residuals, and false negatives; retrain or narrow the validated domain, then test on a new untouched high-spin set.

Continue with the chapter overview.

References

• Doctor, Zoheyr, Ben Farr, Daniel E. Holz, and Michael Pürrer. “Statistical Gravitational Waveform Models: What to Simulate Next?” Physical Review D 96 (2017): 123011. arXiv:1706.05408. DOI.
• Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift.” In Advances in Neural Information Processing Systems 32. 2019. arXiv:1906.02530.