Null Tests and Alternative-Model Tests

A null test asks whether data possess a property predicted by a reference model; an alternative-model test asks which of two specified predictive models describes the data better. These questions overlap, but they are not interchangeable. A statistically significant null residual can reveal a failed model or pipeline without identifying a viable alternative theory, while an alternative can be disfavored even when every simple residual looks innocuous.

Required background. The Einstein Equation supplies the Einstein equations used to define the GR reference prediction. Probability and Inference for Gravity supplies test statistics, likelihoods, sampling distributions, and Bayesian evidence.

Helpful background. Weak Metric Perturbations and Gauge Freedom provides the weak-field expansion behind common null coefficients. The Tests of Gravity overview shows where null and alternative tests enter each observational regime.

Null hypotheses and residual statistics

Let $T(d)$ be a statistic constructed so that the null hypothesis $H_0$ predicts a known distribution $p(T\mid H_0)$. In an idealized coefficient test, $H_0$ may predict $\delta=0$, and an estimator $\hat\delta$ with standard error $\sigma_\delta$ gives

$$z=\frac{\hat\delta}{\sigma_\delta}.$$

This quantity is meaningful only if the estimator is calibrated, the quoted error includes relevant nuisance uncertainty, and the reference distribution is valid. For a standard-normal estimator and a predeclared two-sided test, $p_0=2[1-\Phi_{\rm N}(|z|)]$, where $\Phi_{\rm N}$ is the standard-normal cumulative distribution. A one-sided search for a positive shift instead uses $1-\Phi_{\rm N}(z)$. Thus $z=3$ corresponds to approximately $0.00270$ or $0.00135$, respectively, before any trials correction. More generally, a small tail probability $p_0=P(T\ge T_{\rm obs}\mid H_0)$ measures incompatibility with $H_0$ under its calibrated statistic. It is not the probability that GR is false, nor the probability that a particular rival theory is true.

Many powerful gravity tests are internal consistency tests. One may estimate common source parameters from early and late parts of a signal, infer masses from different post-Keplerian timing observables, or compare lensing and dynamical mass estimates. Under the null, the derived estimates should agree after shared covariance is included. Such a test can be intentionally agnostic about the cause of disagreement.

The null distribution often has to be simulated. Parameter boundaries, weak identifiability, searching across many time windows, and choosing a statistic after viewing the data can invalidate a textbook $\chi^2$ approximation. The look-elsewhere effect belongs to the definition of the search, not to an after-the-fact verbal caveat Particle Data Group 2024, “Statistics,” §§40.2–40.4.

Specified alternatives and likelihood ratios

For two predictive hypotheses, a likelihood-ratio statistic compares their best fits,

$$q=-2\ln\frac{\sup_{\theta_0,\eta_0}p(d\mid H_0,\theta_0,\eta_0)} {\sup_{\theta_1,\eta_1}p(d\mid H_1,\theta_1,\eta_1)}.$$

Here each hypothesis carries its own physical and nuisance parameters. If $H_0$ is nested inside $H_1$, the numerator cannot exceed the denominator and $q\ge0$; for nonnested models this need not hold. Wilks-type asymptotic formulae additionally require regularity, identifiability, an interior null point, and a sufficiently informative sample; otherwise calibration with simulation is safer Cowan et al. 2011, §§2–3. A likelihood ratio rewards fit, but a more flexible alternative can gain likelihood by following noise or unmodeled systematics.

Bayesian comparison instead uses the marginal likelihood

$$Z_H=\int p(d\mid\theta,\eta,H)\,\pi(\theta,\eta\mid H)\,d\theta\,d\eta, \qquad B_{10}=\frac{Z_1}{Z_0}.$$

The evidence averages predictive performance over prior support. This implements an automatic complexity penalty, but it also means that an arbitrary widening of a weakly constrained prior can change $B_{10}$. A Bayes factor is therefore inseparable from its parameterization, prior measure, and stability restrictions. It compares the stated models, not “GR” with the set of all imaginable alternatives.

Choosing the question before the statistic

Scientific question	Appropriate construction	What a positive result establishes	Principal boundary
Is a GR-predicted coefficient zero?	Calibrated null statistic	The composite null is in tension	Does not identify the cause
Do two data subsets infer the same parameters?	Covariant parameter-difference test	The shared-model description is inconsistent	Split choice and shared nuisance terms matter
Does a specified deformation improve prediction?	Likelihood ratio or predictive score	That deformation fits better by the declared rule	Flexible models and trials require calibration
Which complete hypothesis is preferred?	Evidence or posterior odds	Relative support under stated priors	Prior measure and model completeness matter

The statistic should follow the scientific question and be frozen before inspecting the decisive data when practical. Blinding, simulated challenges, and preregistered quality cuts reduce the freedom to tune a test toward an interesting fluctuation. Posterior-predictive checks are valuable for diagnosing how a fitted model fails, but using the same data to fit and check requires the replicated-data distribution to be interpreted correctly.

Dataset-concordance tests add another layer. A goodness-of-fit loss, an evidence ratio, and a parameter-shift statistic answer different questions and need not rank a tension identically Raveri and Hu 2019, §II.C–E. Reporting several named diagnostics is more informative than collapsing them into an undefined “sigma tension.”

What a rejection permits us to say

A rejected null means that the data, the null prediction, and the assumed noise/nuisance model are jointly incompatible at the reported calibration level. The first response is to localize the failure: repeat the analysis with independent calibration, change data subsets, inject the null through the pipeline, and test known environmental channels. Only a complete alternative that survives those checks earns a comparative physical conclusion.

Conversely, failure to reject does not prove GR. The experiment may be insensitive to the relevant direction, an alternative may reduce to GR in that regime, or nuisance degeneracy may erase the signal. Power or expected sensitivity should accompany a null result so that “consistent with GR” is not confused with “capable of excluding meaningful departures.”

Check your understanding

An analysis reports $z=3$ for a coefficient expected to vanish in GR. Give two reasons this is not yet evidence for a specified modified-gravity model, and state what additional comparison is needed.

Solution

First, the quoted reference distribution may be wrong because a boundary, a search over many choices, or an omitted nuisance process changes the probability of a large residual. Second, many physical and instrumental mechanisms can produce the same nonzero coefficient, so the statistic does not identify a theory. Therefore one should calibrate the complete null search, including selections and trials, and compare a predictive alternative with the null using a declared likelihood or evidence rule. Independent data or a second observable should then test a consequence that distinguishes the alternative from the nuisance explanations.

References

• Cowan, G., Cranmer, K., Gross, E., and Vitells, O. (2011), “Asymptotic formulae for likelihood-based tests of new physics,” European Physical Journal C 71, 1554. Cowan et al. 2011, §§2–4.
• Particle Data Group (2024), “Statistics,” in Review of Particle Physics, Physical Review D 110, 030001. Particle Data Group 2024, §§40.1–40.4, PDF.
• Raveri, M., and Hu, W. (2019), “Concordance and discordance in cosmology,” Physical Review D 99, 043506. Raveri and Hu 2019, §II.C–E.