Tensions, Outliers, and Look-Elsewhere Effects

A discrepant gravity result becomes scientifically interesting when it is calibrated against the search that found it, reproduced with independent data and analysis, and connected to a theory that predicts additional effects. An isolated “sigma” value rarely meets those conditions. Outliers can reveal new physics, but they can also reveal underestimated covariance, flexible analysis choices, selection, or a failed source model.

Required background. Probability and Inference for Gravity supplies probability and statistical inference, tail probabilities, posterior comparison, multiple testing, and robust models.

Helpful background. Weak Metric Perturbations and Gauge Freedom supplies weak-field gravity parameters. The Tests of Gravity overview supplies the regime map. Gravitational-Wave Astronomy supplies gravitational-wave data analysis and inference, catalogs, and selection effects.

Local discrepancy and global search

Let $T_j$ be a statistic for the $j$th predeclared test and $T_{\max}=\max_j T_j$. A local tail probability uses

$$p_{\rm local}=P(T_j\ge T_{j,\rm obs}\mid H_0),$$

while the search-level quantity is

$$p_{\rm global}=P(T_{\max}\ge T_{\max,\rm obs}\mid H_0).$$

The global probability depends on the number and correlation of trials. Counting only published tests misses choices over events, frequency bands, sky regions, environmental thresholds, waveform families, quality cuts, and statistics. When those choices are continuous or adaptive, simulation of the full frozen search is generally safer than multiplying a local probability by an informal trial count Particle Data Group 2024, “Statistics,” §§40.2–40.4.

An Asimov dataset gives median expected sensitivity under a model, but it does not calibrate an observed tail after analysis choices have been tuned. Cowan and collaborators define the Asimov construction at Eq. (24) and develop its asymptotic use in Eqs. (24)–(30) and §4.1 Cowan et al. 2011, §3.2 and §4.1.

Tension statistics answer different questions

For two approximately Gaussian parameter estimates, a simple shift statistic is

$$Q_\Delta=(\hat\theta_A-\hat\theta_B)^T C_\Delta^{-1} (\hat\theta_A-\hat\theta_B),$$

where $C_\Delta$ includes cross-covariance. This tests parameter agreement inside a common model. A goodness-of-fit loss asks whether one shared fit degrades relative to separate fits. An evidence ratio averages predictive density under declared priors. These diagnostics can rank the same datasets differently because they measure different aspects of concordance Raveri and Hu 2019, §II.C–E.

The word “tension” should therefore be accompanied by the statistic, reference distribution or prior, parameter subspace, dataset versions, calibration, selections, and shared covariance. Converting an arbitrary posterior-overlap number to “sigma” can create a familiar scale without a calibrated repeated-sampling meaning.

Outliers and robust alternatives

Deleting a discrepant point because it weakens a constraint is outcome-dependent selection. Keeping every point under a fragile Gaussian likelihood can be equally misleading. A robust mixture model makes the alternative explicit,

$$p(y_i\mid\mu,\epsilon) =(1-\epsilon)\,\mathcal N(y_i\mid\mu,\sigma_i^2) +\epsilon\,\mathcal N(y_i\mid\mu,\kappa^2\sigma_i^2),$$

with $\kappa>1$ and a declared prior or calibration for the outlier fraction $\epsilon$. Heavy-tailed likelihoods, leave-one-out predictive checks, and influence diagnostics answer related questions. They should be selected before examining which model favors GR or a deviation.

An outlier may be the most informative datum if a theory predicts its mass, environment, frequency, or redshift dependence. The test is whether the same theory predicts other observations—not whether the point can be made to disappear under some flexible noise model.

Results and limitations through 2026-08-19

At the 2026-08-19 cutoff, the tests discussed in this volume include disputed and model-dependent anomalies. Their interpretation requires the named analyses and their controls; no single “sigma” summarizes the full collection. This page develops a replication procedure rather than claiming to have tested every proposed anomaly or performed an exhaustive comparison of theories.

Evidence or result	Status at the cutoff	Limitation or uncertainty boundary
Mature searches report calibrated null results and occasional local outliers	Essential inputs for cumulative testing	Publication and analysis-choice trials are not always reconstructible from papers alone
Dataset-concordance diagnostics can localize inconsistent parameters	Several principled statistics are available	Results depend on shared covariance, parameter subspace, and the question encoded by the statistic
Robust and leave-one-out models reveal influential measurements	Strong diagnostic against single-point domination	Flexible robust models can also absorb a genuine localized signal
Some regime-specific anomalies motivate continued study	Scientifically useful hypotheses and follow-up targets	Replication, shared inputs, and a theory’s additional predictions must be assessed case by case

The deviation interpretation treats a repeated residual as the first sign of new gravity. The conservative competing interpretation attributes it to noise, calibration, astrophysical modeling, selection, or uncounted trials. Negative evidence for new gravity is failure of the effect in a predeclared replication or absence of a correlated prediction; negative evidence for the conservative account is survival across independent instruments and nuisance models with the theory-predicted scaling.

A replication protocol that can discriminate

Freeze the target observable, sign, scaling, dataset and calibration release, sample selection, statistic, and stopping rule. Hold back an independent dataset or observing period. Analyze null and alternative injections through the same software and quality cuts, and report all predeclared variants rather than only the most favorable.

The adversarial check asks both camps to predict the held-back data. A new-gravity model must predict a second observable or regime without refitting an unconstrained coefficient. The systematic explanation must predict which environmental or calibration channel tracks the residual. A result that simply repeats after reusing the same catalog, ephemeris, simulation, or prior is not independent replication.

The unresolved issue is that the effective number of trials can be unknowable after exploratory work. In that case, present the finding as hypothesis generation and reserve confirmatory significance for new data. This boundary protects genuine anomalies from being dismissed later because their first analysis overclaimed them.

What could change the conclusion. The interpretation could change when a preregistered independent replication confirms or rejects a consequential anomaly, when a corrected covariance changes a tension classification, or when a viable theory successfully predicts an additional regime under the frozen adversarial protocol.

Check your understanding

Twenty correlated residual statistics are inspected and the most extreme has a local probability of $0.01$. Why can neither $0.01$ nor $0.20$ be asserted as the global probability without more work?

Solution

The local value ignores that the most extreme statistic was selected from twenty. For twenty predeclared valid $p$-values, the union bound gives $P(\min_j p_j\le0.01)\le20(0.01)=0.20$ regardless of their dependence. This follows because the event that at least one test crosses the threshold is a union of twenty events; its probability cannot exceed the sum of their probabilities. This is a conservative bound, not generally an equality. If the $p$-values are independent and continuous uniform under the null, the exact result is $1-(1-0.01)^{20}\simeq0.1821$; if they are identical, it is $0.01$. Correlated searches have different possible joint distributions, which an informal trial count does not specify. Simulate the joint null search with the same selection rule or report the conservative bound. Additional unrecorded analysis choices must be included or treated as exploratory.

References

• Cowan, G., Cranmer, K., Gross, E., and Vitells, O. (2011), “Asymptotic formulae for likelihood-based tests of new physics,” European Physical Journal C 71, 1554. Cowan et al. 2011, §3.2 and §4.1.
• Particle Data Group (2024), “Statistics,” in Review of Particle Physics, Physical Review D 110, 030001. Particle Data Group 2024, §§40.1–40.4, PDF.
• Raveri, M., and Hu, W. (2019), “Concordance and discordance in cosmology,” Physical Review D 99, 043506. Raveri and Hu 2019, §II.C–E.