Systematic Bias and Injection Recovery

A single injection can show that an analysis sometimes fails; it cannot separate a repeatable bias from one realization of detector noise. A useful injection campaign pairs controlled truth with repeated noise and changes one consequential analysis assumption at a time. It measures bias, coverage, residual structure, and numerical stability over the parameter region for which the analysis claims validity.

Helpful background. Injection Campaigns and Search Sensitivity explains injection design at the search stage. Here the target is event-level inference, using the nuisance structure in Calibration and Noise Marginalization and the model alternatives in Waveform Marginalization and Model Uncertainty.

Statistical fluctuation and repeatable bias

Let $\theta_0$ be the injected value and $\hat\theta_r$ an estimator or posterior summary in noise realization $r$. Decompose

$$\hat\theta_r-\theta_0 =b(\theta_0)+\epsilon_r, \qquad \mathbb E[\epsilon_r]=0,$$

where $b$ is repeatable bias under the declared injection and recovery ensembles. The sample estimate from $R$ independent realizations is

$$\hat b =\frac1R\sum_{r=1}^R(\hat\theta_r-\theta_0),$$

with standard error determined by the scatter across realizations. A zero-noise injection is valuable because it isolates deterministic mismatch, but it does not reveal noise-dependent selection, non-Gaussian transients, or repeated-sampling coverage.

The local high-signal-to-noise relation between waveform discrepancy and repeatable parameter displacement is derived by Cutler and Vallisneri 2007, §§ II–III; a campaign is needed when that linear regime is unreliable.

For a nominal credible level $\gamma$, empirical coverage is

$$\widehat C_\gamma =\frac1R\sum_{r=1}^R \mathbf 1\!\left[\theta_0\in\mathcal C_{\gamma,r}\right].$$

At fixed $\theta_0$, this measures the frequentist coverage of the chosen credible-set procedure. Bayesian credible sets need not have nominal coverage at every fixed parameter, even with an exact likelihood and sampler. Nominal coverage follows after averaging over a prior-predictive ensemble: draw each $\theta_{0,r}$ from the same prior used in recovery, then draw its data from the likelihood. Indeed,

$$\mathbb E_{d}\!\left[ P\bigl(\theta\in\mathcal C_\gamma(d)\mid d\bigr) \right]=\gamma.$$

The equality assumes each set has posterior probability $\gamma$ and the ensemble uses the same generative model. If only detected injections are retained, selection changes that ensemble and its expected calibration must be derived explicitly. Fixed-parameter injections remain valuable for mapping local bias and coverage; a deviation there should not automatically be diagnosed as a sampler bug. Coverage also needs a precision diagnostic, since conservative, very broad intervals can conceal weak information.

A campaign that can identify causes

The campaign needs two waveform roles:

• the injection family, treated as controlled truth for the test; and
• the recovery family, used in the likelihood.

Using the same implementation for both is a closure test, not an independent waveform-accuracy test. A strong design crosses the following factors:

Factor	Control	Alternative	Diagnostic
Waveform	Same-family recovery	Held-out numerical or independently built family	Parameter shift and coherent residual
Calibration	Nominal response	Drawn amplitude/phase perturbations	Distance, phase, sky, and mass shifts
PSD/noise	Known stationary Gaussian PSD	Estimated, drifting, or contaminated noise	Coverage and residual distribution
Prior	Broad interim measure	Scientifically motivated alternative	Prior sensitivity and support
Sampler	Tight validated settings	Changed seed, tolerance, live points or chain length	Monte Carlo stability
Signal regime	Interior, moderate information	Boundaries, weak features, high information	Nonlinearity and model-error growth

Pairing the same noise realization across two recovery treatments reduces variance in their difference. Randomizing independently would make it harder to attribute a shift to the analysis change.

Metrics with interpretable failure modes

A standardized residual

$$z_r =\frac{\hat\theta_r-\theta_0}{\hat\sigma_r}$$

is a useful scale for comparing offsets, but need not have zero mean or a standard-normal distribution for posterior medians, skewed posteriors, or boundary parameters. Compare it with a closure ensemble using the same estimator and prior. Rank or probability-integral-transform diagnostics test the posterior across a correctly specified prior-predictive ensemble. For waveform or glitch failures, also measure residual coherent signal-to-noise ratio rather than relying only on parameter summaries.

Report at least:

• bias and its Monte Carlo uncertainty;
• coverage with binomial uncertainty at several credible levels;
• posterior width or information gain;
• effective sample size and sampler diagnostics;
• residual tests by detector and frequency region; and
• stratification by masses, spins, orientation, signal strength, and data state.

An aggregate average can hide a serious corner failure. The injection proposal distribution and analysis domain must cover the region where conclusions are claimed.

Reproducibility boundary

As of 2026-08-18, the official current observational reference is GWTC-5.0 Methods, §§ 2–5, tied to the versioned release. A public teaching campaign does not reproduce collaboration production infrastructure. It must instead pin:

• operating system or container, runtime, package and waveform versions;
• injection table, units, detector/source-frame convention, cosmology, and random seeds;
• detector set, sample rate, duration, frequency band, PSD, calibration curves, and data-quality assumptions;
• prior measure, sampler configuration, stopping rule, and numerical precision; and
• expected closure tolerance, coverage interval, and failure interpretation.

Before reporting a campaign result, rerun from a clean environment, vary injection count and sampler tolerance, and preserve machine-readable inputs and summary outputs. Convergence means the bias and coverage conclusions are stable within their Monte Carlo uncertainty—not merely that every sampler terminated.

The distinction between sampler convergence, analytic tests, and injection-based validation is illustrated in Veitch et al. 2015, §§ III–V.

Reading a failed campaign

Symptom	Plausible cause	Discriminating follow-up
Bias in zero noise, absent in closure test	Waveform mismatch	Change held-out injection family and frequency band
Bias only with calibration draws	Insufficient calibration nuisance support	Expand or correct response basis using the epoch product
Undercoverage with correct mean	Posterior too narrow or tails misspecified	Check PSD uncertainty, non-Gaussian likelihood, and sampler tails
Seed-dependent posterior modes	Sampler nonconvergence	Independent algorithm, longer run, mode-specific diagnostics
Bias only near a boundary	Prior/support and nonlinear geometry	Reparameterize and move injections across the boundary
Good parameters but coherent residual	Missing signal physics not aligned with chosen coordinates	Add modes or discrepancy basis; inspect detector coherence

No one row proves a cause. The follow-up changes the factor predicted by the hypothesis while holding the paired injection and noise fixed.

Exercises

1. Paired recoveries. A paired campaign analyzes the same $R=100$ noise realizations with methods A and B. For parameter $x$, the paired differences $\Delta_r=\hat x_{B,r}-\hat x_{A,r}$ have mean $0.12$ and sample standard deviation $0.40$.

1. Estimate the standard error of the mean difference.
2. Express the observed mean in standard-error units.
3. Explain why this does not by itself identify which method is accurate.

Solution

The standard error is

$$\operatorname{SE}(\bar\Delta) =\frac{0.40}{\sqrt{100}}=0.04.$$

The mean difference is $0.12/0.04=3$ standard errors from zero, so the methods differ repeatably under this ensemble. But the comparison contains no truth reference: A, B, or both may be biased. Injection values, closure controls, residuals, and an independently justified injection family are required to assign accuracy.

2. Exact Bayesian inference with non-nominal local coverage. Let $d\mid\theta\sim\mathcal N(\theta,1)$ and use prior $\theta\sim\mathcal N(0,1)$. The exact posterior is $\mathcal N(d/2,1/2)$. Estimate the repeated-noise coverage of its central 90% credible interval if every injection has $\theta_0=3$. Does a non-90% answer indicate a sampler failure?

Solution

The interval is $d/2\pm1.645/\sqrt2$. It contains $3$ when

$$6-1.645\sqrt2\le d\le6+1.645\sqrt2.$$

But at this fixed truth, $d\sim\mathcal N(3,1)$. The probability is therefore approximately $\Phi(5.326)-\Phi(0.674)\simeq0.25$, where $\Phi$ is the standard-normal cumulative distribution. This low coverage follows from the prior’s shrinkage toward zero at a truth in its tail; inference was exact. If truths are instead drawn from the stated $\mathcal N(0,1)$ prior before drawing each datum, average coverage is exactly 90%. Report which ensemble a calibration statement uses.

References

• Cutler, C., and Vallisneri, M. (2007). “LISA Detections of Massive Black Hole Inspirals: Parameter Extraction Errors Due to Inaccurate Template Waveforms.” Physical Review D 76, 104018. DOI.
• LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration (2026). “GWTC-5.0: Methods for Identifying and Characterizing Gravitational-Wave Transients.” arXiv:2605.27224. Open PDF.
• LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration (2026). GWTC-5.0 Data Release. DOI 10.7935/bk00-6a89. Official release.
• Veitch, J., et al. (2015). “Parameter Estimation for Compact Binaries with Ground-Based Gravitational-Wave Observations Using the LALInference Software Library.” Physical Review D 91, 042003. DOI.