Frequentist Intervals and Coverage

A frequentist confidence interval is a rule that maps every possible dataset to a parameter set with controlled repeated-experiment coverage. The probability statement belongs to the procedure before observing the data, not to a fixed parameter after the interval is reported. This page constructs the exact Gaussian interval, relates likelihood-ratio ordering to confidence belts, and explains why boundaries, nuisance parameters, waveform error, and event selection require injection-based coverage checks in gravitational-wave analyses.

Required background. Probability and Statistical Inference supplies sampling distributions, estimators, hypothesis tests, quantiles, and conditional probability.

Helpful background. Gravitational Radiation and Waveforms supplies the waveform parameters and detector responses whose intervals are constructed.

Coverage is a property of a rule

Let $C_\alpha(d)$ be a set computed from data $d$. It is a $(1-\alpha)$ confidence procedure for parameter $\theta$ if

$$P_\theta\!\left[\theta\in C_\alpha(D)\right] \ge1-\alpha$$

for every $\theta$ in the declared model, or equals $1-\alpha$ when exact coverage is attainable. Here $D$ is a random dataset generated repeatedly at fixed $\theta$. Once $d_\mathrm{obs}$ is observed, both $\theta$ and $C_\alpha(d_\mathrm{obs})$ are fixed; the frequentist statement is about the long-run construction.

A Bayesian credible region instead satisfies

$$\int_{C(d)}p(\theta\mid d)\,d\theta=1-\alpha$$

under a declared prior. The two regions can coincide in special Gaussian problems, but their probability statements and guarantees remain different.

The exact Gaussian interval

Suppose

$$X\sim\mathcal N(\mu,\sigma^2)$$

with known $\sigma$. Since $(X-\mu)/\sigma$ is standard normal,

$$P_\mu\!\left[ -z_{1-\alpha/2} \le\frac{X-\mu}{\sigma} \le z_{1-\alpha/2} \right] =1-\alpha.$$

Solving the inequalities for $\mu$ gives

$$C_\alpha(X) =\left[ X-z_{1-\alpha/2}\sigma,\, X+z_{1-\alpha/2}\sigma \right].$$

At 95%, $z_{0.975}=1.959964\ldots$. Coverage is exactly 0.95 for every real $\mu$, not merely on average over a prior.

The same set follows from the likelihood-ratio statistic

$$-2\ln\lambda(\mu) =-2\ln\frac{\mathcal L(\mu)} {\mathcal L(\widehat\mu)} =\frac{(X-\mu)^2}{\sigma^2}$$

by retaining values with $-2\ln\lambda(\mu)\le z_{1-\alpha/2}^2$. This exact chi-square relation depends on the linear Gaussian model with known variance. General waveform likelihoods often rely on asymptotic approximations that must be checked.

Confidence belts and ordering

A Neyman construction begins at each parameter value $\theta$ by selecting an acceptance region $A_\theta$ in data space with probability at least $1-\alpha$. After observing $d$, invert the belt:

$$C_\alpha(d) =\{\theta:d\in A_\theta\}.$$

Different orderings of data at fixed $\theta$ give different intervals with the same nominal coverage. Central, equal-tail, shortest, and likelihood-ratio orderings answer different design choices. Feldman and Cousins give a likelihood-ratio ordering that transitions between two-sided intervals and upper limits while preserving the construction near physical boundaries Feldman and Cousins 1998, §§II–IV, PDF.

An upper confidence limit $U_\alpha(d)$ is defined through

$$P_\theta[\theta\le U_\alpha(D)]\ge1-\alpha.$$

It is not a Bayesian upper credible bound unless a prior calculation independently gives the same numerical endpoint.

Boundaries and nuisance parameters

Physical parameters such as a rate, signal amplitude, eccentricity, or nonnegative deviation norm may lie on a boundary. Boundary handling must be specified precisely. Intersecting an already valid confidence set with $[0,\infty)$ preserves its coverage for every allowed true value, but can leave an empty set. Replacing a negative upper endpoint by zero, or switching to an upper limit only after inspecting the data, defines a different rule whose coverage needs checking. Standard likelihood-ratio asymptotics can also change because the maximum may sit on the boundary and parameters may be unidentified under the null.

Nuisance parameters include calibration, PSD shape, waveform coefficients, sky position, and other source quantities. A profile likelihood,

$$\lambda(\psi) =\frac{\mathcal L(\psi,\widehat{\widehat\eta}(\psi))} {\mathcal L(\widehat\psi,\widehat\eta)},$$

maximizes nuisance parameters $\eta$ conditionally in the numerator. Turning a profile-likelihood threshold into a confidence interval by a chi-square approximation requires regularity, sufficient information, and correct dimension. Marginalizing $\eta$ with a prior is Bayesian and does not by itself guarantee frequentist coverage.

For nonlinear, bounded, or weak signals, simulate the complete estimator and interval rule over a grid of injected parameters and nuisance conditions. This checks coverage at fixed truth. A related Bayesian calibration test draws truths from the prior and checks posterior ranks or credible-set inclusion averaged over those draws. Veitch et al. 2015, §V.C and Figure 10, PDF performs the latter test. Passing a prior-averaged test does not prove fixed-parameter frequentist coverage everywhere.

The distinction follows from the identity

$$\int \pi(\theta)P_\theta[\theta\in C(D)]\,d\theta =\int p(d)\left[\int_{C(d)}p(\theta\mid d)\,d\theta\right]dd =1-\alpha$$

for exact Bayesian credible sets of posterior mass $1-\alpha$. The average is guaranteed by a correctly specified model, while individual values of $P_\theta[\theta\in C(D)]$ may differ from $1-\alpha$.

Detection and analysis selection

If intervals are reported only for events passing a detection threshold, coverage in the selected sample need not equal unconditional coverage. The relevant target must be declared:

$$P_\theta[\theta\in C(D)] \quad\text{or}\quad P_\theta[\theta\in C(D)\mid D\in\mathcal S],$$

where $\mathcal S$ is the selection event. A procedure calibrated before selection can undercover or overcover after conditioning. Choosing a waveform model, parameter range, or interval type after inspecting the event adds further selection.

Coverage also depends on what generates repeated data. Gaussian injections test the code under its likelihood. Signals injected into real noise test non-Gaussian detector conditions. Alternative waveform and calibration injections test robustness to model misspecification. No single ensemble covers every systematic question.

Reproducible coverage studies

Record the data release or simulated-noise generator, detector GPS segments and sampling weights, calibration version or perturbation distribution, quality masks, PSD construction, waveform versions, injection parameter grid and drawing measure, number of trials per point, detection threshold and selected-sample rule, estimator and optimizer settings, interval ordering, nuisance treatment, random seeds, software commits, and failure or nonconvergence criteria.

For each grid point, report the total number of generated trials, successful fits, failures, covered intervals, and the denominator of the reported coverage estimate, together with a binomial uncertainty interval. For unconditional procedural coverage, every generated trial needs a defined output or failure treatment; a result conditioned on successful fits must be labeled separately. Plot coverage against the true parameter and relevant nuisance directions rather than reporting only a global average. Deleting failed fits without a predeclared rule can inflate coverage.

At the 2026-08-18 evidence cutoff, the O4b characterization software, likelihood inputs, and analysis selections are versioned in LVK Collaboration 2026, §§4–5, PDF. No catalog interval is evaluated here. A current application would need to pin the specific event product and rerun its complete estimator over a documented injection ensemble.

Common pitfalls

A 90% confidence interval does not contain a random fixed parameter with probability 0.9. The interval-producing rule covers the fixed parameter in 90% of repeated datasets under the model.

One injection is not a coverage test. Coverage is a binomial property of an ensemble at fixed truth. A single successful recovery checks one realization.

Asymptotic thresholds need validation. Low signal-to-noise ratio, boundaries, degeneracies, and look-elsewhere maximization can invalidate a nominal chi-square approximation.

Check your understanding

Let $X\sim\mathcal N(0,1)$. A researcher reports the usual 95% interval $[X-1.96,X+1.96]$ only when $X\gt1.645$, corresponding to a one-sided 5% detection threshold. Among reported intervals, what fraction cover the true value $\mu=0$?

Solution

The interval covers zero when

$$-1.96\le X\le1.96.$$

Conditioning on reporting leaves the range $1.645\lt X\le1.96$. Therefore

$$P(0\in C(X)\mid X\gt1.645) =\frac{\Phi(1.96)-\Phi(1.645)} {1-\Phi(1.645)} =\frac{0.975-0.950}{0.050} =0.50.$$

The unselected Gaussian interval has 95% coverage, but among threshold-selected reports it covers the null only 50% of the time. Selection-aware construction or an explicitly conditional target is required.

References

• Feldman, Gary J., and Robert D. Cousins. “Unified Approach to the Classical Statistical Analysis of Small Signals.” Physical Review D 57 (1998): 3873–3889. DOI. Open PDF.
• LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration. “GWTC-5.0: Methods for Identifying and Characterizing Gravitational-Wave Transients.” arXiv:2605.27224 [gr-qc] (2026). Open PDF.
• Veitch, John, et al. “Parameter Estimation for Compact Binaries with Ground-Based Gravitational-Wave Observations Using the LALInference Software Library.” Physical Review D 91 (2015): 042003. DOI. Open PDF.