Calibration and Noise Marginalization

Compact-binary parameters are inferred from a calibrated strain channel and a statistical model for its noise. Neither object is known exactly. Frequency-dependent calibration errors alter waveform amplitude and phase, while uncertainty in the power spectrum changes the weighting and normalization of the likelihood. Propagating both means introducing explicit nuisance variables and integrating them out; fixing them at point estimates generally understates uncertainty and can shift correlated source parameters.

Helpful background. Calibration Response and Uncertainty defines the detector response products, and The Gravitational-Wave Likelihood supplies the probability model marginalized here.

Complex calibration uncertainty

Let $d_I(f)$ be the strain reconstructed with nominal detector response $R_{0,I}$. Define the factor that maps physical strain into these nominal coordinates by

$$C_I(f;\lambda_I) =\frac{R_{0,I}(f)}{R_{{\rm true},I}(f)} =\bigl[1+\delta A_I(f;\lambda_I)\bigr] e^{i\delta\phi_I(f;\lambda_I)},$$

so the residual is

$$r_I(f) =d_I(f)-C_I(f;\lambda_I)h_I(f;\theta).$$

The nuisance vector $\lambda_I$ may be values at spline nodes, principal-component coefficients, or physically motivated response parameters. Its prior must come from the calibration measurement for the same detector epoch—not from an arbitrary smoothness choice. To first order,

$$C_Ih_I \simeq h_I+\delta A_Ih_I+i\delta\phi_Ih_I.$$

Amplitude error correlates strongly with luminosity distance and orientation. Phase error can correlate with coalescence phase, timing, masses, and spins; relative detector phase also affects sky and polarization inference. Detector-response provenance and parameter-estimation treatment are described in LIGO and Virgo 2020, §§6.1 and 9.3.

Noise spectra are inferred quantities

For a fixed one-sided PSD $S_{n,I}(f)$, the frequency-domain Gaussian likelihood is

$$\ln\mathcal L =-\frac12\sum_I(r_I\mid r_I)_I -\sum_{I,k\gt0}\ln S_{n,I}(f_k)+C,$$

where $k\gt0$ labels independent positive-frequency complex bins away from DC and Nyquist. Each such bin contains two real quadratures, which gives one $\ln S_n$ rather than one half; an explicit real-quadrature formulation gives the same normalization after both quadratures are counted. The second term matters whenever the PSD itself varies across hypotheses. In real data, the PSD is estimated from finite neighboring or on-source data. Windowing, segment duration, spectral lines, nonstationarity, and signal leakage all affect it.

One approach assigns spectral correction variables $\xi$ and writes

$$S_{n,I}(f;\xi) =S_{0,I}(f)\exp\!\left[\sum_a \xi_{Ia}B_a(f)\right],$$

possibly augmented by explicit line components. Marginalizing a variance with an inverse-gamma prior produces a Student-$t$-like heavy-tailed likelihood in a simple independent-bin model. That construction is robust to finite variance uncertainty, but it does not by itself model glitches, correlations between bins, or time variation.

Joint marginalization

The source posterior is

$$p(\theta\mid d) \propto \pi(\theta) \int d\lambda\,d\xi\, \mathcal L(d\mid\theta,\lambda,\xi) \pi(\lambda)\pi(\xi).$$

The factorization $\pi(\lambda,\xi)=\pi(\lambda)\pi(\xi)$ is an assumption. It can fail if the calibration procedure, line subtraction, and PSD estimation share data or instrument states. A robust analysis either models consequential correlations or demonstrates through injections that ignoring them has negligible effect in the stated regime.

Compact-binary likelihood construction, PSD treatment, sampler validation, and injection tests are developed together in Veitch et al. 2015, §§ II–V.

Three shortcuts answer different questions:

Treatment	What it computes	What it misses
Fix calibration and PSD	Conditional posterior at nominal products	Nuisance uncertainty and possible shifts
Draw calibration/PSD curves after sampling	Approximate sensitivity if reweighting is valid	Strong source–nuisance coupling or missing support
Sample source and nuisance variables jointly	Posterior under the declared joint model	Misspecification of the nuisance family itself

Nonstationarity and outliers

A smooth PSD correction cannot make a transient glitch stationary Gaussian noise. Useful alternatives include explicit glitch components, time-dependent covariance, segmented likelihoods, or heavy-tailed mixture models. Each changes the generative statement. Gating data, subtracting a reconstructed transient, or shortening the analysis band also changes the data being conditioned on and must be reported.

Posterior predictive residuals should be checked in each detector. Whitened residuals that retain coherent chirp structure suggest waveform or calibration failure; isolated excesses may indicate a glitch model failure; broad spectral structure may indicate PSD misspecification. A residual consistent with the assumed distribution does not prove that source parameters are unbiased, but an inconsistent residual falsifies the model more directly than a visually pleasing posterior.

Reproducible validation protocol

The current observational boundary is 2026-08-18. Any event-level use must bind the exact GWTC-5.0 release, detector and GPS segment, strain and calibration version, data-quality mask, PSD construction, waveform family and modes, frequency band, cosmology, priors, sampler version, and random seed. GWTC-5.0 Methods, §§ 4–5 is the official current analysis description through O4b.

A controlled benchmark can isolate the nuisance effects:

1. inject a known compact-binary signal into controlled noise;
2. recover it with fixed nominal calibration and PSD;
3. repeat with known amplitude, phase, and spectral perturbations;
4. enable joint nuisance marginalization;
5. vary spline or basis resolution, PSD segment length, sampler tolerance, and random seed; and
6. report source-parameter bias, coverage, residual diagnostics, effective sample size, and Monte Carlo error.

Recovery should approach the fixed-nuisance result as nuisance priors collapse to delta functions. If increasing nuisance resolution causes unstable evidence or unconstrained oscillations, the basis is too flexible or the data do not identify it. Failure to recover the injected parameters after marginalization can expose a nuisance family with insufficient support rather than a need for wider source priors.

Exercise

Let a measured scalar amplitude obey $d=Ca+n$, where $n\sim\mathcal N(0,\sigma^2)$, $C=1+\epsilon$, and $\epsilon\sim\mathcal N(0,\tau^2)$. Treat $a$ as fixed. Find the likelihood for $d$ after marginalizing calibration error.

Solution

Rewrite the measurement as

$$d=a+a\epsilon+n.$$

The independent Gaussian variables $a\epsilon$ and $n$ have variances $a^2\tau^2$ and $\sigma^2$. Their sum is Gaussian, so

$$p(d\mid a) = \frac{1}{\sqrt{2\pi(\sigma^2+a^2\tau^2)}} \exp\!\left[ -\frac{(d-a)^2}{2(\sigma^2+a^2\tau^2)} \right].$$

Calibration marginalization adds an amplitude-dependent variance. Simply replacing $\sigma^2$ by a constant enlarged error would miss that dependence and can distort the posterior for $a$.

References

• LIGO Scientific Collaboration and Virgo Collaboration (2020). “A Guide to LIGO–Virgo Detector Noise and Extraction of Transient Gravitational-Wave Signals.” Classical and Quantum Gravity 37, 055002. DOI. Open PDF.
• LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration (2026). “GWTC-5.0: Methods for Identifying and Characterizing Gravitational-Wave Transients.” arXiv:2605.27224. Open PDF.
• LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration (2026). GWTC-5.0 Data Release. DOI 10.7935/bk00-6a89. Official release.
• Veitch, J., et al. (2015). “Parameter Estimation for Compact Binaries with Ground-Based Gravitational-Wave Observations Using the LALInference Software Library.” Physical Review D 91, 042003. DOI.