Synthetic-Observation Validation

A synthetic image can be numerically reproducible and still be physically misleading. Confidence comes from a ladder of increasingly coupled tests: exact rays, analytic transfer, identical-model code comparisons, identical simulation snapshots, instrument-domain transforms, and only then comparison with calibrated observations. Each rung answers a different question and must retain its own resolution, calibration, model, and statistical uncertainty.

Required background. Affine and Variational Geodesics supplies the ray geometry underlying every synthetic observable.

Helpful background. Formulations of General Relativity supplies metric and frame data; Differential Equations for Relativity supplies numerical error analysis; Black-Hole Astrophysics supplies source and emission models; and Computational Relativity supplies the shared verification framework.

A validation ladder

The first rung tests null integration against flat, Schwarzschild, and Kerr trajectories, including null and conserved-quantity residuals. The second tests unpolarized and polarized transfer against homogeneous slabs and analytic Faraday solutions. These are code-verification tests: they establish that chosen equations are solved correctly in controlled regimes, not that an astrophysical source model is true.

The third rung gives independent codes exactly the same analytic metric, coefficients, camera, frequency, image grid, units, and masks. The fourth replaces the analytic source with the same versioned GRMHD snapshot and identical electron prescription. Differences now include interpolation and file-interface choices, so the comparison configuration must say how cell centers, primitive variables, floors, and coordinate maps are interpreted.

Gold and collaborators define such a progression in Gold et al. 2020, §§3.1–3.3 and §§5.1–5.5. Their approximately percent-level flux agreement is a result for the named tests and inputs, not a universal pass threshold.

The fifth rung transforms images into instrument-domain products. For an ideal narrow-field intensity image,

$$\mathcal V(u,v)= \int I(\alpha,\beta) e^{-2\pi i(u\alpha+v\beta)}\,d\alpha\,d\beta$$

defines complex visibility under one Fourier convention. Here $(\alpha,\beta)$ are sky angles in radians, $(u,v)$ are projected baseline components measured in wavelengths, and $\mathcal V$ has flux-density units if $I$ is specific intensity. The visibility coordinate $u$ is unrelated to the observer four-velocity. Pixel quadrature, field truncation, scattering, station response, time and bandwidth averaging, and calibration enter before comparison with interferometric data. The sixth rung evaluates a declared likelihood against a versioned data release. That final step is outside this page unless all instrument and inference assumptions are actually supplied.

An image-to-visibility calculation

A circular Gaussian with total flux density $F$ and angular width $\sigma$ has

$$I(\alpha,\beta)=\frac{F}{2\pi\sigma^2} \exp\!\left[-\frac{\alpha^2+\beta^2}{2\sigma^2}\right].$$

Separating the two Gaussian Fourier integrals gives

$$\mathcal V(u,v)=F \exp[-2\pi^2\sigma^2(u^2+v^2)].$$

The zero-baseline value is $F$, and the half-amplitude baseline length is $\sqrt{\ln2}/(\sqrt2\,\pi\sigma)$. Increasing source size makes visibility decay faster with baseline length.

Moving the image center by $(\alpha_0,\beta_0)$ multiplies visibility by $e^{-2\pi i(u\alpha_0+v\beta_0)}$ without changing its amplitude. Together these identities test flux normalization, radians versus other angular units, Fourier sign, origin, and pixel quadrature. A finite image window or pixel grid should approach these infinite-plane results under separate field-of-view and resolution refinement.

Versioned data and comparison metrics

Pin the analytic benchmark suite and common GRMHD inputs to an exact repository revision with checksums. For the polarized suite, retain the Prather et al. input definitions and code revisions, not merely “the 2023 tests.” If a public 2019 M87 product is used, identify EHT data release DOI 10.25739/g85n-f134, its calibration pipeline, flags, and derived-product version. Synthetic images must never be called observations. The evidence cutoff is 2026-08-18.

For image arrays $X_p$ and $Y_p$, one useful declared metric is

$$\operatorname{NMSE}(X,Y) =\frac{\sum_p(X_p-Y_p)^2} {\sum_p X_p^2}.$$

Complex or signed Stokes data require the corresponding magnitude or real-component definition. Also report total-flux difference, maximum absolute residual, and separate $I,Q,U,V$ results. Mask low-intensity pixels before using fractional polarization, with the mask fixed in advance. Visibilities need amplitude and phase residuals at the same sampled baselines rather than a comparison of differently interpolated plots.

Prather and collaborators give analytic and shared-snapshot polarized tests in Prather et al. 2023, §§4.1–4.4, Eqs. (12)–(19), Tables 1–3, and Figs. 3–7. Their Fig. 5 gives a largest thin-disk pairwise image NMSE of about $0.012$; Fig. 7 lists shared-GRMHD maxima $0.022$, $0.042$, $0.042$, and $0.13$ for $I,Q,U,V$. These are cross-code image differences, not errors against an exact image. The analytic transfer tests are separate (§4.1 and Fig. 3). These conditional outcomes are comparison targets, not acceptance limits for another model or release.

Calibration, selections, and inference boundaries

Calibration records camera scale and orientation, tetrad and polarization conventions, frequency and flux units, Fourier sign, angular coordinates, image origin, and any station or scattering response. Selections fix metric and plasma model, snapshot or time interval, inclination, field of view, pixel and ray masks, frequency band, density or magnetization cuts, image alignment, baselines, and quality cuts before inspecting residuals.

Rigid rotation or translation may be applied only when it represents a declared camera or coordinate transformation. Optimizing alignment separately for every code can erase exactly the handedness or origin error that the comparison should find. Flux renormalization similarly removes one test; show unnormalized results first.

Model and statistical assumptions must remain visible. A deterministic code comparison has residual metrics but no likelihood or prior. An instrument comparison requires a noise model, covariance, calibration nuisance parameters, selection function, and declared priors. Numerical uncertainty from ray, transfer, interpolation, pixel, and time resolution is distinct from uncertainty across electron prescriptions, GRMHD states, or instrument calibration. Systematic variations should change these model and calibration choices one controlled group at a time and report which dominates each observable.

Refinement and reproduction

Refine the ladder rather than only the final image. Sweep geodesic tolerance; transfer and Faraday step; metric and fluid interpolation; image and frequency resolution; snapshot cadence; Fourier quadrature; and instrument sampling. Repeat in alternate coordinates and tetrads where possible. Cross-code agreement at one resolution can reflect the same under-resolved input or shared formula error, so retain analytic invariants and direct coefficient integrals as independent checks.

The environment record should pin every input checksum, software and library revision, precision, configuration, random seed if used, and machine-readable output. A clean reconstruction proceeds from metric and fluid data through camera, rays, coefficients, Stokes images, spectra, visibilities, and comparison metrics without manual plot editing. No such absent executable is claimed here.

Reject the finite claim if analytic limits fail, refinement plateaus, results require unreported alignment, coordinate or tetrad changes alter invariant observables, cross-code differences exceed a predeclared target without explanation, or emission and calibration systematics dominate an effect attributed to geometry. The central failure mode is skipping a rung and treating agreement with one processed image as validation of the whole forward model.

Check your understanding

Two codes produce images with NMSE $0.002$ after each image is independently shifted, rotated, and flux-normalized to minimize the residual. Before alignment the images are mirror reflections with a $7\%$ flux difference. Does the quoted NMSE establish agreement?

Solution

No. The fitted operations removed tests of camera handedness, origin, and absolute calibration. First apply only the transformation predicted by the recorded tetrads and coordinate maps, then compare unnormalized flux and orientation. If a mirror reflection remains, it is a convention or implementation failure, not a nuisance alignment. Report any later instrument-level nuisance fit separately from numerical verification.

Continue with the chapter overview.

References

• Event Horizon Telescope Collaboration. 2019. First M87 EHT Results: Calibrated Data. CyVerse Data Commons. Dataset DOI.
• Gold, Roman, et al. 2020. “Verification of Radiative Transfer Schemes for the Event Horizon Telescope.” The Astrophysical Journal 897, 148. DOI.
• Prather, Cora, et al. 2023. “Comparison of Polarized Radiative Transfer Codes Used by the Event Horizon Telescope Collaboration.” The Astrophysical Journal 950, 35. arXiv.