Benchmarks and Independent Code Comparison

Cross-code agreement is strongest when implementations reach the same continuum observable through genuinely different numerical routes. It is weakest when two programs share equations, initial-data generators, libraries, post-processing, and hidden conventions. A useful comparison therefore freezes the physical problem while documenting software lineage and allowing each method to establish its own convergence before results are placed on a common footing.

Required background. Probability and Inference for Gravity supplies residual metrics and interpretation of correlated evidence.

Helpful background. Formulations of General Relativity supplies the continuum systems and gauges; Differential Equations for Relativity supplies method and convergence comparisons; Computational Relativity supplies the shared workflow; and Mathematical Relativity supplies the continuum domain that numerical agreement cannot prove.

Freeze the problem, not the algorithm

A comparison specification fixes the continuum equations, physical parameters, initial data, gauge source, boundary conditions, units, sign conventions, output times, extraction observable, and comparison norm. It also states acceptable coordinate or frame transformations. Each code may retain its native discretization, mesh, time integrator, solver, and data layout; forcing identical algorithms would remove much of the independence.

When exact common initial data cannot be imported, compare invariant or asymptotic quantities and quantify the initial-data difference. A binary waveform comparison must freeze masses and spins, eccentricity or orbital frequency, reference frame, harmonic convention, extraction and extrapolation definitions, and analysis interval. A hydrodynamic comparison must freeze equation of state, primitive/conserved definitions, atmosphere or floors, and Riemann problem.

The Apples-with-Apples suite gives standardized robust-stability, gauge-wave, linear-wave, and related tests. Babiuc and collaborators compare implementations and failure behavior in Babiuc et al. 2008, §§3–8. These tests reveal formulation and boundary differences, but sharing a test does not make participating codes independent.

Establish each continuum estimate first

For every code, run at least three resolutions or tolerances and demonstrate its own asymptotic regime. Extrapolate each result with its own observed order and uncertainty. Only then compare continuum estimates. Rescaling one code’s differences by another code’s formal order can manufacture visual agreement.

Common post-processing should be minimal and transparent. If both codes use the same waveform extrapolation script, a shared sign, time, or radius defect can survive. Prefer independent extraction or extrapolation implementations, then translate their outputs with a small checked converter. Compare unaligned quantities before applying a declared time and phase alignment.

For complex waveforms, define the shifted comparison waveform and a real noise-weighted inner product by

$$h_2^{(\Delta t,\Delta\phi)}(t) =e^{i\Delta\phi}h_2(t+\Delta t), \qquad \langle a,b\rangle_{\mathbb R} =4\operatorname{Re}\!\int_{f_{\min}}^{f_{\max}} \frac{\widetilde a(f)\widetilde b(f)^*}{S_n(f)}\,df.$$

One scoped mismatch is then

$$\mathcal M =1-\max_{\Delta t,\Delta\phi} \frac{\langle h_1,h_2^{(\Delta t,\Delta\phi)}\rangle_{\mathbb R}} {\sqrt{\langle h_1,h_1\rangle_{\mathbb R} \langle h_2^{(\Delta t,\Delta\phi)},h_2^{(\Delta t,\Delta\phi)}\rangle_{\mathbb R}}},$$

where the noise curve $S_n$, frequency interval, Fourier convention, mode content, and maximized transformations are specified. A value below $10^{-3}$ can be a useful target in one declared application, but it is not a universal accuracy threshold.

Independence has several dimensions

Record whether codes share the continuum formulation, gauge, initial-data solver, mesh library, reconstruction or derivative kernels, equation-of-state table, random generator, compiler, analysis package, and developer lineage. Independence can come from a different formulation, discretization family, coordinate system, code base, analytic oracle, or physical balance law. Two wrappers around one kernel are not two implementations.

The Samurai project compared binary-black-hole waveforms from multiple numerical-relativity codes. Its protocol and waveform results appear in Hannam et al. 2009, §II, §III, Figs. 3–7, §IV, Fig. 8, and §V. The lesson is conditional: agreement was established for named configurations, waveform portions, alignments, and errors. It cannot be transferred automatically to another mass ratio, spin, mode, or eccentricity.

When codes disagree, localize the source instead of averaging. Compare initial data, constraints, gauge diagnostics, conserved quantities, extraction radii, raw modes, and post-processing stages. A benchmark is valuable when disagreement identifies a missing term or regime, not only when a summary plot overlaps.

Reproducible comparison package

The environment record should pin every implementation and dependency revision, compiler, precision, hardware-sensitive settings, input checksums, continuum specification, native meshes, solver tolerances, extraction and analysis software, and converter. Each participant retains raw outputs and independent convergence evidence.

A clean rerun would execute the common scalar-wave and Apples-with-Apples cases, then one physical waveform benchmark. It would refine each code separately, compute independent continuum estimates, translate conventions through tested adapters, and emit norms, mismatches, lineage metadata, and unresolved differences. This page defines the sequence without claiming an absent comparison run.

Reject the comparison claim if continuum problems differ, one code lacks resolution evidence, alignment or filtering is optimized after inspecting disagreement, shared dependencies are described as independent, or a scoped threshold is exported beyond its tested regime. The central failure mode is correlated agreement: the same inherited error can make two outputs nearly identical.

Check your understanding

Two evolution codes use different spacetime formulations and grids. Their waveforms agree to mismatch $4\times10^{-4}$, but both import the same initial data and use the same extraction and extrapolation package. What independent evidence does the comparison provide?

Solution

It provides meaningful evidence about the independent evolution formulations within the tested regime, provided each code converges. It does not independently test the shared initial data or waveform pipeline. Compare raw finite-radius quantities, use an independent extraction or characteristic method, and test the initial data against separate constraints and invariants. Report the lineage explicitly rather than calling the entire end-to-end chain independent.

References

• Babiuc, Maria C., et al. 2008. “Implementation of Standard Testbeds for Numerical Relativity.” Classical and Quantum Gravity 25, 125012. arXiv.
• Hannam, Mark, et al. 2009. “The Samurai Project: Verifying the Consistency of Black-Hole-Binary Waveforms for Gravitational-Wave Detection.” Physical Review D 79, 084025. arXiv.