Code Verification and Unit Tests

Code verification asks whether an implementation solves the equations and discrete algorithm that its authors intended. It does not ask whether those equations describe nature or whether a particular production solution is sufficiently accurate. The strongest verification program combines exact algebraic tests, discrete-operator tests, coupled evolution benchmarks, and deliberate defects that demonstrate the tests can actually fail.

Required background. Probability and Inference for Gravity supplies residual summaries, stochastic-test interpretation, and evidence calibration.

Helpful background. Formulations of General Relativity supplies the continuum equations and constraint structure; Differential Equations for Relativity supplies discretization and stability concepts; Computational Relativity supplies the volume-wide workflow; and Mathematical Relativity supplies the continuum hypotheses that tests must not silently replace.

What code verification can establish

Separate four questions. A unit test checks a local function or identity. An operator test applies a discrete derivative, interpolation, tensor contraction, or boundary closure to known data. An integration test checks coupled modules and data movement. A system benchmark evolves a complete problem whose expected behavior is known. Passing one level does not imply the next.

Examples from relativity include symmetry and trace tests for tensor objects, reconstruction of a metric inverse, finite-difference action on polynomials or Fourier modes, recovery of a known curvature tensor, characteristic-speed checks, and constraint propagation for an exact solution. Every comparison needs an absolute or scaled residual and a tolerance derived from truncation and roundoff, not from the current output.

Scientific-software practice also requires tests to accompany the program rather than live only in a researcher’s memory. Wilson and collaborators make automated testing and planning for mistakes explicit in Wilson et al. 2014, Box 1, item 5, and “Plan for Mistakes”. A regression fixture records intentional behavior, but it is not an exact oracle: a wrong result can be preserved perfectly.

A derivative test you can work out by hand

For the centered fourth-order first derivative, write

$$D_h f(x)=\frac1h\sum_{m=-2}^{2}c_m f(x+mh), \qquad (c_{-2},c_{-1},c_0,c_1,c_2) =\frac1{12}(1,-8,0,8,-1).$$

Taylor expansion says the coefficient of $h^{s-1}f^{(s)}$ is $\sum_m c_m m^s/s!$. The stencil therefore differentiates constants through quartics exactly when

$$\sum_m c_m=0,\qquad \sum_m m c_m=1,\qquad \sum_m m^s c_m=0\quad(s=2,3,4).$$

The next moment is $\sum_m m^5c_m=-4$, giving $D_hf=f'-h^4 f^{(5)}/30+O(h^6)$ for a sufficiently smooth function. These are small, exact rational checks, independent of any evolution. Test the stencil on interior points for polynomials; wrapping a nonperiodic polynomial across a periodic boundary changes the test problem. Periodic Fourier modes instead test the wraparound indexing and the modified wavenumber together.

Changing only $c_2=-1/12$ to $+1/12$ makes the constant-field derivative nonzero, so even the simplest test detects that defect. Reversing all coefficients preserves the zero moments but changes $\sum m c_m$ to $-1$: a constant-only test misses the sign error, while a linear-field test detects it. This is why several independent identities are more informative than many repeats of one fixture.

Tests that exercise coupled relativity code

The Apples-with-Apples family supplies standardized strong and weak tests. Robust stability evolves small random perturbations to reveal unstable growth. A linear wave has a known propagating solution and convergence rate. A gauge wave is flat spacetime in nontrivial coordinates, so physical curvature should remain zero even though metric components evolve. Alcubierre and collaborators define the original test logic in Alcubierre et al. 2004, §§II–V, and the later multi-code implementation study refines procedures in Babiuc et al. 2008, §§3–8.

These benchmarks are discriminating only when their regime is respected. A short robust-stability run can miss a slow instability. A tiny linear-wave amplitude may never exercise nonlinear terms. A gauge wave can trigger formulation-dependent continuum instabilities as well as programming errors. Record amplitude, wavelength, domain, gauge, boundary conditions, run time, norms, and expected convergence before classifying a failure.

Add a physical benchmark appropriate to the claimed observable—for example Kerr horizon area or a relativistic shock tube—because a standardized spacetime test may not exercise matter, extraction, or mesh-refinement interfaces. This remains code verification when the expected mathematical result is prescribed.

Testing whether checks detect known faults

A test suite that always passes may never touch the dangerous line. Deliberately introduce one defect at a time in an isolated copy: reverse a curvature sign, exchange a covariant and contravariant index, drop a coupled source term, perturb a stencil coefficient, change a boundary normal, or swap polarization handedness. The relevant test must fail for the predicted reason. Restore the implementation and require it to pass again.

Mutation does not prove the absence of all defects, but it tests the sensitivity of the evidence. Choose mutations from the equation structure and known failure history rather than random text edits. A tensor-unit test should catch the index mutation; a manufactured or exact-solution convergence test should catch the stencil mutation; a boundary benchmark should catch the normal reversal. If only a distant end-to-end image changes, the diagnostic is too coarse.

For floating-point or parallel reductions, distinguish deterministic identities from statistical expectations. Repeat nondeterministic tests, report distributional tolerances, and avoid accepting a wide interval solely to suppress intermittent failures.

Reproducible verification record

The environment record should pin the equations, implementation and dependency revisions, compiler or interpreter, optimization flags, precision, hardware-sensitive math settings, test data checksums, meshes, solvers, and harness. Each test records inputs, oracle, norm, tolerance, expected order, and failure message. Machine-readable output should include the result before and after every declared mutation.

A clean rerun sequence would execute unit and operator tests, the robust-stability, linear-wave, and gauge-wave suite at three resolutions, then apply the sign, index, and stencil mutations separately. It would emit constraint and solution norms, observed convergence, environment metadata, and mutation outcomes. This page specifies that interface; it does not claim an absent repository program performed it.

Reject the verification claim if a deliberate mutation survives, if tolerances merely match the current discrepancy, if a regression value lacks an independent oracle, if refinement does not enter the expected regime, or if a benchmark omits coupled boundaries and interfaces used in production. The primary failure mode is confusing broad execution coverage with sensitivity to scientifically consequential errors.

Check your understanding

A gauge-wave test converges at second order before and after the sign of one constraint-damping term is reversed. Unit tests also pass because they cover only the principal derivative operator. Has the sign been verified?

Solution

No. The surviving mutation shows that the current tests do not excite or independently evaluate that term. Construct a fixture with nonzero damping constraint, test the source term directly, and run a coupled problem long enough for its predicted effect to appear. Passing convergence only shows that both implementations consistently solve their respective discrete equations.

References

• Alcubierre, Miguel, et al. 2004. “Toward Standard Testbeds for Numerical Relativity.” Classical and Quantum Gravity 21, 589–613. arXiv.
• Babiuc, Maria C., et al. 2008. “Implementation of Standard Testbeds for Numerical Relativity.” Classical and Quantum Gravity 25, 125012. arXiv.
• Wilson, Greg, et al. 2014. “Best Practices for Scientific Computing.” PLOS Biology 12, e1001745. DOI.