Verification, Validation, Uncertainty, and Computational Evidence

This chapter closes the volume by asking exactly what a computation has established. It separates implementation correctness, numerical error in a particular solution, comparison across methods, agreement with empirical evidence, uncertainty propagation, and the record needed for another researcher to falsify the result. The sequence prevents a convergent plot or available archive from carrying more scientific weight than it has earned.

Helpful background. Formulations of General Relativity supplies the continuum systems and conventions; Differential Equations for Relativity supplies consistency and convergence; Computational Relativity supplies the volume-wide methods; Probability and Inference for Gravity supplies uncertainty and statistical comparisons; and Mathematical Relativity supplies the continuum hypotheses that numerical evidence does not prove.

Enter this chapter

Begin with Code Verification and Unit Tests, which combines algebraic, operator, integration, and system tests with deliberate mutations. Solution Verification and Convergence then estimates error in a named computed result using residual control, multi-resolution evidence, Richardson analysis, and asymptotic-regime checks.

Method of Manufactured Solutions creates exact forced problems that exercise otherwise dormant terms, boundaries, and mesh interfaces. Benchmarks and Independent Code Comparison asks when different numerical routes provide genuinely independent evidence. Uncertainty Quantification and Propagation separates discretization, input, stochastic, calibration, and model components, including disputed discrepancy models. Reproducing and Sharing Computational Results specifies the immutable, licensed, rerunnable record supporting the final finite claim.

This is a cumulative order. Release packaging cannot compensate for missing verification, and uncertainty propagation cannot repair an unidentified implementation error.

Six questions for one result

Question	Primary evidence	Typical failure
Did the code implement the intended equations?	Exact identities, operator tests, mutations	Tests never excite the faulty term
How accurate is this computed solution?	Residuals and multi-resolution regime	Richardson fit before asymptotic convergence
Were all code paths exercised?	Manufactured forcing and boundary tests	Symmetry makes couplings vanish
Does another route agree?	Independent formulation or implementation	Shared initial data or analysis defect
How does uncertainty reach the observable?	Component-wise propagation and coverage	Correlations or discrepancy are hidden
Can another researcher test the claim?	DOI-tagged clean rerun and negative case	Availability is mistaken for functionality

Validation adds a seventh question: does the verified model agree with relevant observations or experiments closely enough for its intended use? That requires versioned data, calibration, selections, likelihood or comparison metric, and model-discrepancy treatment. Analytic solutions and code comparisons verify computation; they are not empirical validation.

Roy and Oberkampf organize these distinctions in Roy and Oberkampf 2011, § “Uncertainty Framework,” nozzle example, and Conclusions. The boundaries matter because different evidence types cannot be exchanged one for one.

A chapter-wide benchmark chain

Use one scalar-wave problem to connect the pages. Unit tests verify derivatives and tensor operations. A manufactured smooth wave makes every implemented term nonzero and supplies exact $L_1$, $L_2$, and $L_\infty$ errors. Three or more grids establish observed order and a conditional discretization estimate. A second formulation or code solves frozen common data and compares independent continuum estimates. Correlated uncertain inputs are then propagated to a declared wave observable with sampling and coverage checks.

The Apples-with-Apples robust-stability, linear-wave, and gauge-wave tests broaden the chain, while a physical benchmark—such as a Kerr horizon, relativistic shock, or extracted wave—connects it to the production observable. No single test covers every layer. In particular, smooth manufactured convergence does not verify shock capturing, and a physical result without an exact oracle may not isolate code error.

Every rerun record should pin equations, gauges, boundaries, code and dependency revisions, compiler or interpreter, precision, inputs and checksums, meshes, tolerances, norms, random seeds, analysis scripts, expected runtime, and rejection criteria. Wilson and collaborators explain why automated tests and explicit planning for mistakes belong in the scientific workflow in Wilson et al. 2014, § “Plan for Mistakes” and Box 1, item 5.

What not to combine

Keep iterative residual, discretization estimate, model discrepancy, calibration error, and ensemble variability visible until their interpretations and correlations justify combination. A grid-convergence index is not a statistical confidence interval. Spread among model families is not automatically a probability distribution. Agreement between dependent codes is not independent confirmation. A DOI proves persistent identification, not a successful build.

The computational claim should end with its domain and failure criteria. State the parameter region, regularity, resolution sequence, boundary and gauge choices, observable, tolerance, uncertainty components, independent check, and known breakdown. If evidence changes, issue a versioned correction rather than silently replacing the record.

Review the chapter

A team has a clean, DOI-tagged package and two codes whose continuum-extrapolated waveforms agree. Both codes share the same initial-data solver and analysis package, and neither has run a manufactured boundary test. Which evidence is strong, and what remains open?

Solution

The package supplies availability and potentially functional rerun evidence, while the independently evolved continuum estimates support the evolution stage. The shared initial data and analysis remain correlated, and the untested boundary path remains an implementation gap. Add manufactured boundary and mutation tests, verify initial data with independent constraints or construction, and repeat waveform extraction through a separate route before describing the end-to-end result as independently confirmed.

References

• Roy, Christopher J., and William L. Oberkampf. 2011. “A Comprehensive Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing.” Computer Methods in Applied Mechanics and Engineering 200, 2131–2144. DOI.
• Wilson, Greg, et al. 2014. “Best Practices for Scientific Computing.” PLOS Biology 12, e1001745. DOI.