Computational Experiments and Reproducible Workflows

A computational result becomes scientific evidence only when its finite claim can be reconstructed and challenged. This chapter turns a relativistic question into a declared model, algorithm, environment, data contract, workflow, comparison, and preservation package. Its recurring scalar wave is deliberately elementary: because the continuum answer is known, every mismatch can be traced to discretization, arithmetic, conventions, execution, or incomplete provenance rather than uncertain physics.

Helpful background. Formulations of General Relativity supplies the continuum variables and conventions that numerical records must preserve; Differential Equations for Relativity supplies initial-value problems and convergence; Calculus and Linear Algebra for Relativity supplies norms and conditioning; Probability and Inference for Gravity supplies ensemble and uncertainty reasoning.

Enter the computational experiment

The chapter’s central claim has the form

$$Q_{h,\tau,u}(\mathcal M,\mathcal I,\mathcal C;\mathcal A,\mathcal E) =Q\pm\Delta Q$$

over a stated regime. Here $\mathcal M$ is the continuum model, $\mathcal I$ its physical data, $\mathcal C$ the units and conventions, $\mathcal A$ the algorithm, $\mathcal E$ the software and hardware environment, and $(h,\tau,u)$ summarize discretization, solver tolerance, and arithmetic. The uncertainty $\Delta Q$ is not licensed by notation; it must be earned through refinement, residuals, invariants, boundary studies, independent calculations, and—when randomness enters—sampling evidence.

The finite claim is narrower than “the code works.” A smooth periodic wave can verify a fourth-order derivative without testing shocks. A Schwarzschild invariant can expose a sign or unit error without testing a binary evolution. A byte-identical rerun can reproduce a bug. The experiment states both what its benchmark exercises and what it leaves untested.

The National Academies distinguishes computational reproducibility from independently obtained scientific agreement and explains the data, method, parameter, and environment record required for the former in National Academies 2019, Summary pp. 6–8 and Chapter 3. This chapter builds that distinction into the calculation itself.

The six-page route

Read the pages in order when designing a new computation:

1. From Scientific Question to Computational Experiment defines the finite claim, input and output record, numerical controls, falsifiable thresholds, and clean-rerun sequence.
2. Environments, Dependencies, and Precision separates pinned reconstruction from deliberate variation of compilers, libraries, hardware, arithmetic, and reduction order.
3. Units, Conventions, and Data Contracts makes coordinates, units, tensor layout, signature, normalization, schema, and conversion tests machine-checkable.
4. Workflow Automation and Provenance represents every result as a product of declared, checksum-bearing dependencies and tests cache invalidation and hidden manual work.
5. Determinism, Randomness, and Ensemble Runs assigns bitwise, numerical-tolerance, or statistical agreement to each product and maps random draws to immutable member identities.
6. Archiving and Independent Reproduction packages source, inputs, environment, workflow, semantics, evidence, licenses, and version identifiers for an uninvolved clean rerun.

The pages form one chain. Environment metadata cannot repair an ambiguous tensor convention. A complete schema cannot reconstruct a hidden notebook edit. A workflow graph cannot validate an incorrect equation. An archive cannot supply evidence that was never generated.

Testing a scalar-wave calculation step by step

The common benchmark is the right-moving solution

$$u(t,x)=\sin\!\bigl(2(x-t)\bigr), \qquad 0\le x<2\pi,$$

with periodic boundary conditions. A fourth-order centered derivative and fourth-order Runge–Kutta method evolve the equivalent first-order system at a maximum Courant factor $1/4$ to $t=1$ on $N=32,64,128,256$ points; the step is shortened uniformly to reach that final time exactly. Its continuum energy is $4\pi$. Because continuum-exact initial derivatives leave a known fourth-order discrete-constraint residual, the modified wavenumber determines a full analytic semidiscrete field with both propagating and stationary pieces. That prediction checks the field norm without trusting the evolution code.

The first page supplies a runnable program and measured convergence results: the expected fourth-order sequence, the $E_{256}<10^{-6}$ target, temporal refinement, energy drift, and failure status. The second designs an extension across precision, compiler settings, and thread counts, expecting binary32 to reach its arithmetic floor before binary64. The third uses a separate invariant fixture—a circular Schwarzschild orbit at $r=6GM/c^2$—to test serialization in geometrized and SI forms and requires signed quantities to survive the round trip. The fourth returns to the wave, designing independent resolution branches and tests for deletion, mutation, and tamper behavior.

The fifth adds a spectrally defined random perturbation whose coefficients are keyed by member and mode, so grid refinement and parallel scheduling do not change the input field. The final page gives the package to an uninvolved runner, disables the network after retrieval, and requires reconstruction from archived source and checksums. Each exercise isolates a different layer; the orbit supplements the wave where dimensional and tensor conventions need a geometric test.

The environment record for every arm includes source revision, operating system or immutable image, compiler or interpreter and flags, numerical libraries, precision, hardware, threads and processes, workflow version, and input digests. A clean sequence produces machine-readable inputs, raw fields, residuals, invariant values, refinement ratios, thresholds, and comparison results. The supplied serial binary64 wave program implements the first experiment and records its actual results. The cross-environment, serialization, workflow, ensemble, and archival arms remain specifications for the reader to implement; their outcomes are not implied by that one run.

What each reproducibility test shows

A persuasive calculation normally crosses four layers:

• identity and provenance: exact source, inputs, environment, transformations, and version-specific products are recoverable;
• code verification: analytic identities, manufactured solutions, mutations, or independent components test that the algorithm implements its equations;
• solution verification: residual, tolerance, resolution, precision, boundary, and conservation studies estimate numerical error for the actual run;
• scientific interpretation: the observable and uncertainty support only the claim and regime that were declared.

FAIR metadata can improve discovery and reuse without proving executable reconstruction; the principles and their implementation-independent character are stated in Wilkinson et al. 2016, Box 2 and § “The principles precede implementation”. Likewise, a workflow that returns exit status zero has executed its graph but may have failed every physical tolerance.

Several recurring distinctions control the conclusion:

• a checksum establishes byte identity, not physical correctness;
• a container records much of user space, not every hardware or external-service dependency;
• bitwise agreement tests one supported execution path, not discretization convergence;
• a small algebraic solver residual is not a small continuum error;
• a seed without generator, version, and draw mapping does not define an ensemble;
• a concept DOI locates a work family, while the finite claim needs the exact release;
• a plot is a derived view, not a replacement for the raw products and transformation that produced it.

Failure boundaries and stopping rules

The main failure modes are undeclared inputs, mutable dependencies, hidden unit or sign defaults, stale cached descendants, arithmetic sensitivity larger than the error budget, random streams tied to scheduling, convergence assessed outside the asymptotic regime, and an archive that opens but cannot rebuild. A second implementation is not independent if it shares the faulty operator, data conversion, or analysis routine at issue.

Set stopping rules before inspecting the preferred result. For the wave benchmark, success requires the expected refinement regime, subdominant time error, bounded invariant drift, analytic phase agreement, declared cross-environment tolerance, and a clean rerun. Exploration after a failure is valuable, but changing resolutions, exclusions, or tolerances creates a revised experiment record rather than retroactively satisfying the first rule.

Sandve and collaborators give a compact practice-oriented account of recorded transformations, intermediate results, raw data, seeds, and claim-to-result links in Sandve et al. 2013, pp. 1–4, Rules 1–10. This chapter adds the numerical-analysis requirement that every retained product carry its conventions, comparison norm, error controls, and domain of validity.

Review the chapter

A group deposits a binary64 waveform, a container tagged latest, a plotting notebook, and a concept DOI. Two reruns on the same workstation produce the identical figure. The paper reports fourth-order accuracy from $N=128$ and $N=256$, but preserves no raw fields, exact environment digest, units, random-stream map, or boundary study. What has been established, and what is the minimum repair?

Solution

The group has shown repeatability of one local path to a figure, assuming the mutable container tag still resolved to the same image. Two resolutions cannot by themselves demonstrate an observed order without additional information, and an identical plot does not reveal field errors, boundary contamination, or post-processing choices.

Create a versioned release with immutable source, image or rebuild inputs, exact dependencies, canonical physical inputs, units and conventions, raw fields, workflow, and checksums. Add at least a third resolution, time-step and boundary variations, residual and invariant checks, and an independent analytic or numerical comparison. Define any random stream by generator, version, key or seed mapping, and distribution routine. Deposit the exact release under a version DOI, give an uninvolved runner only that package, and require a network-independent clean rebuild whose actual discrepancies meet predeclared thresholds. Preserve failed tests as well as the accepted result.

References

• National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. Washington, DC: National Academies Press, 2019. DOI.
• Sandve, Geir Kjetil, Anton Nekrutenko, James Taylor, and Eivind Hovig. “Ten Simple Rules for Reproducible Computational Research.” PLOS Computational Biology 9, no. 10 (2013): e1003285. DOI.
• Wilkinson, Mark D., et al. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (2016): 160018. DOI.