Reproducing and Sharing Computational Results

A computational result can be checked when another researcher can identify the precise statement, inspect its inputs and assumptions, recreate its environment, rerun the decisive calculation, and observe both success and expected failure. A repository link or archive identifier helps discovery, but neither demonstrates that the deposited materials execute or that the reported result follows from them.

Required background. Probability and Inference for Gravity supplies uncertainty and comparison criteria.

Helpful background. Formulations of General Relativity supplies the equations and conventions being instantiated; Differential Equations for Relativity supplies numerical controls; Computational Relativity supplies the volume-wide workflow; and Mathematical Relativity supplies the hypotheses that a finite computation cannot replace.

Define the numerical result

Begin with a sentence that can fail: for a named equation set, domain, gauge, boundary and initial data, parameter values, implementation revision, precision, and refinement sequence, observable $Q$ agrees with a named benchmark to tolerance $\epsilon$ and exhibits the reported order. A bare number without this domain is not a reusable result.

To make that result reproducible, provide:

• exact source and dependency revisions;
• input data, configurations, units, conventions, and checksums;
• environment construction, compiler or interpreter, precision, and relevant hardware;
• one command or workflow from clean environment to machine-readable outputs;
• raw outputs, analysis code, norms, tolerances, and uncertainty components;
• unit, manufactured-solution, and physical benchmark results;
• licenses, data rights, citation metadata, provenance, and known limitations; and
• a negative case or deliberate mutation that must fail.

Record expected runtime and resource scale so a reviewer can distinguish a stalled calculation from a long one. If a full production run is impractical, supply a short verification case and a documented path to the full result without implying that the short case reproduces every performance or scaling claim.

Accessible files do not guarantee reproducible results

The FAIR principles improve findability, accessibility, interoperability, and reuse. Wilkinson and collaborators stress that the principles precede any one implementation in Wilkinson et al. 2016, pp. 1–9. A FAIR archive may still contain code that does not build, undocumented parameters, or outputs disconnected from the paper. FAIRness is therefore valuable infrastructure, not proof of computational reproduction.

Software itself needs a citable identity. Smith and collaborators specify importance, credit, unique identification, persistence, accessibility, and related principles in Smith et al. 2016, §1 and §§6–7. Use a persistent DOI for an archived release, retain a version-control tag or commit for exact source, and publish checksums for downloaded inputs. The DOI identifies the deposit; the tag and checksums identify the bits used.

An environment lock file or container is useful but incomplete. It may omit drivers, processor behavior, external services, locale, or nondeterministic reductions. Retain a human-readable environment description and test the package on a clean host not used to develop it.

Independent artifact criteria

The current ACM artifact policy, consulted 2026-08-18, distinguishes availability from evaluated functionality and reusability, and distinguishes results reproduced with the authors’ artifacts from results replicated through an independent implementation. The criteria are summarized in the ACM current Artifact Review and Badging policy, badge-definition sections, consulted 2026-08-18. These labels answer different questions and should not be collapsed into “reproducible.”

As an independent reproduction test, a researcher not involved in the original calculation starts from the DOI-tagged package and documented prerequisites on a clean environment. They verify checksums, build or install, run the manufactured scalar-wave and Apples-with-Apples benchmark subset, reproduce the claimed norms within predeclared tolerances, and trigger the negative mutation. Their report records deviations, manual interventions, runtime, hardware, and the exact output comparison.

The negative case is essential. If reversing a forcing sign or stencil coefficient leaves the package green, the workflow may be reading cached output or testing the wrong file. Remove or isolate caches, seed nondeterministic steps, and make provenance explicit from input to final table.

Reporting failed checks and corrections

Release materials should describe known breakdown regimes, unsupported platforms, nondeterministic variation, and tests that do not pass. A failure notice includes affected versions, scope, consequence, workaround if any, and the corrected release when available. Never replace a tagged archive silently; issue a new immutable version and link the relation.

Distinguish numerical reproduction from scientific confirmation. Running the same code on the same data tests the package and reported computation. An independent implementation, analytic relation, conserved balance, or new data can supply stronger evidence about the scientific claim. Both are valuable, but they are not synonyms.

The environment record should pin equations, harness, code and dependencies, compiler or interpreter, precision, meshes, solvers, test data, statistical tools, random seeds, workflow engine, and analysis scripts. Outputs include provenance links, logs, residuals, refinement tables, checksums, comparison results, licenses, and machine-readable claim metadata.

A clean rerun sequence would create the environment from an empty workspace, verify the DOI release and tag, execute scalar-wave and gauge-wave benchmarks at three resolutions, reproduce the table, apply one known mutation, and confirm the expected failure. This page defines that sequence; it does not claim an archive or absent program has completed it.

The result has not been reproduced if files are merely available, versions or rights are ambiguous, results require undocumented manual edits, tolerances were chosen after rerunning, checksums or inputs are missing, an uninvolved clean run fails, or the negative case does not fail. Availability and preservation make reproduction possible; the calculation must still be rerun and checked.

Check your understanding

A project deposits code and data under a DOI with a clear license and complete metadata. No one outside the author team has installed it, and the documented command fails on a clean machine because one dependency is absent. What has been established?

Solution

The files are available, citable, and preserved, but the calculation has not been successfully reproduced. Add the missing dependency and exact environment specification, issue a new immutable release, and have an uninvolved researcher reproduce the benchmark and negative case from a clean environment. Do not overwrite the original deposit or describe the DOI alone as a reproduced result.

References

• Association for Computing Machinery. 2020. “Artifact Review and Badging.” Version 1.1, 24 August 2020; consulted 2026-08-18. Official policy.
• Smith, Arfon M., Daniel S. Katz, Kyle E. Niemeyer, and FORCE11 Software Citation Working Group. 2016. “Software Citation Principles.” PeerJ Computer Science 2, e86. DOI.
• Wilkinson, Mark D., et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3, 160018. DOI.