Workflow Automation and Provenance

Workflow automation turns a scientific derivation into an executable dependency graph: every result is produced from declared inputs by a declared transformation in a declared environment. Provenance is the evidence that connects the graph’s output back to those causes. A command history is insufficient because it rarely records which file a command read, which version it executed, or whether an intermediate product was edited afterward. The goal is reconstruction of the result, not merely recollection of activity.

Required background. Calculus and Linear Algebra for Relativity supplies norms and invariant checks; Differential Equations for Relativity supplies the numerical problem and refinement controls carried through the workflow.

Helpful background. Formulations of General Relativity connects physical variables and conventions to pipeline inputs; Probability and Inference for Gravity supports stochastic workflows and uncertainty summaries.

Provenance as a derivation graph

Represent a finite workflow as a directed acyclic graph $G=(V,E)$. A node in $V$ is either an immutable input, a transformation, or a product; an edge states that a transformation used an input or generated a product. In the W3C provenance model, inputs and products are entities, transformations are activities, and responsible people, organizations, or software are agents. Agents are not the output files. The core relations are specified in Moreau and Missier 2013, PROV-DM §§ 2.1, 5.1–5.3. An iterative solver can fit this acyclic description by representing successive states as distinct entities; the same mutable filename must not stand for every iteration. The terminology supports a simple causal discipline: a product with no derivation edge is unexplained.

For each transformation $v$, record

$$P_v=(C_v,E_v,I_v,\Theta_v,O_v,S_v),$$

where $C_v$ identifies source and command, $E_v$ the execution environment, $I_v$ the ordered or explicitly unordered inputs, $\Theta_v$ the parameters, $O_v$ the outputs, and $S_v$ the exit state and diagnostics. The record includes start and stop times for interpretation, but timestamps are not identities. Content digests and version identifiers establish which bytes and software participated.

A cache key can be constructed schematically as

$$K_v=H(\text{schema},C_v,E_v,\Theta_v,K_{u_1},\ldots,K_{u_m}),$$

where $H$ is a named cryptographic hash and $u_1,\ldots,u_m$ are the direct dependencies in a canonical order. Raw-input keys hash the bytes plus their semantic metadata. If any relevant input, command, environment, parameter, or schema changes, $K_v$ changes and every descendant becomes stale. A timestamp-only rule cannot provide this guarantee: copied or restored files can have misleading times, and an in-place manual edit can be newer without being legitimate.

Provenance is also semantic. A waveform product needs its mode normalization, tetrad, extraction radius, retarded-time definition, and units, not just a path to an array. A scalar-wave norm needs the exact solution, grid, norm convention, and comparison time. The graph can prove that a script transformed a file while still failing to prove that the transformation answered the intended question.

Designing a rerunnable scalar-wave pipeline

Use the periodic benchmark $u(t,x)=\sin(2(x-t))$ from the experiment-design page. A minimal graph contains these stages:

1. specify canonically serializes the physical problem, fourth-order stencil, Runge–Kutta method, maximum Courant factor $\lambda_{\max}=1/4$, final time $T=1$, resolutions $N=32,64,128,256$, precision, and thresholds; it also records the uniform-step rule $m=\lceil T/(\lambda_{\max}\Delta x)\rceil$, $\Delta t=T/m$, so every branch reaches the same final time;
2. build creates or identifies an isolated environment and records the source revision, compiler or interpreter, libraries, flags, hardware, threads, and process counts;
3. four evolve-N activities independently produce fields and diagnostics carrying the specification and environment digests;
4. analyze-N computes the continuum error, energy drift, constraint norm, and comparison with the full analytic semidiscrete field through code that does not call the evolution update;
5. convergence combines the four analysis records, evaluates $p_N=\log_2(E_N/E_{2N})$, and applies the predeclared tolerances;
6. present derives a table or plot only from the checked convergence record.

Each product is immutable. A correction creates a new product and derivation rather than overwriting history. The raw arrays remain available behind any reduced table, because a final figure cannot reveal an incorrect norm, omitted resolution, or post-processing choice. This realizes the practical advice to record how every result was produced and retain intermediates and raw data in Sandve et al. 2013, Rules 1, 5, 7, and 9.

The benchmark acceptance criteria remain physical: the two finest resolution pairs should approach fourth order, $E_{256}\lt10^{-6}$, relative energy drift should be below $10^{-7}$, and halving the time step should show that temporal error is subdominant. Workflow success without these checks means only that the pipeline completed.

Dependency tests that expose hidden work

Four controlled tests distinguish a reconstructible graph from a convenient script collection.

Clean rebuild. Start with only versioned source, checksum-bearing inputs, and the environment definition. Build every target twice in two clean directories. In the same deterministic environment, byte-defined products should have identical SHA-256 digests; numerical products allowed to vary must satisfy a documented tolerance and retain the comparison. A second run that reuses an undeclared cache is not clean.

Deletion test. Delete one derived analysis record, keep its valid ancestors, and request the final presentation. The system should recreate exactly the missing node and its descendants. Workflow systems that infer a dependency graph from declared rules make this behavior systematic; the rule-based design and execution model of one representative system are described in Köster and Rahmann 2012, pp. 2520–2522. The scientific contract must not depend on that particular tool.

Mutation test. Change one physical input, for example $k=2$ to $k=3$, while leaving source and environment fixed. Every evolution, analysis, convergence, and presentation descendant must receive a new key; environment construction should not. The provenance graph must show the new derivation. If the old plot remains “up to date,” invalidation is broken.

Tamper test. Alter one intermediate field file without updating its metadata. Its content digest must no longer match its recorded identity, and downstream execution must stop. Recomputing only the final plot from the altered array and giving the plot a fresh checksum conceals rather than repairs the break.

These tests should be run under binary64 and, for the analytic semidiscrete diagnostic, a higher-precision reference. Repeat with altered thread counts when a reduction enters. Distinguish bitwise, numerical-tolerance, and statistical comparisons; do not impose a byte-level criterion on an explicitly stochastic ensemble.

Capturing execution without confusing it with meaning

System-call capture can discover files, libraries, and programs used by an existing execution. ReproZip, for example, separates tracing, packaging, and reproduction in Chirigati et al. 2013, §§ 2–4. Such capture is valuable for finding a hidden shared library or configuration file, but it cannot infer the scientific role of an input, the correct tensor sign, or the acceptance threshold. It may also capture secrets, personal paths, licensed software, or an accidental network response, so the resulting package requires deliberate review and minimization.

Conversely, a carefully described scientific method without captured execution can omit a mundane but decisive dependency. The robust design combines both views:

• a prospective graph declares what should run and why;
• an execution record states what did run;
• content checksums connect declared and observed entities;
• physical and numerical tests determine whether the outputs support the claim.

An interactive action can be included only by turning it into an explicit activity with recorded inputs, parameters, operator, output digest, and reason. Better still, replace repeatable clicks with a deterministic transformation. “Adjusted until the curve looked smooth” has neither a finite algorithm nor a defensible stopping rule.

Complete reconstruction sequence

A conforming implementation should expose one top-level target for the finite claim. Starting in a clean directory, it must verify archived input and source checksums, construct the pinned environment, execute every graph node, recompute the analytic and invariant diagnostics, apply the resolution and tolerance sweep, and emit both a machine-readable comparison and human-readable result. It must propagate any failed node or scientific threshold as a failed top-level result.

Then perform an independent reconstruction with only the archived source entities and graph description. Compare the new input digests, graph topology, per-node environment records, output digests or numerical discrepancies, convergence orders, and invariant checks. Provenance validity constraints—including generation and derivation ordering—are formalized in Cheney, Missier, and Moreau 2013, PROV-CONSTRAINTS § 6.2.2, Constraints 37 and 41–42, and § 7; passing them establishes a coherent record, not correct physics.

The principal failure modes are an undeclared input, an ignored digest mismatch, a stale descendant that survives mutation, an unreconstructible manual edit, a clean run that reaches the network for mutable data, a failed scientific threshold hidden by workflow success, or a graph that regenerates only a curated plot rather than its numerical basis. Any one is grounds to reject the computational claim. The benchmark is limited to a smooth deterministic scalar wave; adaptive decisions, nondeterministic accelerators, large external datasets, and ensembles require additional policies.

Check your understanding

A pipeline hashes every final figure. A collaborator manually removes noisy points from an intermediate table, reruns the plotting command, and obtains a new valid figure hash. Is the figure now reproducible?

Solution

No. The final hash identifies the edited figure but does not reconstruct the unrecorded table transformation. The collaborator must encode the point-selection rule as an explicit activity, declare its inputs and parameters, produce a new immutable table with a derivation record, and justify the rule scientifically. If the selection was chosen after inspecting the result, the revised analysis and stopping rule must be labeled as a new experiment.

References

• Cheney, James, Paolo Missier, and Luc Moreau, eds. Constraints of the PROV Data Model. W3C Recommendation, 30 April 2013. Specification.
• Chirigati, Fernando, Dennis Shasha, and Juliana Freire. “ReproZip: Using Provenance to Support Computational Reproducibility.” In 5th USENIX Workshop on the Theory and Practice of Provenance. USENIX Association, 2013. Open PDF.
• Köster, Johannes, and Sven Rahmann. “Snakemake—A Scalable Bioinformatics Workflow Engine.” Bioinformatics 28, no. 19 (2012): 2520–2522. DOI.
• Moreau, Luc, and Paolo Missier, eds. PROV-DM: The PROV Data Model. W3C Recommendation, 30 April 2013. Specification.
• Sandve, Geir Kjetil, Anton Nekrutenko, James Taylor, and Eivind Hovig. “Ten Simple Rules for Reproducible Computational Research.” PLOS Computational Biology 9, no. 10 (2013): e1003285. DOI.