# A deliberately inadequate calibration of physical scattering data

This example fits an elementary scattering-angle discrepancy model to public
numerical-relativity results. It is a **failed model**, retained to show why
small validation and test residuals do not establish accuracy in regions those
sets do not sample. It is not a calibrated EOB Hamiltonian or waveform model.

## Reproduce the published experiment

Save `calibrate-scattering.py` and `calibration-design.json` in the same folder.
The recorded environment is Python 3.12.14, NumPy 2.2.6, SciPy 1.18.1 and
mpmath 1.3.0. SciPy used the macOS Accelerate BLAS/LAPACK implementation; the
result records its reported version and compiler identities. A different
platform may change the last digits. No GPU, stochastic optimizer, or seed is
used in the physical fit.

```sh
python3.12 -m venv .venv
.venv/bin/python -m pip install numpy==2.2.6 scipy==1.18.1 mpmath==1.3.0
.venv/bin/python calibrate-scattering.py --fetch --input data/ScatteringAngles.dat --output-dir results
```

The final command verifies the input checksum, recomputes every eligible
geodesic baseline in two independent forms, fits all three candidate bases,
recreates the training-size and regularization checks, records the selected
model, then evaluates its test and out-of-domain results. `--mode train` saves
only the training/validation decision. `--mode test` requires that saved
decision to match the data, design, source-code hashes and fitted candidates.
The default `--mode reproduce` reruns an already published experiment: these
test values are no longer secret and are not a new blind validation.

Compare `results/calibration-test.json` with the supplied
`calibration-results.json`. Counts, partition IDs, selected basis and rejection
must agree exactly; angle RMS values should agree within 0.000001 degree and
coefficients within 0.00001 in the reported scaling. The script also checks
SVD against pivoted QR, the rank of each design, orthonormality of its
whitened columns, and elliptic-integral against regularized quadrature
baselines. Numerical tolerances do not quantify physical model error.

## Data and definitions

The source is Oliver Long et al., “Highly accurate simulations of asymmetric
black-hole scattering and cross validation of effective-one-body models,”
*Physical Review D* **112**, 124039 (2025),
[DOI](https://doi.org/10.1103/p6fx-7798),
[arXiv:2507.08071v2](https://arxiv.org/abs/2507.08071v2).
See §II.D, Eqs. (6)–(14), for angle extraction and the mean individual angle;
§III, Eqs. (26)–(27), for its distinction from the relative angle.
The script downloads the [v1 ancillary table](https://arxiv.org/src/2507.08071v1/anc/ScatteringAngles.dat).
The corresponding v2 table was independently checked to contain identical bytes.
SHA-256:
`4c2a8fef5ec23f12c6014b0d5cdaa5f446ec02aadff4191f21bd8d51a9354823`.

The external table is fetched by the reader, not redistributed here. The
authors' article, simulation data, and EOB implementations retain their own
rights and attribution. This package contains original analysis code and its
derived numerical results; it copies no third-party code, figure, or prose.

The 60 rows contain 19 scalar fields after expanding the two spin vectors.
We use the mean of the two individual scattering angles in the initial
nominal center-of-mass frame, not the published EOB relative-angle columns.
All angles are converted from degrees to radians once. The positive weight
scale is the larger of the two interval widths obtained by averaging the
individual lower and upper widths. Adding interval endpoints makes no
independence assumption. These source estimates are neither asserted to be
rigorous bounds nor interpreted as Gaussian standard deviations.

With G=c=M=1, q=m1/m2, nu=q/(1+q)^2, define
gamma=1+(Gamma^2-1)/(2nu), where Gamma is the table's total energy, and
j=[J_ADM-chi1z(q/(1+q))^2-chi2z/(1+q)^2]/nu. The tiny residual spins of the
nominally nonspinning cases enter this conversion from total to orbital
angular momentum; the scattering model itself is spinless.

The baseline is the exact timelike Schwarzschild scattering angle at
(gamma,j), plus 3*pi*(5*gamma^2-1)*(1/Gamma-1)/(4*j^2). This restores the full
finite-mass 2PM coefficient. Its test-mass and weak-field checks follow
Thibault Damour, “Classical and quantum scattering in post-Minkowskian
gravity,” *Physical Review D* **102**, 024060 (2020),
[DOI](https://doi.org/10.1103/PhysRevD.102.024060),
[arXiv:1912.02139v3](https://arxiv.org/abs/1912.02139v3),
Eqs. (1.4), (1.9)–(1.11), (2.45)–(2.46).
Radiative corrections begin beyond 2PM; differences between mean individual
and relative angles also lie beyond the orders imposed here. The baseline
does **not** impose all currently known higher-PM information. It was chosen
as a deliberately incomplete comparison model, not as a current best EOB
approximation.

We add nu*(4/j)^3 times a polynomial in x=4/j, t=(gamma-1)/0.1, and nu.
The three candidate bases are [1], [1,x,t,nu], and
[1,x,t,nu,x^2,x*t,t^2]. They preserve the exact test-mass limit and full 1PM
and 2PM coefficients. The coefficient errors obtained from SVD curvature
assume an exact model and independent Gaussian errors with the chosen
scales; those assumptions are **not** asserted for this fit. The archived
matrix is therefore a conditioning diagnostic, not a physical confidence
interval.

## Split, result and limitations

The design was fixed using initial-condition metadata. All eligible cases
with nominal q=1,4,7 train the coefficients; q=3,6 select the basis; q=2,5,8,9,10
test it after selection. The numerical q is retained in all physics; rounding
within 0.001 defines only the group label. Eligibility requires residual spin
at most 0.000001, 1.0199 <= gamma <= 1.16, and a scattering Schwarzschild
baseline. There are 18 training, two validation and five test cases.

The original design, data and source were fixed before evaluating the five
test residuals. Later edits only made the already evaluated calculation
portable and added environment metadata. The supplied result is a clean
rerun of that frozen mathematical experiment, not an independent new test.

| Candidate | Training RMS (degrees) | Validation RMS (degrees) |
|---|---:|---:|
| Constant | 73.791934 | 61.888296 |
| Linear | 46.376724 | 22.618076 |
| Quadratic | 25.522674 | 0.013501 |

The selected quadratic has test RMS 0.055791 degree, or 1.047054 source error
scales. Nevertheless its largest training fractional error is about 50.2%,
so it fails the predeclared requirement of at most 1% error on **every**
training, validation and test case. It was already rejectable before opening
the test; the frozen test is shown only to illustrate its narrow coverage.
This is misspecification and inadequate validation coverage, not a general
claim that seven coefficients overfit. The selected scaled
design condition number is about 3716.6, and weak regularization substantially
changes validation predictions. QR and SVD agree on the failure.

Validation and test cover gamma approximately 1.02 and j approximately 4.8;
training spans gamma approximately 1.02–1.09 and j approximately 4.61–7.33.
The withheld mass ratios therefore test a narrow energy/angular-momentum
slice, not the wider training domain. Test q=8,9,10 also extrapolate beyond
the training mass-ratio range. The one otherwise eligible nonspinning
lower-energy stress case has an error of 42.98 degrees (19.9%). Fourteen
spinning rows are unsupported. The Schwarzschild comparison function is
undefined for the other twenty rows; this is **not** a claim that those
comparable-mass binaries are captured.

No fit or threshold was adjusted after inspecting the test. No simulation
was evolved here; published extracted angles are the physical inputs.
The separate SXS trajectory example checks extraction and resolution on two
versioned releases, without assuming their initial data equal ancillary rows.