.. _developer-trustworthiness-workplan: ######################## Trustworthiness workplan ######################## :Last reviewed: 25 July 2026 :Scope: numerical correctness, reproducibility, reference integrity, and release evidence Purpose ======= A green test is evidence only when the test demonstrably inspected the quantity it claims to validate. This workplan turns that rule into executable gates. It complements :doc:`method-validation`, :doc:`engine-validation`, and the adversarial comparison audit in :file:`docs/test_integrity_audit.md`. Status terms ============ ``DONE`` Implemented and exercised by an automated test or a reproducible command. ``IN PROGRESS`` Work is being implemented in the current change set. ``BLOCKED`` The code-side preparation is complete, but reference-grade software, hardware, or a deliberate reference re-baseline is still required. ``TODO`` Required work that has not started. Definition of trustworthy ========================== A release is trustworthy only when all of the following are true: * required unit, integration, method-validation, and external-engine jobs pass; * no comparison passes after inspecting zero rows, zero paths, zero swaps, an empty archive, or an unexpected all-NaN physical quantity; * every numerical reference records its generating engine, version, execution mode, command, tolerance rationale, and content digest; * each central analysis method has an analytical, hand-computed, or independently implemented oracle; * deterministic and stochastic workflows have separate contracts; * restart, load, interruption, and parallel shutdown behavior are tested as distinct semantics; * a file that cannot be read or written stops the run, instead of being absorbed into an empty read or a truncated output that still reports success; * reading a file gives the same result on every machine, independent of the locale; * running the tests does not modify the working tree, so no test can quietly rewrite a committed reference or input; * the release pipeline cannot publish when a required validation was skipped or downgraded to execution-smoke coverage. Workstreams =========== W0. Integrate concurrent review work ------------------------------------ ``DONE`` Review the concurrent agent's path-validation, shooting-weight, documentation, and changelog changes without reverting or duplicating them. Focused path-analysis and replica-exchange tests pass. W1. Honest engine-suite outcomes -------------------------------- ``DONE`` Engine absence must return the conventional skip status (77), and the orchestrator must report ``VALIDATED``, ``SMOKE``, ``SKIPPED``, and ``FAILED`` separately. ``DONE`` Add ``PYRETIS_REQUIRE_VALIDATION=1``. In this mode, a skipped or smoke-only required engine is a failing release gate rather than a green result. ``DONE`` Exercise both permissive developer mode and strict release mode with automated driver-contract tests. W2. Reference integrity and physical coverage --------------------------------------------- ``DONE`` Add a machine-readable provenance manifest for external-engine references, including deterministic tree digests and explicit tolerance rationale. ``DONE`` Require suites to declare energy terms that are intentionally unavailable. Unexpected all-NaN columns must fail, while declared unavailable columns must be checked to remain all-NaN on both sides. ``DONE`` Report finite OpenMM potential energy from the OpenMM context and regenerate its golden data twice with the pinned OpenMM Reference platform. The two runs were byte-identical, and the release-grade suite now requires the generated files to remain byte-identical. ``DONE`` Establish a run-to-run deterministic CP2K reference build: the conda-forge cp2k 2026.1 package (single rank, ``OMP_NUM_THREADS=1``) reproduces the test-retis-load run byte-identically across clean runs (verified 2026-07-19); the golden was re-blessed on it and the suite validates only on that exact version/revision. ``DONE`` Re-baseline the legacy GROMACS 5.1.4 tarballs onto the single current reference build. The whole GROMACS reference suite now targets one era (double-precision 2025.4 on the development reference box, re-blessed 2026-07-19 from a determinism-checked double run), so strict full-scope validation is satisfiable by one executable. ``DONE`` Keep the test run out of the committed fixtures. Unit tests built engines against tracked input directories, so a run wrote ``pyretis.mdp`` and its logs into directories holding committed references, and the example suites left engine scratch in the shared input directory. Tests now work on copies under ``tmp_path`` (the example suites keep running in place and sweep with ``pyretis tools clean``, since their goldens sit alongside). A full ``./test-easy.sh`` now leaves the working tree byte-for-byte unchanged, verified by snapshotting every file before and after, so no test can quietly rewrite a reference or an input. ``BLOCKED`` Calibrate and tighten each external-engine tolerance from repeated runs on its pinned reference environment. Until measured, the manifest must record the current tolerance and why it remains provisional. W3. Independent scientific oracles ---------------------------------- ``DONE`` REPPTIS cutoff occupancy is checked from raw columns with hand-audited counts and two independent probability constructions. ``DONE`` WHAM has hand-built synthetic exact-value tests, including weighted cases. ``DONE`` The method-validation campaign compares rate estimates with the analytical Kramers/Mel'nikov-Meshkov result. ``DONE`` Keep an evidence matrix that maps each public analysis result to at least one independent oracle and names any remaining unsupported result. W4. Stochastic and parallel contracts ------------------------------------- ``DONE`` Single-worker seeded workflows retain byte-reproducibility checks; multi-worker workflows are explicitly completion-ordered and are not falsely required to be byte-identical. ``DONE`` A swap-consistency check that examines no swaps must report zero coverage and fail when the caller explicitly requested swap validation. ``DONE`` Promote the multi-seed method-validation campaign to a scheduled CI job with confidence intervals and stored provenance. W5. Failure and recovery behavior --------------------------------- ``DONE`` Async worker exception recovery, burst submission, forced cancellation, repeated lifecycle cleanup, SIGTERM unwind, and atomic continuation writes have automated coverage. ``DONE`` Add corruption tests for the reference manifest and explicit failures for missing or changed reference artifacts. ``DONE`` Add controlled disk-write failure injection around run-state persistence and verify that the last durable state remains restartable. ``DONE`` Surface file-I/O failures instead of absorbing them. ``FileIO`` used to log a failed open and keep a ``None`` handle, which iterates as end-of-file, so an unreadable or missing file was indistinguishable from an empty one; a failed write was logged and reported through a status value no caller reads, leaving a truncated ``energy.txt`` or ``order.txt`` behind a run that reported success. Both now raise. Callers for which a missing file is a legitimate state check for it first and say so: a path loaded from sparse or external files carries no energies, and that is reported per path while the simulation continues. ``DONE`` Pin text decoding to UTF-8. The encoding was selected by an inverted test, so every text file was decoded with the platform locale, making a read depend on the machine it ran on. The remaining swallowed I/O error is the directory ``fsync`` in ``inout/common.py``, which weakens durability without corrupting data and is documented as such. W6. CI and release policy ------------------------- ``DONE`` Run reference-manifest validation and strict Sphinx warnings-as-errors in normal CI. ``DONE`` Make tagged releases depend on the strict validation jobs. Manual developer smoke jobs remain useful, but cannot satisfy a release dependency. ``DONE`` Coverage is collected on CI Python 3.12, separately from the Python 3.14/NumPy combination whose tracer is currently incompatible. Execution gates =============== The intended commands, from least to most expensive, are: .. code-block:: bash python -m pyretis.testing.reference_manifest \ examples/tests/reference_provenance.json sphinx-build -W --keep-going -E -b html docs /tmp/pyretis-docs ./test-easy.sh ./test-heavy.sh PYRETIS_REQUIRE_VALIDATION=1 ./test-heavy.sh In permissive mode, missing or non-reference engines are reported as ``SKIPPED`` or ``SMOKE`` and do not by themselves fail the driver. The final command is the release gate and may not pass until every required engine is installed and reference-grade. Current blockers and decision points ==================================== * The CP2K determinism blocker is resolved (conda-forge 2026.1, single rank, deterministic; golden re-blessed 2026-07-19). Non-reference CP2K builds still run execution-smoke only. * GROMACS references are unified on a single era (double-precision 2025.4, the development reference box, re-blessed 2026-07-19): the former 5.1.4 archive suite was regenerated on the chosen reference version. * External tolerances other than the byte-exact OpenMM and LAMMPS gates remain provisional until repeated runs on each pinned reference environment provide a measured error envelope. * Reference updates are review events. A digest update must include the engine provenance, generation command, observed error envelope, and reviewer signoff in the merge request.