pyretis.testing package¶

This package defines common methods which are used for testing.

Package structure¶

Modules¶

simulation_comparison.py (pyretis.testing.simulation_comparison): Common methods for comparing results.
helpers.py (pyretis.testing.helpers): Common methods for tests.

List of submodules¶

pyretis.testing.example_compare
pyretis.testing.helpers
pyretis.testing.reference_manifest
pyretis.testing.simulation_comparison
pyretis.testing.systemhelp

pyretis.testing.example_compare module¶

Shared result-comparison for the example run.sh scripts.

The example test folders under examples/tests/ used to each ship a bespoke compare.py that re-implemented the same three checks against the committed results/ reference. That logic lives here now, so a folder’s run.sh calls python -m pyretis.testing.example_compare <config.toml> (via its -C/-cc flag) instead of carrying its own copy.

The checks run for a given config are selected from its simulation task:

golden – every committed per-ensemble reference file (pathensemble.txt always; order.txt / energy.txt when the folder commits them) is compared against the run’s matching file with the file-type-appropriate primitive (compare_ensemble_files_golden()). Always run; a folder selects which files to check simply by which it commits under results/. --energy-skip/--path-skip/--rel-tol tune this for a folder whose reference legitimately differs in a named term (e.g. GROMACS’s restart-dependent long-range dispersion correction) or needs a looser tolerance (e.g. a build-sensitive external engine).
archive – the per-trial trajectory archive (<ensemble>/paths/<pn>/ plus the per-ensemble long-term <ensemble>/archive/<pn>/) is byte-compared (tolerating --energy-skip the same way as the golden check; compare_archive_check()). Opt-in (not part of the per-task default): the two-run restart/load/two-engine equivalence checks that need it request it explicitly via --checks.
paths – every accepted path is validated (start/middle/end interface labels, length, crossings; check_accepted_paths()). Run for the path-sampling tasks.
relaxed-paths – like paths, but the middle/crossing invariants are relaxed for sparse-loaded (ld-move) initial paths (check_accepted_paths_relaxed()). zero_left is read from the config automatically, so a [0^-] path may legitimately end 'L' under that geometry without being flagged.
swaps – accepted s+/s- swap moves are checked for order-parameter continuity with their parent path (check_retis_swaps()). Opt-in (not a per-task default): the unified infinite-swapping scheduler folds swaps into weighted bookkeeping rather than emitting discrete s+/s- moves, so a caller requests this only where discrete swaps exist, and the check fails if it examines none.

--run-dir/--ref-dir need not point at a committed results/ golden – pointing --ref-dir at a second, independently-run directory of the same config turns the golden/archive checks into a two-run equivalence check (e.g. a restart or two-engine comparison that has no committed reference of its own).

A few folders instead check that a restarted (stopped-and-resumed) run reproduces the uninterrupted one, with no committed golden. Their run.sh selects a dedicated subcommand: restart-whole (the restart already holds the whole history; compare_restart_whole()), restart-concat (part 1 + part 2 must equal the full run; compare_restart_concat()) and restart-md (the same, with a trajectory compared by per-snapshot MSE; compare_restart_md()).

Exit status is the number of failing checks (0 == success), so it drops straight into a run.sh && chain.

pyretis.testing.example_compare._ALL_CHECKS = ('golden', 'paths', 'relaxed-paths', 'swaps', 'structural', 'reports', 'archive')¶: The check names a folder may request (see compare_run()).

pyretis.testing.example_compare._GOLDEN_COMPARATORS = {'energy.txt': <function compare_energy_data>, 'order.txt': <function compare_numerical_data>, 'pathensemble.txt': <function compare_path_ensemble_data>, 'traj.txt': <function compare_simulation_files>}¶: Map a committed per-ensemble golden file to the primitive that compares it. The comparison a folder wants is inferred from which of these files it actually commits under results/ – see compare_ensemble_files_golden(). Insertion order fixes the report order (path ensemble first, then the numerical trajectories).

pyretis.testing.example_compare._MISSING = <object object>¶: Unique sentinel used as the zip_longest fill value when pairing two iterators of trajectory blocks/snapshots, so a length mismatch on either side is detected (a real snapshot is never this object).

pyretis.testing.example_compare._PATH_SAMPLING_TASKS = frozenset({'explore', 'pptis', 'repptis', 'retis', 'tis'})¶: Tasks that sample paths (so the accepted-path validity check applies).

pyretis.testing.example_compare._STRUCTURAL_PROFILES = {'explore': {'has_zero_minus': False, 'min_length': 3, 'require_finite_lockstep': True, 'require_maxo_ge_mino': True}, 'pptis': {'has_zero_minus': True, 'min_length': 2, 'require_finite_lockstep': False, 'require_maxo_ge_mino': False}, 'repptis': {'has_zero_minus': True, 'min_length': 2, 'require_finite_lockstep': False, 'require_maxo_ge_mino': False}}¶: Structural-check profiles for the golden-less path-sampling folders, keyed by task. explore produces N - 1 positive ensembles and no [0^-]; repptis/pptis keep [0^-]. The accepted-length threshold and the extra order-parameter checks also differ.

pyretis.testing.example_compare._TRAJ_READER_SPECS = {<class 'pyretis.inout.formats.path.PathIntFile'>: {'nested': True, 'snapshot_diff': <function _snapshot_diff_pathint>}, <class 'pyretis.inout.formats.snapshot.SnapshotFile'>: {'nested': False, 'snapshot_diff': <function _snapshot_diff_snapshot>}}¶: Per-reader trajectory-MSE behaviour. PathIntFile yields path blocks (nested; each block carries a data list of snapshots) and is used for the whole-file restart; SnapshotFile yields snapshots directly (flat) and is used for the concatenated restart. The matching per-snapshot diff function reflects the file format’s snapshot keys.

pyretis.testing.example_compare._archive_subdir_name(base_dir, ens_dir)¶

Return the ensemble’s operational-archive subdirectory name.

Prefers the current default (ARCHIVE_SUBDIR, accepted) and falls back to the legacy paths / trajs names so a new run can still be compared against a reference recorded before the renames. None when the ensemble has no archive.

pyretis.testing.example_compare._assess_mse(error, what, tol)¶

Log and grade one averaged error term against a tolerance.

Parameters:

error (float) – The mean per-snapshot error term.
what (str) – A label for the term (e.g. positions) used in the log message.
tol (float) – The tolerance the error must stay below.

Returns:

int – 0 when abs(error) < tol, else 1.

pyretis.testing.example_compare._chained_restart_snaps(part_files, reader)¶

Chain the restart parts’ snapshots, dropping each overlap frame.

Parameters:

part_files (list of str) – The restart parts, in order.
reader (callable) – The trajectory reader class (SnapshotFile).

Returns:

iterator – The chained snapshots, with every part after the first missing its leading frame (which repeats the previous part’s last frame).

pyretis.testing.example_compare._check_swaps_one(run_dir, paths, accepted, ens, kind)¶

Check the accepted swaps of a single ensemble against its neighbour.

A swap move that is present but whose parent path cannot be found, or whose order-parameter continuity fails, makes the status non-zero. The second returned value is the number of accepted swap moves examined.

pyretis.testing.example_compare._configure_console_logging()¶

Route this module’s log records to the console.

Every check reports what it compared and, on a mismatch, why (which ensemble, which file, which line) through logger – but the module only attaches a logging.NullHandler at import time (so importing it as a library stays silent by default), so run as a script a failed comparison exits non-zero while printing nothing: an opaque, silent failure that defeats the whole point of a detailed mismatch message. Attach a single stdout logging.StreamHandler at INFO so the CLI entry point (main()) is loud about what it checked and why it failed. Idempotent: a second call does not stack a duplicate handler.

pyretis.testing.example_compare._cycle_index(traj)¶: Extract the cycle index from an order.txt block comment.

pyretis.testing.example_compare._default_checks(task)¶

Return the checks a task runs when a folder does not name its own.

Parameters:: task (str) – The (lower-cased) simulation task.
Returns:: list of str – The default check names: golden always and paths for the path-sampling tasks. Swap validation must be requested explicitly because unified infinite swapping does not emit discrete swap moves.

pyretis.testing.example_compare._generated_report_path(run_dir, name)¶

Resolve a committed report name to the run’s generated report file.

Reports are written under <run_dir>/report/ and may carry a _cycles-NNN suffix (the cycle count at write time). Given the plain reference name, return the matching generated path, preferring an exact match and otherwise the highest-numbered _cycles- variant.

Parameters:

run_dir (str) – The directory the simulation ran in.
name (str) – The reference report file name (e.g. foo_report.html).

Returns:

str – The path to the generated report file.

pyretis.testing.example_compare._golden_ensemble_dirs(ref_dir)¶

Return the ensemble subdirectories that carry golden files.

The reference directory holds one zero-padded, three-digit subdirectory per compared ensemble (000, 001 …). Some folders start at 001 (e.g. the make-tis-files examples), so the set is discovered from ref_dir rather than assumed to be 0 .. n_interfaces.

Parameters:: ref_dir (str) – The committed reference directory (results/).
Returns:: list of str – The ensemble directory names present, sorted ascending.

pyretis.testing.example_compare._order_swap_ok(data0, data1, special)¶: Check order-parameter continuity across a swap (or full equality).

pyretis.testing.example_compare._parse_ensemble_energy_terms(specifications, parser)¶: Parse repeated ENSEMBLE:TERM,TERM CLI specifications.

pyretis.testing.example_compare._parse_ensemble_status_counts(pathensemble_file)¶

Count the per-status rows in a pathensemble.txt.

Parameters:: pathensemble_file (str) – Path to an ensemble’s pathensemble.txt.
Returns:: dict – A count for each of ACC/FTX/BTX/FTL/BTL/0-L.

pyretis.testing.example_compare._pathensemble_rows(pathensemble_file)¶: Return the split non-comment rows of a pathensemble.txt.

pyretis.testing.example_compare._read_path_moves(run_dir, ens_number)¶: Read per-step move bookkeeping (and accepted-at map) for an ensemble.

pyretis.testing.example_compare._read_pathensemble_rows(run_dir, ens_number)¶: Yield the non-comment rows of an ensemble’s pathensemble.txt.

pyretis.testing.example_compare._report_file_pair(name, equal, msg)¶

Log a single file comparison result and return its failure count.

Parameters:

name (str) – The file name being compared (for the log message).
equal (bool) – Whether the comparison judged the files equal.
msg (str) – The descriptive message returned by the comparison primitive.

Returns:

int – 0 when equal is True, else 1.

pyretis.testing.example_compare._resolved_ref_dir(ref_dir)¶

Yield a directory-shaped reference, extracting a tarball first.

Every check that reads ref_dir expects a plain directory laid out as <ensemble>/<file> (a committed results/, or a second live run directory). Some folders instead commit a single gzipped tarball with that same layout stored under a results/ prefix inside the archive – e.g. LAMMPS, where keeping the reference to one small binary-diffable file is preferable to a committed tree (generate_golden_tarball() writes it). Transparently extract it to a temporary directory so every check downstream stays unaware of which form the reference took.

Parameters:: ref_dir (str) – Either a plain reference directory, or the path to a .tgz/.tar.gz archive holding the same layout under a results/ prefix.
Yields:: str – A directory laid out as <ensemble>/<file> – ref_dir itself, or the archive’s extracted results/ subdirectory.

pyretis.testing.example_compare._run_compare_subcommand(args, parser)¶: Handle the compare subcommand (single run vs golden/structural).

pyretis.testing.example_compare._set_swap_parents(paths, ens_left, ens_right, acc_left, acc_right)¶: Tag each s+/s- move with the parent path it swapped with.

pyretis.testing.example_compare._snapshot_diff_pathint(snap1, snap2)¶

Return the summed squared position/velocity differences of a snapshot.

For snapshots loaded by pyretis.inout.formats.path.PathIntFile, whose positions and velocities are (n_atoms, dim) arrays under the pos/vel keys.

Parameters:

snap1 (dict) – The first snapshot (keys pos and vel).
snap2 (dict) – The second snapshot, compared against snap1.

Returns:

diff_pos (float) – The summed squared position difference over all atoms.
diff_vel (float) – The summed squared velocity difference over all atoms.

pyretis.testing.example_compare._snapshot_diff_snapshot(snap1, snap2)¶

Return the summed squared position/velocity differences of a snapshot.

For snapshots loaded by pyretis.inout.formats.snapshot.SnapshotFile, whose coordinates live in the per-axis x/y/z and vx/vy/vz keys.

Parameters:

snap1 (dict) – The first snapshot (keys x/y/z and vx/vy/vz).
snap2 (dict) – The second snapshot, compared against snap1.

Returns:

diff_pos (float) – The summed squared position difference over all atoms.
diff_vel (float) – The summed squared velocity difference over all atoms.

pyretis.testing.example_compare._store_is_empty(directory)¶: Return True if directory exists but contains no files at all.

pyretis.testing.example_compare._struct_check_layout(run_dir, ens_dirs, has_zero_minus)¶: Structural check 1: the expected ensemble dirs and files exist.

pyretis.testing.example_compare._struct_check_lockstep(run_dir, ens_dirs, require_finite)¶: Structural check 3: one order.txt block per pathensemble.txt row.

pyretis.testing.example_compare._struct_check_moves(run_dir, ens_dirs)¶: Structural check 4: real shooting/swap moves (not only loaded).

pyretis.testing.example_compare._struct_check_paths(run_dir, ens_dirs, min_length, require_maxo_ge_mino)¶: Structural check 2: every accepted path is a valid sampling path.

pyretis.testing.example_compare._structural_ensemble_dirs(n_interfaces, has_zero_minus)¶

Return the ensemble directory names for a golden-less run.

Parameters:

n_interfaces (int) – The number of interfaces in the config.
has_zero_minus (bool) – Whether the topology includes the [0^-] (000) ensemble.

Returns:

list of str – Ensemble directory names – 000 .. 0(N-1) when [0^-] is present, else 001 .. 0(N-1) (the explore positive ensembles).

pyretis.testing.example_compare._traj_mse_flat(part_files, full_file, reader, snapshot_diff)¶

Accumulate per-snapshot squared errors for a concatenated restart.

The restart is split into part_files written to separate files. They are chained (dropping each overlapping restart frame; see _chained_restart_snaps()) and compared snapshot-for-snapshot against the full run. The residual flag reports a differing total snapshot count.

Parameters:

part_files (list of str) – The restart parts, in order.
full_file (str) – The uninterrupted run’s trajectory file.
reader (callable) – The trajectory reader class (SnapshotFile).
snapshot_diff (callable) – The per-snapshot difference function.

Returns:

error_pos (list of float) – The per-snapshot squared position errors.
error_vel (list of float) – The per-snapshot squared velocity errors.
residual (bool) – True if the chained restart and the full run differ in length.

pyretis.testing.example_compare._traj_mse_nested(part_files, full_file, reader, snapshot_diff)¶

Accumulate per-snapshot squared errors for a whole-file restart.

The restart already holds the whole history, so it is compared block-for-block (each block a path) and, within a block, snapshot-for-snapshot against the full run. The residual flag reports a differing number of path blocks (the trajectory-count check).

Parameters:

part_files (list of str) – The restart trajectory file(s); the whole-file restart passes one.
full_file (str) – The uninterrupted run’s trajectory file.
reader (callable) – The trajectory reader class (PathIntFile).
snapshot_diff (callable) – The per-snapshot difference function.

Returns:

error_pos (list of float) – The per-snapshot squared position errors.
error_vel (list of float) – The per-snapshot squared velocity errors.
residual (bool) – True if the two runs hold a different number of path blocks.

pyretis.testing.example_compare.check_accepted_paths(run_dir, ensembles)¶

Validate every accepted path in every ensemble.

Each accepted path must have length >= 3, a start/middle/end interface label consistent with the ensemble, and cross its two bounding interfaces.

Parameters:

run_dir (str) – The directory the simulation ran in.
ensembles (list) – The ensemble dictionaries from pyretis.setup.createsimulation.create_ensembles().

Returns:

int – The number of ensembles with a suspicious accepted path.

pyretis.testing.example_compare.check_accepted_paths_relaxed(run_dir, ensembles, zero_left=False)¶

Validate accepted paths with the relaxed (sparse-load) rules.

Like check_accepted_paths(), but the middle == 'M' marker and the interface-crossing invariants are relaxed for the sparse-LOADED initial paths ONLY – the rows whose move is ld. A short loaded [0^-] minus path need not reach the midpoint, so those stricter invariants do not apply to it; every genuinely sampled path (any other move) must still satisfy them, exactly as in the strict check. Length, start label and end label are always enforced.

Parameters:

run_dir (str) – The directory the simulation ran in.
ensembles (list) – The ensemble dictionaries from create_ensembles.
zero_left (bool, optional) – Whether the config sets [simulation] zero_left. Under zero_left, the [0^-] (ensemble 0) minus interface sits at the zero_left/interface[0] midpoint rather than at interface[0] itself, so an accepted [0^-] path can legitimately end 'L' (not only 'R') – without this flag, such a (correct) path is indistinguishable from a genuine wrong-end bug and would be flagged as one.

Returns:

int – The number of ensembles with a suspicious accepted path.

pyretis.testing.example_compare.check_retis_swaps(run_dir, ensembles)¶

Check order-parameter continuity of accepted swaps in every ensemble.

Parameters:

run_dir (str) – The directory the simulation ran in.
ensembles (list) – The ensemble dictionaries from create_ensembles.

Returns:

int – The number of failing swap checks (0 == all consistent and at least one accepted swap was examined).

pyretis.testing.example_compare.check_structural_suite(run_dir, n_interfaces, task)¶

Run the structural validity checks for a golden-less run.

The explore and repptis/pptis runs go through the infinite-swapping coordinator with a non-deterministic initiation, so a byte comparison against a stored reference is not meaningful. Instead this checks the run is a VALID sampling run: the expected per-ensemble output exists, every accepted path obeys the (task-specific) length and order-parameter rules, order.txt is in lockstep with pathensemble.txt, and real (non-loaded) moves were committed.

Parameters:

run_dir (str) – The directory the simulation ran in.
n_interfaces (int) – The number of interfaces in the config.
task (str) – The (lower-cased) simulation task; selects the profile from _STRUCTURAL_PROFILES.

Returns:

int – The number of failing structural checks (0 == valid run).

pyretis.testing.example_compare.check_zero_left_contract(nozero_dir, withzero_dir, acc_ratio_max=3.0)¶

Check the two-run behavioural contract of the zero_left feature.

no-zero and with-zero run the same short RETIS simulation and differ only in the [0^-] configuration. zero_left may reject paths with 0-L in [0^-] (and propagate one to [0^+] via a failed swap-zero), but must never (a) introduce 0-L in the inner ensembles [1^+], [2^+] …, nor (b) introduce maxlength (FTX/BTX) rejections in [0^-]. The two runs’ accepted-path counts should also stay within a generous factor.

Parameters:

nozero_dir (str) – The directory holding the plain (no zero_left) run’s ensembles.
withzero_dir (str) – The directory holding the zero_left run’s ensembles.
acc_ratio_max (float, optional) – The largest tolerated ratio between the two runs’ accepted-path counts in an ensemble (default 3.0; the run is short, fixed-seed).

Returns:

int – The number of violated contract clauses (0 == contract holds).

pyretis.testing.example_compare.compare_archive_check(run_dir, ref_dir, energy_skip=None, frames='byte')¶

Compare the two-tier trajectory stores of two runs.

For each ensemble directory present under ref_dir, byte-compares the per-ensemble OPERATIONAL archive (<ensemble>/paths/<pn>/, written by pyretis.inout.scheduler_archive. SchedulerPathStorage; the legacy trajs name is honoured for old references) between the two runs, and likewise the per-ensemble LONG-TERM store (<ensemble>/archive/<pn>/, the replaced paths) when either side has one – via compare_traj_archive_soft() in both cases. This check is independent of the golden check (which only compares the top-level pathensemble.txt/order.txt/ energy.txt): a restart or continuation can reproduce those exactly while still silently diverging in the raw archived frames, which is exactly what this check exists to catch.

Parameters:

run_dir (str) – The directory the simulation ran in.
ref_dir (str) – The committed reference directory, or a second live run directory.
energy_skip (list of str, optional) – Named energy terms to exclude from an archived energy.txt fragment’s comparison (see compare_traj_archive_soft()).

Returns:

int – The number of stores that differ (0 == all equal). An ensemble whose archive exists on only ONE side is a loud failure; one absent on BOTH sides is skipped (no path was born there on either run) – but a check that ends up comparing nothing at all fails loudly rather than passing vacuously.

pyretis.testing.example_compare.compare_ensemble_files_golden(run_dir, ref_dir, energy_skip=None, energy_unavailable=None, energy_unavailable_by_ensemble=None, path_skip=None, rel_tol=1e-05)¶

Compare every committed per-ensemble golden file against the run.

For each ensemble directory present under ref_dir, every golden file it commits (pathensemble.txt / order.txt / energy.txt / traj.txt) is compared against the matching file the run produced, using the file-type-appropriate primitive. A folder therefore selects its golden comparison simply by which files it commits – a pure RETIS folder commits only pathensemble.txt; a permeability or sparse-load folder also commits order.txt (and energy.txt); a folder whose engine cannot be trusted to match a numerical tolerance across builds (e.g. LAMMPS, run at a pinned, reference-grade version) instead commits traj.txt and compares it byte-for-byte.

Parameters:

run_dir (str) – The directory the simulation ran in (holds 000/, 001/ …).
ref_dir (str) – The committed reference directory (results/ – or a second live run directory, e.g. for a restart/continuation-equivalence check that has no committed golden of its own).
energy_skip (list of str, optional) – Named energy terms (as read by EnergyPathFile, e.g. 'vpot') to exclude from energy.txt. When given, the comparison switches from the default NaN-tolerant compare_energy_data() to a by-column comparison that additionally excludes these named terms – for a case where a term is legitimately expected to differ (e.g. GROMACS applying the long-range dispersion correction differently across a restart), not merely absent (all-NaN).
energy_unavailable (list of str, optional) – Named energy terms expected to be entirely NaN in every ensemble. The comparator verifies that they remain unavailable on both sides.
energy_unavailable_by_ensemble (dict, optional) – Per-ensemble unavailable terms, keyed by the three-digit ensemble directory name. This preserves finite coverage in other ensembles.
path_skip (list of int, optional) – Column indices to exclude from pathensemble.txt.
rel_tol (float, optional) – Relative tolerance forwarded to every numerical comparator (default matches each comparator’s own 1e-5 default).

Returns:

int – The number of mismatching golden files (0 == all equal). A missing or empty reference, or a reference that holds none of the recognized golden files, is a loud failure (>= 1) rather than a vacuous “all equal”: a comparison that inspected nothing has validated nothing.

pyretis.testing.example_compare.compare_reports_golden(run_dir, ref_dir)¶

Compare the committed top-level report / generated-config goldens.

For every non-directory file committed at the top of ref_dir: *.toml (a generated per-interface config) is compared line by line ignoring the run-dependent exe_path; *.html / *.rst / *.tex (a report) is compared with version/timestamp normalisation, resolving the generated report’s _cycles-NNN suffix.

Parameters:

run_dir (str) – The directory the simulation ran in.
ref_dir (str) – The committed reference directory (results/).

Returns:

int – The number of mismatching golden files (0 == all equal). A committed top-level reference file with no comparator (an unexpected extension) is a loud failure rather than being silently skipped, and a reports check that compared nothing (no report/config golden under ref_dir) also fails.

pyretis.testing.example_compare.compare_restart_concat(part1_dir, part2_dir, full_dir)¶

Compare a two-part (concatenated) restart against the full run.

Part 1 plus part 2 (dropping the single overlapping record) must equal the full run for each output file. The concatenation-aware primitives from pyretis.testing.simulation_comparison handle the overlap internally:

cross.txt – compare_restarted_cross_files();
energy.txt and order.txt – compare_restarted_text_files().

Parameters:

part1_dir (str) – The first part’s run directory.
part2_dir (str) – The second (restarted) part’s run directory.
full_dir (str) – The uninterrupted (full) run directory.

Returns:

int – The total number of failing checks (0 == the restart reproduces the full run).

pyretis.testing.example_compare.compare_restart_md(traj_files, text_triples, tol=1e-12)¶

Compare a concatenated MD restart against the full run.

The MD restart is split into two parts written with per-run file-name prefixes. The trajectory is compared by per-snapshot MSE (compare_restart_traj_mse() with SnapshotFile, which drops the overlapping restart frame); the remaining text files (thermo, energy) are compared with compare_restarted_text_files().

Parameters:

traj_files (list of str) – The trajectory files [part1, ..., full]; the last is the full run, the rest are the ordered restart parts.
text_triples (list of (list or tuple) of str) – Each entry is a (part1, part2, full) triple of text-file paths (e.g. the thermo and energy files) checked with the concatenation-aware text comparison.
tol (float, optional) – The trajectory MSE tolerance (1e-12).

Returns:

int – The total number of failing checks (0 == the restart reproduces the full run).

pyretis.testing.example_compare.compare_restart_traj_mse(part_files, full_file, reader, tol=1e-12)¶

Compare a restart trajectory against the full run by snapshot MSE.

The restart and full trajectories are read snapshot-by-snapshot with reader and their per-snapshot squared position/velocity differences are averaged; each average must stay below tol. The two must also hold the same number of snapshots (or path blocks, for the whole-file case). The reader selects the whole-file vs concatenated behaviour and the matching snapshot-difference function (see _TRAJ_READER_SPECS): PathIntFile reads a whole-file restart block-for-block, SnapshotFile chains the concatenated parts (dropping each later part’s overlapping first frame).

Parameters:

part_files (list of str) – The restart trajectory file(s). A whole-file restart passes a single file; a concatenated restart passes the ordered parts.
full_file (str) – The uninterrupted (full) run’s trajectory file.
reader (callable) – The trajectory reader class (PathIntFile or SnapshotFile).
tol (float, optional) – The tolerance each averaged error term must stay below (1e-12).

Returns:

int – The number of failing checks: a length mismatch counts as 1, and each out-of-tolerance term (positions, velocities) adds 1.

pyretis.testing.example_compare.compare_restart_whole(config_path, restart_dir, full_dir, tol=1e-12)¶

Compare a whole-file restart against the uninterrupted run.

The restart leg appends its steps onto the existing history, so its per-ensemble output already holds the WHOLE trajectory; it is compared file-for-file against the full run for the single ensemble named by the config’s [tis] ensemble_number:

the two-tier trajectory stores (<ens>/paths/<pn> + the per-ensemble archive/) – byte-compared via compare_archive_check(): every sampled path’s full trajectory, not just the initial one. (The former <ens>/traj.txt MSE compared a file that, since the in-process-loop retirement, only ever held the cycle-0 initiation block – a vacuous check the retired initiation dump masked.)
energy.txt – column comparison skipping the etot/temp terms the internal engine leaves entirely NaN (an all-NaN term tests nothing; ekin/vpot are still compared);
order.txt – numerical comparison.

Parameters:

config_path (str) – The run config (.toml); its [tis] ensemble_number names the single ensemble to compare.
restart_dir (str) – The restarted run directory (holds the whole history).
full_dir (str) – The uninterrupted (full) run directory.
tol (float, optional) – Unused (kept for call compatibility); the archives are byte-compared.

Returns:

int – The total number of failing checks (0 == the restart reproduces the full run).

pyretis.testing.example_compare.compare_run(config_path, run_dir=None, ref_dir=None, checks=None, energy_skip=None, energy_unavailable=None, energy_unavailable_by_ensemble=None, path_skip=None, rel_tol=1e-05, archive_frames='byte')¶

Run the reference comparison for a completed example simulation.

Parameters:

config_path (str) – The simulation config (.toml) the example ran with.
run_dir (str, optional) – Where the run produced its per-ensemble output (default: the config’s directory).
ref_dir (str, optional) – The committed reference directory (default: results/ next to the config). This may instead be a second, independently-run directory of the SAME config – e.g. a restart/load/two-engine equivalence check that has no committed golden of its own; the golden and archive checks only require a matching per-ensemble layout, not that ref_dir be a “reference” in the historical sense.
checks (list of str, optional) – The checks to run (any of _ALL_CHECKS). When None the set is inferred from the task (_default_checks()). A folder overrides this when its faithful verdict is a subset – e.g. the permeability example is task = retis but historically compared only the golden files (its zero_left geometry does not satisfy the generic accepted-path invariants), so it passes checks=['golden'].
energy_skip (list of str, optional) – Named energy terms to exclude from energy.txt (golden check) and any archived energy.txt fragment (archive check) – see compare_ensemble_files_golden() / compare_archive_check().
energy_unavailable (list of str, optional) – Terms expected to be all-NaN in every golden energy.txt file.
energy_unavailable_by_ensemble (dict, optional) – Per-ensemble expected unavailable terms for the golden check.
path_skip (list of int, optional) – Column indices to exclude from pathensemble.txt (golden check only).
rel_tol (float, optional) – Relative tolerance forwarded to the golden check’s numerical comparators (default 1e-5, matching each comparator’s own default).

Returns:

int – The total number of failing checks (0 == success).

pyretis.testing.example_compare.generate_golden_tarball(config_path, output_tgz, run_dir=None, files=None)¶

Pack a run’s own per-ensemble files into a golden reference tarball.

The “re-bless the reference” counterpart to the golden check’s tarball form (_resolved_ref_dir()): for a folder whose engine is only bit-reproducible at a pinned, reference-grade version (e.g. LAMMPS), the committed reference is one small binary-diffable .tgz rather than a committed tree, generated by running the example once on that reference build and packing its own output. Every ensemble’s traj.txt is reconstructed first (a no-op for an ensemble that already writes it directly) via reconstruct_traj_for_run().

Parameters:

config_path (str) – The simulation config (.toml) the run used.
output_tgz (str) – Where to write the .tgz archive.
run_dir (str, optional) – Where the run produced its per-ensemble output (default: the config’s directory).
files (list of str) – The per-ensemble file names to pack (e.g. ['pathensemble.txt', 'order.txt', 'traj.txt']) – deliberately explicit and required, since this writes the reference every future run of this folder is graded against.

Returns:

int – The number of requested files missing from the run (and therefore not packed); 0 means every requested file, for every ensemble, was found and written.

pyretis.testing.example_compare.main(argv=None)¶

Command-line entry point for the shared example comparison.

Subcommands¶

compare <config.toml>: Compare one completed run against its committed results/ reference (golden files + task-appropriate structural checks; override with --checks / --ref-dir).
zero-left <nozero_dir> <withzero_dir>: Check the two-run zero_left behavioural contract.
restart-whole <config> <restart_dir> <full_dir>: Check a whole-file restart reproduces the uninterrupted run.
restart-concat <part1_dir> <part2_dir> <full_dir>: Check a two-part concatenated restart equals the full run.
restart-md --traj P1 P2 FULL --text P1 P2 FULL ...: Check a concatenated MD restart (trajectory + text files).
reconstruct-traj <config.toml>: Reconstruct fragmented traj.txt files before compare.
generate-golden-tarball <config.toml> <output.tgz> --files ...: Re-bless a tarball-form golden reference from the run’s own output.

pyretis.testing.example_compare.reconstruct_traj_for_run(config_path, run_dir=None)¶

Reconstruct every ensemble’s fragmented traj.txt for a run.

A thin per-run driver over reconstruct_trajectory_text(): some engines (e.g. LAMMPS) scatter an ensemble’s trajectory across per-trial fragments instead of writing a single traj.txt at the ensemble root, so a folder that commits traj.txt as a golden file (see compare_ensemble_files_golden()) must reconstruct it before comparing – run this once, right after the simulation and before compare.

Parameters:

config_path (str) – The simulation config (.toml) the run used.
run_dir (str, optional) – Where the run produced its per-ensemble output (default: the config’s directory).

Returns:

int – The number of ensembles whose traj.txt was (re)written (0 is not an error by itself – an ensemble whose engine never fragments its trajectory has nothing to reconstruct).

pyretis.testing.reference_manifest module¶

Validate provenance and content digests for numerical references.

pyretis.testing.reference_manifest._artifact_path(root, relative)¶: Resolve one safe repository-relative artifact path.

pyretis.testing.reference_manifest._content_sha256(path)¶: Hash a file without loading the complete artifact into memory.

pyretis.testing.reference_manifest._required_text(mapping, key, context)¶: Return a required non-empty string from a manifest object.

pyretis.testing.reference_manifest._validate_artifact_digests(artifact_set, name, root)¶: Compare one set’s declared digests against the on-disk artifacts.

pyretis.testing.reference_manifest._validate_artifact_set(artifact_set, name, context, root)¶: Validate one artifact set’s provenance fields and content digests.

pyretis.testing.reference_manifest._validate_energy_term_list(terms, context)¶: Validate one explicit list of unavailable physical quantities.

pyretis.testing.reference_manifest._validate_unavailable_energy_terms(value, context)¶: Validate global or per-ensemble unavailable-energy declarations.

pyretis.testing.reference_manifest.calculate_artifact_set(root, paths)¶

Calculate a deterministic digest and inventory for artifact paths.

Directory inputs are expanded recursively. The tree digest includes each repository-relative path, byte count, and file-content SHA-256 digest, in lexicographic path order. Overlapping path declarations are rejected.

Parameters:

root (path-like) – Repository root used to resolve the relative paths.
paths (sequence of str) – Files or directories included in one reference artifact set.

Returns:

dict – tree_sha256, file_count, and byte_count values suitable for the manifest’s artifacts object.

pyretis.testing.reference_manifest.main(argv=None)¶: Run reference-manifest validation from the command line.

pyretis.testing.reference_manifest.validate_manifest(manifest_path, root=None)¶

Validate manifest schema, provenance fields, and artifact digests.

Parameters:

manifest_path (path-like) – JSON provenance manifest to validate.
root (path-like, optional) – Repository root. By default this is inferred for the canonical examples/tests/reference_provenance.json location.

Returns:

list of str – Names of the artifact sets successfully validated.

pyretis.testing.simulation_comparison module¶

Methods for comparing simulation results.

This module defines methods that can be used for comparing results from different simulations, such as output files, reports, and path ensembles.

pyretis.testing.simulation_comparison._compare_block_comments(comment1, comment2)¶

Compare two block comment lists, tolerating 1-ULP float differences.

Parameters:

comment1 (list of str) – Comment lines from the first file block.
comment2 (list of str) – Comment lines from the second file block.

Returns:

bool – True if the comments are considered equal.

pyretis.testing.simulation_comparison._files_agree_over_common_prefix(file1, file2)¶

Check two binary files agree byte-for-byte over their common prefix.

Engine frame payloads (e.g. GROMACS .trr segments) may carry unreferenced TRAILING surplus frames: the propagation dump interval does not stop exactly at the frame the path keeps, and how much surplus is written is not reproducible between two runs (the ratified init-procedure rule: logic preserved, not byte-identical). The frames a path actually references are pinned by its traj.txt, which is compared exactly and separately. A rewrite, corruption, or genuine divergence of trajectory data shows up inside the common prefix and still fails.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.

Returns:

equal (bool) – True if the files agree over their whole common prefix.
surplus (bool) – True if the files agree but one carries trailing surplus bytes.

pyretis.testing.simulation_comparison._read_file_lines(filepath)¶: Read all lines from a file.

pyretis.testing.simulation_comparison.compare_data_by_columns(file1, file2, file_type, skip=None)¶

Compare two output PyRETIS data files by columns.

This method compares files where numbers are stored in columns and the columns have specific labels. Here, we also compare labels and comments.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
file_type (str) – A string used to determine the file type (e.g., ‘energy’).
skip (list of str, optional) – A list of items from the loaded data we are to skip. This can, for instance, be certain energy terms that are not absolute and can’t easily be compared.

Returns:

equal (bool) – True if the files are deemed to be equal.
msg (str) – A descriptive message of the result of the comparison.

pyretis.testing.simulation_comparison.compare_energy_columns_mse(energy, reference, pairs, tol=1e-05)¶

Per-term MSE check of a pyretis energy.txt array vs an engine’s.

The test-integrate example comparators (cp2k, gromacs) share this: each checks that pyretis reproduces the engine’s own energies term by term, differing only in which energy.txt column maps to which engine-output key.

Parameters:

energy (numpy.ndarray) – The pyretis energy.txt data (as numpy.loadtxt() returns; columns cycle, vpot, ekin, etot, temp, …).
reference (mapping of str to numpy.ndarray) – The engine’s energies keyed by term name (the keys used in pairs).
pairs (sequence of (int, str)) – (column_index_in_energy, reference_key) pairs to compare, e.g. ((1, 'vpot'), (2, 'ekin'), (3, 'etot'), (4, 'temp')).
tol (float, optional) – Per-term MSE tolerance; a falsy tol reports the MSEs without judging them.

Returns:

equal (bool) – True when every compared term matches in length and (if tol) is within tolerance.
msg (str) – The per-term MSEs, plus the failing term / tolerance on failure.

pyretis.testing.simulation_comparison.compare_energy_data(file1, file2, rel_tol=1e-05, unavailable_terms=None)¶

Compare two energy.txt files with explicit unavailable terms.

Energy files always carry a fixed set of columns (time, potential, kinetic, total, temperature), but not every engine or path populates all of them: paths loaded from disk carry no computed energy, and an engine may not report e.g. the total energy or the temperature. Such a term is written as an entirely-NaN column.

This wraps compare_numerical_data() with energy-specific handling of those columns. Unavailable physical quantities must be named by the caller; matching all-NaN columns are not inferred as a success:

A term named in unavailable_terms must be entirely NaN in both files and is then excluded from the numerical comparison.

An all-NaN term that was not declared is an unexpected coverage gap and fails.

Every other column is compared normally. A column that is all-NaN in only one file (a run that suddenly computes – or stops computing – a term) therefore still fails, via the NaN-position check, rather than being silently skipped.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
rel_tol (float, optional) – Relative tolerance forwarded to compare_numerical_data().
unavailable_terms (sequence of str, optional) – Energy terms that this reference intentionally cannot provide. Valid names are vpot, ekin, etot, and temp.

Returns:

equal (bool) – True if the files are deemed to be equal.
msg (str) – A descriptive message of the result of the comparison.

pyretis.testing.simulation_comparison.compare_numerical_data(file1, file2, rel_tol=1e-05, skip_cols=None)¶

Compare two files containing numerical data.

Here, we compare files that contain numerical data. We don’t care about comments here, we just compare the actual numerical data.

A meaningful comparison must not pass on degenerate input. This rejects empty data, shape mismatches, NaNs that sit at different positions in the two files (a divergence that turned a finite value into NaN or vice versa), and data/columns that are entirely NaN (which test nothing). Partial NaN at matching positions – e.g. loaded (ld) frames that legitimately have no computed energy – is allowed; the finite values around them are still compared. Columns that are unavailable for an engine (e.g. a potential energy that engine never reports) must be excluded explicitly via skip_cols so the gap is visible rather than silently “equal”.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
rel_tol (float, optional) – Relative tolerance for the comparison.
skip_cols (list of int, optional) – Column indices to exclude from the comparison (e.g. a column an engine does not populate). Excluding a column is explicit and visible, unlike letting NaN==NaN pass.

Returns:

equal (bool) – True if the files are deemed to be equal.
msg (str) – A descriptive message of the result of the comparison.

pyretis.testing.simulation_comparison.compare_numerical_mse(file1, file2, tol=1e-12)¶

Compare two numerical files using mean squared error.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
tol (float, optional) – Tolerance for the mean squared error.

Returns:

equal (bool) – True if the MSE is below the tolerance.
msg (str) – A descriptive message with the MSE value.

pyretis.testing.simulation_comparison.compare_path_ensemble_data(file1, file2, rel_tol=1e-05, skip=None)¶

Compare two path ensemble files.

We compare line-by-line, but skip comments and we check that numbers are close, as judged by the given relative tolarance.

Parameters:

file1 (str) – The path to the first file to consider in the comparison.
file2 (str) – The path to the second file to consider in the comparison.
rel_tol (float, optional) – A relative tolerance used to determine if numbers are equal.
skip (list of int, optional) – These are columns we are to skip in the comparison.

Returns:

equal (bool) – True if the files are equal, False otherwise.
msg (str) – A message describing the result of the comparison.

pyretis.testing.simulation_comparison.compare_reports_normalized(fil1, fil2)¶

Compare two reports, normalizing common version/time differences.

This function ignores Docutils version meta-data, timestamps, and common spelling variations (grey/gray) in CSS to remain robust against environment differences.

Parameters:

fil1 (str) – The path to the first report to compare.
fil2 (str) – The path to the second report to compare.

Returns:

equal (bool) – True if reports are essentially equal.
msg (str) – Description of mismatch if found.

pyretis.testing.simulation_comparison.compare_restarted_cross_files(file11, file12, file2)¶

Compare CrossFile data from a restarted simulation.

Parameters:

file11 (str) – Path to the first part of the crossing data.
file12 (str) – Path to the second part of the crossing data.
file2 (str) – Path to the full (continuous) crossing data.

Returns:

equal (bool) – True if the crossing data matches.
msg (str) – A descriptive message.

pyretis.testing.simulation_comparison.compare_restarted_text_files(file11, file12, file2)¶

Check if file2 is equal to file11 + file12 minus one overlapping line.

We handle headers (lines starting with ‘#’) by skipping them in the second file part.

Parameters:

file11 (str) – Path to the first part of the restarted simulation output.
file12 (str) – Path to the second part of the restarted simulation output.
file2 (str) – Path to the full (continuous) simulation output.

Returns:

equal (bool) – True if the files match the pattern.
msg (str) – A descriptive message of the result.

pyretis.testing.simulation_comparison.compare_simulation_files(file1, file2, skip=None, mode='line')¶

Top-level function to compare two simulation output files.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
skip (list of str or list of int, optional) – A list of items that are to be skipped in the comparison.
mode (str, optional) – A string used to determine how we do the comparison: ‘numerical’ will select a comparison of numerical blocks; ‘line’ will select a line-by-line text comparison; anything else will perform a literal file comparison.

Returns:

equal (bool) – True if the files were found to be equal, False otherwise.
msg (str) – A string with information about the comparison result.

pyretis.testing.simulation_comparison.compare_text_line_by_line(file1, file2, skip=None, skip_keys=None)¶

Compare two files, line by line.

Parameters:

file1 (str) – The path to the first file to compare.
file2 (str) – The path to the second file to compare.
skip (list of int, optional) – These are 0-indexed line numbers we are to skip.
skip_keys (list of str, optional) – Lines whose first token matches any key in this list are filtered out from both files before comparison. Useful for ignoring settings like exe_path that differ by run directory.

Returns:

equal (bool) – True if the files are deemed to be equal.
msg (str) – A descriptive message of the result of the comparison.

pyretis.testing.simulation_comparison.compare_traj_archive(dir1, dir2)¶

Compare archived trajectories between two directories.

These archives consist of trajectory information such as energies, order parameters and positions. Here, we verify that the output written by PyRETIS is identical in the two cases.

Parameters:

dir1 (str) – The path to the first directory to use in the comparison.
dir2 (str) – The path to the second directory to use in the comparison.

Returns:

errors (list of tuple) – This list contains the files which differed, if any. A directory that is absent on either side is reported as a difference, never silently equated: two non-existent archives would otherwise both yield empty file lists and be judged “equal”.

pyretis.testing.simulation_comparison.compare_traj_archive_soft(dir1, dir2, energy_skip=None, frames='byte')¶

Compare an archived trajectory, tolerating named energy terms.

Like compare_traj_archive() (byte-exact for every archived file), with engine-facing relaxations for the run-vs-run self-consistency suites:

an archived energy.txt fragment is compared by column instead of byte-for-byte, skipping the named terms in energy_skip. Some external engines recompute a term (e.g. GROMACS’s long-range dispersion correction to the potential energy) slightly differently when continuing a run;
with frames='byte' (the default), engine frame payloads (any file that is not one of the PyRETIS-written energy.txt/order.txt/traj.txt) must agree byte-for-byte over their COMMON PREFIX – trailing unreferenced surplus frames are logged and tolerated (see _files_agree_over_common_prefix());
with frames='names', frame payloads are checked by name and count only. This is for comparisons across DIFFERENT engine implementations of the same MD (e.g. the relaunch vs streaming GROMACS engines), whose physics text output is exact-equal while their archived payload framing (frame counts, dump cadence) is implementation-specific and never byte-comparable.

Parameters:

dir1 (str) – The path to the first directory to use in the comparison.
dir2 (str) – The path to the second directory to use in the comparison.
energy_skip (list of str, optional) – Energy terms (as read by EnergyPathFile, e.g. 'vpot') to exclude from the comparison of any archived energy.txt fragment. None compares every term.
frames (str, optional) – 'byte' (default) or 'names'; see above. The PyRETIS-written text files are compared exactly in BOTH modes.

Returns:

errors (list of tuple) – The files which differed, if any (see compare_traj_archive() for the missing-directory behaviour).

pyretis.testing.simulation_comparison.read_files(*files, read_comments=True)¶

Read files into memory.

Here, we assume that we are given small files and that we can read these into memory.

Parameters:

files (tuple of str) – These are the paths to the files we are to read.
read_comments (bool, optional) – If False, we skip lines starting with a “#”.

Returns:

all_data (list of list of str) – The data read from the different files.

pyretis.testing.simulation_comparison.reconstruct_trajectory_text(ensemble_dir)¶

Reconstruct a consolidated traj.txt from per-trial fragments.

Some engines (e.g. LAMMPS) write one traj.txt fragment per accepted trial, nested under <ensemble_dir>/traj/<trial>/, rather than a single running traj.txt at the ensemble root the way most engines do. Each fragment’s first line carries a # Cycle:<n>, ... header; concatenating the fragments in ascending cycle order reconstructs the same <ensemble_dir>/ traj.txt a non-fragmenting engine would have written directly, so the two can be compared with the same golden-file logic.

Parameters:: ensemble_dir (str) – Path to a single ensemble’s output directory (e.g. .../000).
Returns:: written (bool) – True if at least one fragment was found and traj.txt was (re)written; False if <ensemble_dir>/traj does not exist or holds no traj.txt fragment, in which case nothing is written (the ensemble has no fragmented trajectory to reconstruct – not a failure by itself).

pyretis.testing.helpers module¶

Methods that might be useful for testing.

This module defines generic methods for testing.

pyretis.testing.helpers.VALIDATE_ENV = 'PYRETIS_VALIDATE'¶: Environment variable that turns the numerical reference comparison on ("1", the default) or off ("0"). It is set to "0" by the engine test runners when only a non-reference-grade external MD engine is available (for example a single-precision GROMACS build when the reference data was generated in double precision). External MD engines are not bit-for-bit reproducible across build, precision and hardware, so a reference comparison is only meaningful against the exact reference-grade engine.

pyretis.testing.helpers.clean_dir(dirname)¶: Remove ALL files in the given directory.

pyretis.testing.helpers.reference_validation_enabled()¶

Return whether numerical reference comparison should be performed.

Returns:: out (boolean) – True unless the PYRETIS_VALIDATE environment variable is set to "0". The default (variable unset) is True so that the validated path is unchanged when a reference-grade engine is present.

pyretis.testing.helpers.search_for_files(rootdir, match=None)¶

Find files by walking the given directory.

Parameters:

rootdir (string) – The path where we will search from.
match (string, optional) – If given, the method will only return files that are equal to the given match.

Returns:

out (list of strings) – The paths of the found files.

pyretis.testing.helpers.skip_reference_validation(reason)¶

Report that the run was executed but not numerically validated.

This prints a clearly labelled NOT VALIDATED message and exits with status 0. It is used by the engine test scripts as the “execution smoke” outcome: the simulation ran without error, but the numerical comparison against the committed reference was skipped because no reference-grade engine was available. Following the project rule “better a fail than a wrong pass”, a skipped comparison is never reported as a pass.

Parameters:: reason (string) – Why the numerical comparison was skipped (for example "single-precision GROMACS; reference is double precision").

pyretis.testing.systemhelp module¶

Methods that might be useful for testing.

This module defines methods that are useful in connection with systems.

pyretis.testing.systemhelp.create_system_ext(pos=None, vel=False)¶: Create an external system with given positions and velocities.