Monitoring a running simulation

A path-sampling run is long, and the useful question during it is not “what is the rate” – that is the last thing to converge – but “is this run doing anything sensible?” A run that is sampling badly looks wrong within a few hundred cycles, in ways that are visible long before any rate is trustworthy.

Everything on this page can be done while the simulation is still running. PyRETIS writes its per-ensemble output incrementally and appends to it, so the output on disk is always a complete record of the cycles finished so far. Nothing here modifies the run, and nothing here needs the run to stop.

See also

Simulation outputs describes the files these commands read; Calculating the rate explains the rate itself, and Cleaning run artifacts how to reset a directory afterwards.

pyretis status: what each ensemble sampled

Run it in the directory the simulation is running in:

pyretis status

It reads the per-ensemble output in place, computes nothing expensive – no crossing probabilities, no matching, no rate – and prints what the sampler has actually been doing:

Per-ensemble status

  Ens         Cycles    Trials       Acc       Rej    Acc%   Shoot    Swap   HA eff
-----  -------------  --------  --------  --------  ------  ------  ------  -------
  000           1-25         8         7         1    87.5   0.750   1.000      n/a
  001           1-25         5         5         0   100.0   1.000   1.000      n/a
  002           1-25        11         7         4    63.6   0.636     n/a      n/a
  003           1-25         5         5         0   100.0   1.000     n/a      n/a

Moves attempted, per ensemble

  000: s+: 4/4 (1.000)  sh: 3/4 (0.750)
  001: s-: 4/4 (1.000)  sh: 1/1 (1.000)
  002: sh: 7/11 (0.636)
  003: sh: 5/5 (1.000)

  * a high-acceptance move: read its HA eff., not its ratio.

Why trials were rejected

  000: BTL: 1
  002: NCR: 4

  The codes are explained in the documentation: path types and
  rejections.

Because it is cheap and read-only, it is safe to run repeatedly against a live run – for instance under watch:

watch -n 60 pyretis status --no-moves

Reading the table

Column	Meaning
Ens	The ensemble directory (000, 001, …).
Cycles	The first and last cycle this ensemble has recorded.
Trials	Monte Carlo moves attempted in this ensemble.
Acc / Rej	How many of those were accepted, and how many rejected.
Acc%	The overall acceptance percentage over all move types.
Shoot	Acceptance ratio of the moves that shoot (sh, bias, wt) alone: the high-acceptance moves, which would drag it towards 1, and the frequency moves such as time reversal (tr) are left out.
Swap	Acceptance ratio of the swapping moves. n/a where no swap was ever proposed: a per-interface TIS run, for instance, and every ensemble of a retis run but 000 and 001, whose zero swap is the only swap move of the run.
HA eff	For wire fencing and stone skipping: the weight collected per attempt. Those moves accept almost every trial, so their acceptance ratio is near 1 whatever the sampling is doing; the weight is what varies and what matters.

A ratio is n/a when the move was never attempted in that ensemble. A move that was not attempted has no acceptance ratio – it is not a ratio of one.

What to look for

• A very low shooting ratio in one ensemble usually means its interfaces are too far apart, or the shooting move is too aggressive for the system.

• A dominant rejection code is the most useful single diagnostic on the page. Many NCR (no crossing) means trials are not reaching the interface; the codes are listed in Path types and rejection reasons.

• Swap is a ratio only in 000 and 001 of a retis run, the two ensembles of its one swap move, the zero swap. Every other ensemble shows n/a there by design: those ensembles share the live paths, each path’s occupancy spread over the ensembles it is valid in, instead of swapping them. Whether they exchange paths is the Exchange column of the pyretis analyse report (Acceptance ratios): 0 there means they are not exchanging, which defeats the point of RETIS.

• Cycles spans nearly the whole run in every ensemble of a retis run: an ensemble records every cycle in which no worker is running a move of it, so every cycle with one worker. A stalled ensemble shows in its Trials and Shoot columns, not in Cycles.

Options

Argument	Description
-d DIR, --directory DIR	The directory the simulation ran in (default: the current one).
-skipb N, --skip-begin N	Discard the first N cycles of the run, as for pyretis analyse. The single TIS runs of make-tis-files count the cycles of each ensemble’s own file; a directory without a run file, or whose out.rst names no task, counts the cycles of the run.
-skipe N, --skip-end N	Discard the last N cycles of the run, as for pyretis analyse; the cycles are counted as for -skipb.
--no-moves	Leave out the per-move breakdown.
--no-rejections	Leave out the rejection breakdown.
--json	Write the status as JSON, for further processing.

The cycles of the run are the cycles that the pathensemble.txt of any ensemble records, and the skips leave out the first or the last N of them, the same cycles in every ensemble, as pyretis analyse does. The ensembles of a run can record different cycles: with several workers, and in a PyRETIS 3 RETIS run, whose swap and relative-shoots cycles write no row for the ensembles they leave out ([retis] nullmoves unset or false). The attempted moves of moves.txt are counted over the same window: an ensemble that records no cycle in the window keeps the attempts of the cycles of the window, and a skip of every cycle keeps none.

A directory holds one run per ensemble when its run file names single TIS runs. The run file is read in this order:

• output.toml, else the restart.toml an earlier version wrote in place of it: single TIS runs when [simulation] is a tis task with three interfaces, as each input make-tis-files writes, or has single_tis = true (a run file of an earlier version);

• the out.rst of PyRETIS 3, single TIS runs when its [simulation] task is tis or make-tis-files (the out.rst holds the run made last).

Each ensemble of such a directory counts the cycles its own pathensemble.txt records, as pyretis analyse of the make-tis-files input does. Every other run file, an out.rst that names no task (for instance an empty file), and a directory without a run file are read as one run over the cycles of the run, as pyretis analyse reads every task but make-tis-files.

The --json form is the one to build on if you are watching many runs at once:

pyretis status --json | python -c \
    "import json,sys; [print(e['ensemble'], e['rejections']) \
     for e in json.load(sys.stdin)]"

Old runs are handled too: like pyretis analyse, pyretis status reads the attempted moves from every layout PyRETIS has written – moves.txt, attempts recorded inside pathensemble.txt, or a separate trials.txt.

Analysing a run that has not finished

pyretis analyse can be run against a live simulation:

pyretis analyse -i retis.toml

The analysis reads the output files as they stand and reports on the cycles completed so far. It does not touch the running simulation, and the simulation does not wait for it.

Two things are worth knowing:

1. The report is stamped with the cycle count (<name>_report_cycles-NNNNNNNNN.rst), so analysing repeatedly during a run leaves a series of reports rather than overwriting one. That series is itself informative: the rate should stop moving.

2. A rate from an unconverged run is not wrong so much as premature. Use -skipb to drop the equilibration, and read Is it converged? before quoting anything.

The cheapest useful check during a run is whether the rate is still drifting. Analyse twice, a few hundred cycles apart, and compare – if the second is outside the first’s error bar, the run is not converged.

Visualising a run that has not finished

PyVisA can likewise be used on a running simulation. Its GUI does not open the .toml input of a run: compress what the run has written so far, and open the compressed file:

pyvisa -i retis.toml -cmp
pyvisa -i pyvisa_compressed_data.hdf5.zip

The compressor reads the same output files and leaves them as they are; compress again to pick up the cycles finished since. This is the fastest way to see the shape of a problem – a crossing-probability curve with a kink at one interface, or a path-length distribution with a tail that should not be there.

Resetting a directory afterwards

When a run is finished with, pyretis tools clean removes the generated files and leaves the committed inputs:

pyretis tools clean --dry-run    # list what would be removed
pyretis tools clean              # remove it

Always use --dry-run first in a directory that holds anything you staged by hand. The full description, including what counts as an artifact and what is protected, is in Cleaning run artifacts.