The CPU and CUDA passes of the engine suites¶

The engine suites of examples/tests/ (test-gromacs/, test-openmm/ and test-lammps/) run every case on the CPU. A case with a CUDA pass also runs on an NVIDIA GPU when CUDA is available and the CUDA build of its engine is installed. This page lists the switch that selects the passes, when CUDA is available, what counts as an installed CUDA build, which cases have a CUDA pass and what each one checks, and how the passes are reported and graded. The suites are described in examples/tests/README.rst, and Example test status gives the status of each tutorial.

The device switch¶

PYRETIS_TEST_DEVICE selects the passes of every engine suite:

auto (default): The CPU passes, and the CUDA passes of each engine with a CUDA build when CUDA is available. When CUDA is not available, the CPU passes only, whatever CUDA builds are installed.
cpu: The CPU passes only.
cuda: As auto; CUDA not available, or an engine without a CUDA build, fails the suite of the engine, with exit status 1 and before any case, with “CUDA REQUIRED” (the messages are given below).

Any other value stops the suite with exit status 2.

When CUDA is not available¶

CUDA is available on a machine when nvidia-smi lists a GPU and CUDA_VISIBLE_DEVICES leaves one to the CUDA programs. Every suite checks this first, under auto and cuda, before it looks for the CUDA build of its engine. When CUDA_VISIBLE_DEVICES is empty, its first entry is neither a device index nor a GPU or MIG UUID (-1, for example) or an index beyond the GPUs nvidia-smi lists, or nvidia-smi is not found, cannot be run, fails, does not end within 300 s, or lists no GPU, CUDA is not available: under auto the suite runs its CPU passes only, whatever CUDA builds are installed (a CUDA build that is found, an OpenMM CUDA plugin that fails to load, as it does without the NVIDIA driver), and says why in its output, in the device ledger and in the recap, for example:

Device: gromacs CPU passes only (CUDA not available: nvidia-smi ended
with status 9: NVIDIA-SMI has failed because it couldn't communicate
with the NVIDIA driver. Make sure that the latest NVIDIA driver is
installed and running.)

The CUDA passes do not run and are not counted: the grades of the suite are those of its CPU passes. Under cuda each engine suite fails before any case:

CUDA REQUIRED: GROMACS: CUDA is not available (nvidia-smi not found).
Install or repair the NVIDIA driver so that nvidia-smi lists the GPU,
or set PYRETIS_TEST_DEVICE to auto or cpu to run its CPU passes only.

When CUDA is available, the rest of this page applies.

An engine without a CUDA build fails a cuda suite with a message that ends with the way on, for example for GROMACS:

CUDA REQUIRED: GROMACS has no CUDA build (gmx_cuda not found). Set
PYRETIS_GMX_CUDA to a command whose -version reports "GPU support:
CUDA" (default gmx_cuda, looked up on PATH), or set
PYRETIS_TEST_DEVICE to auto or cpu to run its CPU passes only.

The message of LAMMPS names PYRETIS_LMP_CUDA, -h, GPU package API: CUDA and lmp_cuda. That of OpenMM names the platforms it loaded and the variable OPENMM_PLUGIN_DIR, for example:

CUDA REQUIRED: OpenMM has no CUDA platform (platforms: Reference, CPU,
OpenCL). Set OPENMM_PLUGIN_DIR to a plugins folder of version 8.5.1
whose CUDA plugin libOpenMMCUDA.so loads, or set PYRETIS_TEST_DEVICE
to auto or cpu to run its CPU passes only.

The version is that of the installed OpenMM. Under auto such an engine runs its CPU passes only.

What “installed” means¶

The CUDA build of an engine is installed when

GROMACS: the command that PYRETIS_GMX_CUDA names (default gmx_cuda) is found and its -version reports GPU support: CUDA;
LAMMPS: the command that PYRETIS_LMP_CUDA names (default lmp_cuda) is found and its -h reports GPU package API: CUDA;
OpenMM: its CUDA platform is loaded. A private folder of OpenMM plugins is named with OPENMM_PLUGIN_DIR, which OpenMM reads: it then loads its plugins from that folder in place of the plugins folder of its installation (an empty folder leaves the Reference platform only), and the CUDA plugin loads only when the CUDA libraries it needs are found.

PYRETIS_GMX_CUDA and PYRETIS_LMP_CUDA name the command alone, without options and with no space in its path, as an executable file or as a command on PATH. A relative path is taken from the folder of the suite (test-gromacs/, test-lammps/), and, for the CUDA tutorials, from the folder that examples/run-tutorials.sh is started in. Only the default name (the variable unset or empty) that is not found means that the engine has no CUDA build (auto: the CPU passes only; cuda: “CUDA REQUIRED”). A variable that is set names a build the user means to use, so a value that is neither an executable file nor a command on PATH is a broken build when CUDA is available, also under auto:

CUDA BROKEN: GROMACS: PYRETIS_GMX_CUDA names gmx_gpu, which is not an
executable file or a command on PATH. Set PYRETIS_GMX_CUDA to a
command whose -version reports "GPU support: CUDA" (default gmx_cuda,
looked up on PATH), or set PYRETIS_TEST_DEVICE=cpu to run the CPU
passes only.

OPENMM_PLUGIN_DIR is not covered by that rule: it is OpenMM’s own variable, which can name a plugins folder without the CUDA plugin on purpose (an OpenMM installed for the CPU), so a folder that holds no CUDA plugin, or that does not exist, means that OpenMM has no CUDA build (auto: the CPU passes, the device line says why; cuda: “CUDA REQUIRED”); a CUDA plugin there that fails to load, with CUDA available, is a broken build.

A command that is found without CUDA support, and a CUDA plugin of OpenMM that fails to load, are broken builds as well. The message names the build as PYRETIS_GMX_CUDA names <path> when the variable is set, and as gmx_cuda (the default of PYRETIS_GMX_CUDA) is <path> when it is not (likewise lmp_cuda and PYRETIS_LMP_CUDA). An installed build must also pass a preflight before its first case: the engine runs a small input on the GPU with the options that make it stop when no GPU is usable (a 3-atom mdrun -nb gpu -pme gpu, a 4-atom lmp -sf gpu -pk gpu 1, a context of one particle on the CUDA platform in double precision). A broken build, or one that fails its preflight, fails the suite with exit status 1, before any case, with the message CUDA BROKEN: <engine>: <what failed>. <way on>; the CPU is never used in its place, also under auto: with CUDA available, a CUDA build must pass. The way on is one sentence: what to set or fix to give the engine a CUDA build that runs (PYRETIS_GMX_CUDA or PYRETIS_LMP_CUDA naming a command whose -version or -h reports CUDA; for OpenMM OPENMM_PLUGIN_DIR naming a plugins folder of the installed OpenMM version whose CUDA plugin loads; what stops the preflight), and “set PYRETIS_TEST_DEVICE=cpu to run the CPU passes only”.

The requirement of the engine itself comes first: a suite whose engine is missing ends before its device is selected, as SKIPPED (exit status 77) under auto and cpu, and with CUDA REQUIRED (exit status 1) under cuda, which asks for the CUDA passes; another value of PYRETIS_TEST_DEVICE ends it with exit status 2, as it ends the selection of the device.

Which cases have a CUDA pass¶

These cases have a CUDA pass:

GROMACS: test-gromacs, test-integrate/gromacs, test-integrate/gromacs2, test-repptis and test-retis;
OpenMM: test-retis and test-load-sparse;
LAMMPS: test-dump-phasepoint, test-integrate, test-modify-velocities, test-load-sparse and test-streaming.

The other cases of the suites run on the CPU only. For GROMACS they compare two runs exactly or the pinned references, which a GPU run cannot reproduce; the LAMMPS cases test-propagate and test-retis run too long for the tolerances to hold.

What each CUDA pass checks¶

GROMACS test-gromacs and test-integrate (two engines)

mdrun on a GPU does not repeat to the last digit, so a CUDA pass compares what a GPU run can give. test-gromacs runs each of its four runs over at most 50 MD steps (11 frames of 5 subcycles; the CPU pass keeps 20 frames, 95 MD steps); the order parameters agree within a mean squared error of 1e-9, and the kinetic and potential energies within 3e-5 relative at every frame. In test-integrate/gromacs and test-integrate/gromacs2 the energies of the PyRETIS engine agree with its own .edr within the mean squared error of 1e-5, and with an independent GPU mdrun within 3e-5 relative at every frame (compare_energies.py --rel-tol-per-frame).

GROMACS test-repptis

Structural checks only.

GROMACS test-retis

example_compare --checks paths,swaps, run from each variant folder: each accepted path is valid in its ensemble, and each accepted zero swap joins the frames of the paths it started from (see below).

OpenMM test-retis

The run, on the CUDA platform in double precision with deterministic forces, is compared with openmm/results-cuda-double (golden comparison, --rel-tol 1e-5 --abs-tol 2e-6; the two tolerances add up, see “The tolerances of the golden check”). The references were made on OpenMM 8.5.1 (revision f7fa0c2) with its CUDA platform on a Quadro T2000, and the pass is validated only on a build of REFERENCE_OPENMM_CUDA_BUILDS in run-all-openmm.sh (that build). On any other OpenMM build the pass is NOT VALIDATED: it runs as an execution-smoke check, and its device part is graded SMOKE. The run repeats byte for byte on the GPU of the references; on another GPU its numbers differ in the last printed digits: a Quadro K2200 (another driver, the same OpenMM build and CUDA wheels) kept every row count and every integer and text column, and differed from the references by at most 1.0e-6 in energy.txt and order.txt and 4.4e-9 in pathensemble.txt and moves.txt, which the tolerances above allow.

OpenMM test-load-sparse

The relaxed-paths check of the CPU pass, which holds on every platform.

LAMMPS test-dump-phasepoint, test-integrate and test-modify-velocities

The CPU references, compared by the per-file, per-column rules of reference_compare.py --tolerances:

dumps (*.lammpstrj): token by token as text, with the atoms in the order of their ids;
energy.txt: the step exact; the potential, kinetic and total energy and the temperature within a relative 1e-5 plus half a unit of the sixth decimal of each of the two numbers (the file prints six decimals);
the order files (order_*.txt, prev_*_order_*.txt): the step exact; the order parameter within an absolute 1e-5;
thermo.txt: the step exact; the temperature and the three energies within a relative 1e-5 plus half a unit of the sixth significant digit of each of the two numbers; the pressure within an absolute 1e-5 plus the same print term. The temperature and the three energies of thermo.txt are compared up to step 25 (the first 50 MD steps), the pressure over every step (MD 0-100 of test-integrate);
comment lines as text;
a file, or a column, that no rule names fails.

Measured on a Quadro T2000, the margins are: test-integrate energy.txt at least 10.5x; the order files at least 6.9x over 200 MD steps; the pressure of thermo.txt 11x over its 100 MD steps; its temperature 2.1x (the print resolution, also in the first 50 steps) and its three energies 2.6x to 4.5x over 100 MD steps, so these four are compared over the first 50 MD steps; test-gromacs 4.7x to 6.4x.

LAMMPS test-load-sparse

The relaxed-paths check.

LAMMPS test-streaming

The streaming engine against the relaunch engine.

The LAMMPS CUDA passes are graded by the banner of the CUDA build: a build whose version is not the reference version runs execution-smoke checks only, and with PYRETIS_REQUIRE_VALIDATION=1 the suite stops with exit status 78.

How a CUDA pass runs¶

A CUDA pass runs the case in a copy beside it, named <case>-cuda-pass, which is removed when the pass succeeds. The copy runs with PYRETIS_CASE_DEVICE=cuda, which its run.sh reads, and with PYRETIS_VALIDATION_LEDGER set to the validation ledger of the CUDA passes (below). The CPU passes run with PYRETIS_CASE_DEVICE=cpu whatever the caller exported, and the GROMACS CPU passes with GMX_MDRUN_FLAGS empty.

A pass that fails, because its case fails or because its output does not name one GPU, keeps its copy for inspection and prints where it is:

CUDA pass of <case> FAILED (exit <status>). Its files are kept in <path>.

(for an output that names no GPU, FAILED (its output does not name one GPU)). The pass is written to the device ledger as a CUDA pass of the GPU of the preflight, marked (pass failed), and the suite ends with exit status 1 whatever status the case returned: once the build has passed its preflight, a status 77 or 78 of a case is a failure, not an engine that is unavailable or unvalidated. The suite is graded FAILED for every device it has passes on, so the device summary and the recap show CUDA FAILED (and CPU FAILED) for that engine. The copy stays until the next CUDA pass of the case removes it; the LAMMPS suite, which checks that each folder with a run.sh is one of its cases, passes over such a copy in every mode.

GROMACS cases get GMX (the CUDA command) and GMX_MDRUN_FLAGS (the options that run mdrun on the GPU), which replace the placeholders GMXCOMMAND and MDRUNFLAGS of the input.
LAMMPS cases get LMP, the CUDA command with its options.
OpenMM cases stage a copy of the module and of the input for the CUDA platform: python -m pyretis.testing.devices stage-openmm rewrites the two lines PLATFORM and PLATFORM_PROPERTIES of the module (to 'CUDA' and {'Precision': 'double', 'DeterministicForces': 'true'}) and the input that names it.

Reporting and grading¶

PYRETIS_DEVICE_LEDGER names a file that receives one line per pass:

<engine> | <case> | CUDA | <GPU> | driver <version> | <build>
<engine> | <case> | CUDA | <GPU> (pass failed) | driver <version> | <build>
<engine> | <case> | CPU | <reason>

The GPU of a LAMMPS or an OpenMM pass is read from the output of the pass itself (the Device lines of LAMMPS, the OpenMM platform: line of pyretis.log). A GROMACS pass keeps no md.log, and its GPU is read from the md.log of the preflight run.

examples/run-tests.sh gives each suite a validation ledger for its CPU passes and one for its CUDA passes (PYRETIS_CUDA_VALIDATION_LEDGER), and grades the two apart. A suite with CUDA passes that ends with exit status 0, or 78 with a line in a ledger, prints one grade line per device, for example:

VALIDATED | OpenMM example tests (CPU: 2)
SMOKE | OpenMM example tests (CUDA: 2) (not numerically validated)

and the totals count these lines. Any other ending of the suite gives one line, with both counts. The ENGINE DEVICE SUMMARY at the end of the run gives the grade of each device, for example 2 cases on CUDA (SMOKE), 2 on CPU (VALIDATED).

test-heavy.sh sets PYRETIS_DEVICE_GRADES, which receives the grades, and prints them in its recap:

Engine devices         : GROMACS CUDA VALIDATED, CPU VALIDATED; OpenMM CUDA SMOKE, CPU VALIDATED; LAMMPS CPU VALIDATED (no CUDA build: lmp_cuda not found)

and, on a machine where CUDA is not available:

Engine devices         : GROMACS CPU VALIDATED (CUDA not available: nvidia-smi not found); OpenMM CPU VALIDATED (CUDA not available: nvidia-smi not found); LAMMPS CPU VALIDATED (CUDA not available: nvidia-smi not found)

PYRETIS_REQUIRE_VALIDATION=1 fails the driver on any device part graded SMOKE. With PYRETIS_REQUIRE_VALIDATION=1 and a CUDA build of OpenMM that is not a build of REFERENCE_OPENMM_CUDA_BUILDS, whose CUDA passes compare nothing, the OpenMM suite ends with exit status 78. On a build of REFERENCE_OPENMM_CUDA_BUILDS the CUDA pass of test-retis compares with test-retis/openmm/results-cuda-double, and fails when that folder does not exist, as a CPU pass fails without its references.

The tolerances of the golden check¶

python -m pyretis.testing.example_compare compare --abs-tol X sets the absolute tolerance of the golden check (--checks golden; the default is 0); --rel-tol sets the relative one (the default is 1e-5). Both take a finite number of 0 or more: a negative, NaN or infinite value is refused before the comparison, with exit status 2 and a message such as argument --abs-tol: 'nan' is not a tolerance (a finite number of 0 or more). The per-frame relative tolerance --rel-tol-per-frame of test-integrate/compare_energies.py takes the same values.

An energy.txt compared with --energy-skip and a traj.txt take neither tolerance, whatever --abs-tol and --rel-tol say: an energy.txt under --energy-skip compares with the default tolerances of numpy.allclose (relative 1e-5, absolute 1e-8), its step exactly, and traj.txt compares byte for byte. Where both tolerances apply they add up, and a number agrees with its reference when the two differ by at most the sum:

in order.txt and energy.txt: abs_tol + rel_tol * |reference| (numpy.allclose, whose floor of 1e-8 on abs_tol these two keep);
in pathensemble.txt and moves.txt: abs_tol + rel_tol * max(|a|, |b|), the larger magnitude of the two numbers; with abs_tol 0 this is math.isclose.

The integer columns compare exactly, whatever the tolerances: the step of each frame of order.txt and energy.txt (also with the default tolerances and under --energy-skip), and the counters of pathensemble.txt and moves.txt.

In the per-frame relative comparison of the GROMACS CUDA passes (the energies of test-gromacs and of test-integrate), a NaN or an infinity in either of the two series, and a NaN tolerance, fail the comparison.

Checks of the zero swaps¶

--checks swaps examines the zero swaps of a RETIS run of the scheduler. Each accepted swap (a s+ in \([0^-]\) and a s- in \([0^+]\) of moves.txt, in the same cycle) must join the frames of the paths held when it started: the rows of the cycle before, or the cycle-0 paths for cycle 1, with the order parameters from accepted/<pn> and archive/<pn>. The first two frames of the new \([0^+]\) path equal the last two of the old \([0^-]\) path, and the last two of the new \([0^-]\) path equal the first two frames of a \([0^+]\) path that was held, exactly. For a swap of cycle 1 the output does not record which initial path \([0^+]\) held, so the check takes any cycle-0 path of an ensemble other than \([0^-]\). The check fails when it examines no swap, for a swap recorded in one ensemble only, and for a run with more than one worker.

The CUDA tutorials¶

examples/run-tutorials.sh runs the CUDA variants of three tutorials: gromacs-hydrate-retis-cuda, lammps-wca-retis-cuda and openmm-argon-retis-cuda. A variant checks the optional commands and files of the case first, and a missing one skips the case as it skips its CPU twin. When CUDA is not available, without a CUDA build of the engine, or with PYRETIS_TEST_DEVICE=cpu, a variant is a device skip, with its reason (SKIP <case>: CUDA not available: <why>): it is listed apart (“CUDA cases skipped for the device”) and PYRETIS_TUTORIAL_STRICT does not count it. With PYRETIS_TEST_DEVICE=cuda, CUDA not available or no build fails a variant with “CUDA REQUIRED”; with CUDA available, a broken build fails it.

gromacs-hydrate-retis-cuda runs until its first mdrun run with MD steps has ended (mdrun -g gpu-mdrun/md.log, watched by python -m pyretis.testing.devices watch-mdrun) and is then stopped. Its log must map the run to a GPU: “Device: GROMACS gromacs-hydrate-retis-cuda ran an MD run of mdrun on CUDA, <GPU>.” run_case takes the condition under which a command stops on its own as its seventh argument, OWN_STOP; the command then runs without the PYRETIS_TUTORIAL_TIMEOUT stop. lammps-wca-retis-cuda and openmm-argon-retis-cuda run PYRETIS_TUTORIAL_STEPS cycles when it is set (test-heavy.sh sets 20), are analysed, and check their device line. run_case takes that check as its eighth argument. With PYRETIS_TUTORIAL_TIMEOUT and without PYRETIS_TUTORIAL_STEPS, such a variant that the timeout stops passes only when its output so far names exactly one GPU, and otherwise fails with “stopped at <timeout> before its output named exactly one GPU (set PYRETIS_TUTORIAL_STEPS or a longer PYRETIS_TUTORIAL_TIMEOUT)”.