.. _developer-engine-device-tier: ############################################ The CPU and CUDA passes of the engine suites ############################################ The engine suites of :file:`examples/tests/` (:file:`test-gromacs/`, :file:`test-openmm/` and :file:`test-lammps/`) run every case on the CPU. A case with a CUDA pass also runs on an NVIDIA GPU when CUDA is available and the CUDA build of its engine is installed. This page lists the switch that selects the passes, when CUDA is available, what counts as an installed CUDA build, which cases have a CUDA pass and what each one checks, and how the passes are reported and graded. The suites are described in :file:`examples/tests/README.rst`, and :ref:`example-test-status` gives the status of each tutorial. The device switch ================= ``PYRETIS_TEST_DEVICE`` selects the passes of every engine suite: ``auto`` (default) The CPU passes, and the CUDA passes of each engine with a CUDA build when CUDA is available. When CUDA is not available, the CPU passes only, whatever CUDA builds are installed. ``cpu`` The CPU passes only. ``cuda`` As ``auto``; CUDA not available, or an engine without a CUDA build, fails the suite of the engine, with exit status 1 and before any case, with "CUDA REQUIRED" (the messages are given below). Any other value stops the suite with exit status 2. When CUDA is not available ========================== CUDA is available on a machine when ``nvidia-smi`` lists a GPU and ``CUDA_VISIBLE_DEVICES`` leaves one to the CUDA programs. Every suite checks this first, under ``auto`` and ``cuda``, before it looks for the CUDA build of its engine. When ``CUDA_VISIBLE_DEVICES`` is empty, its first entry is neither a device index nor a GPU or MIG UUID (``-1``, for example) or an index beyond the GPUs ``nvidia-smi`` lists, or ``nvidia-smi`` is not found, cannot be run, fails, does not end within 300 s, or lists no GPU, CUDA is not available: under ``auto`` the suite runs its CPU passes only, whatever CUDA builds are installed (a CUDA build that is found, an OpenMM CUDA plugin that fails to load, as it does without the NVIDIA driver), and says why in its output, in the device ledger and in the recap, for example:: Device: gromacs CPU passes only (CUDA not available: nvidia-smi ended with status 9: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.) The CUDA passes do not run and are not counted: the grades of the suite are those of its CPU passes. Under ``cuda`` each engine suite fails before any case:: CUDA REQUIRED: GROMACS: CUDA is not available (nvidia-smi not found). Install or repair the NVIDIA driver so that nvidia-smi lists the GPU, or set PYRETIS_TEST_DEVICE to auto or cpu to run its CPU passes only. When CUDA is available, the rest of this page applies. An engine without a CUDA build fails a ``cuda`` suite with a message that ends with the way on, for example for GROMACS:: CUDA REQUIRED: GROMACS has no CUDA build (gmx_cuda not found). Set PYRETIS_GMX_CUDA to a command whose -version reports "GPU support: CUDA" (default gmx_cuda, looked up on PATH), or set PYRETIS_TEST_DEVICE to auto or cpu to run its CPU passes only. The message of LAMMPS names ``PYRETIS_LMP_CUDA``, ``-h``, ``GPU package API: CUDA`` and ``lmp_cuda``. That of OpenMM names the platforms it loaded and the variable ``OPENMM_PLUGIN_DIR``, for example:: CUDA REQUIRED: OpenMM has no CUDA platform (platforms: Reference, CPU, OpenCL). Set OPENMM_PLUGIN_DIR to a plugins folder of version 8.5.1 whose CUDA plugin libOpenMMCUDA.so loads, or set PYRETIS_TEST_DEVICE to auto or cpu to run its CPU passes only. The version is that of the installed OpenMM. Under ``auto`` such an engine runs its CPU passes only. What "installed" means ====================== The CUDA build of an engine is installed when * GROMACS: the command that ``PYRETIS_GMX_CUDA`` names (default ``gmx_cuda``) is found and its ``-version`` reports ``GPU support: CUDA``; * LAMMPS: the command that ``PYRETIS_LMP_CUDA`` names (default ``lmp_cuda``) is found and its ``-h`` reports ``GPU package API: CUDA``; * OpenMM: its ``CUDA`` platform is loaded. A private folder of OpenMM plugins is named with ``OPENMM_PLUGIN_DIR``, which OpenMM reads: it then loads its plugins from that folder in place of the plugins folder of its installation (an empty folder leaves the ``Reference`` platform only), and the ``CUDA`` plugin loads only when the CUDA libraries it needs are found. ``PYRETIS_GMX_CUDA`` and ``PYRETIS_LMP_CUDA`` name the command alone, without options and with no space in its path, as an executable file or as a command on ``PATH``. A relative path is taken from the folder of the suite (:file:`test-gromacs/`, :file:`test-lammps/`), and, for the CUDA tutorials, from the folder that :file:`examples/run-tutorials.sh` is started in. Only the default name (the variable unset or empty) that is not found means that the engine has no CUDA build (``auto``: the CPU passes only; ``cuda``: "CUDA REQUIRED"). A variable that is set names a build the user means to use, so a value that is neither an executable file nor a command on ``PATH`` is a broken build when CUDA is available, also under ``auto``:: CUDA BROKEN: GROMACS: PYRETIS_GMX_CUDA names gmx_gpu, which is not an executable file or a command on PATH. Set PYRETIS_GMX_CUDA to a command whose -version reports "GPU support: CUDA" (default gmx_cuda, looked up on PATH), or set PYRETIS_TEST_DEVICE=cpu to run the CPU passes only. ``OPENMM_PLUGIN_DIR`` is not covered by that rule: it is OpenMM's own variable, which can name a plugins folder without the ``CUDA`` plugin on purpose (an OpenMM installed for the CPU), so a folder that holds no ``CUDA`` plugin, or that does not exist, means that OpenMM has no CUDA build (``auto``: the CPU passes, the device line says why; ``cuda``: "CUDA REQUIRED"); a ``CUDA`` plugin there that fails to load, with CUDA available, is a broken build. A command that is found without CUDA support, and a CUDA plugin of OpenMM that fails to load, are broken builds as well. The message names the build as ``PYRETIS_GMX_CUDA names `` when the variable is set, and as ``gmx_cuda (the default of PYRETIS_GMX_CUDA) is `` when it is not (likewise ``lmp_cuda`` and ``PYRETIS_LMP_CUDA``). An installed build must also pass a preflight before its first case: the engine runs a small input on the GPU with the options that make it stop when no GPU is usable (a 3-atom ``mdrun -nb gpu -pme gpu``, a 4-atom ``lmp -sf gpu -pk gpu 1``, a context of one particle on the CUDA platform in double precision). A broken build, or one that fails its preflight, fails the suite with exit status 1, before any case, with the message ``CUDA BROKEN: : . ``; the CPU is never used in its place, also under ``auto``: with CUDA available, a CUDA build must pass. The way on is one sentence: what to set or fix to give the engine a CUDA build that runs (``PYRETIS_GMX_CUDA`` or ``PYRETIS_LMP_CUDA`` naming a command whose ``-version`` or ``-h`` reports CUDA; for OpenMM ``OPENMM_PLUGIN_DIR`` naming a plugins folder of the installed OpenMM version whose CUDA plugin loads; what stops the preflight), and "set PYRETIS_TEST_DEVICE=cpu to run the CPU passes only". The requirement of the engine itself comes first: a suite whose engine is missing ends before its device is selected, as SKIPPED (exit status 77) under ``auto`` and ``cpu``, and with ``CUDA REQUIRED`` (exit status 1) under ``cuda``, which asks for the CUDA passes; another value of ``PYRETIS_TEST_DEVICE`` ends it with exit status 2, as it ends the selection of the device. Which cases have a CUDA pass ============================ These cases have a CUDA pass: * GROMACS: ``test-gromacs``, ``test-integrate/gromacs``, ``test-integrate/gromacs2``, ``test-repptis`` and ``test-retis``; * OpenMM: ``test-retis`` and ``test-load-sparse``; * LAMMPS: ``test-dump-phasepoint``, ``test-integrate``, ``test-modify-velocities``, ``test-load-sparse`` and ``test-streaming``. The other cases of the suites run on the CPU only. For GROMACS they compare two runs exactly or the pinned references, which a GPU run cannot reproduce; the LAMMPS cases ``test-propagate`` and ``test-retis`` run too long for the tolerances to hold. What each CUDA pass checks ========================== **GROMACS** ``test-gromacs`` and ``test-integrate`` (two engines) ``mdrun`` on a GPU does not repeat to the last digit, so a CUDA pass compares what a GPU run can give. ``test-gromacs`` runs each of its four runs over at most 50 MD steps (11 frames of 5 subcycles; the CPU pass keeps 20 frames, 95 MD steps); the order parameters agree within a mean squared error of 1e-9, and the kinetic and potential energies within 3e-5 relative at every frame. In ``test-integrate/gromacs`` and ``test-integrate/gromacs2`` the energies of the PyRETIS engine agree with its own ``.edr`` within the mean squared error of 1e-5, and with an independent GPU ``mdrun`` within 3e-5 relative at every frame (``compare_energies.py --rel-tol-per-frame``). **GROMACS** ``test-repptis`` Structural checks only. **GROMACS** ``test-retis`` ``example_compare --checks paths,swaps``, run from each variant folder: each accepted path is valid in its ensemble, and each accepted zero swap joins the frames of the paths it started from (see below). **OpenMM** ``test-retis`` The run, on the CUDA platform in double precision with deterministic forces, is compared with ``openmm/results-cuda-double`` (golden comparison, ``--rel-tol 1e-5 --abs-tol 2e-6``; the two tolerances add up, see "The tolerances of the golden check"). The references were made on OpenMM 8.5.1 (revision f7fa0c2) with its CUDA platform on a Quadro T2000, and the pass is validated only on a build of ``REFERENCE_OPENMM_CUDA_BUILDS`` in :file:`run-all-openmm.sh` (that build). On any other OpenMM build the pass is NOT VALIDATED: it runs as an execution-smoke check, and its device part is graded SMOKE. The run repeats byte for byte on the GPU of the references; on another GPU its numbers differ in the last printed digits: a Quadro K2200 (another driver, the same OpenMM build and CUDA wheels) kept every row count and every integer and text column, and differed from the references by at most 1.0e-6 in ``energy.txt`` and ``order.txt`` and 4.4e-9 in ``pathensemble.txt`` and ``moves.txt``, which the tolerances above allow. **OpenMM** ``test-load-sparse`` The relaxed-paths check of the CPU pass, which holds on every platform. **LAMMPS** ``test-dump-phasepoint``, ``test-integrate`` and ``test-modify-velocities`` The CPU references, compared by the per-file, per-column rules of ``reference_compare.py --tolerances``: * dumps (``*.lammpstrj``): token by token as text, with the atoms in the order of their ids; * ``energy.txt``: the step exact; the potential, kinetic and total energy and the temperature within a relative 1e-5 plus half a unit of the sixth decimal of each of the two numbers (the file prints six decimals); * the order files (``order_*.txt``, ``prev_*_order_*.txt``): the step exact; the order parameter within an absolute 1e-5; * ``thermo.txt``: the step exact; the temperature and the three energies within a relative 1e-5 plus half a unit of the sixth significant digit of each of the two numbers; the pressure within an absolute 1e-5 plus the same print term. The temperature and the three energies of ``thermo.txt`` are compared up to step 25 (the first 50 MD steps), the pressure over every step (MD 0-100 of ``test-integrate``); * comment lines as text; * a file, or a column, that no rule names fails. Measured on a Quadro T2000, the margins are: ``test-integrate`` ``energy.txt`` at least 10.5x; the order files at least 6.9x over 200 MD steps; the pressure of ``thermo.txt`` 11x over its 100 MD steps; its temperature 2.1x (the print resolution, also in the first 50 steps) and its three energies 2.6x to 4.5x over 100 MD steps, so these four are compared over the first 50 MD steps; ``test-gromacs`` 4.7x to 6.4x. **LAMMPS** ``test-load-sparse`` The relaxed-paths check. **LAMMPS** ``test-streaming`` The streaming engine against the relaunch engine. The LAMMPS CUDA passes are graded by the banner of the CUDA build: a build whose version is not the reference version runs execution-smoke checks only, and with ``PYRETIS_REQUIRE_VALIDATION=1`` the suite stops with exit status 78. How a CUDA pass runs ==================== A CUDA pass runs the case in a copy beside it, named ``-cuda-pass``, which is removed when the pass succeeds. The copy runs with ``PYRETIS_CASE_DEVICE=cuda``, which its :file:`run.sh` reads, and with ``PYRETIS_VALIDATION_LEDGER`` set to the validation ledger of the CUDA passes (below). The CPU passes run with ``PYRETIS_CASE_DEVICE=cpu`` whatever the caller exported, and the GROMACS CPU passes with ``GMX_MDRUN_FLAGS`` empty. A pass that fails, because its case fails or because its output does not name one GPU, keeps its copy for inspection and prints where it is:: CUDA pass of FAILED (exit ). Its files are kept in . (for an output that names no GPU, ``FAILED (its output does not name one GPU)``). The pass is written to the device ledger as a CUDA pass of the GPU of the preflight, marked ``(pass failed)``, and the suite ends with exit status 1 whatever status the case returned: once the build has passed its preflight, a status 77 or 78 of a case is a failure, not an engine that is unavailable or unvalidated. The suite is graded FAILED for every device it has passes on, so the device summary and the recap show ``CUDA FAILED`` (and ``CPU FAILED``) for that engine. The copy stays until the next CUDA pass of the case removes it; the LAMMPS suite, which checks that each folder with a :file:`run.sh` is one of its cases, passes over such a copy in every mode. * GROMACS cases get ``GMX`` (the CUDA command) and ``GMX_MDRUN_FLAGS`` (the options that run ``mdrun`` on the GPU), which replace the placeholders ``GMXCOMMAND`` and ``MDRUNFLAGS`` of the input. * LAMMPS cases get ``LMP``, the CUDA command with its options. * OpenMM cases stage a copy of the module and of the input for the CUDA platform: ``python -m pyretis.testing.devices stage-openmm`` rewrites the two lines ``PLATFORM`` and ``PLATFORM_PROPERTIES`` of the module (to ``'CUDA'`` and ``{'Precision': 'double', 'DeterministicForces': 'true'}``) and the input that names it. Reporting and grading ===================== ``PYRETIS_DEVICE_LEDGER`` names a file that receives one line per pass:: | | CUDA | | driver | | | CUDA | (pass failed) | driver | | | CPU | The GPU of a LAMMPS or an OpenMM pass is read from the output of the pass itself (the ``Device`` lines of LAMMPS, the ``OpenMM platform:`` line of ``pyretis.log``). A GROMACS pass keeps no ``md.log``, and its GPU is read from the ``md.log`` of the preflight run. :file:`examples/run-tests.sh` gives each suite a validation ledger for its CPU passes and one for its CUDA passes (``PYRETIS_CUDA_VALIDATION_LEDGER``), and grades the two apart. A suite with CUDA passes that ends with exit status 0, or 78 with a line in a ledger, prints one grade line per device, for example:: VALIDATED | OpenMM example tests (CPU: 2) SMOKE | OpenMM example tests (CUDA: 2) (not numerically validated) and the totals count these lines. Any other ending of the suite gives one line, with both counts. The ``ENGINE DEVICE SUMMARY`` at the end of the run gives the grade of each device, for example ``2 cases on CUDA (SMOKE), 2 on CPU (VALIDATED)``. ``test-heavy.sh`` sets ``PYRETIS_DEVICE_GRADES``, which receives the grades, and prints them in its recap:: Engine devices : GROMACS CUDA VALIDATED, CPU VALIDATED; OpenMM CUDA SMOKE, CPU VALIDATED; LAMMPS CPU VALIDATED (no CUDA build: lmp_cuda not found) and, on a machine where CUDA is not available:: Engine devices : GROMACS CPU VALIDATED (CUDA not available: nvidia-smi not found); OpenMM CPU VALIDATED (CUDA not available: nvidia-smi not found); LAMMPS CPU VALIDATED (CUDA not available: nvidia-smi not found) ``PYRETIS_REQUIRE_VALIDATION=1`` fails the driver on any device part graded SMOKE. With ``PYRETIS_REQUIRE_VALIDATION=1`` and a CUDA build of OpenMM that is not a build of ``REFERENCE_OPENMM_CUDA_BUILDS``, whose CUDA passes compare nothing, the OpenMM suite ends with exit status 78. On a build of ``REFERENCE_OPENMM_CUDA_BUILDS`` the CUDA pass of ``test-retis`` compares with ``test-retis/openmm/results-cuda-double``, and fails when that folder does not exist, as a CPU pass fails without its references. The tolerances of the golden check ================================== ``python -m pyretis.testing.example_compare compare --abs-tol X`` sets the absolute tolerance of the golden check (``--checks golden``; the default is 0); ``--rel-tol`` sets the relative one (the default is 1e-5). Both take a finite number of 0 or more: a negative, NaN or infinite value is refused before the comparison, with exit status 2 and a message such as ``argument --abs-tol: 'nan' is not a tolerance (a finite number of 0 or more)``. The per-frame relative tolerance ``--rel-tol-per-frame`` of :file:`test-integrate/compare_energies.py` takes the same values. An ``energy.txt`` compared with ``--energy-skip`` and a ``traj.txt`` take neither tolerance, whatever ``--abs-tol`` and ``--rel-tol`` say: an ``energy.txt`` under ``--energy-skip`` compares with the default tolerances of ``numpy.allclose`` (relative 1e-5, absolute 1e-8), its step exactly, and ``traj.txt`` compares byte for byte. Where both tolerances apply they add up, and a number agrees with its reference when the two differ by at most the sum: * in ``order.txt`` and ``energy.txt``: ``abs_tol + rel_tol * |reference|`` (``numpy.allclose``, whose floor of 1e-8 on ``abs_tol`` these two keep); * in ``pathensemble.txt`` and ``moves.txt``: ``abs_tol + rel_tol * max(|a|, |b|)``, the larger magnitude of the two numbers; with ``abs_tol`` 0 this is ``math.isclose``. The integer columns compare exactly, whatever the tolerances: the step of each frame of ``order.txt`` and ``energy.txt`` (also with the default tolerances and under ``--energy-skip``), and the counters of ``pathensemble.txt`` and ``moves.txt``. In the per-frame relative comparison of the GROMACS CUDA passes (the energies of ``test-gromacs`` and of ``test-integrate``), a NaN or an infinity in either of the two series, and a NaN tolerance, fail the comparison. Checks of the zero swaps ======================== ``--checks swaps`` examines the zero swaps of a RETIS run of the scheduler. Each accepted swap (a ``s+`` in :math:`[0^-]` and a ``s-`` in :math:`[0^+]` of ``moves.txt``, in the same cycle) must join the frames of the paths held when it started: the rows of the cycle before, or the cycle-0 paths for cycle 1, with the order parameters from ``accepted/`` and ``archive/``. The first two frames of the new :math:`[0^+]` path equal the last two of the old :math:`[0^-]` path, and the last two of the new :math:`[0^-]` path equal the first two frames of a :math:`[0^+]` path that was held, exactly. For a swap of cycle 1 the output does not record which initial path :math:`[0^+]` held, so the check takes any cycle-0 path of an ensemble other than :math:`[0^-]`. The check fails when it examines no swap, for a swap recorded in one ensemble only, and for a run with more than one worker. The CUDA tutorials ================== :file:`examples/run-tutorials.sh` runs the CUDA variants of three tutorials: ``gromacs-hydrate-retis-cuda``, ``lammps-wca-retis-cuda`` and ``openmm-argon-retis-cuda``. A variant checks the optional commands and files of the case first, and a missing one skips the case as it skips its CPU twin. When CUDA is not available, without a CUDA build of the engine, or with ``PYRETIS_TEST_DEVICE=cpu``, a variant is a device skip, with its reason (``SKIP : CUDA not available: ``): it is listed apart ("CUDA cases skipped for the device") and ``PYRETIS_TUTORIAL_STRICT`` does not count it. With ``PYRETIS_TEST_DEVICE=cuda``, CUDA not available or no build fails a variant with "CUDA REQUIRED"; with CUDA available, a broken build fails it. ``gromacs-hydrate-retis-cuda`` runs until its first ``mdrun`` run with MD steps has ended (``mdrun -g gpu-mdrun/md.log``, watched by ``python -m pyretis.testing.devices watch-mdrun``) and is then stopped. Its log must map the run to a GPU: "Device: GROMACS gromacs-hydrate-retis-cuda ran an MD run of mdrun on CUDA, ." ``run_case`` takes the condition under which a command stops on its own as its seventh argument, ``OWN_STOP``; the command then runs without the ``PYRETIS_TUTORIAL_TIMEOUT`` stop. ``lammps-wca-retis-cuda`` and ``openmm-argon-retis-cuda`` run ``PYRETIS_TUTORIAL_STEPS`` cycles when it is set (:file:`test-heavy.sh` sets 20), are analysed, and check their device line. ``run_case`` takes that check as its eighth argument. With ``PYRETIS_TUTORIAL_TIMEOUT`` and without ``PYRETIS_TUTORIAL_STEPS``, such a variant that the timeout stops passes only when its output so far names exactly one GPU, and otherwise fails with "stopped at before its output named exactly one GPU (set PYRETIS_TUTORIAL_STEPS or a longer PYRETIS_TUTORIAL_TIMEOUT)".