Skip to content

Update Documentation for Suite Directories #850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 12 commits into
base: develop
Choose a base branch
from
22 changes: 12 additions & 10 deletions docs/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ In addition to host config files, you can provide mode config files that
you can apply to any test when you run it. They have the same format as
the host configs, but multiple can be provided per test.

Unlike host configs, mode configs apply _last_ in the process, overriding
values set be host configs and the test itself.
Unlike host configs, mode configs apply *last* in the process, overriding
values set by host configs and the test itself.

For example, the following mode file could be used to set a particular
set of slurm vars:
Expand All @@ -38,7 +38,7 @@ Test Series
A test series is a well defined group of tests that are designated to be
run together. A series is created automatically whenever you use ``pav run``,
but you can also explicitly define series using a
``configs/series/<series_name>.yaml`` file. This allows you defined
``configs/series/<series_name>.yaml`` file. This allows you to define
relationships and dependencies between the tests, among other things.

See :ref:`tutorials.series`.
Expand All @@ -57,13 +57,15 @@ Variables
Test configs can contain *expressions* within their config values that
reference and manipulate variables.

These variables come from a variety of sources (this is also the
resolution order):
These variables come from a variety of sources, each of which is associated
with a particular *variable category*, shown in parentheses:

- The test config's variables section (var)
- System Plugins (sys)
- Pavilion hardcoded variables (pav)
- The selected scheduler (sched)
1. The test config's variables section (`var`)
2. System Plugins (`sys`)
3. Pavilion hardcoded variables (`pav`)
4. The selected scheduler (`sched`)

The above ordering is also the resolution order for variables.

Variable names must be in lowercase and start with a letter, but may
contain number and underscores.
Expand All @@ -86,7 +88,7 @@ contain number and underscores.
- Use double curly brackets ``{{var.myvar}}``.
- Variable category is optional. ``{{myvar}}`` is fine.
- Name conflicts are resolved in the order of categories listed above.
- In fact, it's recommended to not use the category component unless
- In fact, it's recommended not to use the category component unless
you need to make the reference explicit.
- You'll also see ``{{myvar.2}}`` list references, ``{{myvar.foo}}``
attribute references, and the combination of the two
Expand Down
19 changes: 13 additions & 6 deletions docs/basics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,7 @@ Pavilion doesn't come with any tests itself (though it does come with
example tests in ``examples/``); it's just a system for
running them on HPC clusters. Each test needs a configuration script,
and most will need some source files. Both of these will live in one of
your :ref:`config.config_dirs` under the ``tests/`` and ``test_src/``
sub-directories.
your :ref:`config.config_dirs` under the ``suites/`` sub-directory.

Test configs tell pavilion what environment it needs to build and run
your test, the commands to run it, how to schedule it on a cluster, and
Expand Down Expand Up @@ -144,6 +143,11 @@ current host, which is provided via a built-in
providing your own sys\_var plugin, typically to use more colloquial names for
systems.

.. note::
It is also possible to specify host configs on a per-suite basis using a slightly
different format. For more information on suite-specific host configs, see the
section on :ref:`tests.suites`.

Running tests
~~~~~~~~~~~~~

Expand All @@ -165,7 +169,7 @@ suite). Did you forget what you named them? That's ok! Just ask Pavilion.
1 tests started as test series s33.

If you want to run every test in the suite, you can just give the suite
name. You can also run whatever combinations of tests you want. You also
name. You can also run whatever combinations of tests you want. You can also
list tests in a file and have Pavilion read that.

.. code:: bash
Expand All @@ -183,7 +187,7 @@ Test Status
^^^^^^^^^^^

If you want to know what's going on with your tests, just use the
``pav status`` command.
``pav status`` command.

.. code:: bash

Expand All @@ -201,7 +205,7 @@ Test Series and ID's
~~~~~~~~~~~~~~~~~~~~

From the above, you may have noticed that each test gets a series id
like ``s24`` and a test id like ``41``. You can use these id's to
like ``s24`` and a test id like ``41``. You can use these ID's to
reference tests or suites of tests to get their status, results, and
logs through the pavilion interface. The ID's are unique for a given
Pavilion :ref:`config.working_dir` but they will
Expand All @@ -214,6 +218,9 @@ Pavilion builds a mapping of result keys and values for every test that
runs. You can view the results of any tests using the ``pav results``
command.

runs. You can view the results of any tests using the ``pav results``
command.

.. code:: bash

$ pav results
Expand All @@ -239,7 +246,7 @@ Every test has a results object that contains a variety of useful,
automatically populated keys. Additional keys can be defined through
:ref:`result parsing and result evaluations <results.basics>`.

Results are saved alongside each test, as well being written to a
Results are saved alongside each test, as well as being written to a
central result log that is suitable for import into Splunk or other
tools.

Expand Down
4 changes: 2 additions & 2 deletions docs/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ Finding the Pavilion.yaml file
Pavilion looks for its main ``pavilion.yaml`` config in the following hierarchy
and uses the first one it finds.

- The user's `~/.pavilion` directory.
- The user's ``~/.pavilion`` directory.
- The directory given via the ``PAV_CONFIG_DIR`` environment variable.

The pavilion.yaml file can configure additional locations to look for test,
The ``pavilion.yaml`` file can configure additional locations to look for test,
mode, and host configs, as well as plugins using the ``config_dirs`` option.
The ``~/.pavilion`` directory is only searched for ``pavilion.yaml`` by
default, but searches for other configs there can be turned on in ``pavilion
Expand Down
2 changes: 1 addition & 1 deletion docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ We recommend the following directory structure for Pavilion.
The ``<root>/pavilion/config`` directory will contain all of your site
specific tests, plugins, and test source files. If you organize your tests in
separate repositories, you may check them out here, and simply link the
appropriate files into ``config/tests`` and ``config/test_src``.
appropriate files into ``config/suites``.
Alternatively, you can make the entire config directory, source tarballs and
all, its own repository.

Expand Down
30 changes: 15 additions & 15 deletions docs/test_run_lifecycle.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ You'll often run a test using a test command like this:

$ pav run mytests.last_login

In this case, we have a ``mytests.yaml`` test suite file in our tests
directory, that looks like this:
In this case, we have a ``mytests.yaml`` test suite file in our ``suites``
directory that looks like this:

.. code-block:: yaml

Expand All @@ -37,7 +37,7 @@ directory, that looks like this:
# Five hours in seconds
time_limit: '{{ 60^2 * hours }}'

# Always skip gary.
# Always skip jerry.
not_if:
'{{user}}': 'jerry'

Expand Down Expand Up @@ -109,7 +109,7 @@ Finding Suite, Host, and Mode Configs

Pavilion can be configured to look in multiple places for test configs
(see :ref:`config.config_dirs`), and uses the first matching file found. In
case, we need a suite config named ``tests/mytests.yaml`` and a host config
our case, we need a suite config named ``suites/mytests.yaml`` and a host config
named ``hosts/tester.yaml``. Pavilion will load the yaml from each of these
files, and use it to construct our raw test config.

Expand Down Expand Up @@ -239,7 +239,7 @@ permutations, and generates test run objects and directories.
1. The available variable values are collected for each of the four variable
types and put in a single *variable manager* for each test
(:ref:`tests.variables`).
2. These, along with the ``permute_on`` value for a test, is used to compute
2. These, along with the ``permute_on`` value for a test, are used to compute
a unique collection of variable values for each
:ref:`Test Permutation<tests.permutations>`. Each of these will result
in a separate *Test Run*.
Expand Down Expand Up @@ -301,7 +301,7 @@ Deferred Variable Errors
########################

You may also see errors involving :ref:`tests.variables.deferred`. Some
sections, like 'build' and scheduler configuration sections, don't allow them.
sections, like ``build`` and ``scheduler`` configuration sections, don't allow them.

.. code-block:: yaml

Expand Down Expand Up @@ -329,8 +329,8 @@ that object and run the test.
:alt: Creating a Test Run

1. The *Test Run* object is created from the config, which immediately grabs
the next available test_id number. The test run directory is then created
in a directory named for that number under `<working_dir>/test_runs/`.
the next available test ID number. The test run directory is then created
in a directory named for that number under ``<working_dir>/test_runs/``.
2. Everything needed to create the test run object is saved to the test's run
directory, including the config, test variables, and any other attributes
of the test.
Expand Down Expand Up @@ -365,7 +365,7 @@ The following steps will be taken:

1) A ``kickoff`` script will be generated by the scheduler for each test run.

1) The kickoff script will run `pav _run <test_run_id>` and set up
1) The kickoff script will run ``pav _run <test_run_id>`` and set up
the basic Pavilion environment.
#) The extension of the kickoff script is scheduler dependent.
2) The scheduler plugin will run the kickoff script such that
Expand Down Expand Up @@ -398,9 +398,9 @@ Generating the Run Script
Run scripts are generated mostly identically to build scripts, and consist
of the same basic components.

1) Manage modules as described in the 'run.modules' options.
2) Manipulate environment variables as set in the 'run.env' options.
3) Run all the commands in `run.cmds`.
1) Manage modules as described in the ``run.modules`` options.
2) Manipulate environment variables as set in the ``run.env`` options.
3) Run all the commands in ``run.cmds``.

For example:

Expand Down Expand Up @@ -441,7 +441,7 @@ At this point, Pavilion simply runs the test's ``run.sh`` script. Like with
building, Pavilion will only timeout a test if it doesn't produce output at
least once every ``run.timeout`` seconds.

The return value from the ``run.sh`` script is saved in the 'return_value'
The return value from the ``run.sh`` script is saved in the ``return_value``
result, and may be used later when we gather the test results.

If this step seems overly simple, it is! Most of the work running Pavilion
Expand All @@ -455,6 +455,6 @@ This is described fully in :ref:`results.basics`.
Mark the Test as Complete
~~~~~~~~~~~~~~~~~~~~~~~~~

Finally, the test is marked as complete by saving a 'RUN_COMPLETE' file in the
Finally, the test is marked as complete by saving a ``RUN_COMPLETE`` file in the
test's run directory. Pavilion uses this to quickly determine which tests
might still be running.
might still be running.
10 changes: 7 additions & 3 deletions docs/tests/build.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,13 @@ This build option tells Pavilion when to download test source.
Finding Source Files
~~~~~~~~~~~~~~~~~~~~

The primary way to specify source in Pavilion is with the ``source_path``
build option. Additionally you can bring additional files into your build
directory with the ``extra files`` option.
If a suite is organized as directory, all references to files within a test are interpreted as
relative to the suite directory. If your tests requires source code or other files, this is the
recommended method of organizing your test suite.

It is also possible to use source code outside of the suite directory using the ``source_path``
build option. Additionally you can bring additional files into your build directory with the
``extra files`` option.

.. code:: yaml

Expand Down
Loading