Pass registered suite name to Jinja2 environment

[TomekTrzeciak]

In my team we tend to develop suites that get configured and deployed from a common source code into running instances. It could be useful to know the name of the suite, so that the relevant configuration could be picked automatically rather than require extra command line options or source code changes.

[oliver-sanders]

Unfortunately due to "offline commands" such as validate the registration of a workflow is not always known.

We could leave the title blank for these cases, however, that would leave you with a workflow which would fail validation until it is installed. Probably not a big issue for your use case though I would guess?

[hjoliver]

For offline (I.e. by file path not reg name) validate etc. we could set a default name - UNKNOWN or whatever - to avoid the undefined variable error, and users could deal with the consequences of that: either handle the offline case explicitly in name switches or don’t validate by file path.

[TomekTrzeciak]

Unfortunately due to "offline commands" such as validate the registration of a workflow is not always known.

We could leave the title blank for these cases, however, that would leave you with a workflow which would fail validation until it is installed. Probably not a big issue for your use case though I would guess?

Not a big issue indeed, although you could potentially pass the directory name of suite.rc file in this case, as this would be the default name for registering the suite, but leaving it blank, set to None or even undefined (this can be tested for in Jinja) would be fine too.

[TomekTrzeciak]

Come to think about it, having other suite variables available in Jinja (e.g. CYLC_SUITE_INITIAL_CYCLE_POINT) could be useful too.

[oliver-sanders]

Could just expose the CLI options object (or a dict form of it). Again there is the risk of workflow logic being developed which can only work when executed by cylc run and not the static analysis commands.

[TomekTrzeciak]

Turns out CYLC_SUITE_NAME is available via environ already at template processing time, at least on Cylc 7. Is this going to stay the same for Cylc 8?

[oliver-sanders]

Yes, this will still be available in Cylc8, we will probably add something like CYLC_FLOW_NAME to match new terminology but the old var will be supported for a while.

However, using the suite name as a programming interface is going to be fragile as you have no control over how the person who runs the flow decides to register it. For example they might decide to register their flows hierarchically (e.g. OS??/name).

With Cylc8 it will be much more likely that the registration will not match basename $PWD due to:

• Changes in the UI/CLI making hierarchical registrations more useful for organisation and control.
• Changes in the installation process making changing the registration easier.
• CLI changes designed to make it easier to control flows from arbitrary locations in the filesystem.
• The addition of (optional) numbered run directories for each workflow run.

The last point means that the name you want might not actually be the lowest level in the hierarchy.

[TomekTrzeciak]

Yes, this will still be available in Cylc8, we will probably add something like CYLC_FLOW_NAME to match new terminology but the old var will be supported for a while.

Great!

However, using the suite name as a programming interface is going to be fragile as you have no control over how the person who runs the flow decides to register it. For example they might decide to register their flows hierarchically (e.g. OS??/name).

Duly noted and I'm well aware of it.

With Cylc8 it will be much more likely that the registration will not match basename $PWD due to:

• Changes in the UI/CLI making hierarchical registrations more useful for organisation and control.
• Changes in the installation process making changing the registration easier.

Does this means that it will be possible to rename/re-register suites without re-installation? (That would be awesome.)

• CLI changes designed to make it easier to control flows from arbitrary locations in the filesystem.
• The addition of (optional) numbered run directories for each workflow run.

The last point means that the name you want might not actually be the lowest level in the hierarchy.

Thanks for listing out possible problems with this. Of course, I wouldn't go with an implementation that requires suite registration to be such and such in order to work - you would be still able to supply the configuration name independently from the suite name. It's more of a convenience feature for devs/operators. Say, you run a particular configuration by selecting an optional config rose suite-run -O xyz or setting a suite variable rose suite-run -S SUITE_CONFIG=xyz. You are likely to name the suite after the config anyway, say, rose suite-run -n xyz, so why specify that twice. If someone would try to run without a config option and using some dodgy name, they would be simply greeted with an error message.

The other use case I'm thinking of is option-less suite start (which is required by our operational tooling). Unless you rely on something like suite name, $PWD or environment setting, this cannot be done without modifying the suite source in some way to pick up the right configuration. This makes deployment of multiple suites from a single source more awkward.

The alternative solution I'm thinking of:

• At the top of rose-suite.conf put:

opts=(main)

• For XYZ suite deployment under `opt/' add a symlink:

rose-suite-main.conf -> rose-suite-XXX.conf

This does require some changes, but at least no edits to the source. And these symlinks could be made permanent, e.g. in dedicated deployment branches (at the cost of maintenance of said branches).

I'm still mulling over my options here, so if you have any other ideas I'm all ears.

[oliver-sanders]

Does this means that it will be possible to rename/re-register suites without re-installation? (That would be awesome.)

Not what I was getting at, but if you mv a run dir you change its reg, Cylc8 is ok with that (so long as the flow is stopped at the time). It's not officially supported or tested functionality and you do need to remember to move any run dirs on remote hosts. (We would need to ensure all paths/symlinks are relative were we to support this properly).

What I meant is that a run dir is aware of its installation source now that cylc is in charge of the installation process (i.e. you wont necessarily have to specify the rose -n option repeatably if you decide to change the the reg in the install process).

The other use case I'm thinking of is option-less suite start

I wonder if the install-run separation at Cylc8 will help here. The convolution of rose suite-run has caused a few problems. I see it as meaning rose suite-install --run.

In Cylc8 Rose optional configs will be applied by cylc install, so any subsequent cylc run or cylc restart commands will pick up previously specified configs:

# installing multiple flows with different optional configs from a single source directory
# roughly equivalent to rose suite-run -l
cylc install --flow-name abc -O 123 -O x
cylc install --flow-name bcd -O 234 -O y

Then operate on them without specifying the opt conf key again for the lifetime of the run:

# opt conf keys are recorded during installation and do not need to be specified again
cylc run abc
cylc stop abc
cylc restart abc
cylc run --rerun abc

An outstanding question I'm not quite sure of the answer to is what should happen with re-installation.

# equivalent to rose suite-run --reload
cylc reinstall abc
cylc reload abc

Should we continue with the previously specified opt conf keys, that would certainly be nice, especially for this use case. Though presumably we should permit the opt conf keys to change during re-installation.

cylc reinstall --no-opt-conf-keys -O abc  # ?

[oliver-sanders]

Update:

We think reinstall should preserve optional configs from previous installations, however, will look to provide additional options to remove specified optional configs or rebuild from scratch.

Added to the Rose plugin issue - #3819 (comment)

[dwsutherland]

Turns out CYLC_SUITE_NAME is available via environ already at template processing time, at least on Cylc 7. Is this going to stay the same for Cylc 8?

Yes, I remember getting this change in a couple of years ago, as I wanted it for NIWA operations... And use it in Jinja2, i.e.

Jinja2Filters/exclude_current_suite.py:       current_suite = os.environ['CYLC_SUITE_NAME']

(this is for a suite that monitors our suites)
(of course environ is available directly in Jinja2 also)

[TomekTrzeciak]

For a while now we've been using the following snippet in our single-source multiple-suites set-up:

{% if SUITE_NAME is undefined %}
    {% set SUITE_NAME = environ['CYLC_SUITE_NAME'].split('/')[-1].split('_')[0] %}
{% endif %}

where SUITE_NAME defines the configuration to run. With all the caveats @oliver-sanders listed above, this has proved to work quite well for us so far.

Moving to Cylc 8 in the future we could possibly use --run-name for that purpose instead (depends a bit on what convention our operations are willing to adopt). I would like to suggest that CYLC_SUITE_RUN_NAME is added to the suite environment. This should enable reliable deconstruction of the suite name into its constituent components and it might be useful for other things too (e.g., run-specific custom messages).