Skip to content

Commit 788cf42

Browse files
committed
evolve nv templating
1 parent 9c17454 commit 788cf42

5 files changed

Lines changed: 144 additions & 56 deletions

File tree

docs/TEMPLATES.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Templates
2+
CM Services uses [jinja](https://jinja.palletsprojects.com/en/stable/templates/) templates when writing BPS submit YAML files and submit scripts for various operations. These templates enforce a common structure for these files while allowing some flexibility at the campaign or step level. Generally, the `data` object associated with a campaign element is used to populate the environment used to render these templates.
3+
4+
The template files used by CM Service are in the `src/lsst/cmservice/templates` directory and are part of the CM Service Python package namespace.
5+
6+
## WMS Submit Script
7+
The template `wms_submit_sh.j2` is used to render the shell script used, among other things, to call a BPS submit command along with a rendered BPS submit YAML. When the campaign's `script_method` is "htcondor", the execution environment of this script is expected to be an HTCondor interactive node.
8+
9+
This template is structured as follows with a number of named phases. Those phases indicated by square brackets (`[...]`) are "baked into" the template and will always be present. Those phases indicated by angle bracket (`<...>`) are available for campaign designers to insert or override values as appropriate by including an object with a matching name in a `data` object mapping at a step or campaign level.
10+
11+
```
12+
[shebang]
13+
<prepend>
14+
[LSST setup]
15+
<custom_lsst_setup>
16+
[WMS setup]
17+
<custom_wms_setup>
18+
[command]
19+
<append>
20+
[EOF]
21+
```
22+
23+
Designers are free to add any directives to the override blocks that are appropriate to their placement in the script (always executed top-to-bottom). Examples of directives that could be added to these phases include the following.
24+
25+
- `prepend`. Setting extra environment variables or calling commands prior to the setup of the LSST stack.
26+
- `custom_lsst_setup`. Extra `setup -j ...` commands or environment variable manipulation following the Conda environment activation and EUPS setup.
27+
- `custom_wms_setup`. Setting or changing WMS-specific environment variables.
28+
- `append`. Calling commands after the primary work of the script is complete.
29+
30+
## BPS Submit YAML
31+
The template `bps_submit_yaml.j2` is used to render the YAML file used with `bps submit` for a Step in a Campaign.
32+
33+
This template is structured with some general headings and has the potential to be quite complex. In most cases, the template assumes that the complete `data` dictionary for each Step is complete and correct.
34+
35+
The template sections are divided thusly:
36+
37+
```
38+
[Header]
39+
[LSST Setup]
40+
[BPS Variables]
41+
[Pipeline Configuration]
42+
[Submission Environment Variables]
43+
[BPS Payload Options]
44+
[Clustering Configuration]
45+
[Site WMS Configuration]
46+
```
47+
48+
### Header
49+
The header section includes metadata values such as the description and Jira link for the campaign.
50+
51+
### LSST Setup
52+
The LSST Setup section includes top-level BPS configuration related to the LSST Stack, notably the stack version used for the workflow.
53+
54+
### BPS Variables
55+
This section defines key-value pairs for BPS variable definitions, i.e., `{var_name}` that may be used throughout the workflow file.
56+
57+
### Pipeline Configuration
58+
This section defines the `pipelineYaml` used for the workflow and any `includeConfigs` consumed by the workflow.
59+
60+
This section may also include arbitrary top-level bps configuration parameters that don't otherwise fit in another section. Steps may declare these parameters with a `data.bps_literals` object.
61+
62+
### Submission Environment Variables
63+
This section populates a top-level `environments` object with key-value pairs from a `data.bps_environment` object. If no such object is defined, this section is not included.
64+
65+
### BPS Payload Options
66+
This section populates a top-level `payload` object with key-value pairs from a `data.payload` object, and defines values for BPS pass-through arguments using `extra*Options` parameters.
67+
68+
### Clustering Configuration
69+
This section includes any inline `clustering` directives based on the literal contents of a `data.cluster` object. If no such object is defined, this section is not included.
70+
71+
### Site WMS Configuration
72+
This section includes configuration specific to a WMS and a Compute Site. This includes inline resource and memory management configurations. A `data.compute_site` object is used as a literal in this section.

examples/template_LSSTCam_NV.yaml

Lines changed: 39 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,10 @@ Campaign:
5050
data_query: >-
5151
instrument='LSSTCam'
5252
AND skymap='lsst_cells_v1'
53-
AND (exposure.day_obs>=${NIGHTLY_START} and exposure.day_obs<${NIGHTLY_END})
53+
AND exposure.day_obs=${NIGHTLY_START}
5454
AND exposure.observation_type IN ('science', 'acq')
55+
AND detector NOT IN (120, 121, 122, 78, 79, 80)
56+
AND detector < 189
5557
---
5658
# Base Campaign SpecBlock for Nightly Validation
5759
- SpecBlock:
@@ -104,51 +106,58 @@ Campaign:
104106
prerequisites: ["step1d"]
105107
---
106108
# Campaign Step SpecBlocks
109+
- SpecBlock:
110+
name: nv_step
111+
includes: ["step"]
112+
data:
113+
prepend: |-
114+
export LSST_S3_USE_THREADS=False
115+
custom_lsst_setup: |-
116+
setup -j -r /sdf/data/rubin/shared/campaigns/LSSTCam-Nightly-Validation/lsst_devel/pipe_base
117+
extra_run_quantum_options: >-
118+
--no-raise-on-partial-outputs
119+
extra_qgraph_options: >-
120+
--dataset-query-constraint off
121+
--skip-existing-in LSSTCam/runs/nightlyValidation
122+
-c parameters:sasquatch_dataset_identifier=LSSTCam/nightlyValidation
123+
-c parameters:sasquatch_timestamp_version=explicit_timestamp:{SASQ_TIMESTAMP}T000000Z
124+
child_config:
125+
split_method: no_split
107126
- SpecBlock:
108127
name: step1a
109-
includes: ["nv_step", "step"]
128+
includes: ["nv_step", "step", "usdf"]
110129
data:
111130
pipeline_yaml: "${DRP_PIPE_DIR}/pipelines/LSSTCam/nightly-validation.yaml#step1a-single-visit-detectors"
131+
numberOfRetries: 2
132+
retryUnlessExit:
133+
- 2
112134
- SpecBlock:
113135
name: step1b
114-
includes: ["nv_step", "step"]
136+
includes: ["nv_step", "step", "usdf"]
115137
data:
116138
pipeline_yaml: "${DRP_PIPE_DIR}pipelines/LSSTCam/nightly-validation.yaml#step1b-single-visit-visits"
139+
bps_literals:
140+
numberOfRetries: 2
141+
retryUnlessExit:
142+
- 2
117143
- SpecBlock:
118144
name: step1c
119-
includes: ["nv_step", "step"]
145+
includes: ["nv_step", "step", "usdf"]
120146
data:
121147
pipeline_yaml: "${DRP_PIPE_DIR}pipelines/LSSTCam/nightly-validation.yaml#step1c-single-visit-tracts"
122148
- SpecBlock:
123149
name: step1d
124-
includes: ["nv_step", "step"]
150+
includes: ["nv_step", "step", "usdf"]
125151
data:
126152
pipeline_yaml: "${DRP_PIPE_DIR}pipelines/LSSTCam/nightly-validation.yaml#step1d-single-visit-global"
127153
- SpecBlock:
128154
name: stage3
129-
includes: ["nv_step", "step"]
155+
includes: ["nv_step", "step", "usdf"]
130156
data:
131157
prepend: |-
132158
export LSST_S3_USE_THREADS=False
133159
unset DAF_BUTLER_CACHE_DIRECTORY
134160
pipeline_yaml: "${DRP_PIPE_DIR}pipelines/LSSTCam/nightly-validation.yaml#stage3-coadd"
135-
- SpecBlock:
136-
name: nv_step
137-
includes: ["step"]
138-
data:
139-
prepend: |-
140-
export LSST_S3_USE_THREADS=False
141-
custom_lsst_setup: |-
142-
setup -j -r /sdf/data/rubin/shared/campaigns/LSSTCam-Nightly-Validation/lsst_devel/pipe_base
143-
extra_run_quantum_options: >-
144-
--no-raise-on-partial-outputs
145-
extra_qgraph_options: >-
146-
--dataset-query-constraint off
147-
--skip-existing-in LSSTCam/runs/nightlyValidation
148-
-c parameters:sasquatch_dataset_identifier=LSSTCam/nightlyValidation
149-
-c parameters:sasquatch_timestamp_version=explicit_timestamp:{SASQ_TIMESTAMP}T000000Z
150-
child_config:
151-
split_method: no_split
152161
# Specifications provide mapping options for different campaign flavors, such
153162
# as htcondor vs. panda.
154163
- Specification:
@@ -163,6 +172,13 @@ Campaign:
163172
campaign: nightly_validation
164173
bps_submit_script: bps_htcondor_submit_script
165174
bps_report_script: bps_htcondor_report_script
175+
# SPECBLOCKS for COMPUTE SITES
176+
- SpecBlock:
177+
name: usdf
178+
data:
179+
compute_site:
180+
computeCloud: US
181+
computeSite: SLAC
166182
---
167183
# COMMON SPECBLOCKS for SCRIPTS
168184
# define the null Script

src/lsst/cmservice/handlers/jobs.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,13 +99,16 @@ async def _write_script(
9999
workflow_config["lsst_version"] = lsst_version
100100
workflow_config["lsst_distrib_dir"] = lsst_distrib_dir
101101
workflow_config["wms"] = self.wms_method.name
102+
# TODO can we lookup any kind of default compute_site specblock by
103+
# a non-namespaced name?
104+
workflow_config["compute_site"] = data_dict.get("compute_site", None)
102105
workflow_config["script_method"] = script.run_method.name
103-
workflow_config["compute_site"] = data_dict.get("compute_site", self.default_compute_site.name)
104106
workflow_config["clustering"] = data_dict.get("cluster", None)
105107
workflow_config["extra_qgraph_options"] = data_dict.get("extra_qgraph_options", None)
106108
workflow_config["extra_run_quantum_options"] = data_dict.get("extra_run_quantum_options", None)
107109
workflow_config["bps_environment"] = data_dict.get("bps_environment", None)
108-
# Script Phases
110+
workflow_config["bps_literals"] = data_dict.get("bps_literals", {})
111+
# Script Phases - Do not use with BPS YAML
109112
workflow_config["prepend"] = data_dict.get("prepend", None)
110113
workflow_config["custom_lsst_setup"] = data_dict.get("custom_lsst_setup", None)
111114
workflow_config["custom_wms_setup"] = data_dict.get("custom_wms_setup", None)
@@ -141,6 +144,10 @@ async def _write_script(
141144
# `bps_wms_extra_files` instead of being specific keywords. The only
142145
# reason they are kept separate is to support overrides of their
143146
# specific role
147+
# FIXME there shouldn't be any INCLUDES in a BPS submit YAML at all
148+
# unless they are unique to the submission and separated for
149+
# readability. The use of any kind of "shared" or "global" config
150+
# items breaks provenance for all campaigns that reference them.
144151
bps_wms_extra_files = data_dict.get("bps_wms_extra_files", [])
145152
bps_wms_clustering_file = data_dict.get("bps_wms_clustering_file", None)
146153
bps_wms_resources_file = data_dict.get("bps_wms_resources_file", None)

src/lsst/cmservice/templates/bps_submit_yaml.j2

Lines changed: 23 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,16 @@ project: {{ project }}
1010
campaign: {{ campaign }}
1111
submitPath: {{ submit_path }}
1212
{# BLANK LINE #}
13+
########################################
14+
# LSST SETUP
15+
########################################
16+
LSST_VERSION: {{ lsst_version }}
17+
{# BLANK LINE #}
1318
{%- if bps_variables|length > 0 %}
1419
########################################
15-
# ADDITIONAL BPS DIRECTIVES
20+
# BPS VARIABLES
1621
########################################
17-
{%- for key, value in bps_variables|items %}
22+
{%- for key, value in bps_variables | items %}
1823
{{ key }}: "{{ value }}"
1924
{%- endfor %}
2025
{# BLANK LINE #}
@@ -26,14 +31,14 @@ pipelineYaml: {{ pipeline_yaml }}
2631
{%- if include_configs | length > 0 %}
2732
includeConfigs:
2833
{%- for config in include_configs %}
29-
- {{ config }}
34+
- {{ config }}
35+
{%- endfor %}
36+
{%- endif %}
37+
{%- if bps_literals | length > 0 %}
38+
{%- for key, value in bps_literals | items %}
39+
{{ key }}: {{ value }}
3040
{%- endfor %}
3141
{%- endif %}
32-
{# BLANK LINE #}
33-
########################################
34-
# LSST SETUP
35-
########################################
36-
LSST_VERSION: {{ lsst_version }}
3742
{# BLANK LINE #}
3843
{%- if bps_environment %}
3944
########################################
@@ -58,10 +63,16 @@ extraQgraphOptions: {{ extra_qgraph_options | replace("\n", " ") | trim }}
5863
{%- if extra_run_quantum_options %}
5964
extraRunQuantumOptions: {{ extra_run_quantum_options | replace("\n", " ") | trim }}
6065
{%- endif %}
66+
{%- if extra_init_options %}
67+
extraInitOptions: {{ extra_init_options | replace("\n", " ") | trim }}
68+
{%- endif %}
69+
{%- if extra_update_qgraph_options %}
70+
extraUpdateQgraphOptions: {{ extra_update_qgraph_options | replace("\n", " ") | trim }}
71+
{%- endif %}
6172
{# BLANK LINE #}
6273
{%- if clustering %}
6374
########################################
64-
# ADDITIONAL CLUSTERING CONFIGURATION
75+
# CLUSTERING CONFIGURATION
6576
########################################
6677
clustering:
6778
{{ clustering | toyaml | indent(2) -}}
@@ -72,28 +83,10 @@ clustering:
7283
########################################
7384
{%- if wms == "htcondor" %}
7485
wmsServiceClass: lsst.ctrl.bps.htcondor.HTCondorService
75-
{%- if compute_site == "usdf" %}
76-
site:
77-
s3df:
78-
profile:
79-
condor:
80-
+Walltime: 7200
81-
memoryMultiplier: 4.
82-
numberOfRetries: 3
83-
memoryLimit: 400000
84-
{%- endif %}
85-
{%- endif %}
86-
{%- if wms == "panda" %}
86+
{%- elif wms == "panda" %}
8787
wmsServiceClass: lsst.ctrl.bps.panda.PanDAService
88-
{%- if compute_site == "usdf" %}
89-
computeCloud: US
90-
computeSite: SLAC
91-
requestMemory: 4000
92-
memoryMultiplier: 1.2
93-
{%- elif compute_site == "lanc" %}
94-
computeSite: LANCS
95-
{%- elif compute_site == "in2p3" %}
96-
computeSite: CC-IN2P3
9788
{%- endif %}
89+
{%- if compute_site %}
90+
{{ compute_site | toyaml -}}
9891
{%- endif %}
9992
{# BLANK LINE #}

src/lsst/cmservice/templates/wms_submit_sh.j2

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
set -eux
2+
set -e
33
{# BLANK LINE #}
44
{#- PHASE: HEADER =========================================================== #}
55
{%- if prepend %}

0 commit comments

Comments
 (0)