add config-driven scripts for htcondor#44
add config-driven scripts for htcondor#44gsharma99 wants to merge 7 commits intoiris-hep:hpc_workflow_managementfrom
Conversation
…definitions to config.pipeline.yaml, issue iris-hep#12
gsharma99
force-pushed
the
feature/config-driven-pipeline
branch
from
fb4656f to
d2168f6
Compare
…efinitions to config.pipeline.yaml, issue iris-hep#12
…pipeline.yaml, issue iris-hep#12
…nd the config.pipeline.yaml, issue iris-hep#12
…s/nerual_likelihood_ratio_estimation, issue iris-hep#12
…the config.pipeline.yaml, issue iris-hep#12
gsharma99
force-pushed
the
feature/config-driven-pipeline
branch
from
c2333e6 to
aab2c1b
Compare
gsharma99
force-pushed
the
feature/config-driven-pipeline
branch
from
b4bdfab to
d58dae1
Compare
JaySandesara
left a comment
JaySandesara
left a comment
There was a problem hiding this comment.
Preliminary changes suggested to fix bugs in .dag file
| JOB LOAD FAIR_universe_Higgs_tautau/htcondor/job.sub | ||
| VARS LOAD CONFIG="config.pipeline.yaml" STEP="data_loader" CPUS="12" MEM="128GB" GPUS="0" DISK="64GB" No newline at end of file | ||
| # Global Config Variable | ||
| CONFIG = config.pipeline.yaml |
There was a problem hiding this comment.
Defining a user variable doesnt seem to work in DAG files - need to use the file name everywhere explicitly, or find some solution that works
| # --- Step 1: Data Loading --- | ||
|
|
||
| JOB data_loader FAIR_universe_Higgs_tautau/htcondor/job.sub | ||
| VARS data_loader STEP="data_loader" CONFIG="$(CONFIG)" CPUS="1" MEM="8GB" GPUS="0" DISK="32GB" |
There was a problem hiding this comment.
$(CONFIG) -> config.pipeline.yaml since the variable thing doesnt work. Same for all steps
| # --- Step 1: Data Loading --- | ||
|
|
||
| JOB data_loader FAIR_universe_Higgs_tautau/htcondor/job.sub | ||
| VARS data_loader STEP="data_loader" CONFIG="$(CONFIG)" CPUS="1" MEM="8GB" GPUS="0" DISK="32GB" |
There was a problem hiding this comment.
| VARS data_loader STEP="data_loader" CONFIG="$(CONFIG)" CPUS="1" MEM="8GB" GPUS="0" DISK="32GB" | |
| VARS data_loader STEP="data_loader" CONFIG="$(CONFIG)" CPUS="1" MEM="64GB" GPUS="0" DISK="64GB" |
| transfer_input_files = pyproject.toml, src, FAIR_universe_Higgs_tautau, README.md | ||
| transfer_output_files = FAIR_universe_Higgs_tautau/saved_datasets | ||
| transfer_output_remaps = "saved_datasets = ./FAIR_universe_Higgs_tautau/saved_datasets" | ||
| transfer_output_files = FAIR_universe_Higgs_tautau/saved_datasets, FAIR_universe_Higgs_tautau/output |
There was a problem hiding this comment.
Since FAIR_universe_Higgs_tautau/output is not saved in all steps, both this and needs to be and FAIR_universe_Higgs_tautau/saved_datasets and any other outputs need to be passed as a list to the submit file. If not that, we need to have a separate .sub file for each step (maybe easiest). Could you create a unique job.sub file for each step, with names like job_data_loader.sub, etc. ?
There was a problem hiding this comment.
The workflow.dag file can then have entries without the STEP input. E.g.
JOB data_loader FAIR_universe_Higgs_tautau/htcondor/job_data_loader.sub
VARS data_loader CONFIG="config.pipeline.yaml" CPUS="1" MEM="64GB" GPUS="0" DISK="64GB"
and in the job_data_loader.sub for example, we can pass the STEP argument explicitly depending on the job submit file:
executable = FAIR_universe_Higgs_tautau/htcondor/run_step.sh
arguments = data_loader $(CONFIG)
| + input_features_1Jets \ | ||
| + input_features_2Jets \ | ||
| + input_features_nJets | ||
| cfg_full = load_config(args.config) |
There was a problem hiding this comment.
| cfg_full = load_config(args.config) | |
| cfg_full = load_config(args.config)["data_preprocessing"] |
and do similar adjustments to all the other files as well. NO need to load the full config - more error prone
| if "data_preprocessing" not in cfg_full: | ||
| raise KeyError("Config file missing 'data_preprocessing' section.") | ||
|
|
||
| feats = cfg_full["data_preprocessing"]["features"] |
There was a problem hiding this comment.
| feats = cfg_full["data_preprocessing"]["features"] | |
| feats = cfg_full["features"] |
with the above change, this is how we use the config. Could you also rename cfg_full to something like config_workflow?
| # Section for Data Preprocessing | ||
| data_preprocessing: | ||
|
|
||
| config_path: "./config.yaml" |
There was a problem hiding this comment.
| config_path: "./config.yaml" | |
| config_path: "./config.yml" |
There was a problem hiding this comment.
And everywhere else too - change config.yaml to config.yml
2 participants
Related issue #12
cc @JaySandesara