Skip to content

Commit 818b1b3

Browse files
tellieretelliere
committed
adding documentation about client and server cli
1 parent 5be9d8b commit 818b1b3

File tree

6 files changed

+177
-5
lines changed

6 files changed

+177
-5
lines changed

‎client/job_preparation/utils/cli/cli.py

+4-4
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ def parse_arguments() -> argparse.Namespace:
88
Returns:
99
ArgumentParser: the ArgumentParser produced
1010
"""
11-
parser = argparse.ArgumentParser(description="CLI Optinons")
11+
parser = argparse.ArgumentParser(description="CLI Options")
1212

1313
parser.add_argument(
1414
"--config",
@@ -21,14 +21,14 @@ def parse_arguments() -> argparse.Namespace:
2121
"--job-name",
2222
"-J",
2323
type=str,
24-
help="name of job",
24+
help="name of the job",
2525
)
2626
parser.add_argument(
2727
"--nodes",
2828
"-N",
2929
type=str,
3030
required=True,
31-
help="number of nodes on which to run",
31+
help="count of nodes on which to run",
3232
)
3333
parser.add_argument(
3434
"--partition",
@@ -49,7 +49,7 @@ def parse_arguments() -> argparse.Namespace:
4949
"-A",
5050
type=str,
5151
required=True,
52-
help="charge job to specified account",
52+
help="account to bill the job to",
5353
)
5454
parser.add_argument(
5555
"--nodelist",

‎docs/README.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,8 @@ In-depth documentations are available in this directory.
55
## Architecture
66

77
An overview and an in-depth diagram of the architecture is available [here](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/architecture.md).
8-
You can also see sequence diagrams for the [container preparation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/container_preparation.md), [data preparation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/data_preparation.md) and [job preparation (and runtime)](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/job_preparation.md).
8+
You can also see sequence diagrams for the [container preparation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/container_preparation.md), [data preparation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/data_preparation.md) and [job preparation (and runtime)](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/architecture/job_preparation.md).
9+
10+
## CLI
11+
12+
Documentation of the CLI of the server and the client are available under `cli/`. Those documentations are exports of the `python3 code --help` with further explanation if needed.

‎docs/cli/container_preparation.md

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Container preparation
2+
3+
Using the cli directly isn't recommended, the supported way is through docker's entrypoint.
4+
5+
The container preparation cli allows the user to create/encrypt/ship a HPCS ready image based on any OCI image.
6+
7+
```
8+
usage: prepare_container.py [-h] --base-oci-image BASE_OCI_IMAGE --sif-path SIF_PATH [--encrypted] [--docker-path DOCKER_PATH]
9+
10+
CLI Options
11+
12+
options:
13+
-h, --help show this help message and exit
14+
--base-oci-image BASE_OCI_IMAGE, -b BASE_OCI_IMAGE
15+
Base OCI image
16+
--sif-path SIF_PATH, -s SIF_PATH
17+
Path for the final SIF image (please use "$(pwd)" instead of ".")
18+
--encrypted, -e Encrypt final SIF image or not (default : False)
19+
--docker-path DOCKER_PATH, -d DOCKER_PATH
20+
Path to the docker socket (default : /var/run/docker.sock)
21+
```
22+
23+
I.e
24+
25+
```bash
26+
# Show the help above
27+
python3 ./client/container_preparation/prepare_container.py --help
28+
29+
# Run the container preparation while specifying every parameters
30+
python3 ./client/container_preparation/prepare_container.py --base-oci-image talinx/jp2a --sif-path $(pwd) --encrypted --docker-path /var/run/docker.sock
31+
32+
# Run the container preparation while specifying every parameters (shortened version)
33+
python3 ./client/container_preparation/prepare_container.py -b talinx/jp2a -s $(pwd) -e -d /var/run/docker.sock
34+
35+
# Run the container preparation specifying minimum flags
36+
python3 ./client/container_preparation/prepare_container.py -b talin/jp2a -s $(pwd)
37+
```

‎docs/cli/data_preparation.md

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Data preparation
2+
3+
Using the cli directly isn't recommended, the supported way is through docker's entrypoint.
4+
5+
The data preparation cli allows the user to encrypt/ship an encrypted archive based on any source directory.
6+
7+
```
8+
usage: prepare_data.py [-h] --input-path INPUT_PATH --output-path OUTPUT_PATH
9+
10+
CLI Options
11+
12+
options:
13+
-h, --help show this help message and exit
14+
--input-path INPUT_PATH, -i INPUT_PATH
15+
Path to the input data
16+
--output-path OUTPUT_PATH, -o OUTPUT_PATH
17+
Path to the output encrypted data
18+
```
19+
20+
I.e
21+
22+
```bash
23+
# Show the help above
24+
python3 ./client/data_preparation/prepare_data.py --help
25+
26+
# Run the data preparation while specifying every parameters
27+
python3 ./client/data_preparation/prepare_data.py --input-path $(pwd)/input_data --output-path $(pwd)
28+
29+
# Run the data preparation while specifying every parameters (shortened version)
30+
python3 ./client/data_preparation/prepare_data.py -i $(pwd)/input_data -o $(pwd)
31+
```

‎docs/cli/job_preparation.md

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Job preparation
2+
3+
Using the cli directly isn't recommended, the supported way is through docker's entrypoint.
4+
5+
The job preparation cli allows the user to run the secure job. It will gather the necessary informations using the info files before connecting to the supercomputer to run a template-generated sbatch file.
6+
7+
```
8+
usage: prepare_job.py [-h] --config CONFIG [--job-name JOB_NAME] --nodes NODES --partition PARTITION --time TIME --account ACCOUNT --nodelist NODELIST --workdir WORKDIR --application-info APPLICATION_INFO --data-info
9+
DATA_INFO --supplementary-input-scripts SUPPLEMENTARY_INPUT_SCRIPTS --supplementary-output-scripts SUPPLEMENTARY_OUTPUT_SCRIPTS
10+
[--singularity-supplementary-flags SINGULARITY_SUPPLEMENTARY_FLAGS] [--arguments ARGUMENTS] [--follow]
11+
12+
CLI Options
13+
14+
options:
15+
-h, --help show this help message and exit
16+
--config CONFIG Configuration file (INI Format) (default: /tmp/hpcs-client.conf)
17+
--job-name JOB_NAME, -J JOB_NAME
18+
name of the job
19+
--nodes NODES, -N NODES
20+
count of nodes on which to run
21+
--partition PARTITION, -p PARTITION
22+
partition requested
23+
--time TIME, -t TIME time limit
24+
--account ACCOUNT, -A ACCOUNT
25+
account to bill the job to
26+
--nodelist NODELIST, -w NODELIST
27+
request a specific list of hosts
28+
--workdir WORKDIR, -s WORKDIR
29+
directory to work in (LUMI-SD specific -> different from chdir)
30+
--application-info APPLICATION_INFO, -ai APPLICATION_INFO
31+
path to the info file for the image to run on supercomputer
32+
--data-info DATA_INFO, -di DATA_INFO
33+
path to the info file for the dataset to use on supercomputer
34+
--supplementary-input-scripts SUPPLEMENTARY_INPUT_SCRIPTS, -i SUPPLEMENTARY_INPUT_SCRIPTS
35+
path to your input verification scripts directory
36+
--supplementary-output-scripts SUPPLEMENTARY_OUTPUT_SCRIPTS, -o SUPPLEMENTARY_OUTPUT_SCRIPTS
37+
path to your output verification scripts directory
38+
--singularity-supplementary-flags SINGULARITY_SUPPLEMENTARY_FLAGS, -flags SINGULARITY_SUPPLEMENTARY_FLAGS
39+
supplementary arguments to pass to singularity
40+
--arguments ARGUMENTS, -args ARGUMENTS
41+
supplementary arguments to pass to the application
42+
--follow, -f Follow job's output (default : False)
43+
```
44+
45+
The standard command format would be like this :
46+
47+
```bash
48+
python3 ./client/job_preparation/prepare_job.py [SLURM OPTIONS] --workdir [WORKDIR] --application-info [PATH TO INFO FILE] --data-info [PATH TO INFO FILE] --supplementary-input-scripts [PATH TO SUPPLEMENTARY SCRIPTS TO RUN BEFORE APPLICATION] --supplementary-output-scripts [PATH TO SUPPLEMENTARY SCRIPTS TO RUN AFTER APPLICATION] --singularity-supplementary-flags [FLAGS TO MANUALLY PASS TO SINGULARITY] --arguments [ARGUMENTS FOR THE FINAL APPLICATION]
49+
```
50+
51+
I.e :
52+
53+
```bash
54+
python3 ./prepare_job.py -u etellier -N 1 -p standard -t "00:60:00" -A project_462000031 --nodelist nid001791 --workdir /scratch/project_462000031/etellier -ai /pfs/lustrep4/scratch/project_462000031/etellier/encrypted_prepared_jp2a.sif.info.yaml -di /pfs/lustrep4/scratch/project_462000031/etellier/encrypted_jp2a_input.tgz.info.yaml -args "\" /sd-container/encrypted/jp2a_input/image.png --width=100 --output=/sd-container/output/result \"" -i /pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts -o /pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts -flags "--env TERM=xterm-256color" -f
55+
```
56+
57+
Will create a slurm job with the following configuration :
58+
- 1 node (nid001791)
59+
- On partition "standard"
60+
- As etellier
61+
- for an hour
62+
63+
Also :
64+
- HPCS will run the job using `/scratch/project_462000031/etellier` as its workdir
65+
- The application will be handled using info file at `/pfs/lustrep4/scratch/project_462000031/etellier/encrypted_prepared_jp2a.sif.info.yaml`
66+
- The application will be handled using info file at `/pfs/lustrep4/scratch/project_462000031/etellier/encrypted_jp2a_input.tgz.info.yaml`
67+
- "`--env TERM=xterm-256color`" will be passed to `singularity run` when creating the final container
68+
- The scripts available under the `/pfs/lustrep4/scratch/project_462000031/etellier/verification_scripts` directory will be mounted in the final container in order to run them before and after the container's application
69+
- `\" /sd-container/encrypted/jp2a_input/image.png --width=100 --output=/sd-container/output/result \"` will be passed to the `singularity run` command, i.e : `singularity run prepared_jp2a.sif -- \" /sd-container/encrypted/jp2a_input/image.png --width=100 --output=/sd-container/output/result \"`
70+
- The client will follow the output of the job, because of the `-f` flag
71+
72+
For more information about the server configuration, see the [associated documentation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/configuration/hpcs-client.md).

‎docs/cli/server.md

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Server
2+
3+
Using the cli directly isn't recommended, the supported way is through docker's entrypoint.
4+
5+
The server cli allows the user to invoke the server, passing the hpcs server configuration.
6+
7+
```
8+
usage: app.py [-h] [--config CONFIG]
9+
10+
CLI Options
11+
12+
options:
13+
-h, --help show this help message and exit
14+
--config CONFIG, -c CONFIG
15+
Configuration file (INI Format) (default: /tmp/hpcs-server.conf)
16+
```
17+
18+
I.e
19+
20+
```bash
21+
# Show the help above
22+
python3 ./server/app.py --help
23+
24+
# Run server, using ~/.config/hpcs-server.conf config file
25+
python3 ./server/app.py --config ~/.config/hpcs-server.conf
26+
```
27+
28+
For more information about the server configuration, see the [associated documentation](https://github.com/CSCfi/HPCS/tree/doc/readme_and_sequence_diagrams/docs/configuration/hpcs-server.md).

0 commit comments

Comments
 (0)