Skip to content

Commit aca5e43

Browse files
CrazyglueCrazyglue
authored
Expand README to list out available parameters for async job (#1298)
Signed-off-by: Eric Dobroveanu <[email protected]>
1 parent 000d57c commit aca5e43

File tree

2 files changed

+64
-0
lines changed

2 files changed

+64
-0
lines changed

‎jobs/async-upload/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,69 @@ docker push quay.io/<org>/async-upload-job:<tag>
1919
# TODO: Run locally...
2020
```
2121

22+
## Job Configuration
23+
24+
The async job is able to take a number of configuration parameters and environment variables which can be consumed to perform the job of synchronizing a model from a source to a destination.
25+
26+
When using environment variables to configure the job, you will need to provide them in the Kubernetes Job manifest in the `spec.template.spec.containers[*].env` section. From the job's perspective, these will become standard env vars that it can read. See the [samples directory](./samples/) for typical usage.
27+
28+
When using a parameter-based approach, the configuration variables will need to be passed in as `args` to the `command`.
29+
30+
When providing parameters in a mixed-fashion to the job, the job will prioritize certain sources of those parameters over others. The order of priority is below:
31+
32+
1. Command-line arguments (`args: []` in the manifest)
33+
2. Environment variables (`env: { ... }` in the manifest)
34+
3. Credentials files (read from the parameter \*\_CREDENTIALS_PATH env/arg)
35+
4. Default values
36+
37+
Below is a table of the configuration that can be passed into the job
38+
39+
See asterisks below table for details
40+
41+
| Environment Variable | Arg | Default Value | Required | Description |
42+
| -------------------------------------------- | ----------------------------------- | ----------------- | -------- | ---------------------------------------------------------------- |
43+
| MODEL_SYNC_SOURCE_TYPE | --source-type | s3 | ✅ | |
44+
| MODEL_SYNC_SOURCE_S3_CREDENTIALS_PATH | --source-s3-credentials-path | | | |
45+
| MODEL_SYNC_SOURCE_OCI_CREDENTIALS_PATH | --source-oci-credentials-path | | | |
46+
| MODEL_SYNC_SOURCE_AWS_BUCKET | --source-aws-bucket | | ✅\* | When --source-type is "s3" |
47+
| MODEL_SYNC_SOURCE_AWS_KEY | --source-aws-key | | ✅\* | " |
48+
| MODEL_SYNC_SOURCE_AWS_REGION | --source-aws-region | | | " |
49+
| MODEL_SYNC_SOURCE_AWS_ACCESS_KEY_ID | --source-aws-access-key-id | | ✅\* | " |
50+
| MODEL_SYNC_SOURCE_AWS_SECRET_ACCESS_KEY | --source-aws-secret-access-key | | ✅\* | " |
51+
| MODEL_SYNC_SOURCE_AWS_ENDPOINT | --source-aws-endpoint | | | " |
52+
| MODEL_SYNC_SOURCE_OCI_URI | --source-oci-uri | | ✅\+ | When --source-type is "oci" |
53+
| MODEL_SYNC_SOURCE_OCI_USERNAME | --source-oci-username | | ✅\+ | " |
54+
| MODEL_SYNC_SOURCE_OCI_PASSWORD | --source-oci-password | | ✅\+ | " |
55+
| MODEL_SYNC_DESTINATION_TYPE | --destination-type | oci | ✅ | |
56+
| MODEL_SYNC_DESTINATION_S3_CREDENTIALS_PATH | --destination-s3-credentials-path | | | |
57+
| MODEL_SYNC_DESTINATION_OCI_CREDENTIALS_PATH | --destination-oci-credentials-path | | | |
58+
| MODEL_SYNC_DESTINATION_AWS_BUCKET | --destination-aws-bucket | | ✅\* | When --destination-type is "s3" |
59+
| MODEL_SYNC_DESTINATION_AWS_KEY | --destination-aws-key | | ✅\* | " |
60+
| MODEL_SYNC_DESTINATION_AWS_REGION | --destination-aws-region | | | " |
61+
| MODEL_SYNC_DESTINATION_AWS_ACCESS_KEY_ID | --destination-aws-access-key-id | | ✅\* | " |
62+
| MODEL_SYNC_DESTINATION_AWS_SECRET_ACCESS_KEY | --destination-aws-secret-access-key | | ✅\* | " |
63+
| MODEL_SYNC_DESTINATION_AWS_ENDPOINT | --destination-aws-endpoint | | | " |
64+
| MODEL_SYNC_DESTINATION_OCI_URI | --destination-oci-uri | | ✅\+ | When --destination-type is "oci" |
65+
| MODEL_SYNC_DESTINATION_OCI_USERNAME | --destination-oci-username | | ✅\+ | " |
66+
| MODEL_SYNC_DESTINATION_OCI_PASSWORD | --destination-oci-password | | ✅\+ | " |
67+
| MODEL_SYNC_MODEL_ID | --model-id | | ✅ | The `RegisteredModel.id` |
68+
| MODEL_SYNC_VERSION_ID | --model-version-id | | ✅ | The `ModelVersion.id` |
69+
| MODEL_SYNC_ARTIFACT_ID | --model-artifact-id | | ✅ | The `ModelArtifact.id` |
70+
| MODEL_SYNC_STORAGE_PATH | --storage-path | `/tmp/model-sync` | ✅ | Temporary storage, must be large enough to hold the entire model |
71+
| MODEL_SYNC_REGISTRY_SERVER_ADDRESS | --registry-server-address | | ✅ | Server address for the model-registry client to connect to |
72+
| MODEL_SYNC_REGISTRY_PORT | --registry-port | 443 | | |
73+
| MODEL_SYNC_REGISTRY_IS_SECURE | --registry-is-secure | True | | |
74+
| MODEL_SYNC_REGISTRY_AUTHOR | --registry-author | | | |
75+
| MODEL_SYNC_REGISTRY_USER_TOKEN | --registry-user-token | | | |
76+
| MODEL_SYNC_REGISTRY_USER_TOKEN_ENVVAR | --registry-user-token-envvar | | | |
77+
| MODEL_SYNC_REGISTRY_CUSTOM_CA | --registry-custom-ca | | | |
78+
| MODEL_SYNC_REGISTRY_CUSTOM_CA_ENVVAR | --registry-custom-ca-envvar | | | |
79+
| MODEL_SYNC_REGISTRY_LOG_LEVEL | --registry-log-level | | | |
80+
81+
✅\*: Must be present in some form when the source/destination is `s3`. This might be from the parameter in the table, or from the credentials file(s) that was specified/provided.
82+
83+
✅\+: Must be present in some from when the source/destination is `oci`. This might be from the parameter in the table, or from the credentials file(s) that was specified/provided.
84+
2285
## References
2386

2487
- Issue thread : [https://github.com/kubeflow/model-registry/issues/1108](https://github.com/kubeflow/model-registry/issues/1108)

‎jobs/async-upload/job/config.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ def _parser() -> cap.ArgumentParser:
6464
p.add("--registry-custom-ca-envvar", default=None)
6565
p.add("--registry-log-level", default=logging.WARNING)
6666

67+
# TODO: The type of credential should be inferrable from the `type` specified in the source/destination
6768
p.add(
6869
"--source-s3-credentials-path",
6970
metavar="PATH",

0 commit comments

Comments
 (0)