diff --git a/Development/v4.x/backend/configuration/jobconfig.md b/Development/v4.x/backend/configuration/jobconfig.md index 206bafcf..2098a4c0 100644 --- a/Development/v4.x/backend/configuration/jobconfig.md +++ b/Development/v4.x/backend/configuration/jobconfig.md @@ -11,9 +11,11 @@ Example jobs include: page](https://github.com/SciCatProject/LandingPageServer)) - Run maintenance tasks such as emailing users -If you just plan to use SciCat for cataloging data and don't plan to use its data management features then you may not need any job types. -If no job types are configured then SciCat will reject any backend requests to create jobs. -In this case [frontend features](../../frontend/configuration.md) for archiving (`archiveWorkflowEnabled: false`) and retrieval should be disabled. +If you just plan to use SciCat for cataloging data and don't plan to use its data +management features then you may not need any job types. If no job types are configured +then SciCat will reject any backend requests to create jobs. In this case [frontend +features](../../frontend/configuration.md) for archiving (`archiveWorkflowEnabled: +false`) and retrieval should be disabled. ### Migration Notes @@ -21,7 +23,7 @@ In v3.x the `archive`, `retrieve`, and `public` jobs were hard-coded. In v4.x th types can be arbitrary strings; however we recommend using the standard job names to avoid confusion. -Also note that some checks that were preformed by default in v3.x for certain job types +Also note that some checks that were performed by default in v3.x for certain job types must now be configured explicitly as actions. These are included in the provided `jobConfig.example.yaml` file and are also noted below. @@ -29,10 +31,11 @@ must now be configured explicitly as actions. These are included in the provided Jobs follow a standard Create-Read-Update-Delete (CRUD) lifecycle: -1. Jobs are _created_ via a `POST` request to `/jobs`. This can be the result of a frontend +1. Jobs are *created* via a `POST` request to `/jobs`. This can be the result of a frontend interaction (eg selecting a dataset for publishing) or through the REST API. The body of the request should follow the CreateJobDto (Data Transfer Object): + ```json { "type": "archive", @@ -42,25 +45,28 @@ Jobs follow a standard Create-Read-Update-Delete (CRUD) lifecycle: "jobParams": {} } ``` -2. Jobs can be _read_ via a `GET` request to `/jobs/:id` or through the `/jobs/fullquery` + +2. Jobs can be *read* via a `GET` request to `/jobs/:id` or through the `/jobs/fullquery` search endpoint. The frontend uses this to display the list of jobs. -3. Jobs are _updated_ via a `PATCH` or `PUT` request to `/jobs/:id`. This is usually +3. Jobs are *updated* via a `PATCH` or `PUT` request to `/jobs/:id`. This is usually used by facility services to update the job status and provide feedback. The body of the request should follow the UpdateJobDto: - ``` + + ```json { "statusCode": "string", "statusMessage": "string", "jobResultObject": {} } ``` -4. Jobs may be _deleted_ periodically during maintenance. This is usually not done by + +4. Jobs may be *deleted* periodically during maintenance. This is usually not done by users. ### Actions -After _create_ and _update_ stages, a series of actions can be performed by SciCat. This +After *create* and *update* stages, a series of actions can be performed by SciCat. This can be things like sending an email, posting a message to a message broker, or calling an API. The `jobParams` and `jobResultObject` are used to provide additional information that the actions may need, such as the list of datasets the job refers to. @@ -76,14 +82,15 @@ actions based on the job status. The available jobs are configured in the file `jobConfig.yaml` (or can be overridden with the `JOB_CONFIGURATION_FILE` [environment -variable](../configuration.md#environment-variables)). An example `jobConfig.example.yaml` file -is available +variable](../configuration.md#environment-variables)). An example +`jobConfig.example.yaml` file is available [here](https://github.com/SciCatProject/scicat-backend-next/blob/master/jobConfig.example.yaml). ### Configuration overview + The top-level configuration is structured like this: -``` +```yaml configVersion: v1.0 jobs: - jobType: archive @@ -107,15 +114,20 @@ jobs: configured jobType will be accepted by the backend. - `create` and `update` correspond to `POST` and `PATCH` requests to the `/jobs` endpoint. These configure 'actions' which are run when at different phases of the job - lifecycle. The actions are defined in the [job actions section](#job-actions). + lifecycle. The actions are defined in the [job actions + section](#actions-configuration). - `auth` configures the roles authorized to use the endpoint for each job operation. - `actions` give a list of actions to run when the endpoint is called. ### Authorization -Values for `auth` are described in [Jobs Authorization](../authorization/authorization_jobs.md). Some authorization values may require certain information to be passed in the request body; for instance, `"#datasetOwner"` requires that a dataset be passed. +Values for `auth` are described in [JobsAuthorization](../authorization/authorization_jobs.md). +Some authorization values may require certain information to be passed in the request body; +for instance, `"#datasetOwner"` requires that a dataset be passed. -> **Caution** Setting `auth` to a permissive value (eg `#all`) could expose archiving services to external users. Please consider the security model carefully when configuring jobs. +> **Caution** Setting `auth` to a permissive value (eg `#all`) could expose archiving +> services to external users. Please consider the security model carefully when +> configuring jobs. ### Actions Configuration @@ -126,11 +138,12 @@ The following actions are built-in to SciCat and can be included in the `actions The `URL` action responds to a Job event by making an HTTP call. **Configuration**: -The _URL action_, per job stage (_create_, _update_) if defined, must be configured in `jobConfig.yaml`. -It supports templating, by extracting the values from the job schema. +The *URL action*, per job stage (*create*, *update*) if defined, must be configured in +`jobConfig.yaml`. It supports templating, by extracting the values from the job schema. For example: -```yml + +```yaml - actionType: url url: http://localhost:3000/api/v3/health?jobid={{id}} method: GET @@ -139,20 +152,22 @@ For example: ``` Where: - - `url` (required): The URL for the request.This can include template variables. - - `method` (optional): The HTTP method for the request, e.g. "GET", "POST". - - `headers` (optional): An object containing HTTP headers to be included in the request. - - `body` (optional): The body of the request, for methods like "POST" or "PUT". + +- `url` (required): The URL for the request.This can include template variables. +- `method` (optional): The HTTP method for the request, e.g. "GET", "POST". +- `headers` (optional): An object containing HTTP headers to be included in the request. +- `body` (optional): The body of the request, for methods like "POST" or "PUT". #### Validate -The `validate` action is used to validate requests to the job endpoints. It is -used to enforce custom constraints on `jobParams` or `jobResultObject` for each job -type. If other actions rely on custom fields in their templates they should first be -validated with this action. +The `validate` action is used to validate requests to the job endpoints. It is used to +enforce custom constraints on `jobParams` or `jobResultObject` for each job type. If +other actions rely on custom fields in their templates they should first be validated +with this action. ValidateAction is configured with a series of `: ` pairs which describe a constraint to be validated. These can be applied to different contexts: + - **`request`** - Checks the incoming request body (aka the DTO). - **`datasets`** - (CREATE only) requires that a list of datasets be included in `jobParams.datasetList`. Checks are applied to each dataset @@ -185,7 +200,8 @@ In this case an `email` action would be configured using handlebars to insert th catch errors early where the subject is not specified: `jobConfig.yaml`: -``` + +```yaml jobs: - jobType: email_demo create: @@ -231,7 +247,8 @@ The following validate actions are recommended for `archive`, `retrieve` and `pu jobs: `jobConfig.yaml`: -```yml + +```yaml jobs: - jobType: archive create: @@ -270,7 +287,7 @@ jobs: **Configuration**: The config file for a validate action will look like this: -```yml +```yaml - actionType: validate request: : @@ -288,7 +305,7 @@ The `` expression is a JSON Schema. While complicated schemas are pos the combination with JSONPath makes common type checks very concise and legible. Here are some example `` expressions: -``` +```yaml - actionType: validate request: # applies to the request body jobParams.stringVal: # match simple types @@ -312,7 +329,7 @@ or if any values matching the path do not validate against the provided schema. The `Email` action responds to a Job event by sending an email. **Configuration**: -The _Mail service_ must first be configured through environmental variables, as described in the [configuration](../configuration.md). +The *Mail service* must first be configured through environmental variables, as described in the [configuration](../configuration.md). There you can define the `EMAIL_TYPE`, which can be either `smtp` or `ms365`, along with the type's respective configuration values. While SMTP is the default, MS365 adds support for [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0) for sending emails. This is an alternative to SMTP for MS365 accounts that would otherwise require interactive OAuth logins, making it useful for automated emails. @@ -320,11 +337,12 @@ To use MS365, you (or your azure admin) will need to generate a `tenantId`, `cli The process is described [here](https://docs.emailengine.app/setting-up-oauth2-with-outlook/). Upon instantiation, the service will create the configured transporter, ready to sent emails upon request. -The _Email action_, per job stage (_create_, _update_) if defined, must be configured in `jobConfig.yaml`. +The *Email action*, per job stage (*create*, *update*) if defined, must be configured in `jobConfig.yaml`. It supports templating, by extracting the values from the job schema. Example: -``` + +```yaml - actionType: email to: "{{contactEmail}}" from: "sender@example.com", @@ -333,13 +351,15 @@ Example: ``` Where: + - `to`: The recipient's email address. This can include template variables. - `from`: The sender's email address. If no value is provided, then the default sender email address will be used, as defined in [configuration](../configuration.md). - `subject`: The subject of the email. This can include template variables. - `bodyTemplateFile`: The path to the HTML template file for the email body. You can create your own template for the email's body, which should be a valid html file - for example: -``` + +```html