Skip to content

Commit 7a308d0

Browse files
jnt0rjnt0r
committed
feat(persistence): allow setting optional schema for postgres and mysql database
Signed-off-by: Jonathan Pollert <38696668+jnt0r@users.noreply.github.com>
1 parent 1fb87a0 commit 7a308d0

71 files changed

Lines changed: 787 additions & 804 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.features/pending/persistence-schema.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
Description: You can now configure a specific database schema for Argo Persistence in the controller config map.
2-
Authors: [Jonathan Pollert](https://github.com/jnt0r)
3-
Component: General
4-
Issues: 2452
5-
6-
This provides better data isolation and security in shared database environments by allowing Argo to operate within a designated logical schema rather than the default one.
1+
Description: You can now configure a specific postgres database schema for Argo Persistence in the controller config map.
2+
Authors: [Jonathan Pollert](https://github.com/jnt0r)
3+
Component: General
4+
Issues: 2452
5+
6+
Added support for custom database schemas to improve data isolation and security in shared environments. This allows Argo to operate within a designated logical schema rather than the default. Note: This feature is not applicable to MySQL, as it does not support logical schemas; for MySQL users, database names should continue to be used for application separation.

‎config/config.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -371,8 +371,6 @@ type DatabaseConfig struct {
371371
Port int `json:"port,omitempty"`
372372
// Database is the name of the database to connect to
373373
Database string `json:"database"`
374-
// Schema is the name of the schema to use in the database. If not set, the default schema of the database will be used
375-
Schema string `json:"schema"`
376374
// TableName is the name of the table to use, must be set
377375
TableName string `json:"tableName,omitempty"`
378376
// UsernameSecret references a secret containing the database username
@@ -391,6 +389,8 @@ func (c DatabaseConfig) GetHostname() string {
391389
// PostgreSQLConfig contains PostgreSQL-specific database configuration
392390
type PostgreSQLConfig struct {
393391
DatabaseConfig
392+
// Schema is the name of the schema to use in the database. If not set, the default schema of the database will be used
393+
Schema string `json:"schema"`
394394
// SSL enables SSL connection to the database
395395
SSL bool `json:"ssl,omitempty"`
396396
// SSLMode specifies the SSL mode (disable, require, verify-ca, verify-full)

‎docs/workflow-archive.md‎

Lines changed: 108 additions & 99 deletions
Original file line numberDiff line numberDiff line change
@@ -1,99 +1,108 @@
1-
# Workflow Archive
2-
3-
> v2.5 and after
4-
5-
If you want to keep completed workflows for a long time, you can use the workflow archive to save them in a Postgres (>=9.4) or MySQL (>= 5.7.8) database.
6-
The workflow archive stores the status of the workflow, which pods have been executed, what was the result etc.
7-
The job logs of the workflow pods will not be archived.
8-
If you need to save the logs of the pods, you must setup an [artifact repository](artifact-repository-ref.md) according to [this doc](configure-artifact-repository.md).
9-
10-
The quick-start deployment includes a Postgres database server.
11-
In this case the workflow archive is already enabled.
12-
Such a deployment is convenient for test environments, but in a production environment you must use a production quality database service.
13-
14-
## Enabling Workflow Archive
15-
16-
To enable archiving of the workflows, you must configure database parameters in the `persistence` section of [your configuration](workflow-controller-configmap.yaml) and set `archive:` to `true`.
17-
18-
Example:
19-
20-
persistence:
21-
archive: true
22-
postgresql:
23-
host: localhost
24-
port: 5432
25-
database: postgres
26-
tableName: argo_workflows
27-
userNameSecret:
28-
name: argo-postgres-config
29-
key: username
30-
passwordSecret:
31-
name: argo-postgres-config
32-
key: password
33-
34-
You must also create the secret with database user and password in the namespace of the workflow controller.
35-
36-
Example:
37-
38-
kubectl create secret generic argo-postgres-config -n argo --from-literal=password=mypassword --from-literal=username=argodbuser
39-
40-
Note that IAM-based authentication is not currently supported. However, you can start your database proxy as a sidecar
41-
(e.g. via [CloudSQL Proxy](https://github.com/GoogleCloudPlatform/cloud-sql-proxy) on GCP) and then specify your local
42-
proxy address, IAM username, and an empty string as your password in the persistence configuration to connect to it.
43-
44-
The following tables will be created in the database when you start the workflow controller with enabled archive:
45-
46-
* `argo_workflows`
47-
* `argo_archived_workflows`
48-
* `argo_archived_workflows_labels`
49-
* `schema_history`
50-
51-
## Automatic Database Migration
52-
53-
Every time the Argo workflow-controller starts with persistence enabled, it tries to migrate the database to the correct version.
54-
If the database migration fails, the workflow-controller will also fail to start.
55-
In this case you can delete all the above tables and restart the workflow-controller.
56-
57-
If you know what are you doing you also have an option to skip migration:
58-
59-
persistence:
60-
skipMigration: true
61-
62-
## Required database permissions
63-
64-
### Postgres
65-
66-
The database user/role must have `CREATE` and `USAGE` permissions on the `public` schema of the database so that the tables can be created during the migration.
67-
68-
## Archive TTL
69-
70-
You can configure the time period to keep archived workflows before they will be deleted by the archived workflow garbage collection function.
71-
The default is forever.
72-
73-
Example:
74-
75-
persistence:
76-
archiveTTL: 10d
77-
78-
The `ARCHIVED_WORKFLOW_GC_PERIOD` variable defines the periodicity of running the garbage collection function.
79-
The default value is documented [here](environment-variables.md).
80-
When the workflow controller starts, it sets the ticker to run every `ARCHIVED_WORKFLOW_GC_PERIOD`.
81-
It does not run the garbage collection function immediately and the first garbage collection happens only after the period defined in the `ARCHIVED_WORKFLOW_GC_PERIOD` variable.
82-
83-
## Cluster Name
84-
85-
Optionally you can set a unique name of your Kubernetes cluster. This name will populate the `clustername` field in the `argo_archived_workflows` table.
86-
87-
Example:
88-
89-
persistence:
90-
clusterName: dev-cluster
91-
92-
## Disabling Workflow Archive
93-
94-
To disable archiving of the workflows, set `archive:` to `false` in the `persistence` section of [your configuration](workflow-controller-configmap.yaml).
95-
96-
Example:
97-
98-
persistence:
99-
archive: false
1+
# Workflow Archive
2+
3+
> v2.5 and after
4+
5+
If you want to keep completed workflows for a long time, you can use the workflow archive to save them in a Postgres (>=9.4) or MySQL (>= 5.7.8) database.
6+
The workflow archive stores the status of the workflow, which pods have been executed, what was the result etc.
7+
The job logs of the workflow pods will not be archived.
8+
If you need to save the logs of the pods, you must setup an [artifact repository](artifact-repository-ref.md) according to [this doc](configure-artifact-repository.md).
9+
10+
The quick-start deployment includes a Postgres database server.
11+
In this case the workflow archive is already enabled.
12+
Such a deployment is convenient for test environments, but in a production environment you must use a production quality database service.
13+
14+
## Enabling Workflow Archive
15+
16+
To enable archiving of the workflows, you must configure database parameters in the `persistence` section of [your configuration](workflow-controller-configmap.yaml) and set `archive:` to `true`.
17+
18+
Example:
19+
20+
persistence:
21+
archive: true
22+
postgresql:
23+
host: localhost
24+
port: 5432
25+
database: postgres
26+
tableName: argo_workflows
27+
userNameSecret:
28+
name: argo-postgres-config
29+
key: username
30+
passwordSecret:
31+
name: argo-postgres-config
32+
key: password
33+
34+
You must also create the secret with database user and password in the namespace of the workflow controller.
35+
36+
Example:
37+
38+
kubectl create secret generic argo-postgres-config -n argo --from-literal=password=mypassword --from-literal=username=argodbuser
39+
40+
Note that IAM-based authentication is not currently supported. However, you can start your database proxy as a sidecar
41+
(e.g. via [CloudSQL Proxy](https://github.com/GoogleCloudPlatform/cloud-sql-proxy) on GCP) and then specify your local
42+
proxy address, IAM username, and an empty string as your password in the persistence configuration to connect to it.
43+
44+
The following tables will be created in the database when you start the workflow controller with enabled archive:
45+
46+
* `argo_workflows`
47+
* `argo_archived_workflows`
48+
* `argo_archived_workflows_labels`
49+
* `schema_history`
50+
51+
## Automatic Database Migration
52+
53+
Every time the Argo workflow-controller starts with persistence enabled, it tries to migrate the database to the correct version.
54+
If the database migration fails, the workflow-controller will also fail to start.
55+
In this case you can delete all the above tables and restart the workflow-controller.
56+
57+
If you know what are you doing you also have an option to skip migration:
58+
59+
persistence:
60+
skipMigration: true
61+
62+
## Required database permissions
63+
64+
### Postgres
65+
66+
The database user/role must have `CREATE` and `USAGE` permissions on the `public` schema of the database so that the tables can be created during the migration.
67+
68+
## Archive TTL
69+
70+
You can configure the time period to keep archived workflows before they will be deleted by the archived workflow garbage collection function.
71+
The default is forever.
72+
73+
Example:
74+
75+
persistence:
76+
archiveTTL: 10d
77+
78+
The `ARCHIVED_WORKFLOW_GC_PERIOD` variable defines the periodicity of running the garbage collection function.
79+
The default value is documented [here](environment-variables.md).
80+
When the workflow controller starts, it sets the ticker to run every `ARCHIVED_WORKFLOW_GC_PERIOD`.
81+
It does not run the garbage collection function immediately and the first garbage collection happens only after the period defined in the `ARCHIVED_WORKFLOW_GC_PERIOD` variable.
82+
83+
## Cluster Name
84+
85+
Optionally you can set a unique name of your Kubernetes cluster. This name will populate the `clustername` field in the `argo_archived_workflows` table.
86+
87+
Example:
88+
89+
persistence:
90+
clusterName: dev-cluster
91+
92+
## Disabling Workflow Archive
93+
94+
To disable archiving of the workflows, set `archive:` to `false` in the `persistence` section of [your configuration](workflow-controller-configmap.yaml).
95+
96+
Example:
97+
98+
persistence:
99+
archive: false
100+
101+
## Customizing PostgreSQL Database Schema
102+
103+
To change the schema for the tables in PostgreSQL, set `schema:` to the desired schema in the `persistence` section of [your configuration](workflow-controller-configmap.yaml). Only available on PostgreSQL.
104+
105+
Example:
106+
107+
persistence:
108+
schema: argo

‎docs/workflow-controller-configmap.md‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -202,10 +202,10 @@ PostgreSQLConfig contains PostgreSQL-specific database configuration
202202
| `Host` | `string` | Host is the database server hostname |
203203
| `Port` | `int` | Port is the database server port |
204204
| `Database` | `string` | Database is the name of the database to connect to |
205-
| `Schema` | `string` | Schema is the name of the schema to use in the database. If not set, the default schema of the database will be used |
206205
| `TableName` | `string` | TableName is the name of the table to use, must be set |
207206
| `UsernameSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | UsernameSecret references a secret containing the database username |
208207
| `PasswordSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | PasswordSecret references a secret containing the database password |
208+
| `Schema` | `string` | Schema is the name of the schema to use in the database. If not set, the default schema of the database will be used |
209209
| `SSL` | `bool` | SSL enables SSL connection to the database |
210210
| `SSLMode` | `string` | SSLMode specifies the SSL mode (disable, require, verify-ca, verify-full) |
211211
| `AzureToken` | [`AzureTokenConfig`](#azuretokenconfig) | AzureToken specifies if the password should be fetched as an Azure token |
@@ -235,16 +235,15 @@ MySQLConfig contains MySQL-specific database configuration
235235

236236
### Fields
237237

238-
| Field Name | Field Type | Description |
239-
|------------------|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
240-
| `Host` | `string` | Host is the database server hostname |
241-
| `Port` | `int` | Port is the database server port |
242-
| `Database` | `string` | Database is the name of the database to connect to |
243-
| `Schema` | `string` | Schema is the name of the schema to use in the database. If not set, the default schema of the database will be used |
244-
| `TableName` | `string` | TableName is the name of the table to use, must be set |
245-
| `UsernameSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | UsernameSecret references a secret containing the database username |
246-
| `PasswordSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | PasswordSecret references a secret containing the database password |
247-
| `Options` | `Map<string,string>` | Options contains additional MySQL connection options |
238+
| Field Name | Field Type | Description |
239+
|------------------|-----------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
240+
| `Host` | `string` | Host is the database server hostname |
241+
| `Port` | `int` | Port is the database server port |
242+
| `Database` | `string` | Database is the name of the database to connect to |
243+
| `TableName` | `string` | TableName is the name of the table to use, must be set |
244+
| `UsernameSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | UsernameSecret references a secret containing the database username |
245+
| `PasswordSecret` | [`apiv1.SecretKeySelector`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core) | PasswordSecret references a secret containing the database password |
246+
| `Options` | `Map<string,string>` | Options contains additional MySQL connection options |
248247

249248
## ConnectionPool
250249

0 commit comments

Comments
 (0)