You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: website/docs/components/data-connectors/dremio/deployment.md
+2-3Lines changed: 2 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,14 +15,13 @@ Production operating guide for the Dremio data connector covering authentication
15
15
16
16
## Authentication & Secrets
17
17
18
-
The Dremio connector connects over [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) with username/password or personal-access-token (PAT) authentication.
18
+
The Dremio connector connects over [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) with username/password authentication.
|`open`| Absolute path to the DuckDB database file. |
23
-
|`duckdb_connection_string`| Alternative: DuckDB connection URI with options. |
22
+
|`duckdb_open`| Absolute path to the DuckDB database file. If omitted, uses in-memory mode. |
24
23
25
24
Protect the DuckDB file with filesystem permissions. Store it on encrypted storage (LUKS/dm-crypt, EBS encryption, etc.) for data-at-rest protection. For data loaded from cloud object stores inside DuckDB, configure AWS/Azure/GCS credentials via DuckDB extensions rather than Spice parameters.
|`graphql_auth_header`| Authorization header. Typically `"Bearer ${secrets:api_token}"`.|
22
+
|`graphql_auth_header`| Custom authorization header name. The value of `graphql_auth_token` is sent as this header's value.|
23
+
|`graphql_auth_token`| Bearer token for GraphQL requests. Typically `"${secrets:api_token}"`. |
24
24
|`graphql_query`| The GraphQL query to execute. |
25
-
|`graphql_json_pointer`| RFC-6901 JSON pointer to the row collection inside the response (e.g. `/data/repository/issues/nodes`). |
26
-
|`graphql_pagination_parameters`| Cursor / page-size configuration for pagination (see the connector reference). |
25
+
|`json_pointer`| RFC-6901 JSON pointer to the row collection inside the response (e.g. `/data/repository/issues/nodes`). |
27
26
28
27
Tokens must be sourced from a [secret store](../../secret-stores/) in production.
29
28
@@ -39,7 +38,7 @@ HTTP-level retries follow the shared `resilient_http` policy: 408/429/5xx plus t
39
38
40
39
### Pagination
41
40
42
-
The connector supports cursor-based pagination. Each page is a separate HTTP request; pagination errors mid-sequence cause the entire refresh to fail. Use `graphql_json_pointer` to select the row collection and configure the pagination variables to match the upstream schema's cursor fields.
41
+
The connector supports cursor-based pagination. Each page is a separate HTTP request; pagination errors mid-sequence cause the entire refresh to fail. Use `json_pointer` to select the row collection and configure the pagination variables to match the upstream schema's cursor fields.
43
42
44
43
### Server Rate Limits
45
44
@@ -77,7 +76,7 @@ GraphQL requests participate in [task history](../../../reference/task_history)
|`401 Unauthorized`| Wrong or expired token in `graphql_auth_header`. | Rotate the token; verify the header format (`Bearer` prefix, etc.). |
80
-
| Rows missing from the dataset | Wrong `graphql_json_pointer`. | Inspect the response payload; JSON pointer must navigate to the array of rows. |
79
+
| Rows missing from the dataset | Wrong `json_pointer`. | Inspect the response payload; JSON pointer must navigate to the array of rows. |
81
80
| Refresh fails mid-pagination | Rate-limit or transient network failure. | Reduce refresh frequency; the connector will retry on retriable errors. Narrow the query. |
82
81
| Query cost exceeded | Query requests too many nested fields. | Simplify the query; fetch only required fields. |
83
82
| Inferred schema differs between refreshes | Optional fields appear/disappear in responses. | Provide an explicit dataset `schema` to lock down types. |
Copy file name to clipboardExpand all lines: website/docs/components/data-connectors/mysql/deployment.md
+12-13Lines changed: 12 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,9 +37,8 @@ TLS is controlled via `mysql_sslmode`:
37
37
|`disabled`| No TLS. |
38
38
|`preferred`| Try TLS, fall back to plaintext. Not recommended for production. |
39
39
|`required`| Require TLS; do **not** verify the server certificate. |
40
-
|`verify_ca`| Require TLS and verify the CA chain. |
41
40
42
-
For production, use `verify_ca` with `mysql_sslrootcert` pointing to the CA bundle. The default is `required`, which encrypts the connection but does not validate the server's identity.
41
+
For production, use `required` with `mysql_sslrootcert` pointing to the CA bundle. The default is `required`, which encrypts the connection but does not validate the server's identity.
43
42
44
43
## Resilience Controls
45
44
@@ -49,18 +48,18 @@ The connector uses a per-dataset connection pool with the following defaults:
|`connection_pool_min_idle`|`1`| Minimum idle connections held by the pool. |
53
-
|`connection_pool_size`|`5`| Maximum connections the pool will open. |
51
+
|`mysql_pool_min`|`1`| Minimum idle connections held by the pool. |
52
+
|`mysql_pool_max`|`5`| Maximum connections the pool will open. |
54
53
55
-
Invalid values (non-integers) are logged as a warning and silently replaced with the defaults. Size the pool to match the concurrent query and refresh load for the dataset; the upper bound should respect the MySQL server's `max_connections` budget shared across all Spice datasets and external clients.
54
+
Invalid values (non-integers) are logged as a warning and silently replaced with the defaults. Size the pool to match the concurrent query and refresh load for the dataset; the upper bound should respect the MySQL server's `max_connections` budget shared across all Spice datasets and external clients.`mysql_pool_min` must be less than or equal to `mysql_pool_max`; conflicting values are rejected at startup.
56
55
57
56
### Retry Behavior
58
57
59
58
Transient query failures are not automatically retried at the connector layer. Dataset refresh retries are controlled by the acceleration refresh policy (see [Data Refresh](../../../features/data-acceleration/data-refresh)). Connection failures surface to the caller and to the connection pool metrics below.
60
59
61
60
## Capacity & Sizing
62
61
63
-
-**Network**: MySQL traffic is TCP. Plan for the sum of `connection_pool_size` across all Spice datasets targeting the same MySQL instance when sizing database `max_connections`.
62
+
-**Network**: MySQL traffic is TCP. Plan for the sum of `mysql_pool_max` across all Spice datasets targeting the same MySQL instance when sizing database `max_connections`.
64
63
-**Memory**: Each pooled connection holds a small amount of client-side state. Result sets stream in batches; memory footprint for federated reads is bounded by DataFusion's record batch size (8192 rows default).
65
64
-**Throughput**: For full-table materialization (acceleration refresh), query latency scales with the source table size and the presence of indexes on the refresh partitioning/ordering columns.
66
65
@@ -74,15 +73,15 @@ The MySQL connector exposes observable metrics for its connection pool. Enable t
74
73
|`connections_in_pool`| ObservableGauge | Idle connections sitting in the pool. |
75
74
|`active_wait_requests`| ObservableGauge | Requests waiting for a connection (saturation signal). |
76
75
|`create_failed`| Counter | Connections that failed to be created. |
77
-
|`discarded_excess_idle_connection`| Counter | Connections closed because the pool already had enough idle connections. |
76
+
|`discarded_superfluous_connection`| Counter | Connections closed because the pool already had enough idle connections. |
78
77
|`discarded_unestablished_connection`| Counter | Connections closed because they could not be established. |
79
78
|`dirty_connection_return`| Counter | Connections returned to the pool in a dirty state (open transactions, pending queries, etc.). |
80
79
81
80
Metric instruments are exposed with the prefix `dataset_mysql_`. Each instrument carries a `name` attribute set to the dataset name.
82
81
83
82
Key signals to alert on:
84
83
85
-
-`active_wait_requests > 0` sustained → pool is saturated, increase `connection_pool_size` or the server's `max_connections`.
84
+
-`active_wait_requests > 0` sustained → pool is saturated, increase `mysql_pool_max` or the server's `max_connections`.
86
85
-`create_failed` increasing → credentials, network, or server availability problem.
87
86
-`dirty_connection_return` increasing → a query is not cleaning up its transaction state; investigate long-running or aborted queries.
88
87
@@ -93,7 +92,7 @@ MySQL operations participate in Spice [task history](../../../reference/task_his
93
92
## Known Limitations
94
93
95
94
- Only TCP connections are supported. Unix socket connections are not exposed through Spice configuration.
96
-
- TLS with full hostname verification (`verify_identity`) is not a documented option; use `verify_ca` with a trusted CA bundle.
95
+
- TLS with certificate verification (`verify_ca`, `verify_identity`) is not supported; only `disabled`, `preferred`, and `required` modes are available.
97
96
- Large text/blob columns are fetched in their entirety per row; consider selecting only the columns you need when federating.
98
97
-`mysql_sslmode: preferred` silently downgrades to plaintext on TLS negotiation failure and is not recommended for production.
99
98
@@ -102,7 +101,7 @@ MySQL operations participate in Spice [task history](../../../reference/task_his
|`Access denied for user`| Incorrect credentials or user lacks `SELECT` on the DB. | Verify credentials; confirm the user has read access on the required tables. |
105
-
|`Too many connections`| Sum of Spice pool sizes + other clients exceeds server `max_connections`. | Reduce `connection_pool_size` or raise the server limit. |
|`SSL connection error`| Certificate mismatch with `mysql_sslmode: verify_ca`.| Verify `mysql_sslrootcert` matches the server's issuing CA. Use `openssl s_client -connect` to inspect. |
108
-
| Silent plaintext connection |`mysql_sslmode: preferred` falling back. | Switch to `required` or `verify_ca`.|
104
+
|`Too many connections`| Sum of Spice pool sizes + other clients exceeds server `max_connections`. | Reduce `mysql_pool_max` or raise the server limit. |
|`SSL connection error`| Certificate mismatch or TLS negotiation failure. | Verify `mysql_sslrootcert` matches the server's issuing CA. Use `openssl s_client -connect` to inspect. |
|`pg_connection_pool_min_idle`|`1`| Minimum idle connections held by the pool.|
55
+
|`connection_pool_size`|`5`| Maximum connections the pool will open.|
56
56
57
-
`connection_pool_min_idle` must be less than or equal to `connection_pool_size`; conflicting values are rejected as configuration errors at startup.
57
+
`pg_connection_pool_min_idle` must be less than or equal to `connection_pool_size`; conflicting values are rejected as configuration errors at startup.
58
58
59
59
Size the pool to match concurrent query and refresh load for the dataset. The server's `max_connections` (default 100) is a shared budget across Spice datasets, other clients, and server-side background workers — plan accordingly, or front Postgres with PgBouncer.
60
60
61
61
### Application Name
62
62
63
-
`pg_application_name` defaults to the Spice.ai version string, which surfaces in `pg_stat_activity.application_name`. Override this to distinguish traffic from multiple Spice instances or environments.
63
+
The connector automatically sets `application_name`to the Spice.ai version string, which surfaces in `pg_stat_activity.application_name`. This value is not configurable.
|`FATAL: password authentication failed`| Incorrect credentials. | Verify credentials via the secret store; test with `psql` using the same credentials. |
107
107
|`FATAL: too many clients already`| Pool size + other clients exceeds server `max_connections`. | Reduce `connection_pool_size` or raise `max_connections` / front the server with PgBouncer. |
108
-
|`connection_pool_min_idle must be <= connection_pool_size` at startup | Misconfiguration. | Correct the values so `min_idle <= size`. |
108
+
|`pg_connection_pool_min_idle must be <= connection_pool_size` at startup | Misconfiguration. | Correct the values so `pg_connection_pool_min_idle <= connection_pool_size`.|
109
109
| Sustained `active_wait_requests > 0`| Pool saturation. | Increase `connection_pool_size` or reduce concurrent refreshes. |
110
110
|`certificate verify failed`|`pg_sslmode: verify-ca` / `verify-full` with wrong CA or hostname. | Verify `pg_sslrootcert` matches the server's issuing CA; with `verify-full` ensure hostname matches SAN. |
111
-
| Sessions lingering with the default app name | Multiple Spice instances share the same name. | Set `pg_application_name` per deployment for clear `pg_stat_activity` attribution. |
111
+
| Sessions lingering with the default app name | Multiple Spice instances share the same version-based name. | The `application_name` is auto-set to the Spice.ai version and is not currently configurable.|
0 commit comments