Skip to content

Commit dcbaf74

Browse files
authored
docs: Updated postgres source setup (#395)
* Removed optional parameter from wal_sender_timeout in postgres * added ssl mode for postgres * added the issue in troubleshooting section
1 parent 35d41b1 commit dcbaf74

5 files changed

Lines changed: 18 additions & 6 deletions

File tree

docs/connectors/postgres/index.mdx

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ For users who previously set up jobs using the wal2json plugin: [wal2json plugin
7777
| Replication Slot `required for CDC` | Logical replication slot (pgoutput) that retains WAL segments until read, ensuring consistent CDC. | `olake-repl-slot` |
7878
| Publication `required for CDC` | Name of the PostgreSQL publication that defines which tables and operations (INSERT, UPDATE, DELETE, TRUNCATE) to replicate. Must exist in the same database as the replication slot and include all tables for CDC capture. | `postgres_pub` |
7979
| JDBC URL Parameters | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` |
80-
| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | `disable` |
80+
| SSL Mode | SSL configuration for the database connection. Contains details such as the SSL mode. | <ul> <li>`require`</li> <li>`disable`</li> <li>`verify-ca`</li> <li>`verify-full`</li> </ul> |
8181
| SSH Config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for the list of supported parameters. | <ul> <li>`No Tunnel`</li> <li>`SSH Key Authentication`</li> <li>`SSH Password Authentication`</li> </ul> |
8282

8383
:::warning Important: Unique Replication Slot Required
@@ -120,7 +120,7 @@ An example `source.json` file will look like this:
120120
| max_threads | Maximum number of worker threads the connector can spin up for parallel tasks. | `10` | Integer |
121121
| retry_count | Number of times the retry will take place incase of timeout before failing the sync. | `3` | Integer |
122122
| jdbc_url_params | Extra [JDBC URL parameters](https://jdbc.postgresql.org/documentation/use/) for fine-tuning the connection. | `{"connectTimeout":"20"}` | Object |
123-
| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | `{"mode": "disable"}` | Object |
123+
| ssl | SSL configuration for the database connection. Contains details such as the SSL mode. | <ul> <li>`require`</li> <li>`disable`</li> <li>`verify-ca`</li> <li>`verify-full`</li> </ul> | Object |
124124
| ssh_config | Configure OLake to connect through an SSH tunnel. See [SSH Config details](/docs/understanding/terminologies/olake#1-ssh-configuration) for parameter descriptions. | `{"host": "my-tunnel-host", "port": 22 , "username": "my-tunnel-user", "password": "tunnel-password"}` | Object |
125125

126126

@@ -226,6 +226,18 @@ To allow a specific role to bypass RLS entirely, execute the following command (
226226
ALTER USER <rolename> WITH BYPASSRLS;
227227
```
228228

229+
### 3. Failed with non retryable error: LSN not updated after `X` mins
230+
231+
```logs
232+
Failed with non retryable error: LSN not updated after `X` mins
233+
```
234+
235+
This issue occurs when `wal_sender_timeout` is set too low (for example, `30s`). If CDC processing takes longer than that, PostgreSQL closes the replication connection before LSN acknowledgment is completed.
236+
237+
It is more likely to happen on databases with concurrent writes or heavier CDC load.
238+
239+
**Solution**: Set `wal_sender_timeout` to `0` (recommended) or a sufficiently large value.
240+
229241
**If the issue is not listed here, post the query on Slack to get it resolved within a few hours.**
230242

231243
## Changelog

docs/connectors/postgres/setup/aurora.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ Configure these key parameters:
2929
- **`rds.logical_replication = 1`** - Enables logical replication (required for pgoutput)
3030
- **`wal_level = logical`** - May be set automatically by the above parameter
3131
- **`max_wal_senders`** and **`max_replication_slots`** - Set to accommodate your CDC connections
32-
- **`wal_sender_timeout = 0`** - *(Optional)* Prevents timeouts during long snapshots
32+
- **`wal_sender_timeout = 0`** - Prevents timeouts during long snapshots
3333

3434
Save the changes after configuration.
3535

docs/connectors/postgres/setup/azure.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ In the Azure Portal, navigate to your PostgreSQL Flexible Server. Go to `Setting
3535
| `wal_level` | `logical` | Enable logical replication |
3636
| `max_replication_slots` | `≥5` | Number of concurrent CDC connections |
3737
| `max_wal_senders` | `≥7` | Should exceed replication slots |
38-
| `wal_sender_timeout` | `0` | *(Optional)* Prevents snapshot timeouts |
38+
| `wal_sender_timeout` | `0` | Prevents snapshot timeouts |
3939

4040
![azure-server-parameters](/img/docs/cdc/postgres/azure-server-parameters.webp)
4141

docs/connectors/postgres/setup/generic.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ wal_level = logical
4949
max_replication_slots = 10
5050
max_wal_senders = 10
5151
52-
# Optional: Prevent timeouts during long snapshots
52+
# Prevent timeouts during long snapshots
5353
wal_sender_timeout = 0
5454
5555
# Optional: Control WAL retention

docs/connectors/postgres/setup/rds.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ In the AWS RDS console, go to `Parameter Groups` and create a new parameter grou
3232
| `wal_level` | `logical` | Usually set automatically by above |
3333
| `max_replication_slots` | `≥5` | Number of concurrent CDC connections |
3434
| `max_wal_senders` | `≥7` | Should exceed replication slots |
35-
| `wal_sender_timeout` | `0` | *(Optional)* Prevents snapshot timeouts |
35+
| `wal_sender_timeout` | `0` | Prevents snapshot timeouts |
3636

3737
Save the parameter group changes.
3838

0 commit comments

Comments
 (0)