You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs:Bulk configuration section added in Ingestion Properties (#407)
* docs:Bulk configuration section added in Ingestion Properties
* modified the edge case for data filter and cursor field
* corrected the bulk config definition and refurbished the edge cases
Copy file name to clipboardExpand all lines: docs/understanding/terminologies/olake.mdx
+45Lines changed: 45 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -364,6 +364,50 @@ These modes control how records are written to the destination for CDC and Incre
364
364
365
365
---
366
366
367
+
### 9. Bulk Configure
368
+
369
+
Bulk Configure lets users apply the same configuration to multiple selected streams at once. When working with sources containing multiple streams, users often need to apply the same configuration across several streams. Bulk Configuration significantly reduces setup time and improves usability.
370
+
371
+
#### Supported Bulk Configurations
372
+
373
+
When multiple streams are selected using checkboxes, users can trigger a bulk configuration action to apply the following properties for all selected streams at once:
374
+
375
+
-**Sync Mode**
376
+
-**Cursor Fields:** Shown only if Sync mode is *Full Refresh + Incremental*. Only columns that are common across all selected streams are available for cursor field selection.
377
+
-**Data Filter (Column and Values):** Only columns that are common across all selected streams and have similar data types are available for data filter configuration.
378
+
-**Partition Regex Value**
379
+
380
+
::::info Important Considerations
381
+
-**Overwriting Existing Configurations:** Existing configurations are overwritten only for the modified properties. If a user uses the bulk configure option to change only a few properties of a previously configured stream, properties configured outside the bulk configure feature remain unchanged.
382
+
-**Warning on Reconfiguration:** If a few streams are already configured and the user configures them again via Bulk Configuration, a warning modal will appear to confirm the action. Changing an already configured stream property will trigger a clear destination run.
383
+
::::
384
+
385
+
#### Step 1: Initiate Bulk Configuration
386
+
387
+
From the Streams page, the **Bulk Configure** button can be used to apply the same configuration across multiple streams at once.
After clicking **Bulk Configure**, a modal opens where the target streams can be selected for bulk configuration. Once stream selection is complete, click **Configure Streams** to continue.
The next window displays the selected streams for bulk configuration. Configure the properties to be applied across those streams. The properties that are configured are marked with a warning sign, and only those marked properties are applied.
400
+
401
+

402
+
403
+
#### Step 4: Review and Apply Configuration
404
+
405
+
In the final window, the selected configuration can be reviewed before submission. After confirmation, clicking **Apply Changes** applies the configuration to the selected streams.
406
+
407
+

408
+
409
+
---
410
+
367
411
## OLake-Generated Columns
368
412
369
413
When OLake replicates data from source (like PostgreSQL, MySQL, Oracle or MongoDB) to destination formats (Apache Iceberg, Parquet), it automatically adds several metadata columns to track the lifecycle and processing history of each record. These columns help users understand how and when each row was captured and written to the destination.
@@ -540,3 +584,4 @@ When a job is inactive, certain job-level features are unavailable:
540
584
541
585
To use these features, resume the job to move it back to active status.
0 commit comments