Skip to content

Commit 7254ad4

Browse files
authored
docs:Bulk configuration section added in Ingestion Properties (#407)
* docs:Bulk configuration section added in Ingestion Properties * modified the edge case for data filter and cursor field * corrected the bulk config definition and refurbished the edge cases
1 parent 0152e6f commit 7254ad4

5 files changed

Lines changed: 45 additions & 0 deletions

File tree

docs/understanding/terminologies/olake.mdx

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -364,6 +364,50 @@ These modes control how records are written to the destination for CDC and Incre
364364

365365
---
366366

367+
### 9. Bulk Configure
368+
369+
Bulk Configure lets users apply the same configuration to multiple selected streams at once. When working with sources containing multiple streams, users often need to apply the same configuration across several streams. Bulk Configuration significantly reduces setup time and improves usability.
370+
371+
#### Supported Bulk Configurations
372+
373+
When multiple streams are selected using checkboxes, users can trigger a bulk configuration action to apply the following properties for all selected streams at once:
374+
375+
- **Sync Mode**
376+
- **Cursor Fields:** Shown only if Sync mode is *Full Refresh + Incremental*. Only columns that are common across all selected streams are available for cursor field selection.
377+
- **Data Filter (Column and Values):** Only columns that are common across all selected streams and have similar data types are available for data filter configuration.
378+
- **Partition Regex Value**
379+
380+
::::info Important Considerations
381+
- **Overwriting Existing Configurations:** Existing configurations are overwritten only for the modified properties. If a user uses the bulk configure option to change only a few properties of a previously configured stream, properties configured outside the bulk configure feature remain unchanged.
382+
- **Warning on Reconfiguration:** If a few streams are already configured and the user configures them again via Bulk Configuration, a warning modal will appear to confirm the action. Changing an already configured stream property will trigger a clear destination run.
383+
::::
384+
385+
#### Step 1: Initiate Bulk Configuration
386+
387+
From the Streams page, the **Bulk Configure** button can be used to apply the same configuration across multiple streams at once.
388+
389+
![Bulk Configuration Button](/img/docs/terminologies/bulk-configure-button.webp)
390+
391+
#### Step 2: Select Streams
392+
393+
After clicking **Bulk Configure**, a modal opens where the target streams can be selected for bulk configuration. Once stream selection is complete, click **Configure Streams** to continue.
394+
395+
![Bulk Configuration Streams Selection](/img/docs/terminologies/bulk-configure-select-streams.webp)
396+
397+
#### Step 3: Select Configuration
398+
399+
The next window displays the selected streams for bulk configuration. Configure the properties to be applied across those streams. The properties that are configured are marked with a warning sign, and only those marked properties are applied.
400+
401+
![Bulk Configuration on the streams](/img/docs/terminologies/configure-bulk-streams.webp)
402+
403+
#### Step 4: Review and Apply Configuration
404+
405+
In the final window, the selected configuration can be reviewed before submission. After confirmation, clicking **Apply Changes** applies the configuration to the selected streams.
406+
407+
![Review and Apply Bulk Configuration](/img/docs/terminologies/review-bulk-config.webp)
408+
409+
---
410+
367411
## OLake-Generated Columns
368412

369413
When OLake replicates data from source (like PostgreSQL, MySQL, Oracle or MongoDB) to destination formats (Apache Iceberg, Parquet), it automatically adds several metadata columns to track the lifecycle and processing history of each record. These columns help users understand how and when each row was captured and written to the destination.
@@ -540,3 +584,4 @@ When a job is inactive, certain job-level features are unavailable:
540584

541585
To use these features, resume the job to move it back to active status.
542586
:::
587+
31.4 KB
Loading
20 KB
Loading
27.8 KB
Loading
23 KB
Loading

0 commit comments

Comments
 (0)