Skip to content

Commit 873526f

Browse files
committed
fusion bulk edit documentation
1 parent 7598aa8 commit 873526f

11 files changed

Lines changed: 30 additions & 34 deletions

File tree

docs/core/architecture.mdx

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,10 @@ sidebar_label: Architecture
55
sidebar_position: 1
66
---
77

8-
# Architecture
8+
# OLake Go Architecture
99

1010
<!-- TODO: Ingestion architecture needs to be updated. -->
1111

12-
## OLake Go Architecture
13-
1412
For an in-depth look at OLake Go's architecture, including chunking strategies, concurrency models, and state management, explore our blog: [**A Deep Dive into OLake Go Architecture and Inner Workings**](/blog/olake-architecture-deep-dive).
1513

1614
<div className="w-full lg:w-[80%] mx-auto">

docs/fusion/compaction/configuration.mdx

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,56 +3,62 @@ title: Configuration
33
sidebar_position: 2
44
---
55

6-
# Configuring Table Compaction in Olake
6+
# Configuring Table Compaction
77

8-
Each table in Olake can have its **own compaction schedule and advanced settings**.
9-
Follow these steps to configure compaction for a specific table:
8+
Each table in OLake Fusion can have its **own compaction schedule and advanced settings**.
109

10+
## 1. Single Table Configuration
1111

12-
### 1. Click the Configure Button
12+
### Step 1. Click the Configure Button
1313

1414
Click the **Configure** button next to the table you want to compact.
1515
This opens a modal where you can schedule **Lite, Medium, and Full compactions**.
1616

1717
![Configure button](pathname:///img/docs/iceberg-maintenance/compaction/configure-button.webp)
1818

1919

20-
### 2. Set the Compaction Schedule
20+
### Step 2. Set the Compaction Schedule
2121

2222
- Select a schedule from the **predefined dropdown options** or choose **Custom** to specify your own cron expression.
2323
- Compaction will run automatically according to the schedule set for that table.
2424

2525
![Compaction Schedule](pathname:///img/docs/iceberg-maintenance/compaction/configuration.webp)
2626

27-
### 3. Advanced Config: Target File Size
27+
### Step 3. Advanced Config: Target File Size
2828

2929
- Expand the **Advanced Config** panel in the modal.
3030
- Specify the **Target File Size** for the table (default **512 MB** if you leave it unchanged).
3131

32-
For how **target file size** affects **Lite**, **Medium**, and **Full** compaction, see [Types of Compaction Supported in OLake](/docs/iceberg-maintenance/compaction/overview).
32+
For how **target file size** affects **Lite**, **Medium**, and **Full** compaction, see [Types of Compaction Supported in OLake Fusion](/docs/fusion/compaction/types-of-compaction/#types-of-compaction).
3333

3434
> **Tip:** Choose a target size based on your query patterns and table size. Larger files can improve scan efficiency based on the query but may increase the cost of rewriting files.
3535
3636
![Target File Size](pathname:///img/docs/iceberg-maintenance/compaction/target-file-size.webp)
3737

38-
### 4. Save the Configuration
38+
### Step 4. Save the Configuration
3939

4040
- Click **Save**.
4141

4242
![Save Configuration](pathname:///img/docs/iceberg-maintenance/compaction/save-configuration.webp)
4343

4444
- A dialog box confirms that the configuration was successful.
4545

46-
![Save Configuration](pathname:///img/docs/iceberg-maintenance/compaction/configuration-successful.webp)
46+
![Save successful](/img/docs/iceberg-maintenance/compaction/configuration-successful.webp)
4747

48+
- Once saved, the **Status** toggle for that table turns on automatically and compaction runs on the schedule you configured. Confirm this in the **Status** column as the toggle should appear active.
4849

49-
### 5. Enable the Table for Compaction
50+
![status active](/img/docs/iceberg-maintenance/compaction/active-status-single.webp)
5051

51-
- After saving, you will be redirected to the **Tables** page.
52-
- Locate the table and **toggle the Status switch** to activate scheduled compaction for that specific table.
52+
## 2. Bulk Configuration
5353

54-
> **Important:** The **Status toggle must be switched on**. Even if a cron schedule is configured, compaction will not execute unless the table is enabled.
54+
Bulk configuration lets you apply the same compaction schedule and settings to multiple tables in one go, instead of configuring each table individually.
5555

56-
![Enable compaction](pathname:///img/docs/iceberg-maintenance/compaction/enable-compaction.webp)
56+
To get started, select the tables you want to configure from the **Tables** page using their checkboxes, then click **Bulk Configure**. The same configuration modal opens — set the schedule, target file size, and save, exactly as described in Steps 2–4 of [Single Table Configuration](#1-single-table-configuration).
5757

58+
![bulk select tables](/img/docs/iceberg-maintenance/compaction/bulk-select-tables.webp)
5859

60+
:::note Default Configuration
61+
62+
If any configuration is left unset, the following defaults apply: **24 hours** as the frequency for each compaction type and **512 MB** as the target file size.
63+
64+
:::

docs/fusion/compaction/types-of-compaction.mdx

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,19 +17,21 @@ OLake supports three types of compaction:
1717

1818
### 1. Lite Compaction
1919

20-
Lite compaction is the lightest and most frequently run type. It focuses on two things, **merging fragment into larger ones** and **converting Equality Delete Files into Position Delete Files**. Position Delete Files are cheaper for query engines to process, so this conversion alone improves read performance without doing a heavy rewrite. Since streaming writes and high-frequency CDC constantly produce small fragment files, Minor compaction is typically scheduled to run frequentlly to keep the table tidy before the clutter builds up.
20+
Lite compaction is the lightest and most frequently run type. It focuses on two things, **merging fragment into larger ones** and **converting Equality Delete Files into Position Delete Files**. Position Delete Files are cheaper for query engines to process, so this conversion alone improves read performance without doing a heavy rewrite. Since streaming writes and high-frequency CDC constantly produce small fragment files, Lite compaction is typically scheduled to run frequentlly to keep the table tidy before the clutter builds up.
2121

2222
### 2. Medium Compaction
2323

24-
Medium compaction goes a step further. It **merges segment files up to the target file size**, and when too many Position Delete Files have accumulated, it merges them directly into the corresponding Data Files that is physically removing deleted rows from the table. This is more thorough than Minor compaction but still does not rewrite the entire table. The Medium Compaction is typically scheduled less frequently than the Lite Compaction to keep the table efficient and not spend too much compute.
24+
Medium compaction goes a step further. It **merges segment files up to the target file size**, and when too many Position Delete Files have accumulated, it merges them directly into the corresponding Data Files that is physically removing deleted rows from the table. This is more thorough than Lite compaction but still does not rewrite the entire table. The Medium Compaction is typically scheduled less frequently than the Lite Compaction to keep the table efficient and not spend too much compute.
2525

2626
### 3. Full Compaction
2727

2828
Full compaction is the deepest and most comprehensive type. It rewrites all data files fragments, segments, and delete files into optimally sized files that exactly match the configured target file size. Because it rewrites the entire table, it is the most compute-intensive option and is typically run less frequently. Use it when tables have accumulated heavy fragmentation over time or when you need the best possible query performance.
2929

3030
:::info Compaction precedence
3131

32-
When more than one type of compaction is scheduled for a table to run at the same time, **only the highest runs**: **Full** overrides **Medium** and **Lite**; **Medium** overrides **Lite**. For example, if **Full**, **Medium**, and **Lite** are all due together, **Full** runs alone; if **Medium** and **Lite** are due together, **Medium** runs alone.
32+
When more than one type is scheduled at the same time, only the highest-priority type runs:
33+
34+
**Full** > **Medium** > **Lite**
3335

3436
:::
3537

docs/fusion/core/architecture.mdx

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,7 @@ title: Architecture
33
sidebar_label: Architecture
44
---
55

6-
# Architecture
7-
8-
## OLake Fusion Architecture
6+
# OLake Fusion Architecture
97

108
<div className="w-full lg:w-[80%] mx-auto">
119
![OLake fusion architecture diagram](/img/docs/fusion-architecture.webp)

docs/fusion/getting-started/configure-first-compaction.mdx

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -132,18 +132,10 @@ After saving, a **Configuration Successful** modal appears. It **closes automati
132132

133133
![Configuration Successful modal](pathname:///img/docs/iceberg-maintenance/compaction/configuration-successful.webp)
134134

135-
136-
## Step 6: Enable Compaction
137-
138-
Saving the configuration does not start compaction automatically. You must enable it:
139-
140-
1. On the **Tables** page, locate the **Status** column next to the **Configure** button for your table.
141-
2. Use the **toggle** in the **Status** column to enable the compaction configuration.
135+
Once saved, the **Status** toggle for that table turns on automatically and compaction runs on the schedule you configured. Confirm this in the **Status** column as the toggle should appear active.
142136

143137
![Enable Configuration](pathname:///img/docs/iceberg-maintenance/compaction/enable-compaction.webp)
144138

145-
Once enabled, OLake Fusion will start running compaction for that table according to the schedule you configured.
146-
147139
## Health Score and Last Run Status
148140

149141
With a catalog and database selected, the **Tables** page shows one row per table. The sections below explain **Health Score** (overall table health) and **Last Run status** (per-type status for Lite, Medium, and Full compaction).

docs/fusion/getting-started/overview.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ You should consider running maintenance in the following scenarios:
3434
Iceberg Maintenance is available starting from v0.4.0. Upgrade OLake UI to access the **Maintenance** module.
3535

3636
- **Existing users (Docker):** If you are already using OLake Go for Ingestion, follow the [upgrade guide](/docs/fusion/install/olake-ui/?setup-mode=configuration#updating-olake-ui-version) to access the Maintenance module.
37-
- **Existing users (Helm / Kubernetes):** If you are running OLake Go on Kubernetes, follow the [chart upgrade guide](http://localhost:3000/docs/fusion/install/kubernetes-compaction/?setup-mode=installation#upgrading-chart-version) to access the Maintenance module.
37+
- **Existing users (Helm / Kubernetes):** If you are running OLake Go on Kubernetes, follow the [chart upgrade guide](/docs/fusion/install/kubernetes-compaction/?setup-mode=installation#upgrading-chart-version) to access the Maintenance module.
3838
- **New Users (Docker):** Follow the [quickstart guide](/docs/fusion/install/olake-ui/?setup-mode=installation&quick-start=Ingestion+%2B+Maintenance#one-command-setup) to get started.
3939
- **New Users (Helm / Kubernetes):** Follow the [quickstart guide](/docs/fusion/install/kubernetes-compaction/?setup-mode=installation#quick-start) to get started.
4040

docs/intro.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ import Head from '@docusaurus/Head'
9999
</a>
100100
</h1>
101101

102-
<p align="center">Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Visit <a href="https://olake.io" target="_blank">olake.io</a> for the full documentation, and benchmarks</p>
102+
<p align="center">Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics.</p>
103103

104104
import Tabs from '@theme/Tabs';
105105
import TabItem from '@theme/TabItem';
26.2 KB
Loading
26.7 KB
Loading
-184 KB
Loading

0 commit comments

Comments
 (0)