Skip to content

Commit e1f8894

Browse files
JingsongLiJingsongLi
committed
[python] Add block-level local disk cache for file reads
Introduce a CachingFileIO wrapper that transparently caches remote file reads at block granularity on local disk. Files are classified by FileType (ported from Java) and only META, GLOBAL_INDEX, BUCKET_INDEX types are cached; DATA and FILE_INDEX are read directly. Enable via table.copy({"file-cache.enabled": "true"}).
1 parent f9eb502 commit e1f8894

17 files changed

Lines changed: 2652 additions & 2 deletions

File tree

‎docs/content/program-api/file-cache.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: "Local Disk Cache"
3+
weight: 8
4+
type: docs
5+
aliases:
6+
- /program-api/file-cache.html
7+
- /pypaimon/file-cache.html
8+
---
9+
<!--
10+
Licensed to the Apache Software Foundation (ASF) under one
11+
or more contributor license agreements. See the NOTICE file
12+
distributed with this work for additional information
13+
regarding copyright ownership. The ASF licenses this file
14+
to you under the Apache License, Version 2.0 (the
15+
"License"); you may not use this file except in compliance
16+
with the License. You may obtain a copy of the License at
17+
18+
http://www.apache.org/licenses/LICENSE-2.0
19+
20+
Unless required by applicable law or agreed to in writing,
21+
software distributed under the License is distributed on an
22+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
23+
KIND, either express or implied. See the License for the
24+
specific language governing permissions and limitations
25+
under the License.
26+
-->
27+
28+
# Local Disk Cache
29+
30+
When reading files from remote storage (S3, OSS, HDFS, etc.), each seek+read goes over the network. Paimon provides a block-level local disk cache that transparently caches file reads on local disk, significantly reducing remote I/O for repeated access patterns.
31+
32+
## Cached File Types
33+
34+
The cache classifies files by type. By default, only `meta` and `global-index` types are cached. You can customize this via the `file-cache.whitelist` option.
35+
36+
| File Type | Config Name | Examples | Default Cached |
37+
|-----------|-------------|----------|----------------|
38+
| META | meta | snapshot, schema, manifest, statistics, tag | Yes |
39+
| GLOBAL_INDEX | global-index | BTree, Lumina, Tantivy index files | Yes |
40+
| BUCKET_INDEX | bucket-index | Hash, deletion vector index files | No |
41+
| DATA | data | Data files (ORC, Parquet, etc.) | No |
42+
| FILE_INDEX | file-index | Data-file level bloom filter, bitmap | No |
43+
44+
All file types can be added to the whitelist. The default whitelist is `meta,global-index`.
45+
46+
## Enable Cache
47+
48+
Use `table.copy()` to pass cache options as dynamic parameters:
49+
50+
{{< tabs "enable-cache" >}}
51+
52+
{{< tab "Java" >}}
53+
54+
```java
55+
import org.apache.paimon.table.Table;
56+
57+
import java.util.HashMap;
58+
import java.util.Map;
59+
60+
Table table = catalog.getTable(Identifier.create("my_db", "my_table"));
61+
62+
Map<String, String> options = new HashMap<>();
63+
options.put("file-cache.enabled", "true");
64+
// optional: customize cache directory and limits
65+
options.put("file-cache.dir", "/tmp/paimon-file-cache");
66+
options.put("file-cache.max-size", "2gb");
67+
options.put("file-cache.block-size", "1mb");
68+
69+
// All subsequent reads on this table instance will use the cache
70+
table = table.copy(options);
71+
```
72+
73+
{{< /tab >}}
74+
75+
{{< tab "Python" >}}
76+
77+
```python
78+
table = catalog.get_table("db.my_table")
79+
80+
# Enable cache with dynamic options
81+
table = table.copy({
82+
"file-cache.enabled": "true",
83+
# optional: customize cache directory and limits
84+
"file-cache.dir": "/tmp/paimon-file-cache",
85+
"file-cache.max-size": "2gb",
86+
"file-cache.block-size": "1mb",
87+
})
88+
89+
# All subsequent reads on this table instance will use the cache
90+
```
91+
92+
{{< /tab >}}
93+
94+
{{< /tabs >}}
95+
96+
## Cache Options
97+
98+
| Option | Type | Default | Description |
99+
|--------|------|---------|-------------|
100+
| `file-cache.enabled` | Boolean | false | Whether to enable local disk block cache. |
101+
| `file-cache.dir` | String | `<tmpdir>/paimon-file-cache` | Directory for storing cached blocks. |
102+
| `file-cache.max-size` | MemorySize | unlimited | Maximum total size of the cache. When exceeded, the least recently used blocks are evicted. |
103+
| `file-cache.block-size` | MemorySize | 1 mb | Block size for caching. Files are logically divided into fixed-size blocks and cached independently. |
104+
| `file-cache.whitelist` | String | meta,global-index | Comma-separated list of file types to cache. Supported values: `meta`, `global-index`, `bucket-index`, `data`, `file-index`. |
105+
106+
## How It Works
107+
108+
- Files are logically divided into fixed-size blocks (default 1 MB).
109+
- On the first read, blocks are downloaded from remote storage and saved to local disk.
110+
- Subsequent reads of the same block are served from local disk, skipping remote I/O.
111+
- Cache files are keyed by remote file path and block offset, so they persist across process restarts and can be reused.
112+
- When the cache exceeds `max-size`, the least recently used blocks are evicted automatically.

‎docs/content/pypaimon/global-index.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
title: "Global Index"
3+
weight: 6
4+
type: docs
5+
aliases:
6+
- /pypaimon/global-index.html
7+
---
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
# Global Index
28+
29+
PyPaimon supports querying global indexes built on Data Evolution (append) tables. Three index types are available:
30+
31+
- **BTree Index**: B-tree based index for scalar column lookups. Supports equality, IN, range, and combined predicates.
32+
- **Vector Index (Lumina)**: Approximate nearest neighbor (ANN) index for vector similarity search.
33+
- **Full-Text Index (Tantivy)**: Full-text search index for text retrieval with relevance scoring.
34+
35+
> Global indexes must be built beforehand (e.g., via Spark or Flink). See [Global Index]({{< ref "append-table/global-index" >}}) for how to create indexes.
36+
37+
## BTree Index
38+
39+
BTree index is automatically used during scan when a filter predicate matches the indexed column. No special API is needed — just set a filter on the read builder.
40+
41+
```python
42+
import pypaimon
43+
44+
catalog = pypaimon.create_catalog(...)
45+
table = catalog.get_table("db.my_table")
46+
47+
# BTree index is used automatically when filtering on indexed columns
48+
read_builder = table.new_read_builder()
49+
read_builder = read_builder.with_filter(
50+
pypaimon.PredicateBuilder(table.fields)
51+
.in_("name", ["a200", "a300"])
52+
)
53+
54+
scan = read_builder.new_scan()
55+
read = read_builder.new_read()
56+
splits = scan.plan().splits
57+
data = read.to_arrow(splits)
58+
```
59+
60+
Supported predicates: `equal`, `not_equal`, `less_than`, `less_or_equal`, `greater_than`, `greater_or_equal`, `in_`, `not_in`, `between`, `is_null`, `is_not_null`.
61+
62+
## Vector Index (Lumina)
63+
64+
Use `VectorSearchBuilder` to perform approximate nearest neighbor search on a vector column, then read the matched rows.
65+
66+
```python
67+
table = catalog.get_table("db.my_table")
68+
69+
# Step 1: vector search to get matching row IDs
70+
builder = table.new_vector_search_builder()
71+
index_result = (
72+
builder
73+
.with_vector_column("embedding")
74+
.with_query_vector([1.0, 2.0, 3.0, ...])
75+
.with_limit(10)
76+
.execute_local()
77+
)
78+
79+
# Step 2: read actual data for matched rows
80+
read_builder = table.new_read_builder()
81+
scan = read_builder.new_scan()
82+
scan.with_global_index_result(index_result)
83+
read = read_builder.new_read()
84+
data = read.to_arrow(scan.plan().splits)
85+
```
86+
87+
You can also add a scalar filter to pre-filter rows before vector search:
88+
89+
```python
90+
predicate = (
91+
pypaimon.PredicateBuilder(table.fields)
92+
.equal("category", "electronics")
93+
)
94+
95+
index_result = (
96+
table.new_vector_search_builder()
97+
.with_vector_column("embedding")
98+
.with_query_vector([1.0, 2.0, 3.0, ...])
99+
.with_limit(10)
100+
.with_filter(predicate)
101+
.execute_local()
102+
)
103+
104+
read_builder = table.new_read_builder()
105+
scan = read_builder.new_scan()
106+
scan.with_global_index_result(index_result)
107+
read = read_builder.new_read()
108+
data = read.to_arrow(scan.plan().splits)
109+
```
110+
111+
## Full-Text Index (Tantivy)
112+
113+
Use `FullTextSearchBuilder` to perform full-text search on a text column, then read the matched rows.
114+
115+
```python
116+
table = catalog.get_table("db.my_table")
117+
118+
# Step 1: full-text search to get matching row IDs
119+
builder = table.new_full_text_search_builder()
120+
index_result = (
121+
builder
122+
.with_text_column("content")
123+
.with_query_text("search keywords")
124+
.with_limit(20)
125+
.execute_local()
126+
)
127+
128+
# Step 2: read actual data for matched rows
129+
read_builder = table.new_read_builder()
130+
scan = read_builder.new_scan()
131+
scan.with_global_index_result(index_result)
132+
read = read_builder.new_read()
133+
data = read.to_arrow(scan.plan().splits)
134+
```
135+
136+
For better performance when reading from remote storage, consider enabling the [Local Disk Cache]({{< ref "program-api/file-cache" >}}).

‎docs/layouts/shortcodes/generated/core_configuration.html‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -566,6 +566,36 @@
566566
<td>String</td>
567567
<td>Default aggregate function of all fields for partial-update and aggregate merge function.</td>
568568
</tr>
569+
<tr>
570+
<td><h5>file-cache.block-size</h5></td>
571+
<td style="word-wrap: break-word;">1 mb</td>
572+
<td>MemorySize</td>
573+
<td>Block size for local disk cache.</td>
574+
</tr>
575+
<tr>
576+
<td><h5>file-cache.dir</h5></td>
577+
<td style="word-wrap: break-word;">(none)</td>
578+
<td>String</td>
579+
<td>Directory for file block cache. Defaults to a 'paimon-file-cache' subdirectory under the system temp directory.</td>
580+
</tr>
581+
<tr>
582+
<td><h5>file-cache.enabled</h5></td>
583+
<td style="word-wrap: break-word;">false</td>
584+
<td>Boolean</td>
585+
<td>Whether to enable local disk block cache for file reads.</td>
586+
</tr>
587+
<tr>
588+
<td><h5>file-cache.max-size</h5></td>
589+
<td style="word-wrap: break-word;">(none)</td>
590+
<td>MemorySize</td>
591+
<td>Maximum total size of the local disk block cache. Unlimited by default.</td>
592+
</tr>
593+
<tr>
594+
<td><h5>file-cache.whitelist</h5></td>
595+
<td style="word-wrap: break-word;">"meta,global-index"</td>
596+
<td>String</td>
597+
<td>Comma-separated list of file types to cache. Supported values: meta, global-index, bucket-index, data, file-index.</td>
598+
</tr>
569599
<tr>
570600
<td><h5>file-index.in-manifest-threshold</h5></td>
571601
<td style="word-wrap: break-word;">500 bytes</td>

‎paimon-api/src/main/java/org/apache/paimon/CoreOptions.java‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -710,6 +710,41 @@ public InlineElement getDescription() {
710710
.defaultValue(MemorySize.parse("64 kb"))
711711
.withDescription("Memory page size for caching.");
712712

713+
public static final ConfigOption<Boolean> FILE_CACHE_ENABLED =
714+
key("file-cache.enabled")
715+
.booleanType()
716+
.defaultValue(false)
717+
.withDescription("Whether to enable local disk block cache for file reads.");
718+
719+
public static final ConfigOption<String> FILE_CACHE_DIR =
720+
key("file-cache.dir")
721+
.stringType()
722+
.noDefaultValue()
723+
.withDescription(
724+
"Directory for file block cache. "
725+
+ "Defaults to a 'paimon-file-cache' subdirectory under the system temp directory.");
726+
727+
public static final ConfigOption<MemorySize> FILE_CACHE_MAX_SIZE =
728+
key("file-cache.max-size")
729+
.memoryType()
730+
.noDefaultValue()
731+
.withDescription(
732+
"Maximum total size of the local disk block cache. Unlimited by default.");
733+
734+
public static final ConfigOption<MemorySize> FILE_CACHE_BLOCK_SIZE =
735+
key("file-cache.block-size")
736+
.memoryType()
737+
.defaultValue(MemorySize.ofMebiBytes(1))
738+
.withDescription("Block size for local disk cache.");
739+
740+
public static final ConfigOption<String> FILE_CACHE_WHITELIST =
741+
key("file-cache.whitelist")
742+
.stringType()
743+
.defaultValue("meta,global-index")
744+
.withDescription(
745+
"Comma-separated list of file types to cache. "
746+
+ "Supported values: meta, global-index, bucket-index, data, file-index.");
747+
713748
public static final ConfigOption<MemorySize> TARGET_FILE_SIZE =
714749
key("target-file-size")
715750
.memoryType()
@@ -2887,6 +2922,28 @@ public int cachePageSize() {
28872922
return (int) options.get(CACHE_PAGE_SIZE).getBytes();
28882923
}
28892924

2925+
public boolean fileCacheEnabled() {
2926+
return options.get(FILE_CACHE_ENABLED);
2927+
}
2928+
2929+
@Nullable
2930+
public String fileCacheDir() {
2931+
return options.get(FILE_CACHE_DIR);
2932+
}
2933+
2934+
@Nullable
2935+
public MemorySize fileCacheMaxSize() {
2936+
return options.get(FILE_CACHE_MAX_SIZE);
2937+
}
2938+
2939+
public MemorySize fileCacheBlockSize() {
2940+
return options.get(FILE_CACHE_BLOCK_SIZE);
2941+
}
2942+
2943+
public String fileCacheWhitelist() {
2944+
return options.get(FILE_CACHE_WHITELIST);
2945+
}
2946+
28902947
public MemorySize lookupCacheMaxMemory() {
28912948
return options.get(LOOKUP_CACHE_MAX_MEMORY_SIZE);
28922949
}

0 commit comments

Comments
 (0)