You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/usage/working-with-partitions.md
+79-40
Original file line number
Diff line number
Diff line change
@@ -7,8 +7,7 @@ Below, we demonstrate how to create, query, and update partitioned Delta tables,
7
7
8
8
## Creating a Partitioned Table
9
9
10
-
To create a partitioned Delta table, specify one or more partition columns when writing the data. If you’re using Python, pass `partition_by=[<column>]` to the [write_deltalake()][deltalake.write_deltalake] function. In Rust, you can use `with_partition_columns(...)` on the builder when creating the table.
11
-
10
+
To create a partitioned Delta table, specify one or more partition columns when creating the table. Here we partition by the country column.
12
11
```python
13
12
from deltalake import write_deltalake
14
13
import pandas as pd
@@ -22,7 +21,8 @@ df = pd.DataFrame({
22
21
# Create a table partitioned by the "country" column
The structure in the “tmp/partitioned-table” folder is showing how Delta Lake organizes data by the partition column. The “_delta_log” folder holds transaction metadata, while each “country=<value>” subfolder contains the Parquet files for rows matching that partition value. This layout allows efficient queries and updates on partitioned data.
24
+
25
+
The structure in the "tmp/partitioned-table" folder shows how Delta Lake organizes data by the partition column. The "_delta_log" folder holds transaction metadata, while each "country=<value>" subfolder contains the Parquet files for rows matching that partition value. This layout allows efficient queries and updates on partitioned data.
26
26
```plaintext
27
27
tmp/partitioned-table/
28
28
├── _delta_log/
@@ -37,14 +37,13 @@ tmp/partitioned-table/
37
37
38
38
### Filtering by partition columns
39
39
40
-
Because partition columns are part of the storage path, queries that filter on those columns can skip reading unneeded partitions. You can specify partition filters when reading data with [DeltaTable.to_pandas()][deltalake.table.DeltaTable.to_pandas], [DeltaTable.to_pyarrow_table()][deltalake.table.DeltaTable.to_pyarrow_table], or [DeltaTable.to_pyarrow_dataset()][deltalake.table.DeltaTable.to_pyarrow_dataset].
40
+
Because partition columns are part of the storage path, queries that filter on those columns can skip reading unneeded partitions. You can specify partition filters when reading data with [DeltaTable.to_pandas()](../../delta_table/#deltalake.DeltaTable.to_pandas).
41
41
42
-
```python
43
-
from deltalake import DeltaTable
44
42
43
+
In this example we restrict our query to the `country="US"` partition.
44
+
```python
45
45
dt = DeltaTable("tmp/partitioned-table")
46
46
47
-
# Only read files from partitions where country = 'US'
48
47
pdf = dt.to_pandas(partitions=[("country", "=", "US")])
49
48
print(pdf)
50
49
```
@@ -56,11 +55,9 @@ print(pdf)
56
55
57
56
### Partition Columns in Table Metadata
58
57
59
-
Partition columns can also be inspected via metadata:
58
+
Partition columns can also be inspected via metadata on a `DeltaTable`.
You can simply write additional data with mode="append" and the partition columns will be used to place data in the correct partition directories.
73
+
You can write additional data to partitions (or create new partitions) with `mode="append"` and the partition columns will be used to place data in the correct partition directories.
You can overwrite a specific partition, leaving the other partitions intact. Pass in `mode="overwrite"` together with a predicate string.
102
+
103
+
In this example we overwrite the `DE` paritition with new data.
90
104
91
-
You can overwrite a specific partition, leaving the other partitions intact. Pass in mode="overwrite" together with partition_filters.
92
105
```python
93
106
df_overwrite = pd.DataFrame({
94
107
"num": [900, 1000],
95
108
"letter": ["m", "n"],
96
109
"country": ["DE", "DE"]
97
110
})
98
111
99
-
from deltalake import DeltaTable, write_deltalake
100
-
101
112
dt = DeltaTable("tmp/partitioned-table")
102
113
write_deltalake(
103
114
dt,
104
115
df_overwrite,
105
-
partition_filters=[("country", "=", "DE")],
116
+
predicate="country = 'DE'",
106
117
mode="overwrite",
107
118
)
108
-
```
109
-
This will remove only the `country=DE` partition files and overwrite them with the new data.
110
119
111
-
### Overwriting Parts of the Table Using a Predicate
112
-
113
-
If you have a more fine-grained predicate than a partition filter, you can use the [predicate argument][deltalake.write_deltalake] (sometimes called replaceWhere) to overwrite only rows matching a specific condition.
120
+
dt = DeltaTable("tmp/partitioned-table")
121
+
pdf = dt.to_pandas()
122
+
print(pdf)
123
+
```
114
124
115
-
(See the “Overwriting part of the table data using a predicate” section in the Writing Delta Tables docs for more details.)
125
+
```plaintext
126
+
num letter country
127
+
0 900 m DE
128
+
1 1000 n DE
129
+
2 10 x CA
130
+
3 3 c CA
131
+
4 1 a US
132
+
5 2 b US
133
+
```
116
134
117
135
## Updating Partitioned Tables with Merge
118
136
119
137
You can perform merge operations on partitioned tables in the same way you do on non-partitioned ones. Simply provide a matching predicate that references partition columns if needed.
120
138
121
-
You can match on both the partition column (country) and some other condition. This example shows a merge operation that checks both the partition column (“country”) and a numeric column (“num”) when merging:
122
-
- The table is partitioned by “country,” so underlying data is physically split by each country value.
123
-
- The merge condition (predicate) matches target rows where both “country” and “num” align with the source.
124
-
- When a match occurs, it updates “letter”; otherwise, it inserts the new row.
125
-
- This approach ensures that only rows in the relevant partition (“US”) are processed, keeping operations efficient.
139
+
You can match on both the partition column (country) and some other condition. This example shows a merge operation that checks both the partition column ("country") and a numeric column ("num") when merging:
140
+
- The merge condition (predicate) matches target rows where both "country" and "num" align with the source.
141
+
- When a match occurs, it updates the "letter" column; otherwise, it inserts the new row.
126
142
127
143
```python
128
-
from deltalake import DeltaTable
129
-
import pyarrow as pa
130
-
131
144
dt = DeltaTable("tmp/partitioned-table")
132
145
133
-
# New data that references an existing partition "US"
This approach ensures that only rows in the relevant partition ("US") are processed, keeping operations efficient.
179
+
151
180
## Deleting Partition Data
152
181
153
182
You may want to delete all rows from a specific partition. For example:
154
183
```python
155
184
dt = DeltaTable("tmp/partitioned-table")
156
185
157
-
# Delete all rows from the 'US' partition:
158
186
dt.delete("country = 'US'")
187
+
188
+
dt = DeltaTable("tmp/partitioned-table")
189
+
pdf = dt.to_pandas()
190
+
print(pdf)
191
+
```
192
+
193
+
```plaintext
194
+
num letter country
195
+
0 900 m DE
196
+
1 1000 n DE
197
+
2 10 x CA
198
+
3 3 c CA
159
199
```
160
200
This command logically deletes the data by creating a new transaction.
161
201
162
202
## Maintaining Partitioned Tables
163
203
164
204
### Optimize & Vacuum
165
205
166
-
Partitioned tables can accummulate many small files if a partition is frequently appended to. You can compact these into larger files on a specific partition:
206
+
Partitioned tables can accummulate many small files if a partition is frequently appended to. You can compact these into larger files on a specific partition with [`optimize.compact`](../../delta_table/#deltalake.DeltaTable.optimize).
Then optionally vacuum the table to remove older, unreferenced files.
211
+
Then optionally [`vacuum`](../../delta_table/#deltalake.DeltaTable.vacuum) the table to remove older, unreferenced files.
172
212
173
213
### Handling High-Cardinality Columns
174
214
175
-
Partitioning can be very powerful, but be mindful of using high-cardinality columns (columns with too many unique values). This can create an excessive number of directories and can hurt performance. For example, partitioning by date is typically better than partitioning by user_id if user_id has
176
-
215
+
Partitioning can be very powerful, but be mindful of using high-cardinality columns (columns with too many unique values). This can create an excessive number of directories and can hurt performance. For example, partitioning by date is typically better than partitioning by user_id if user_id has millions of unique values.
0 commit comments