Skip to content

Commit 1ab08b4

Browse files
authored
Storage README clarifications and corrections.
Signed-off-by: Patrick Ames <833518+pdames@users.noreply.github.com>
1 parent 3bac4f6 commit 1ab08b4

File tree

1 file changed

+35
-31
lines changed

1 file changed

+35
-31
lines changed

deltacat/storage/README.md

Lines changed: 35 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ DeltaCAT uses a combination of **Immutable IDs** and human-readable **Aliases**
4040
| **Table** | `name` | UUID | ✅ Yes |
4141
| **Table Version** | `version` | `version` | ❌ No |
4242
| **Stream** | `format` | UUID | ❌ No |
43-
| **Partition** | `partition_values` + `scheme` | UUID | ❌ No |
43+
| **Partition** | `partition_values` + `scheme_id` | UUID | ❌ No |
4444
| **Delta** | `stream_position` | `stream_position`| ❌ No |
4545

4646
### Canonical Strings
@@ -61,7 +61,7 @@ SHA-1 hexdigests of canonical strings are used to create **Name Resolution Direc
6161
DeltaCAT uses two types of directories to separate immutable data storage from mutable name resolution,
6262
**Immutable ID Directories** and **Name Resolution Directories**.
6363

64-
## Immutable ID Directories (Data Storage)
64+
## Immutable ID Directories (Metadata Storage)
6565

6666
**Purpose**: Metadata and child object storage.
6767

@@ -71,41 +71,45 @@ Every metadata object has a directory named after its **Immutable ID** (UUID or
7171
This directory structure **never changes**, even when objects are renamed.
7272

7373
```
74-
${CATALOG_ROOT}/${namespace_uuid}/ # Namespace data directory
74+
${CATALOG_ROOT}/${namespace_uuid}/ # Namespace root directory
7575
├── rev/ # Namespace metadata revisions
7676
│ ├── 00000000000000000001_create_<txn_id>.mpk
7777
│ └── 00000000000000000002_update_<txn_id>.mpk
78-
├── ${table_uuid_1}/ # Table 1 data directory
78+
├── ${table_uuid_1}/ # Table root directory
7979
│ ├── rev/ # Table metadata revisions
80-
│ ├── ${table_version_id_1}/ # Table Version 1
81-
│ │ ├── rev/ # Table Version metadata
82-
│ │ ├── ${stream_id_1}/ # Stream (e.g., Parquet format)
83-
│ │ │ ├── rev/ # Stream metadata
84-
│ │ │ └── ${partition_id_1}/ # Partition
85-
│ │ │ ├── rev/ # Partition metadata
86-
│ │ │ └── ${delta_id_1}/ # Delta (data change)
87-
│ │ │ └── rev/ # Delta metadata
88-
│ │ └── ${stream_id_2}/ # Another stream (e.g., Delta format)
89-
│ └── ${table_version_id_2}/ # Table Version 2
90-
└── ${table_uuid_2}/ # Table 2 data directory
80+
│ ├── ${table_version_id_1}/ # Table Version root directory
81+
│ │ ├── rev/ # Table Version metadata revisions
82+
│ │ ├── ${stream_id_1}/ # Stream root directory (e.g., DeltaCAT format)
83+
│ │ │ ├── rev/ # Stream metadata revisions
84+
│ │ │ ├── ${partition_id_1}/ # Partition root directory
85+
│ │ │ │ ├── rev/ # Partition metadata revisions
86+
│ │ │ │ ├── ${delta_id_1}/ # Delta root directory (data change)
87+
│ │ │ │ │ └── rev/ # Delta metadata revisions
88+
│ │ │ │ └── ${delta_id_2}/ # Another delta
89+
│ │ │ └── ${partition_id_2}/ # Another partition
90+
│ │ └── ${stream_id_2}/ # Another Stream (e.g., Iceberg format)
91+
│ └── ${table_version_id_2}/ # Another Table Version
92+
└── ${table_uuid_2}/ # Another Table
9193
```
9294

9395
### Metadata Revision Files
9496

9597
Each `rev/` directory contains **Metadata Revision Files** stored in partitioned subdirectories using a **2-level exponential partition transform with base 1000**:
9698

97-
**Directory Structure**: `rev/<base^1>/<base^0>/<revision_file>`
99+
**Directory Structure**: `rev/<N*base^2>/<K*base^1>/<revision_file>`
98100
- Level 1: Multiples of 1000000 (1000^2): `1000000/`, `2000000/`, `3000000/`...
99101
- Level 2: Multiples of 1000 (1000^1): `1000/`, `2000/`, `3000/`...
100-
- **Example**: Revision 1234567 → `rev/2000000/1235000/00000000000001234567_create_<txn_id>`
102+
103+
**Example**: Revision 1234567 → `rev/2000000/1235000/00000000000001234567_create_<txn_id>`
101104

102105
**File Format**: `<revision_number>_<operation>_<transaction_id>`
103106
- `revision_number`: Zero-padded 20 digits (e.g., `00000000000000000001`)
104107
- `operation`: `create`, `update`, or `delete`
105108
- `transaction_id`: Unique transaction UUID
106-
- **Content Type**: File contents are written using **MessagePack**
107109

108-
Simultaneous updates to the same `rev/` partition directory result in a concurrent modification conflict, but any number of different `rev/` partition directories may be modified simultaneously.
110+
**File Content Type**: [MessagePack](https://github.com/msgpack/msgpack)
111+
112+
**Isolation**: Simultaneous updates to the same `rev/` partition directory result in a concurrent modification conflict, but any number of different `rev/` partition directories may be modified simultaneously.
109113

110114
## Name Resolution Directories (Mutable Name Mapping)
111115

@@ -127,7 +131,7 @@ ${CATALOG_ROOT}/<namespace_uuid>/<table_name_sha1>/ # Maps "my_table" → table
127131
└── ...
128132
```
129133

130-
Simultaneous updates to the same name resolution directory results in a concurrent modification conflict, but any number of different name resolution directories may be modified simultaneously.
134+
**Isolation**: Simultaneous updates to the same name resolution directory results in a concurrent modification conflict, but any number of different name resolution directories may be modified simultaneously.
131135

132136
### Name Mapping Files
133137

@@ -171,12 +175,12 @@ ${CATALOG_ROOT}/<namespace_uuid>/<customers_sha1>/ # New name
171175
**Purpose**: Track all successful, failed, running, and paused catalog transactions.
172176

173177
**Location**:
174-
- `${CATALOG_ROOT}/txn/success/`: successful transactions
175-
- `${CATALOG_ROOT}/txn/failed/`: failed transactions
176-
- `${CATALOG_ROOT}/txn/running/`: running transactions
177-
- `${CATALOG_ROOT}/txn/paused/`: paused transactions
178+
- `${CATALOG_ROOT}/txn/success/`: successful transactions
179+
- `${CATALOG_ROOT}/txn/failed/`: failed transactions
180+
- `${CATALOG_ROOT}/txn/running/`: running transactions
181+
- `${CATALOG_ROOT}/txn/paused/`: paused transactions
178182

179-
**Sucess** and **Failed** transaction files are stored in directories partitioned by each transaction's **UTC start time**:
183+
Transaction files are stored in directories partitioned by each transaction's **UTC start time**:
180184

181185
- **Directory Structure**: `txn/<status>/<year>/<month>/<day>/<hour>/<minute>/<transaction_id>`
182186
- Year: 4 digits (e.g., `2025/`)
@@ -185,6 +189,12 @@ ${CATALOG_ROOT}/<namespace_uuid>/<customers_sha1>/ # New name
185189
- Hour: 2 digits (e.g., `14/`)
186190
- Minute: 2 digits (e.g., `30/`)
187191

192+
**Transaction Log File Format by Status**:
193+
- **Success**: Transaction ID directories contain a single file with transaction metadata whose name is the transaction end time.
194+
- `${CATALOG_ROOT}/txn/success/<year>/<month>/<day>/<hour>/<minute>/<transaction_id>/<transaction_end_time_epoch_ns>`
195+
- **Failed/Running/Paused**: Transaction ID files with transaction metadata.
196+
- `${CATALOG_ROOT}/txn/{status}/<year>/<month>/<day>/<hour>/<minute>/<transaction_id>`
197+
188198
**Transaction ID Format**: `<start_time_epoch_ns>_<uuid>`
189199

190200
```
@@ -206,12 +216,6 @@ ${CATALOG_ROOT}/txn/
206216
│ └── 1754104438234567000_e2f3g4h5-6i7j-8k9l-0m1n-o2p3q4r5s6t7u8
207217
```
208218

209-
**Transaction Log File Format by State**:
210-
- **Success**: Transaction ID directories contain a single file with transaction metadata whose name is the transaction end time.
211-
- `<transaction_id>/<transaction_end_time_epoch_ns>`
212-
- **Failed/Running/Paused**: Transaction ID files with transaction metadata.
213-
- `<transaction_id>`
214-
215219
DeltaCAT transactions rely on MVCC snapshot isolation, with conflicts isolated to concurrent operations against the same object ID. Each transaction log contains details about the operations performed by that transaction.
216220

217221
Transactions transition from RUNNING → SUCCESS/PAUSED/FAILED states, with the transaction stored in an equivalently named parent directory at `${CATALOG_ROOT}/txn/${state}/`.

0 commit comments

Comments
 (0)