Skip to content

Commit eb7e390

Browse files
ekkolonekkolon
authored
Update README.md file (#18)
1 parent 0f2361c commit eb7e390

1 file changed

Lines changed: 46 additions & 43 deletions

File tree

‎README.md‎

Lines changed: 46 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,34 @@
11
# minikv
22

3-
Minimal, S3-compatible distributed key-value store. Keys and metadata are stored in LevelDB; values (object bytes) live on nginx WebDAV volume servers.
3+
A distributed key-value store written in Rust.
4+
5+
**minikv** stores keys and metadata in LevelDB and object bytes on nginx WebDAV volume servers. Each object is replicated across a configurable number of volumes. The server handles routing, replication, and metadata; nginx handles the data.
46

57
## Architecture
68

79
```txt
810
Client
9-
│
10-
▼
11-
frontend nginx (port 8080) <== X-Accel-Redirect proxy
12-
│ proxy_pass =>
13-
▼
14-
minikv coordinator (port 3000) <== metadata, routing, replication
15-
│ replicates to =>
16-
├── volume1 nginx (port 8080) <== nginx DAV object storage
17-
├── volume2 nginx (port 8080)
18-
└── volume3 nginx (port 8080)
11+
|
12+
v
13+
frontend nginx (port 8080) <- X-Accel-Redirect proxy
14+
| proxy_pass ->
15+
v
16+
minikv server (port 3000) <- metadata, routing, replication
17+
| replicates to ->
18+
+-- volume1 nginx (port 8080) <- nginx DAV object storage
19+
+-- volume2 nginx (port 8080)
20+
+-- volume3 nginx (port 8080)
1921
```
2022

21-
**GET/HEAD flow:** The coordinator looks up the key in LevelDB, probes volume servers to find a live replica, then returns `X-Accel-Redirect` to the frontend nginx. nginx fetches the object body directly from the volume server and streams it to the client - the coordinator is never in the data path. Response headers (`Content-Type`, `Content-Blake3`, `Key-Balance`) come from coordinator metadata.
23+
**GET/HEAD flow:** The server looks up the key in LevelDB, probes volume servers to find a live replica, then returns `X-Accel-Redirect` to the frontend nginx. nginx fetches the object body directly from the volume server and streams it to the client. The server is not in the data path for reads. Response headers (`Content-Type`, `Content-Blake3`, `Key-Balance`) come from stored metadata.
2224

23-
**PUT flow:** The coordinator writes a soft-delete sentinel to LevelDB, replicates the object body to all replica volumes, optionally computes a BLAKE3 checksum, then marks the key as fully present.
25+
**PUT flow:** The server writes a soft-delete sentinel to LevelDB, replicates the object body to all replica volumes, optionally computes a BLAKE3 checksum, then marks the key as fully present.
2426

2527
## Hashing
2628

27-
This project uses **BLAKE3** for all content-addressing and volume selection.
29+
BLAKE3 is used for all content-addressing (`key_to_path`) and volume selection (`key_to_volume`).
2830

29-
> ⚠️ The hash function used for `key_to_path` and `key_to_volume` determines the physical layout of all stored data. Changing it after data is written is a **breaking change** requiring a full rebalance.
31+
> The hash function determines the physical layout of all stored data. Changing it after data is written is a breaking change and requires a full rebalance.
3032
3133
## Record Wire Format
3234

@@ -43,19 +45,20 @@ Each LevelDB value encodes object metadata as a compact byte string:
4345

4446
## HTTP API
4547

46-
| Method | Path | Description |
47-
| ----------- | -------------------- | ------------------------------------------------------------------------ |
48-
| `PUT` | `/<key>` | Store an object. Supply `Content-Type` header for correct MIME metadata. |
49-
| `GET` | `/<key>` | Retrieve an object (via X-Accel-Redirect or 302). |
50-
| `HEAD` | `/<key>` | Returns metadata headers without body. |
51-
| `DELETE` | `/<key>` | Hard delete. Requires prior `UNLINK` when `--protect` is set. |
52-
| `UNLINK` | `/<key>` | Soft delete. |
53-
| `REBALANCE` | `/<key>` | Move a single key to its ideal volume set. |
54-
| `GET` | `/<prefix>?list` | List active keys under prefix. |
55-
| `GET` | `/<prefix>?unlinked` | List soft-deleted keys under prefix. |
56-
| `POST` | `/<key>?uploads` | Initiate S3-style multipart upload. |
57-
| `POST` | `/<key>?uploadId=X` | Complete multipart upload. |
58-
| `POST` | `/<key>?delete` | Batch delete. |
48+
| Method | Path | Description |
49+
| ----------- | -------------------------------- | ----------------------------------------------------------------------- |
50+
| `PUT` | `/<key>` | Store an object. Supply `Content-Type` for correct MIME metadata. |
51+
| `GET` | `/<key>` | Retrieve an object (via X-Accel-Redirect or 302). |
52+
| `HEAD` | `/<key>` | Returns metadata headers without body. |
53+
| `DELETE` | `/<key>` | Hard delete. Requires prior `UNLINK` when `--protect` is set. |
54+
| `UNLINK` | `/<key>` | Soft delete. |
55+
| `REBALANCE` | `/<key>` | Move a single key to its ideal volume set. |
56+
| `GET` | `/<prefix>?list` | List active keys under prefix. Accepts `&start=X` and `&limit=N`. |
57+
| `GET` | `/<prefix>?unlinked` | List soft-deleted keys under prefix. Accepts `&start=X` and `&limit=N`. |
58+
| `GET` | `/<prefix>?list-type=2&prefix=X` | S3-style XML key listing. |
59+
| `POST` | `/<key>?uploads` | Initiate multipart upload. Returns an upload ID. |
60+
| `POST` | `/<key>?uploadId=X` | Complete multipart upload. |
61+
| `POST` | `/<key>?delete` | Batch delete (XML body). |
5962

6063
### Response Headers
6164

@@ -72,9 +75,9 @@ Each LevelDB value encodes object metadata as a compact byte string:
7275
docker compose up --build
7376
```
7477

75-
All services start in dependency order: volume nodes => coordinator => frontend nginx.
78+
Services start in dependency order: volume nodes -> server -> frontend nginx.
7679

77-
The only externally exposed port is **8080** (frontend nginx). Volume nodes and the coordinator are internal to the Docker network.
80+
The only externally exposed port is **8080** (frontend nginx). Volume nodes and the server are internal to the Docker network.
7881

7982
### PUT an object
8083

@@ -84,7 +87,7 @@ curl -X PUT -H "Content-Type: image/png" \
8487
http://localhost:8080/mybucket/photo
8588
```
8689

87-
> Always supply `Content-Type` on PUT. It is stored in LevelDB and returned on all subsequent GET/HEAD requests. Objects stored without `Content-Type` will be served as `application/octet-stream`.
90+
It is recommended to supply a `Content-Type` header on **PUT**. It is stored in LevelDB and returned on subsequent GET/HEAD requests, allowing clients and browsers to handle the response correctly without guessing the format. Objects stored without one will be served as `application/octet-stream`.
8891

8992
### GET an object
9093

@@ -111,9 +114,9 @@ curl -X DELETE http://localhost:8080/mybucket/photo
111114
minikv <COMMAND>
112115
113116
Commands:
114-
server Run the HTTP metadata coordinator
115-
rebuild Reconstruct LevelDB from volume server autoindex
116-
rebalance Move all keys to their ideal volume set
117+
server Run the HTTP server
118+
rebuild Reconstruct LevelDB from volume server autoindex
119+
rebalance Move all keys to their ideal volume set
117120
```
118121

119122
### server
@@ -125,7 +128,7 @@ Commands:
125128
--subvolumes <N> Shard count (default: 10) [env: MINIKV_SUBVOLUMES]
126129
--voltimeout <duration> Volume probe timeout [env: MINIKV_VOLTIMEOUT]
127130
--port <N> Listen port (default: 3000) [env: MINIKV_PORT]
128-
--public-volumes <host:port,…> External volume addresses [env: MINIKV_PUBLIC_VOLUMES]
131+
--public-volumes <host:port,> External volume addresses [env: MINIKV_PUBLIC_VOLUMES]
129132
--fallback <host:port> Fallback for missing keys [env: MINIKV_FALLBACK]
130133
--protect Require UNLINK before DELETE [env: MINIKV_PROTECT]
131134
--checksum Store BLAKE3 digest on PUT [env: MINIKV_CHECKSUM]
@@ -137,31 +140,31 @@ All flags can be set via environment variables. Duration values accept `1s`, `50
137140

138141
### rebuild
139142

140-
Reconstructs LevelDB by scanning nginx autoindex JSON listings on all volume servers. This is a **destructive** operation. It clears the existing DB before scanning. Use when LevelDB is lost but volume data is intact.
143+
Reconstructs LevelDB by scanning nginx autoindex JSON listings on all volume servers. **Destructive**: clears the existing DB before scanning. Use when LevelDB is lost but volume data is intact.
141144

142-
> `Content-Type` metadata cannot be recovered during rebuild. It exists only in LevelDB, never on volume servers. Objects will be served as `application/octet-stream` until re-PUT.
145+
`Content-Type` metadata cannot be recovered during rebuild. It exists only in LevelDB, never on volume servers. Affected objects will be served as `application/octet-stream` until re-PUT with a `Content-Type` header.
143146

144147
### rebalance
145148

146149
Moves all keys to their ideal volume set as computed by the current `--volumes` list. Run after adding or removing volume servers.
147150

148151
## Consistency Model
149152

150-
- **PUT** is atomic at the record level. The key is marked soft-deleted (in-progress sentinel) before any volume write and marked fully present only after all replicas succeed. A crash mid-write leaves a soft-deleted key that can be cleaned up manually.
153+
- **PUT** is atomic at the record level. The key is marked soft-deleted (in-progress sentinel) before any volume write, and marked fully present only after all replicas succeed. A crash mid-write leaves a soft-deleted key that can be cleaned up manually.
151154
- **No read-after-write guarantee across replicas.** GET probes volumes in random order and returns the first live replica.
152155
- **Rebalance** clears the stored hash for the moved object. The body is not re-verified during rebalance.
153156
- **Soft delete (UNLINK)** removes the key from client visibility immediately. The object bytes remain on volume servers until a hard DELETE is issued.
154157

155158
## Content-Type and X-Accel-Redirect
156159

157-
When `--accel-redirect` is enabled the coordinator returns `X-Accel-Redirect` instead of `302`. The frontend nginx intercepts this, fetches the object body from the volume server internally, and sends it to the client. Because the body comes from nginx's internal subrequest (not the coordinator response), headers are injected via nginx variable persistence:
160+
When `--accel-redirect` is enabled the server returns `X-Accel-Redirect` instead of `302`. The frontend nginx intercepts this, fetches the object body from the volume server internally, and sends it to the client. Because the body comes from nginx's internal subrequest rather than the server response, headers are carried across via nginx variable persistence:
158161

159-
1. Coordinator sets `X-Content-Type: image/png` on its response.
160-
2. nginx captures this as `$upstream_http_x_content_type` - a variable that persists across the internal redirect.
162+
1. The server sets `X-Content-Type: image/png` on its response.
163+
2. nginx captures this as `$upstream_http_x_content_type`, a variable that persists across the internal redirect.
161164
3. The `/accel/` location suppresses the volume's `Content-Type` and replaces it with `$upstream_http_x_content_type`.
162165

163-
In plain `302` mode, the coordinator sets `Content-Type` directly and the client receives it on the HEAD response. The GET redirect goes to the volume server which returns `application/octet-stream` - this is a known limitation of redirect mode.
166+
In plain `302` mode, the server sets `Content-Type` on the redirect response and the client receives it on HEAD. The GET redirect goes to the volume server which returns `application/octet-stream` regardless of stored metadata. This is a known limitation of redirect mode.
164167

165168
## License
166169

167-
This project is licensed under the **GNU General Public License v2 (GPLv2)**. See [`LICENSE`](./LICENSE).
170+
GNU General Public License v2 (GPLv2). See [`LICENSE`](./LICENSE).

0 commit comments

Comments
 (0)