Skip to content

Commit 58043d0

Browse files
lukekimlukekim
committed
v0.2.0: SMB2 compounding, small-file fast paths, tests, and benchmarks
SMB2 compound operations batch multiple SMB operations into single TCP round trips, cutting latency for small file reads (3 RT -> 1 RT), writes (5+ RT -> 2 RT), head/delete (2 RT -> 1 RT), and directory creation (2N RT -> 1 RT). Router fast paths serve small GETs inline via compound Create+Read+Close and small PUTs via compound Create+Write+Close, bypassing the streaming path overhead. Additional changes: - Add --version flag to the binary - Add port-in-use handling to test scripts - Add CI step to verify binary version and kill stale processes - Add extended integration test building spiceai repo through sccache - Add unit tests across all modules (100 total) - Fix parse_http_date misrouting RFC 7231 dates to ISO 8601 parser - Add criterion benchmarks (crypto, protocol, s3) - Optimize epoch_to_iso8601/http_date from 150ns to 18ns (8x) - Add zero-copy decode_read_response_owned for the hot read path - Bump version to 0.2.0, license to Apache 2.0 - Rewrite README with architecture, use cases, and sccache example
1 parent 65a27c1 commit 58043d0

23 files changed

Lines changed: 3446 additions & 329 deletions

‎.github/workflows/ci.yml‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
name: CI
2+
3+
on:
4+
pull_request:
5+
types: [opened, synchronize, reopened]
6+
7+
concurrency:
8+
group: ${{ github.workflow }}-${{ github.ref }}
9+
cancel-in-progress: true
10+
11+
env:
12+
CARGO_TERM_COLOR: always
13+
CARGO_REGISTRIES_CRATES_IO_PROTOCOL: sparse
14+
CARGO_NET_GIT_FETCH_WITH_CLI: true
15+
16+
jobs:
17+
ci:
18+
name: Format, lint, build, and test
19+
# Self-hosted macOS runner required for CommonCrypto FFI and NAS access.
20+
# Skip fork PRs to prevent untrusted code execution on self-hosted runners.
21+
if: github.event.pull_request.head.repo.full_name == github.repository
22+
runs-on: spiceai-macos
23+
permissions:
24+
contents: read
25+
26+
steps:
27+
- name: Check out repository
28+
uses: actions/checkout@v4
29+
30+
- name: Set up Rust toolchain
31+
uses: dtolnay/rust-toolchain@stable
32+
with:
33+
components: rustfmt,clippy
34+
35+
- name: Cache Rust build artifacts
36+
uses: Swatinem/rust-cache@v2
37+
38+
- name: Check formatting
39+
run: cargo fmt --all --check
40+
41+
- name: Run cargo check
42+
run: cargo check --locked --all-targets --all-features
43+
44+
- name: Run clippy
45+
run: cargo clippy --locked --all-targets --all-features -- -D warnings -D clippy::all -D clippy::cargo -A clippy::cargo-common-metadata
46+
47+
- name: Check rustdoc
48+
run: RUSTDOCFLAGS="-D warnings" cargo doc --locked --workspace --no-deps --document-private-items
49+
50+
- name: Build debug binary
51+
run: cargo build --locked
52+
53+
- name: Run unit tests
54+
run: cargo test --locked
55+
56+
- name: Check SMB credentials
57+
run: |
58+
if [[ -n "$SPICEIO_SMB_PASS" ]]; then
59+
echo "HAS_SMB_PASS=true" >> "$GITHUB_ENV"
60+
fi
61+
env:
62+
SPICEIO_SMB_PASS: ${{ secrets.UNAS_SMB_PASS }}
63+
64+
- name: Install AWS CLI
65+
if: ${{ env.HAS_SMB_PASS == 'true' }}
66+
run: |
67+
if ! command -v aws &>/dev/null; then
68+
brew install awscli
69+
fi
70+
71+
- name: Verify spiceio binary version
72+
run: |
73+
BUILT_VERSION=$(./target/debug/spiceio --version)
74+
echo "Built: ${BUILT_VERSION}"
75+
echo "SPICEIO_BUILT_VERSION=${BUILT_VERSION}" >> "$GITHUB_ENV"
76+
77+
# If an older spiceio is already listening (e.g. from a previous run),
78+
# kill it so the test starts the freshly-built binary.
79+
for PORT in 18333 18334; do
80+
STALE_PID=$(lsof -i ":${PORT}" -sTCP:LISTEN -t 2>/dev/null || true)
81+
if [[ -n "$STALE_PID" ]]; then
82+
echo "Stale process on :${PORT} (pid ${STALE_PID}), killing..."
83+
kill "$STALE_PID" 2>/dev/null || true
84+
fi
85+
done
86+
87+
- name: Run sccache integration test
88+
# Skipped when UNAS_SMB_PASS secret is not configured (e.g. fork PRs)
89+
if: ${{ env.HAS_SMB_PASS == 'true' }}
90+
env:
91+
SPICEIO_SMB_SERVER: ${{ vars.SPICEIO_SMB_SERVER || '192.168.3.148' }}
92+
SPICEIO_SMB_USER: ${{ vars.SPICEIO_SMB_USER || 'runner' }}
93+
SPICEIO_SMB_PASS: ${{ secrets.UNAS_SMB_PASS }}
94+
SPICEIO_SMB_SHARE: ${{ vars.SPICEIO_SMB_SHARE || 'ai_platform_dev' }}
95+
SPICEIO_BUCKET: ${{ vars.SPICEIO_BUCKET || 'spiceio' }}
96+
SPICEIO_REGION: ${{ vars.SPICEIO_REGION || 'us-west-1' }}
97+
run: ./scripts/test-sccache.sh
98+
99+
- name: Build release artifact
100+
run: cargo build --release --locked --bin spiceio
101+
102+
- name: Upload release artifact
103+
uses: actions/upload-artifact@v4
104+
with:
105+
name: spiceio-${{ runner.os }}-${{ runner.arch }}
106+
path: target/release/spiceio
107+
if-no-files-found: error

‎CLAUDE.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
44

5-
## What is spio
5+
## What is spiceio
66

7-
spio is an S3-compatible API proxy that translates S3 HTTP requests into SMB 3.1.x file operations. It speaks the SMB wire protocol directly over TCP (no mount, no libsmbclient) and uses macOS CommonCrypto via FFI for all cryptographic primitives (NTLMv2 auth, SHA-256, HMAC). Targets macOS 26+ only.
7+
spiceio is an S3-compatible API proxy that translates S3 HTTP requests into SMB 3.1.x file operations. It speaks the SMB wire protocol directly over TCP (no mount, no libsmbclient) and uses macOS CommonCrypto via FFI for all cryptographic primitives (NTLMv2 auth, SHA-256, HMAC). Targets macOS 26+ only.
88

99
## Design principles
1010

@@ -19,27 +19,27 @@ spio is an S3-compatible API proxy that translates S3 HTTP requests into SMB 3.1
1919
make # fmt + lint + test + build (default target)
2020
make release # optimized release build
2121
make lint # fmt-check + check + strict clippy + rustdoc warnings
22-
make test # sccache integration test (requires SPIO_SMB_USER/PASS)
22+
make test # sccache integration test (requires SPICEIO_SMB_USER/PASS)
2323
make fmt # auto-format
2424
make clean # cargo clean
2525
```
2626

2727
The binary requires these environment variables:
28-
- `SPIO_SMB_SERVER` (required) — SMB server hostname or IP
29-
- `SPIO_SMB_USER` (required) — SMB username
30-
- `SPIO_SMB_PASS` (required) — SMB password
31-
- `SPIO_SMB_SHARE` (required) — SMB share name
32-
- `SPIO_BIND` — listen address (default `0.0.0.0:8333`)
33-
- `SPIO_SMB_PORT` — SMB port (default `445`)
34-
- `SPIO_SMB_DOMAIN` — SMB domain (default empty)
35-
- `SPIO_BUCKET` — virtual S3 bucket name (defaults to `SPIO_SMB_SHARE`)
36-
- `SPIO_REGION` — AWS region to advertise (default `us-east-1`)
28+
- `SPICEIO_SMB_SERVER` (required) — SMB server hostname or IP
29+
- `SPICEIO_SMB_USER` (required) — SMB username
30+
- `SPICEIO_SMB_PASS` (required) — SMB password
31+
- `SPICEIO_SMB_SHARE` (required) — SMB share name
32+
- `SPICEIO_BIND` — listen address (default `0.0.0.0:8333`)
33+
- `SPICEIO_SMB_PORT` — SMB port (default `445`)
34+
- `SPICEIO_SMB_DOMAIN` — SMB domain (default empty)
35+
- `SPICEIO_BUCKET` — virtual S3 bucket name (defaults to `SPICEIO_SMB_SHARE`)
36+
- `SPICEIO_REGION` — AWS region to advertise (default `us-east-1`)
3737

3838
## Architecture
3939

4040
The codebase has three modules:
4141

42-
- **`s3`** — HTTP layer. Parses incoming S3 API requests and produces XML responses. `router.rs` is the central dispatch (path-style bucket routing). Covers GetObject, PutObject, CopyObject, DeleteObject, HeadObject, ListObjectsV1/V2, multipart uploads, and stub endpoints for ACL/tagging/versioning. `xml.rs` is a hand-rolled XML builder. `multipart.rs` manages upload state in-memory, with parts stored as temp files under `.spio-uploads/` on the SMB share. `body.rs` implements `SpioBody`, a zero-copy streaming response body (channel-backed for large reads, inline for XML/errors).
42+
- **`s3`** — HTTP layer. Parses incoming S3 API requests and produces XML responses. `router.rs` is the central dispatch (path-style bucket routing). Covers GetObject, PutObject, CopyObject, DeleteObject, HeadObject, ListObjectsV1/V2, multipart uploads, and stub endpoints for ACL/tagging/versioning. `xml.rs` is a hand-rolled XML builder. `multipart.rs` manages upload state in-memory, with parts stored as temp files under `.spiceio-uploads/` on the SMB share. `body.rs` implements `SpiceioBody`, a zero-copy streaming response body (channel-backed for large reads, inline for XML/errors).
4343

4444
- **`smb`** — Wire protocol client. `protocol.rs` defines SMB 3.1.x packet structures (little-endian). `client.rs` manages the TCP connection, negotiate/session-setup handshake, and exposes operations (tree connect, create, read, write, close, query directory). `auth.rs` implements NTLMv2 challenge-response. `ops.rs` provides the high-level `ShareSession` abstraction the S3 layer consumes (list, read, write, delete, stat, copy).
4545

@@ -51,7 +51,7 @@ The codebase has three modules:
5151

5252
- Zero external crypto dependencies — all crypto goes through `crypto::ffi` to CommonCrypto.
5353
- No `async-trait` — the SMB client uses `tokio::sync::Mutex` around the TCP stream with manual `async` methods.
54-
- GetObject streams SMB read chunks directly to the HTTP response via `SpioBody::channel` — no full-file buffering.
54+
- GetObject streams SMB read chunks directly to the HTTP response via `SpiceioBody::channel` — no full-file buffering.
5555
- PutObject streams HTTP request body chunks directly to SMB write calls — no full-body collection.
5656
- Body is collected into `Bytes` only for operations that require the full payload (multi-delete, multipart complete, upload-part for ETag hashing).
5757
- S3 path-style addressing only (no virtual-hosted-style).

0 commit comments

Comments
 (0)