Skip to content

Commit 65a27c1

Browse files
committed
Add streaming S3 transfers and sccache test flow
SMB protocol: - Fix SPNEGO DER encoding for payloads >127 bytes - Fix Create/Write request offset calculations (off-by-one) - Fix Create/Write response minimum size checks - Handle STATUS_PENDING by looping for final response - Offer SMB 3.0.0/3.0.2 dialect fallbacks in negotiate - Add NTLMSSP Version field and echo server negotiate flags - Cap max read/write to 64KB to avoid oversized messages SMB3 message signing: - SHA-512 and AES-128-CMAC via CommonCrypto FFI - SP800-108 KDF signing key derivation from session base key - Preauth integrity hash tracking through handshake - Sign all post-auth messages with AES-128-CMAC S3 layer: - Map SMB NotFound → S3 NoSuchKey (404), PermissionDenied → 403 - Streaming PutObject via channel-backed body Test harness: - make test runs sccache integration against SMB share - Cold build populates cache, warm build verifies 100% hit rate
1 parent 3caccc2 commit 65a27c1

16 files changed

Lines changed: 487 additions & 678 deletions

File tree

CLAUDE.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ spio is an S3-compatible API proxy that translates S3 HTTP requests into SMB 3.1
1818
```bash
1919
make # fmt + lint + test + build (default target)
2020
make release # optimized release build
21-
make lint # cargo fmt --check + cargo clippy
21+
make lint # fmt-check + check + strict clippy + rustdoc warnings
2222
make test # sccache integration test (requires SPIO_SMB_USER/PASS)
2323
make fmt # auto-format
2424
make clean # cargo clean
@@ -39,11 +39,11 @@ The binary requires these environment variables:
3939

4040
The codebase has three modules:
4141

42-
- **`s3`** — HTTP layer. Parses incoming S3 API requests and produces XML responses. `router.rs` is the central dispatch (path-style bucket routing). Covers GetObject, PutObject, CopyObject, DeleteObject, HeadObject, ListObjectsV1/V2, multipart uploads, and stub endpoints for ACL/tagging/versioning. Auth is SigV4 (header + presigned URL) in `auth.rs`. `xml.rs` is a hand-rolled XML builder. `multipart.rs` manages upload state in-memory, with parts stored as temp files under `.spio-uploads/` on the SMB share. `body.rs` implements `SpioBody`, a zero-copy streaming response body (channel-backed for large reads, inline for XML/errors).
42+
- **`s3`** — HTTP layer. Parses incoming S3 API requests and produces XML responses. `router.rs` is the central dispatch (path-style bucket routing). Covers GetObject, PutObject, CopyObject, DeleteObject, HeadObject, ListObjectsV1/V2, multipart uploads, and stub endpoints for ACL/tagging/versioning. `xml.rs` is a hand-rolled XML builder. `multipart.rs` manages upload state in-memory, with parts stored as temp files under `.spio-uploads/` on the SMB share. `body.rs` implements `SpioBody`, a zero-copy streaming response body (channel-backed for large reads, inline for XML/errors).
4343

44-
- **`smb`** — Wire protocol client. `protocol.rs` defines SMB 3.1.x packet structures (little-endian). `client.rs` manages the TCP connection, negotiate/session-setup handshake, and exposes operations (tree connect, create, read, write, close, query directory, query info). `auth.rs` implements NTLMv2 challenge-response. `ops.rs` provides the high-level `ShareSession` abstraction the S3 layer consumes (list, read, write, delete, stat, copy).
44+
- **`smb`** — Wire protocol client. `protocol.rs` defines SMB 3.1.x packet structures (little-endian). `client.rs` manages the TCP connection, negotiate/session-setup handshake, and exposes operations (tree connect, create, read, write, close, query directory). `auth.rs` implements NTLMv2 challenge-response. `ops.rs` provides the high-level `ShareSession` abstraction the S3 layer consumes (list, read, write, delete, stat, copy).
4545

46-
- **`crypto`** — FFI bindings to macOS CommonCrypto (`Security.framework`/`libcommonCrypto`). Exposes MD4, SHA-256, HMAC-MD5, HMAC-SHA256. No Rust crypto crates.
46+
- **`crypto`** — FFI bindings to macOS CommonCrypto (`Security.framework`/`libcommonCrypto`). Exposes MD4, SHA-256, and HMAC-MD5. No Rust crypto crates.
4747

4848
**Request flow:** HTTP request → `s3::router::handle_request` → S3 operation → `smb::ops::ShareSession` method → `smb::client::SmbClient` wire operations → TCP to SMB server.
4949

Cargo.toml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,17 @@ http = "1"
2020
bytes = "1"
2121
percent-encoding = "2"
2222

23+
[lints.rust]
24+
warnings = "deny"
25+
unsafe_op_in_unsafe_fn = "deny"
26+
unused_lifetimes = "deny"
27+
unused_qualifications = "deny"
28+
29+
[lints.clippy]
30+
all = { level = "deny", priority = -1 }
31+
cargo = { level = "deny", priority = -1 }
32+
cargo_common_metadata = "allow"
33+
2334
[profile.release]
2435
opt-level = 3
2536
lto = "fat"

Makefile

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,29 @@
1-
.PHONY: build release check fmt lint clippy test clean all
1+
.PHONY: build release check fmt fmt-check clippy doc lint test clean all
22

33
all: fmt lint test build
44

55
build:
6-
cargo build
6+
cargo build --locked --all-targets --all-features
77

88
release:
9-
cargo build --release
9+
cargo build --release --locked
1010

1111
check:
12-
cargo check
12+
cargo check --locked --all-targets --all-features
1313

1414
fmt:
15-
cargo fmt
15+
cargo fmt --all
1616

1717
fmt-check:
18-
cargo fmt --check
18+
cargo fmt --all --check
1919

2020
clippy:
21-
cargo clippy -- -W clippy::all
21+
cargo clippy --locked --all-targets --all-features -- -D warnings -D clippy::all -D clippy::cargo -A clippy::cargo-common-metadata
2222

23-
lint: fmt-check clippy
23+
doc:
24+
RUSTDOCFLAGS="-D warnings" cargo doc --locked --workspace --no-deps --document-private-items
25+
26+
lint: fmt-check check clippy doc
2427

2528
test: build
2629
./scripts/test-sccache.sh

README.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,6 @@ S3-compatible API proxy to SMB file shares. Translates S3 HTTP requests into SMB
1010

1111
GetObject (range + conditional), PutObject, CopyObject, DeleteObject, HeadObject, ListObjectsV1/V2, ListBuckets, multipart uploads (create/upload-part/complete/abort/list-parts/list-uploads), HeadBucket, GetBucketLocation, and stubs for ACL, tagging, and versioning.
1212

13-
Auth: AWS Signature V4 (Authorization header and presigned URLs).
14-
1513
## Build
1614

1715
Requires Rust (edition 2024) and macOS 26+.
@@ -20,7 +18,7 @@ Requires Rust (edition 2024) and macOS 26+.
2018
make # fmt + lint + test + build
2119
make release # optimized release build
2220
make test # run tests
23-
make lint # cargo fmt --check + cargo clippy
21+
make lint # fmt-check + check + strict clippy + rustdoc warnings
2422
```
2523

2624
## Configuration

src/crypto/ffi.rs

Lines changed: 132 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,33 @@ use std::os::raw::{c_uint, c_void};
66
const CC_MD4_DIGEST_LENGTH: usize = 16;
77
const CC_MD5_DIGEST_LENGTH: usize = 16;
88
const CC_SHA256_DIGEST_LENGTH: usize = 32;
9+
const CC_SHA512_DIGEST_LENGTH: usize = 64;
910

1011
// HMAC algorithm identifiers (from CommonCrypto/CommonHMAC.h)
1112
const K_CC_HMAC_ALG_MD5: c_uint = 1;
1213
const K_CC_HMAC_ALG_SHA256: c_uint = 2;
1314

15+
// CCCrypt constants
16+
const K_CC_ENCRYPT: u32 = 0;
17+
const K_CC_ALGORITHM_AES128: u32 = 0;
18+
const K_CC_OPTION_ECB_MODE: u32 = 2;
19+
20+
const AES_BLOCK_SIZE: usize = 16;
21+
1422
unsafe extern "C" {
1523
// CC_MD4
1624
fn CC_MD4(data: *const c_void, len: c_uint, md: *mut u8) -> *mut u8;
1725

1826
// CC_MD5
27+
#[cfg(test)]
1928
fn CC_MD5(data: *const c_void, len: c_uint, md: *mut u8) -> *mut u8;
2029

2130
// CC_SHA256
2231
fn CC_SHA256(data: *const c_void, len: c_uint, md: *mut u8) -> *mut u8;
2332

33+
// CC_SHA512
34+
fn CC_SHA512(data: *const c_void, len: c_uint, md: *mut u8) -> *mut u8;
35+
2436
// CCHmac
2537
fn CCHmac(
2638
algorithm: c_uint,
@@ -30,6 +42,21 @@ unsafe extern "C" {
3042
data_length: usize,
3143
mac_out: *mut c_void,
3244
);
45+
46+
// CCCrypt — single-shot encrypt/decrypt
47+
fn CCCrypt(
48+
op: u32,
49+
alg: u32,
50+
options: u32,
51+
key: *const c_void,
52+
key_length: usize,
53+
iv: *const c_void,
54+
data_in: *const c_void,
55+
data_in_length: usize,
56+
data_out: *mut c_void,
57+
data_out_available: usize,
58+
data_out_moved: *mut usize,
59+
) -> i32;
3360
}
3461

3562
/// Compute MD4 digest. Used in NTLM password hashing.
@@ -45,7 +72,8 @@ pub fn md4(data: &[u8]) -> [u8; 16] {
4572
out
4673
}
4774

48-
/// Compute MD5 digest. Used in NTLMv2 computations.
75+
#[cfg(test)]
76+
/// Compute MD5 digest. Used in tests for the CommonCrypto binding.
4977
pub fn md5(data: &[u8]) -> [u8; 16] {
5078
let mut out = [0u8; CC_MD5_DIGEST_LENGTH];
5179
unsafe {
@@ -87,7 +115,20 @@ pub fn sha256(data: &[u8]) -> [u8; 32] {
87115
out
88116
}
89117

90-
/// Compute HMAC-SHA256. Used in AWS Signature V4.
118+
/// Compute SHA-512 digest. Used for SMB 3.1.1 preauth integrity hash.
119+
pub fn sha512(data: &[u8]) -> [u8; 64] {
120+
let mut out = [0u8; CC_SHA512_DIGEST_LENGTH];
121+
unsafe {
122+
CC_SHA512(
123+
data.as_ptr() as *const c_void,
124+
data.len() as c_uint,
125+
out.as_mut_ptr(),
126+
);
127+
}
128+
out
129+
}
130+
131+
/// Compute HMAC-SHA256. Used in signing key derivation (SP800-108 KDF).
91132
pub fn hmac_sha256(key: &[u8], data: &[u8]) -> [u8; 32] {
92133
let mut out = [0u8; CC_SHA256_DIGEST_LENGTH];
93134
unsafe {
@@ -103,6 +144,95 @@ pub fn hmac_sha256(key: &[u8], data: &[u8]) -> [u8; 32] {
103144
out
104145
}
105146

147+
/// AES-ECB encrypt a single 16-byte block.
148+
fn aes128_ecb_block(key: &[u8; 16], block: &[u8; 16]) -> [u8; 16] {
149+
let mut out = [0u8; 16];
150+
let mut out_len: usize = 0;
151+
unsafe {
152+
CCCrypt(
153+
K_CC_ENCRYPT,
154+
K_CC_ALGORITHM_AES128,
155+
K_CC_OPTION_ECB_MODE,
156+
key.as_ptr() as *const c_void,
157+
16,
158+
std::ptr::null(),
159+
block.as_ptr() as *const c_void,
160+
16,
161+
out.as_mut_ptr() as *mut c_void,
162+
16,
163+
&mut out_len,
164+
);
165+
}
166+
out
167+
}
168+
169+
/// Compute AES-128-CMAC (RFC 4493). Used for SMB 3.x message signing.
170+
pub fn aes128_cmac(key: &[u8; 16], data: &[u8]) -> [u8; 16] {
171+
// Step 1: Generate subkeys
172+
let zero_block = [0u8; AES_BLOCK_SIZE];
173+
let l = aes128_ecb_block(key, &zero_block);
174+
175+
let k1 = dbl_block(&l);
176+
let k2 = dbl_block(&k1);
177+
178+
// Step 2: Determine number of blocks
179+
let n = if data.is_empty() {
180+
1
181+
} else {
182+
(data.len() + AES_BLOCK_SIZE - 1) / AES_BLOCK_SIZE
183+
};
184+
let complete = !data.is_empty() && data.len() % AES_BLOCK_SIZE == 0;
185+
186+
// Step 3: Build the last block
187+
let mut last = [0u8; AES_BLOCK_SIZE];
188+
if complete {
189+
let start = (n - 1) * AES_BLOCK_SIZE;
190+
last.copy_from_slice(&data[start..start + AES_BLOCK_SIZE]);
191+
xor_block(&mut last, &k1);
192+
} else {
193+
// Pad with 10*
194+
let start = (n - 1) * AES_BLOCK_SIZE;
195+
let remaining = data.len() - start;
196+
last[..remaining].copy_from_slice(&data[start..]);
197+
last[remaining] = 0x80;
198+
// rest is already zero
199+
xor_block(&mut last, &k2);
200+
}
201+
202+
// Step 4: CBC-MAC
203+
let mut x = [0u8; AES_BLOCK_SIZE];
204+
for i in 0..n - 1 {
205+
let start = i * AES_BLOCK_SIZE;
206+
let mut block = [0u8; AES_BLOCK_SIZE];
207+
block.copy_from_slice(&data[start..start + AES_BLOCK_SIZE]);
208+
xor_block(&mut block, &x);
209+
x = aes128_ecb_block(key, &block);
210+
}
211+
xor_block(&mut last, &x);
212+
aes128_ecb_block(key, &last)
213+
}
214+
215+
/// Double a block in GF(2^128) with the AES-CMAC polynomial.
216+
fn dbl_block(block: &[u8; 16]) -> [u8; 16] {
217+
let mut out = [0u8; 16];
218+
let mut carry = 0u8;
219+
for i in (0..16).rev() {
220+
out[i] = (block[i] << 1) | carry;
221+
carry = block[i] >> 7;
222+
}
223+
if block[0] & 0x80 != 0 {
224+
out[15] ^= 0x87; // Rb for 128-bit block
225+
}
226+
out
227+
}
228+
229+
/// XOR two 16-byte blocks in place.
230+
fn xor_block(a: &mut [u8; 16], b: &[u8; 16]) {
231+
for i in 0..16 {
232+
a[i] ^= b[i];
233+
}
234+
}
235+
106236
/// Encode bytes as lowercase hex string.
107237
pub fn hex_encode(bytes: &[u8]) -> String {
108238
const HEX: &[u8; 16] = b"0123456789abcdef";

src/crypto/mod.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@
55
66
mod ffi;
77

8-
pub use ffi::{hex_encode, hmac_md5, hmac_sha256, md4, sha256};
8+
pub use ffi::{aes128_cmac, hex_encode, hmac_md5, hmac_sha256, md4, sha256, sha512};

0 commit comments

Comments
 (0)