Skip to content

Commit ff572c7

Browse files
frristclaude
andcommitted
Merge origin/main (Hilt client + per-bucket spaces) into fil-588-blob-remove
Ports the deferred-park multipart and blob-removal flow onto main's Hilt architecture. The load-bearing resolutions: - Space is per-bucket and threaded as a parameter (main's model): the DeferredBodyUploader seams (UploadBlobParked/ConcludeBlob/AbortBlob), BlobRemover, and the s3frontend park/conclude/cleanup paths all take the bucket's space, resolved via the new Backend.bucketSpace helper (the session-expiry sweeper resolves it per session). - Proof chains are request-scoped (main's model): park and upload require the IAM-attached reqscope store and capture it as the space's write authority; abort and remove resolve reqscope-first with the captured store as the off-request fallback (the sweeper path). forgeclient.BlobRemove/BlobAbort accept WithProofStore like BlobAdd. - forgeclient.BlobAddParked/BlobConclude re-derived from main's BlobAdd: per-call proof store, no /blob/accept re-delegation (sprue owns accept), conclude carries no space proof. - seedSpaceDelegations and standaloneApp stay deleted (main): space authority is Hilt-issued per access key. NOTE: hilt's s3perm map does not yet delegate /blob/abort — multipart abort needs a hilt follow-up. - MultipartSessionTTL ported into config.ServerConfig; the sweeper keeps running under main's config package. - Fork-era s3err constants: ErrInvalidPartNumberRange for the Complete part-number range check, ErrInvalidRequest for invalid list markers. - versity partition: kept the fil-520 placements (ListParts implemented, Complete validation fixed) and moved main's two XFail cases (etag_quoting_consistency, missing_part_fields) into the passing lists; forge itests provision via hiltProvisionTenant. - go.mod: libforge at the RFC-alignment pin, smelt at main's Hilt-era pin — the smelt fil-588 branch (/blob/release delegation) must rebase onto smelt main before the removal itests can run end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 parents 43d6482 + 02af9fc commit ff572c7

107 files changed

Lines changed: 7391 additions & 1939 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,10 @@ ingot depends only on these — it must **never** import `fil-forge/sprue` or
5252
`commands/{blob,content,http,assert,ucan,index,provider,access}`, `blobindex`
5353
(sharded-dag-index), `ucan` (ProofStore), `ucan/retrieval`, `didmailto`, `receipt`.
5454
- **`indexing-service/pkg/{client,types}`** — indexer query client.
55-
- **`versity/versitygw`** — the S3 REST front end (we implement `backend.Backend`).
55+
- **`fil-forge/versitygw`** — our fork of versity/versitygw, the S3 REST front
56+
end (we implement `backend.Backend`). The fork adds externally derived SigV4
57+
signing keys (`auth.Account.SigningKey`, `middlewares.RequestIAMService`) for
58+
the Hilt flow.
5659
- Plumbing: `go-cid`, `go-block-format`, `whyrusleeping/cbor-gen` (**not**
5760
go-ipld-prime), `multiformats/*`, `pgx/v5`, `goose/v3`, `spf13/{cobra,viper}`,
5861
`uber-go/fx`, `zap`.
@@ -96,10 +99,21 @@ Internal:
9699
- **`forgeclient/`** — carried-from-guppy edge client: `/blob/add`,
97100
`/ucan/conclude`, `/index/add`, `/provider/add`, `/access/delegate`, and the
98101
`/access` login flow.
102+
- **`iam/`** — the Hilt (auth-service) integration over the external
103+
`github.com/fil-forge/hilt/pkg/client` (RFC: forge-s3-tenant-management):
104+
versitygw `IAMService`/`RequestIAMService` authorizing each non-root
105+
request via `/s3/request/authorize` (derived SigV4 key), plus
106+
`DelegationCache` — a TTL cache (go-cache) of Hilt-issued delegations
107+
(authorize re-delegations + `/s3/bucket/info` chains) that the network
108+
read tier consumes as its per-space `/content/retrieve` proof store.
109+
**Forge mode requires Hilt** (`auth_service_url`/`auth_service_did`): the
110+
postgres registry forwards bucket create/delete/list to it, recovering
111+
the signed S3 request from the method's ctx.
99112
- **`tokenstore/`** — carried-from-guppy delegation store (`tokens.cbor`).
100113
- **`bucket/`** — per-object model: `manifest.go` (`ObjectManifest`, `Body`),
101114
`chunker.go` (`BodyCodec`/`FixedChunker`), `cbor_gen.go`.
102-
- **`mst/`** — the forked MST (only dep: go-cid).
115+
- **`mst/`** — the forked MST (deps: go-cid, blockstore, ucantone/did — trees
116+
carry their bucket's space for network-backed reads).
103117
- **`inmem/`**`MemStore` (Registry+Meta), `NopBaseReader`, `NopUploader`; backs
104118
standalone mode (slated for removal).
105119
- **`cars/`**, **`migrations/`**, **`internal/ucanexec/`**, **`gen/`**,
@@ -141,7 +155,9 @@ Viper/yaml-bindable. Key fields: `Enabled`, `Addr`, `DataDir`, `Region`,
141155
`DataPlane`/`CatalogPlane` `{SealBytes, SealAge, Ship, Retain}` overrides,
142156
`IndexerEndpoint`/`IndexerDID`, `ReadCacheBytes` (0 → 256 MiB, <0 → off),
143157
`UploadServiceURL`/`UploadServiceDID`/`UploadReceiptsURL`, `TokenStoreDir`
144-
(→ `DataDir`). `Config.ServerConfig()` is the single mapping site. The daemon's
158+
(→ `DataDir`), `HiltURL`/`HiltDID`/`HiltProofs` (tenant-management service;
159+
proofs = file path or string-encoded UCAN container, optional).
160+
`Config.ServerConfig()` is the single mapping site. The daemon's
145161
`DaemonConfig` (cmd/config.go) embeds `Config` + `Mode`/`PostgresDSN`/`Identity`.
146162

147163
## Testing

Dockerfile

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,7 @@
2424
# /usr/bin/ingot -- serve --config /etc/ingot/config.yaml
2525
# then attach your client on the mapped 2345 (smelt's compose.debug.yml pattern).
2626

27-
# ---- build base: runs natively on the builder, cross-compiles to target ----
28-
FROM --platform=$BUILDPLATFORM golang:1.25-bookworm AS build
27+
FROM --platform=$BUILDPLATFORM golang:1.26-bookworm AS build
2928
ARG TARGETOS=linux
3029
ARG TARGETARCH
3130
WORKDIR /src

blockstore/cache.go

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"io"
77
"sync"
88

9+
"github.com/fil-forge/ucantone/did"
910
block "github.com/ipfs/go-block-format"
1011
"github.com/ipfs/go-cid"
1112
mh "github.com/multiformats/go-multihash"
@@ -61,21 +62,23 @@ var (
6162
// Body blobs are large and streamed; caching one whole would defeat streaming
6263
// (and a single blob can exceed the whole budget). Small catalog blocks still
6364
// cache through GetBlock.
64-
func (c *Cached) OpenBlob(ctx context.Context, digest mh.Multihash) (io.ReadCloser, error) {
65+
func (c *Cached) OpenBlob(ctx context.Context, space did.DID, digest mh.Multihash) (io.ReadCloser, error) {
6566
if br, ok := c.base.(BlobReader); ok {
66-
return br.OpenBlob(ctx, digest)
67+
return br.OpenBlob(ctx, space, digest)
6768
}
6869
return nil, ErrNotFound
6970
}
7071

7172
// GetBlock returns the cached block if present, otherwise fetches it from the
72-
// base reader and caches it (subject to the byte budget).
73-
func (c *Cached) GetBlock(ctx context.Context, k cid.Cid) (block.Block, error) {
73+
// base reader and caches it (subject to the byte budget). The LRU is keyed by
74+
// CID only — blocks are content-addressed, so a hit is valid whatever space
75+
// originally fetched it.
76+
func (c *Cached) GetBlock(ctx context.Context, space did.DID, k cid.Cid) (block.Block, error) {
7477
key := k.KeyString()
7578
if blk, ok := c.get(key); ok {
7679
return blk, nil
7780
}
78-
blk, err := c.base.GetBlock(ctx, k)
81+
blk, err := c.base.GetBlock(ctx, space, k)
7982
if err != nil {
8083
return nil, err
8184
}

blockstore/cache_test.go

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package blockstore
22

33
import (
44
"context"
5+
"github.com/fil-forge/ucantone/did"
56
"testing"
67

78
block "github.com/ipfs/go-block-format"
@@ -18,7 +19,7 @@ type countingReader struct {
1819
calls map[string]int
1920
}
2021

21-
func (r *countingReader) GetBlock(_ context.Context, c cid.Cid) (block.Block, error) {
22+
func (r *countingReader) GetBlock(_ context.Context, _ did.DID, c cid.Cid) (block.Block, error) {
2223
if r.calls == nil {
2324
r.calls = map[string]int{}
2425
}
@@ -50,7 +51,7 @@ func TestCached_Hit(t *testing.T) {
5051
k := mkcid(7)
5152

5253
for i := 0; i < 3; i++ {
53-
if _, err := c.GetBlock(context.Background(), k); err != nil {
54+
if _, err := c.GetBlock(context.Background(), did.Undef, k); err != nil {
5455
t.Fatalf("get: %v", err)
5556
}
5657
}
@@ -66,20 +67,20 @@ func TestCached_EvictsByBytes(t *testing.T) {
6667

6768
first, second, third := mkcid(1), mkcid(2), mkcid(3)
6869
for _, k := range []cid.Cid{first, second, third} {
69-
if _, err := c.GetBlock(context.Background(), k); err != nil {
70+
if _, err := c.GetBlock(context.Background(), did.Undef, k); err != nil {
7071
t.Fatalf("get: %v", err)
7172
}
7273
}
7374

7475
// first was evicted -> refetched (2 base hits).
75-
if _, err := c.GetBlock(context.Background(), first); err != nil {
76+
if _, err := c.GetBlock(context.Background(), did.Undef, first); err != nil {
7677
t.Fatalf("get: %v", err)
7778
}
7879
if got := base.calls[first.KeyString()]; got != 2 {
7980
t.Fatalf("expected first block evicted and refetched (2 base hits), got %d", got)
8081
}
8182
// third is the most-recently-used and still cached (1 base hit).
82-
if _, err := c.GetBlock(context.Background(), third); err != nil {
83+
if _, err := c.GetBlock(context.Background(), did.Undef, third); err != nil {
8384
t.Fatalf("get: %v", err)
8485
}
8586
if got := base.calls[third.KeyString()]; got != 1 {

0 commit comments

Comments
 (0)