fee: cross-implementation test vectors vs foc-encryption (FIL-473)

[Peeja]

tl;dr: Checks our implementation against the foc-encryption reference implementation, including Kubuxu/foc-encryption-demo#2.

---

What

Implements FIL-473. Adds fixed, deterministic FEE cross-implementation test vectors that pin the wire format across the Go implementation (fee/cose, fee/aesstream, fee/ecdhkw, fee/aeskw) and the TypeScript reference foc-encryption (Kubuxu/foc-encryption-demo, packages/foc-encryption), pinned to master the head of PR #2 (158571ae…, the RFC 9052 §5.3 context fix) — not master, which still seals tag-96 bodies under the buggy "Encrypt0" context. Re-pin to the merge commit once PR #2 lands.

Per the reopened issue, the reference is the source of truth — the Go side was reconciled to it, then the vectors were written.

Acceptance criteria

All three met and verified in both directions against the real pinned reference:

• ✅ Single-chunk in Go → decrypt in TS — single-chunk-go (tag 16); foc-encryption decrypts it.
• ✅ Multi-chunk in TS → decrypt in Go — multi-chunk-ts; TestVectors decrypts it.
• ✅ Multi-recipient in Go → unwrap each CEK in TS — multi-recipient-go (tag 96) with real ECDH-ES+A256KW (X25519) and A256KW recipients. The reference carries wrappedKey opaquely and has no unwrap code, so it parses every recipient descriptor and decrypts the body from the shared CEK; the actual per-recipient CEK unwrap is asserted Go-side in TestVectors.
• ✅ Deterministic, checked-in fixtures — testdata/<name>/{blob.bin,plaintext.bin,meta.json}; tests only read fixed files and run deterministic decrypt/unwrap.

(Plus multi-chunk-go for extra Go→TS multi-chunk coverage.)

Go implementation changes (reconcile to the reference)

Confined to fee/cose:

• A single Envelope type for both COSE forms — the reference uses tag 16 (COSE_Encrypt0) for a no-recipient file and tag 96 (COSE_Encrypt) with recipients; cose previously modeled only tag 96. Rather than carry a parallel Encrypt0 type, cose now has one Envelope{Headers, Recipients} whose form — the CBOR tag (96/16), the array shape (4/3 elements), and the Enc_structure context — is computed from recipient presence. RFC 9052 requires a COSE_Encrypt to carry ≥1 recipient, so a recipient-less envelope is exactly a COSE_Encrypt0; deriving the tag from the recipients makes "an Encrypt0 with recipients" unrepresentable and removes the duplicated encode/AAD paths. The two byte decoders collapse into one tag-dispatching Decode returning *Envelope; PeekTag stays for callers that want the raw on-wire tag without decoding. encrypt.go + encrypt0.go merge into envelope.go.
• Body AAD context per form — the reference (as fixed in PR docs: write architecture document #2) builds the body AEAD's AAD with the RFC 9052 §5.3 context that matches the envelope: "Encrypt" when recipients are present, "Encrypt0" when not. Envelope.EncStructure selects between them by recipient presence; the context strings stay internal to cose, and the vectors take the AAD from the envelope's own EncStructure (no exported context constants or standalone builder).

fee/aesstream's STREAM framing (base[7] ‖ idx[4 BE] ‖ lastFlag[1], tag 16, constant per-chunk AAD) already matched byte-for-byte and is unchanged. The FEE profile constants (typ = application/vnd.foc-envelope+cose, alg -65793, unprotected labels chunkSize -65790 / chunkCount -65791) live in the vectors package (the caller), matching the reference's src/cose/headers.ts.

For #14 (FIL-569): #14 (the top-level fee composer) is stacked on this branch and has been reconciled both to the format pinned here (typ = application/vnd.foc-envelope+cose, unprotected chunkSize -65790, form-tracked AAD context) and to the unified cose.Envelope — its cose.DecodeReader returns *Envelope and shares the decodeEnvelope validation core with Decode. This PR stays independent (it targets main); #14 retargets to main automatically once this lands.

Verifying

GOWORK=off go test ./fee/...                 # Go decrypts every fixture + unwraps recipients
./fee/vectors/pull-foc-encryption.sh         # real foc-encryption regenerates multi-chunk-ts + decrypts all (needs bun)

pull-foc-encryption.sh vendors the pinned reference into ts/vendor/ (gitignored) via git clone + git fetch refs/pull/2/head, checks out the pinned SHA, and falls back to raw.githubusercontent.com where git egress is blocked. The TS driver runs under bun. See fee/vectors/README.md.

Testing

• GOWORK=off go build ./..., go vet ./fee/..., gofmt -l fee/ — clean.
• GOWORK=off go test ./fee/... -count=1 — green (incl. fee/cose and fee/vectors).
• ./fee/vectors/pull-foc-encryption.sh — cross-implementation check OK (4/4 fixtures decrypt under the real foc-encryption).
• The red go-test checks are the pre-existing flaky ./testing/ smoke suite (its TestSmokeXFail_* cases fail the run on an unexpected pass), unrelated to this fee/-only diff — testing/ has no dependency on fee/*.

Notes

• All key material (CEKs, the X25519 tenant key, the A256KW KEK, base nonces) is non-secret, deterministically derived from fixed labels, and recorded in each meta.json — it exists only to pin these vectors.
• Downstream: FIL-567 (security review of the hand-rolled fee/* crypto) is gated on this reconciliation settling.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HB9xeMxQFD67u61tgyoKHS

---

Generated by Claude Code

[Copilot]

Pull request overview

Adds deterministic, checked-in FEE cross-implementation fixtures and harnesses to pin the COSE+STREAM wire format against the pinned TypeScript foc-encryption reference, and extends the Go fee/cose package with COSE_Encrypt0 (tag 16) support plus tag dispatch.

Changes:

• Add fee/vectors package with deterministic fixture generation, validation tests, and committed testdata/<name>/{blob.bin,plaintext.bin,meta.json} vectors.
• Add a TypeScript harness (fee/vectors/ts) and a vendoring script to pull a pinned foc-encryption reference and verify/regenerate TS-produced fixtures.
• Extend fee/cose with COSE_Encrypt0 encode/decode and PeekTag for tag-based dispatch.

Reviewed changes

Copilot reviewed 21 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
fee/vectors/vectors_test.go	Loads all fixtures and verifies Go decryption + recipient unwrap assertions for tag-96 vectors.
fee/vectors/helpers_test.go	Implements deterministic key derivation, fixture IO, envelope composition, decode, and body decryption helpers.
fee/vectors/doc.go	Package-level documentation of the vector purpose, wire shape, and regeneration flow.
fee/vectors/README.md	Human-readable documentation of fixtures, wire format, and regeneration steps.
fee/vectors/pull-foc-encryption.sh	Pulls a pinned TS reference into ts/vendor/, installs deps, and runs the TS driver to generate/verify fixtures.
fee/vectors/ts/package.json	Declares the minimal TS harness dependency set (cborg).
fee/vectors/ts/driver.ts	Drives the pinned TS reference to generate multi-chunk-ts and verify decryption of all committed fixtures.
fee/vectors/ts/.gitignore	Ensures vendored TS reference and node_modules are not committed.
fee/vectors/testdata/single-chunk-go/blob.bin	Committed Go-produced tag-16 fixture blob (envelope
fee/vectors/testdata/single-chunk-go/plaintext.bin	Plaintext for single-chunk-go fixture.
fee/vectors/testdata/single-chunk-go/meta.json	Metadata sidecar for single-chunk-go fixture (tag/typ/alg/CEK/etc).
fee/vectors/testdata/multi-chunk-go/blob.bin	Committed Go-produced multi-chunk tag-16 fixture blob.
fee/vectors/testdata/multi-chunk-go/plaintext.bin	Plaintext for multi-chunk-go fixture.
fee/vectors/testdata/multi-chunk-go/meta.json	Metadata sidecar for multi-chunk-go fixture.
fee/vectors/testdata/multi-chunk-ts/blob.bin	Committed TS-produced multi-chunk tag-16 fixture blob.
fee/vectors/testdata/multi-chunk-ts/plaintext.bin	Plaintext for multi-chunk-ts fixture.
fee/vectors/testdata/multi-chunk-ts/meta.json	Metadata sidecar for multi-chunk-ts fixture.
fee/vectors/testdata/multi-recipient-go/blob.bin	Committed Go-produced tag-96 multi-recipient fixture blob.
fee/vectors/testdata/multi-recipient-go/plaintext.bin	Plaintext for multi-recipient-go fixture.
fee/vectors/testdata/multi-recipient-go/meta.json	Metadata sidecar for multi-recipient-go fixture incl. recipient descriptors and unwrap inputs.
fee/cose/encrypt0.go	Introduces COSE_Encrypt0 (tag 16) detached-payload envelope with encode + Enc_structure AAD construction.
fee/cose/encrypt0_test.go	Adds tests for tag-16 encode/decode shape, Enc_structure correctness, and error cases.
fee/cose/encrypt.go	Refactors Enc_structure building through a shared helper used by Encrypt and Encrypt0.
fee/cose/decode.go	Adds DecodeEncrypt0 for tag-16 envelopes and PeekTag for lightweight tag dispatch.
fee/cose/cose.go	Updates package docs/scope and defines tag-16 constant + Enc_structure context strings.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bajtos]

Moved the foc-encryption-demo pin to the latest master, since our fix was landed.

[bajtos]

I think this is good to land 👍🏻

[Copilot]

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

fee/vectors/vectors_test.go:105

• verifyRecipients indexes recipients by algorithm only (byAlg[alg] = r). If an envelope ever contains multiple recipients that share the same alg (e.g., two A256KW recipients with different kids), later entries overwrite earlier ones and the test can unwrap/validate the wrong recipient or miss one entirely. Index by (alg,kid) instead, and optionally fail on duplicate (alg,kid) pairs to avoid silent overwrites.

	byAlg := map[int64]*cose.Recipient{}
	for _, r := range p.recipients {
		alg, ok := r.Headers.Protected.Int(cose.HeaderLabelAlg)
		require.True(t, ok, "recipient algorithm present")
		byAlg[alg] = r
	}

	for _, rm := range f.meta.Recipients {
		r := byAlg[rm.Algorithm]
		require.NotNilf(t, r, "envelope missing recipient alg %d", rm.Algorithm)

		// The kid binds the descriptor to its key.
		kid, ok := r.Headers.Unprotected.Bytes(cose.HeaderLabelKID)
		require.True(t, ok, "recipient kid present")
		require.Equal(t, rm.KidHex, hex.EncodeToString(kid), "recipient kid")

[bajtos]

Code review (written by Claude)

This review and its line comments were written by Claude, posted from a Claude Code session at a maintainer's request.

Overview

The PR does two related things: (1) refactors fee/cose to a single Envelope type covering COSE_Encrypt (tag 96) and COSE_Encrypt0 (tag 16), with the form — tag, array shape, and RFC 9052 §5.3 Enc_structure context — computed from recipient presence; and (2) adds fee/vectors: four checked-in deterministic fixtures, a Go suite that decrypts every fixture and actually unwraps each recipient CEK (ECDH-ES+A256KW/X25519 and A256KW), and pull-foc-encryption.sh + a bun/TS driver verifying everything against the pinned foc-encryption reference.

What's solid

• The "form follows recipients" design is correct and its invariant is enforced, not assumed. RFC 9052 requires ≥1 recipient for COSE_Encrypt, and the unchanged decodeRecipients rejects an empty tag-96 recipients array — so a decoded tag-96 always carries recipients, which is exactly what EncStructure's context selection relies on. "An Encrypt0 with recipients" is unrepresentable.
• Blast radius is contained: nothing outside fee/ imports these packages, so the Encrypt → Envelope API break is confined; #14 is stated to be reconciled.
• Test quality is high: round-trip, wire-shape (tag byte, arity, null detached body), AAD divergence between forms for an identical header, RawProtected AAD stability, a negative unwrap check, and the coreVectors completeness map so a dropped fixture fails the suite.
• Security/supply-chain posture is right: all committed key material is deterministically derived, explicitly non-secret, and exists only to pin the vectors; the reference is pinned to a fixed SHA, vendored source is gitignored, and the raw fallback fetches by SHA.
• CI is green at head (unit on 3 OSes, itest, go-check, both Docker build checks) — the flaky-smoke-suite caveat in the PR body no longer applies.

Findings (details in line comments)

1. Stale pin docs — fee/vectors/README.md and fee/vectors/doc.go still say master is buggy and not used, contradicting the script's refs/heads/master re-pin. Should fix before merge.
2. CEK + base-nonce reuse across fixtures — single-chunk-go and multi-recipient-go collide on chunk-0 key+nonce; harmless for public vectors but worth a one-line per-fixture CEK derivation ahead of FIL-567.
3. Go encode path isn't pinned in CI — suggestion: byte-equality re-compose for the deterministic tag-16 Go fixtures in TestVectors.
4. PeekTag vs Decode sentinel inconsistency — nit.

Informational

A recipient-less Envelope.Encode() previously failed with ErrNoRecipients and now silently emits a valid tag-16 envelope. That's the intended design and no in-repo callers are affected, but it's worth remembering when #14 lands: a caller that forgets to attach recipients now gets a valid envelope instead of an error.

Verdict

Well-engineered PR — the unification is a genuine simplification with its central invariant enforced on both encode and decode paths, and the vector suite is rigorous in both directions. Item 1 actively misleads and should be fixed before merge; item 2 is best done while fixture regeneration is still cheap; items 3–4 are non-blocking.

---

Generated by Claude Code

[bajtos]

Stale pin documentation (should fix before merge). The pin has since been moved to master (pull-foc-encryption.sh now sets REF_FETCH="refs/heads/master", per the earlier PR comment), but three doc spots still describe the pre-merge state and now contradict the script:

• this paragraph: "Master is deliberately not used: it still seals tag-96 bodies under the buggy "Encrypt0" context. Re-pin to the merge commit once the PR lands."
• line 80 below: "git clone + git fetch refs/pull/2/head" — the script no longer fetches the PR ref
• fee/vectors/doc.go (the paragraph starting "The reference is the source of truth"): "…rather than master, which still seals tag-96 bodies under "Encrypt0""

Since the upstream fix landed on master, anyone reading these docs will wrongly conclude master is unusable. Suggest updating all three in the same spirit as the PR description's strikethrough edit.

---

Generated by Claude Code

[bajtos]

Addressed in 56de988

[bajtos]

Minor (crypto hygiene): shared CEK + base nonce across fixtures. All three Go-produced fixtures reuse the same CEK and base nonce. single-chunk-go and multi-recipient-go are both single-chunk, so their chunk-0 nonces are identical (baseNonce ‖ 0 ‖ lastFlag=0x01) under the same key with different plaintexts — textbook AES-GCM key+nonce reuse. (The differing AAD context doesn't help: the GCM keystream depends only on key+nonce, so the two ciphertexts XOR to the plaintexts' XOR.)

This is harmless here since all the material is public by design, but these are reference vectors feeding the FIL-567 security review, and it would be better not to enshrine a nonce-reuse pattern in them. Deriving a per-fixture CEK — include the fixture name in the derivation label, as the TS side already does with fil-473-fee-cek-ts-v1 — is a one-line change while regeneration is still cheap.

---

Generated by Claude Code

[bajtos]

Addressed in 6cfa69e

[bajtos]

Suggestion (test coverage): the Go encode path isn't pinned in CI. This test decrypts the committed blobs, which covers TS→Go and Go's decode path. The Go→TS direction and Go's encode-path bytes are only checked when someone manually runs pull-foc-encryption.sh — an encode regression (e.g. a header-serialization change) would keep CI green because the committed blobs still decrypt.

For the tag-16 Go fixtures every input is deterministic (fixed CEK, base nonce, chunk size, committed plaintext), so this test could re-run composeFEE for producer == "go" tag-16 fixtures and assert byte equality with blob.bin, pinning the encode path in CI for free.

(The tag-96 fixture can't be byte-pinned — ecdhkw.Wrap mints a random ephemeral key — which is also the actual reason a regenerated Go blob can differ from the committed one; the README's regeneration note attributes that drift only to base nonces.)

---

Generated by Claude Code

[bajtos]

I don't know what to think about this. Seems like the solution is non-trivial; can we defer this for later? I don't expect the FEE implementation to change much, so maybe it's okay not to run full test coverage on every Ingot PR.

[alanshaw]

This should be added to CI when the code is extracted as a library.

[bajtos]

Nit: sentinel inconsistency with Decode. For the same non-tag input, PeekTag returns ErrMalformed while Decode classifies it as ErrNotEncrypt (see the "bare integer, not a tag" case in decode_test.go). Both behaviors are documented, but callers classifying with errors.Is get different sentinels for the same bytes depending on entry point; aligning PeekTag on ErrNotEncrypt would be more predictable.

---

Generated by Claude Code

[bajtos]

Addressed in ed98ec6

[alanshaw]

I think this whole implementation and the test vectors should be extracted as a standalone library in a separate repo.

[alanshaw]

nit: I know this was not added here but typically we define a package level function for Encode and pass in the value to encode. This is also more idiomatic Go.

[alanshaw]

This should be added to CI when the code is extracted as a library.