[Phase 1] Canonical Request Signing — Deterministic HMAC Across SDKs

[RachanaB5]

Context

Currently, payloads are signed as raw bytes. While this is technically valid, it leads to inconsistent HMAC verification across SDKs when JSON key ordering differs.

This issue proposes introducing canonical JSON signing to ensure deterministic HMAC across SDKs.

Example:

• Python SDK sends: {"b":2,"a":1}
• Go sidecar receives: {"a":1,"b":2}
• Result: ❌ HMAC mismatch — despite semantically identical payloads

Relates to #18, #23.

---

Solution

Canonicalize JSON before HMAC computation in both the Go sidecar and Python SDK:

1. Parse payload as JSON (reject if invalid)
2. Re-encode with sorted keys, no whitespace
3. Sign the canonical bytes

This guarantees identical HMAC input regardless of SDK or serialization order.

---

Changes Required

Go (sidecar/internal/transport/)

• frame.go — Canonicalize in SignedMessage(), update return signature to ([]byte, error)
• frame.go — Propagate errors in EncodeRequest() / DecodeRequest()
• listener.go — Handle canonicalization errors
• frame_test.go — Add tests for key order, whitespace, invalid JSON, cross-SDK interop

Python (sdk/python/acf/frame.py)

• frame.py — Canonicalize in signed_message() using json.dumps(sort_keys=True)
• test_frame.py — Mirror Go test cases

---

Acceptance Criteria

• SignedMessage() / signed_message() canonicalize JSON with sorted keys
• Invalid JSON is rejected with a clear error
• Python and Go produce identical signatures for semantically identical JSON
• All existing tests pass
• New tests cover: key order, whitespace, cross-language interop, error cases
• phase1.md documentation updated
• No breaking public API changes

[shayankashif123]

Hi — I've been working in this codebase and would like to take this on.My understanding of the problem: payloads are currently
signed as raw serialized bytes, which means key ordering differences between SDKs produce different HMAC inputs for semantically identical JSON. This will become a hard blocker the moment the TypeScript SDK sends requests to
the Go sidecar.
Proposed fix: canonicalize JSON (sorted keys, no whitespace) before HMAC computation in both the Python SDK and Go sidecar, so the signature is always computed over the same bytes regardless of serialization order.I'll implement both sides (Python frame.py + Go frame.go) with matching test cases including cross-language interop verification.
One question: should the canonicalization error in Go propagate as a hard rejection (close the connection immediately) or return a structured error response to the SDK? I'm leaning toward hard rejection since an invalid JSON payload at the signing stage
indicates a malformed request, but want to confirm this aligns with the existing error handling philosophy in listener.go.

I'll have an implementation up within a few days.

[RachanaB5]

Hi, I’ve already opened a PR for this issue (#33) and implemented the proposed changes.
Happy to incorporate any feedback or suggestions from maintainers.

[shayankashif123]

I see this has been picked up — no problem at all. I had already started an implementation (Python side complete
with canonicalization tests and cross-SDK interop verification).
Happy to share notes or test cases if useful to whoever is working on it.

[RachanaB5]

Thanks, appreciate that!

Would be great if you could share any notes or test cases — especially around cross-SDK interop. That would be really helpful to validate and improve the current implementation.

Happy to incorporate anything useful.

[shayankashif123]

Happy to share — here are the test cases I had written for the Python side, particularly the cross-SDK interop section.The key insight that shaped these tests: two separate encode_request calls will always produce different HMACs because each generates a
fresh nonce — that is replay protection working correctly. The cross-SDK guarantee is narrower and more precise: given the same
nonce, two SDKs signing semantically identical JSON must produce identical signed messages and therefore identical HMACs.

This distinction matters for how you structure the interop tests — you cannot compare HMACs from two separate encode_request calls directly. You need to fix the nonce and test at the signed_message level.

canonical_payload unit tests

class TestCanonicalPayload:

    def test_sorts_keys_alphabetically(self):
        a = canonical_payload(b'{"b":2,"a":1}')
        b = canonical_payload(b'{"a":1,"b":2}')
        assert a == b

    def test_output_is_correctly_sorted(self):
        result = canonical_payload(b'{"z":3,"a":1,"m":2}')
        assert result == b'{"a":1,"m":2,"z":3}'

    def test_removes_whitespace(self):
        pretty  = b'{\n  "a": 1,\n  "b": 2\n}'
        compact = b'{"a":1,"b":2}'
        assert canonical_payload(pretty) == canonical_payload(compact)

    def test_nested_object_keys_sorted(self):
        result = canonical_payload(b'{"z":{"b":2,"a":1},"a":0}')
        assert result == b'{"a":0,"z":{"a":1,"b":2}}'

    def test_empty_object(self):
        assert canonical_payload(b"{}") == b"{}"

    def test_invalid_json_raises(self):
        with pytest.raises(CanonicaliseError, match="not valid JSON"):
            canonical_payload(b"not json at all")

    def test_empty_bytes_raises(self):
        with pytest.raises(CanonicaliseError):
            canonical_payload(b"")

    def test_partial_json_raises(self):
        with pytest.raises(CanonicaliseError):
            canonical_payload(b'{"key": ')

    def test_canonicalise_error_is_subclass_of_frame_error(self):
        """CanonicaliseError must subclass FrameError for transparent catching."""
        with pytest.raises(FrameError):
            canonical_payload(b"not json")

    def test_idempotent(self):
        """Canonicalizing an already-canonical payload must be a no-op."""
        original = b'{"z":3,"a":1}'
        once     = canonical_payload(original)
        twice    = canonical_payload(once)
        assert once == twice

    def test_riskcontext_keys_sorted(self):
        """Real RiskContext payload must canonicalize to sorted key order."""
        raw = json.dumps({
            "score":      0.0,
            "signals":    [],
            "provenance": "user",
            "session_id": "",
            "hook_type":  "on_prompt",
            "payload":    "hello world",
            "state":      None,
        }).encode("utf-8")
        canonical = canonical_payload(raw)
        parsed    = json.loads(canonical)
        keys      = list(parsed.keys())
        assert keys == sorted(keys), f"keys not sorted: {keys}"

Cross-SDK interop tests

These are the most important ones. The _hmac_for helper fixes the
nonce so we can compare HMAC outputs in isolation.

class TestCrossSDKInterop:
    """
    Verifies the core guarantee: two SDKs serializing the same
    RiskContext in different key order produce identical HMACs.

    Nonce is fixed in these tests to isolate the HMAC computation.
    In production each request has a unique nonce — the guarantee
    is that given the same nonce, both SDKs sign identical bytes.
    """

    _KEY   = b"test-key-32-bytes-long-padded!!!"
    _NONCE = b"\x01" * 16

    def _hmac_for(self, payload: bytes) -> bytes:
        canonical = canonical_payload(payload)
        length    = len(canonical)
        msg       = signed_message(VERSION, length, self._NONCE, canonical)
        return hmac.new(self._KEY, msg, hashlib.sha256).digest()

    def test_different_key_order_same_hmac(self):
        """
        The exact failure mode described in the issue.
        Python SDK and TypeScript SDK serializing the same payload
        in different key order must produce identical HMACs.
        """
        payload_python     = b'{"hook_type":"on_prompt","score":0.0,"payload":"hello"}'
        payload_typescript = b'{"payload":"hello","score":0.0,"hook_type":"on_prompt"}'

        assert self._hmac_for(payload_python) == self._hmac_for(payload_typescript), (
            "HMAC mismatch for semantically identical JSON with different key order"
        )

    def test_whitespace_differences_same_hmac(self):
        compact = b'{"a":1,"b":2}'
        spaced  = b'{"a": 1, "b": 2}'
        assert self._hmac_for(compact) == self._hmac_for(spaced)

    def test_pretty_printed_same_hmac(self):
        compact = b'{"hook_type":"on_prompt"}'
        pretty  = b'{\n  "hook_type": "on_prompt"\n}'
        assert self._hmac_for(compact) == self._hmac_for(pretty)

    def test_full_riskcontext_interop(self):
        """
        Full RiskContext — simulates Python and TypeScript both sending
        the same struct with different serialization order.
        """
        fields = {
            "score":      0.0,
            "signals":    [],
            "provenance": "user",
            "session_id": "abc123",
            "hook_type":  "on_prompt",
            "payload":    "hello world",
            "state":      None,
        }
        python_order     = json.dumps(fields, separators=(",", ":")).encode()
        typescript_order = json.dumps(
            dict(reversed(list(fields.items()))),
            separators=(",", ":"),
        ).encode()

        # Sanity check — payloads must actually differ for this to be a
        # valid interop test
        assert python_order != typescript_order, \
            "test setup error: payloads must differ"

        assert self._hmac_for(python_order) == self._hmac_for(typescript_order)

    def test_canonical_bytes_identical(self):
        """
        Intermediate check — canonical bytes must match before HMAC.
        Useful for pinpointing failures: if this passes but HMAC fails,
        the bug is in signing, not canonicalization.
        """
        assert canonical_payload(b'{"b":2,"a":1}') == canonical_payload(b'{"a":1,"b":2}')

    def test_signed_messages_identical_given_same_nonce(self):
        """
        Given the same nonce, signed_message must produce identical bytes
        for semantically identical JSON. This is the direct HMAC input —
        if this passes, HMAC must also match.
        """
        payload_a = b'{"b":2,"a":1}'
        payload_b = b'{"a":1,"b":2}'
        canon_a   = canonical_payload(payload_a)
        canon_b   = canonical_payload(payload_b)
        length    = len(canon_a)
        msg_a     = signed_message(VERSION, length, self._NONCE, canon_a)
        msg_b     = signed_message(VERSION, length, self._NONCE, canon_b)
        assert msg_a == msg_b

    def test_two_encode_request_calls_differ_by_nonce(self):
        """
        Documents expected behaviour: two encode_request calls for the
        same payload produce different HMACs because each generates a
        fresh nonce. This is replay protection — not a bug.
        The interop guarantee operates at the signed_message level,
        not the encode_request level.
        """
        key     = self._KEY
        frame_a = encode_request(b'{"a":1}', key)
        frame_b = encode_request(b'{"a":1}', key)
        nonce_a = frame_a[6:22]
        nonce_b = frame_b[6:22]
        hmac_a  = frame_a[22:54]
        hmac_b  = frame_b[22:54]
        assert nonce_a != nonce_b,  "nonces must differ between calls"
        assert hmac_a  != hmac_b,   "HMACs differ because nonces differ — expected"

---

One edge case worth validating on the Go side too: after canonicalization
the length field in the signed message must reflect the canonical byte
length, not the raw input length. These can differ when whitespace is
stripped — e.g. {"a": 1} (8 bytes) canonicalizes to {"a":1}
(7 bytes). If the length field uses the raw length but the payload
uses canonical bytes, verification will fail.

Hope these are useful — happy to answer questions about any of them.

[RachanaB5]

This is super helpful, thanks a lot for sharing!
The distinction around nonce vs HMAC comparison makes sense, and I’ll align the tests on the Go side accordingly (especially testing at the signed_message level with a fixed nonce).
I’ll also make sure to validate the canonical length vs raw input length case - that’s a great catch.

Really appreciate you sharing these, will incorporate similar coverage in the PR.

[RachanaB5]

Thanks again for sharing those details — I’ve incorporated similar canonicalization and fixed-nonce interop tests in my PR (#33).
Would love any feedback if you get a chance to take a look!

[shayankashif123]

A few observations after reading carefully:

Strongest parts:
TestNegativeBehavior_DifferentNonces paired with
TestNegativeBehavior_FixedNonceComparison is exactly the right way to document the nonce distinction. These two tests together explain the cross-SDK mental model better than any comment could.

TestCrossSDK_CanonicalLengthInSignedMessage precisely validates the length field edge case — this is the test I flagged in the earlier comment and the implementation here is correct.

TestCanonicalizeJSON_TableDriven with the array preserved case correctly documents that canonicalization sorts keys but preserves array order — an important and non-obvious invariant.

One test to fix:
TestFrameHeader_CanonicalVsRaw only uses t.Logf in its main body — t.Logf never fails a test regardless of what the code does. The three-branch if/else block that logs the relationship between raw and canonical length always passes even if the implementation is completely broken. Only the final assertion at the end does real verification. The test should be restructured to assert the two invariants directly: frame header length == raw payload length, signed message length == canonical length, and the two must differ for a whitespace payload. Happy to share a suggested rewrite if useful.

One wrapper to remove:
FixedNonceSignedMessage is a pass-through wrapper around SignedMessage with no additional logic. Every call site could
call SignedMessage directly. The name implies something special about the nonce handling but there is none — SignedMessage already accepts any nonce. FixedNonceHMAC is justified because it combines two operations, but the wrapper above it adds only indirection.

One potential cross-SDK gap:
TestCanonicalizeJSON_SpecialCharacters only verifies idempotency — it does not assert the exact canonical output. The \u0041 case is worth asserting explicitly: Go normalizes \u0041 → A during unmarshal/marshal. If Python's canonicalization path uses ensure_ascii=True, it might not normalize the same way, breaking
the cross-SDK guarantee for payloads containing Unicode escapes.
Worth a specific assertion and a note in the test comment.

Overall this is solid coverage — the interop tests are well-designed and the table-driven section follows Go best practices. The
TestCrossSDK_ComplexNested test with real-world nested structures
is particularly good for catching regressions. Nice work.

[RachanaB5]

Thanks for taking the time to go through this — really appreciate the detailed feedback!

I’ve incorporated your suggestions:

• replaced log-based checks with assertions
• removed the unnecessary wrapper
• added a Unicode normalization test for cross-SDK consistency

Would be great to hear your thoughts on the updated version.

[shayankashif123]

Read through the updated file — all three changes are correctly
implemented.

TestFrameHeader_CanonicalVsRaw is now properly assertive. The three named invariants with t.Fatalf are exactly the right
structure — if any invariant fails, the test stops rather than continuing with broken state. Clean fix.
FixedNonceSignedMessage is gone and FixedNonceHMAC calls
SignedMessage directly. No unnecessary indirection remaining.

TestCanonicalizeJSON_UnicodeNormalization is well done — asserting the exact output, verifying idempotency, and verifying that escaped and literal forms produce identical canonical output covers the cross-SDK concern precisely. The comment explaining why this matters for cross-SDK consistency is a nice touch.
All three issues from the previous review are resolved. This is in good shape — nice work incorporating the feedback quickly.

[RachanaB5]

Thanks a lot for the detailed feedback earlier, really helped improve the tests.
Glad the updates look good!

[tharindupr]

If you are reporting a bug, use [Bug] to create the issue. And provide evidence that this bug exists through various means.

[RachanaB5]

That makes sense but this was intended more as a proposal to explain the current behavior and suggest canonical signing, rather than a bug report.
Happy to update it if you'd prefer a different format.