Skip to content

Give a nested stringref namespace its own index space - #335

Open
dylanpulver wants to merge 2 commits into
agronholm:masterfrom
dylanpulver:fix-nested-stringref-namespace
Open

Give a nested stringref namespace its own index space#335
dylanpulver wants to merge 2 commits into
agronholm:masterfrom
dylanpulver:fix-nested-stringref-namespace

Conversation

@dylanpulver

@dylanpulver dylanpulver commented Sep 1, 2026

Copy link
Copy Markdown

Changes

encode_semantic() (rust/encoder.rs:915) turned string_referencing on for tag
256 but left the outer string_references / bytes_references maps in place —
the two TODOs in that function marked exactly this. References emitted inside a
nested namespace were therefore numbered against the outer namespace, and
strings registered inside it leaked back out. The decoder
(rust/decoder.rs:1735) already handles it correctly: it pushes a fresh
namespace and resolves tag 25 against the innermost one.

So the encoder emits output its own decoder cannot read — and sometimes reads
back as different data, with no exception:

>>> v = ["aaa", CBORTag(256, ["bbb", "ccc", "ddd", "bbb"])]
>>> loads(dumps(v, string_referencing=True))
['aaa', ['bbb', 'ccc', 'ddd', 'ccc']]        # 'bbb' came back as 'ccc'

The stringref spec requires the reset:

Since stringref-namespace tags can be nested, the decoder needs to save and
restore the outer array
before starting and after ending the decoding of the
tagged value.

The fix sets the outer maps aside on entry and restores them on exit.

Oracle: the spec's own worked example

That page publishes a nested example with its bytes and its decoding. On master,
loads() reproduces the documented value exactly, but dumps() of that same
value produces different bytes, and loads() then fails on them:

master this PR
decode spec bytes → documented value
encode documented value → spec bytes
round trip through cbor2's own decoder string reference 2 not found

The new test asserts against those published bytes rather than against anything
this implementation produced.

Test runs (same command and environment each time)

python -m pytest tests -q

result
clean master 464 passed
master + the new tests 2 failed, 465 passed
this PR 467 passed

I also built the half-fix — clear the maps on entry, don't restore them — and it
still fails both new tests, which is why the restore case is tested separately.
test_decode_..._matches_spec_example passes either way; it is there as a
control showing the decoder was never the faulty side.

Not a security issue as far as I can tell: it is encoder-side, string_referencing
is opt-in, and it needs a CBORTag(256, …) in the caller's own data. I'll add the
changelog PR link once this has a number.

Checklist

  • You've added tests (in tests/) which would fail without your patch
  • You've updated the documentation (in docs/), in case of behavior changes or new features
  • You've added a new changelog entry (in docs/versionhistory.rst)

AI disclosure: found and drafted with Claude Code (model Claude Opus 5,
claude-opus-5) — differential fuzzing of encode→decode round trips, then the
spec's published vector as the oracle. Reviewed before submission.

encode_semantic() turned string_referencing on for tag 256 but left the outer
string_references/bytes_references maps in place -- the two TODOs in that
function marked exactly this. So references emitted inside a nested namespace
were numbered against the outer namespace, and strings registered inside it
leaked back out afterwards. The decoder already does this correctly: it pushes
a fresh namespace and resolves tag 25 against the innermost one.

The result is that the encoder produces output the decoder cannot read, and
sometimes reads as different data:

    >>> v = ["aaa", CBORTag(256, ["bbb", "ccc", "ddd", "bbb"])]
    >>> loads(dumps(v, string_referencing=True))
    ['aaa', ['bbb', 'ccc', 'ddd', 'ccc']]        # 'bbb' came back as 'ccc'

The stringref spec (http://cbor.schmorp.de/stringref) requires it:

    Within a value tagged with stringref-namespace, every string that is
    encoded with a definite length and has a minimum length is assigned an
    implicit index, starting from zero.

    Since stringref-namespace tags can be nested, the decoder needs to save
    and restore the outer array before starting and after ending the decoding
    of the tagged value.

Set the outer maps aside on entry and restore them on exit. With this, the
encoder reproduces the nested worked example published in the spec byte for
byte; the test asserts against those published bytes.
@coveralls

coveralls commented Sep 1, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 94.939% (+0.2%) from 94.766% — dylanpulver:fix-nested-stringref-namespace into agronholm:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants