Unable to interpret varint-encoded int32 protobuf message field when value is negative

[swang373]

After recently perusing the AwkwardForth docs and seeing mentions of protobuf support as a possibility, I started messing around with ForthMachine32 and some simple message schemas to see how far I could get. While doing so, I ran into an issue with decoding int32 fields when they are assigned a negative value, so I'd like to bring that up for discussion and help. For context, I'm working off of an M4 Macbook Pro running macOS Tahoe 26.2 and running an environment that has Python 3.13.11, protobuf 6.33.2, protoc 33.2, and awkward 2.8.11 installed.

Decoding with Protobuf

1. I created a file named test.proto and populated it with the following definition for the Test message.

edition = "2024";

message Test {
  int32 a = 1;
}

2. I used protoc to generate the module needed to use the Test message in Python.

# Generate the test_pb2.py file
protoc --python_out=. test.proto

3. I ran the following in a Python console.

>>> from test_pb2 import Test
>>> message = Test(a=-1)
>>> print(message)
a: -1

>>> encoded_message = message.SerializeToString()
>>> print(encoded_message)
b'\x08\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01'
>>> decoded_message = Test.FromString(encoded_message)
>>> print(decoded_message)
a: -1

Nothing unexpected here, just establishing a baseline for myself that I can create a Test message with its field "a" set to -1, serialize that message, and then call a static method with the serialized data to generate a new Test message instance which recovers the value of -1. The encoded message looks as I expect too: a tag followed by the two's complement of -1 encoded as a varint that takes up the last ten bytes.

Decoding with AwkwardForth

The first two steps have already been completed, so we can skip straight to the Python console and reproduce my problem.

>>> from awkward.forth import ForthMachine32
>>> vm = ForthMachine32("""
... input source
... \\ Read the tag to get to the value
... source varint-> stack
... \\ Decode the value back into two's complement
... source varint-> stack
... \\ I don't get to take the two's complement and recover
... \\ the original value because the read above raises an error
... """)
>>> from test_pb2 import Test
>>> message = Test(a=-1)
>>> print(message)
a: -1

>>> encoded_message = message.SerializeToString()
>>> print(encoded_message)
b'\x08\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01'
>>> vm.run({'source': encoded_message})
Traceback (most recent call last):
  File "<python-input-7>", line 1, in <module>
    vm.run({'source': encoded_message})
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: 'varint too big' in AwkwardForth runtime: variable-length integer is too big to represent as a fixed-width integer

That's the ValueError I keep running into. The first part of the exception states that the varint is too big for the AwkwardForth runtime. I don't know enough about its implementation to comment on that. The second part of the exception says that the varint is too big to represent as a fixed-width integer, but I can see the unsigned 64-bit representation of the two's complement of -1 peeking out of the last ten bytes.

Points of Discussion

• Is such a varint too big for the AwkwardForth runtime? I noticed that in the definition of ForthInputBuffer::read_varint(), there's a condition that checks whether shift == 7 * 9, which I suspect is what's holding AwkwardForth back from continuing to the tenth byte of the varint.

awkward/awkward-cpp/src/libawkward/forth/ForthInputBuffer.cpp

Lines 104 to 106 in c4a2e57

	if (shift == 7 * 9) {
	err = util::ForthError::varint_too_big;
	return 0;

• Using sint32 values which have their negative values zigzag encoded for more compactness might sidestep this issue for negative integer values, but that would ignore the fact that protobuf can handle the test case for negative value int32's, which sets up the expectation that we should be able to as well.

I'd love to hear your thoughts on this. I wish you a merry Christmas, and a huge thank you to the awkward development team for making one of my favorite libraries on the planet ;]

[ianna]

@swang373 — Thank you — that’s a great direction to explore.

Firstly, Protobuf encodes negative int32 values as 10‑byte varints. This is a quirk of Protobuf’s varint design:

• Protobuf varints are unsigned.
• int32 fields are not zigzag‑encoded.
• So negative values are encoded as their two’s‑complement 64‑bit representation, which always requires 10 bytes.

For -1, the varint is:

ff ff ff ff ff ff ff ff ff 01

This is exactly what you saw.

Protobuf’s decoder is perfectly happy with this because it always reads varints into a uint64 and then casts down to the declared type.

Secondly, AwkwardForth’s varint-> instruction is intentionally limited. AwkwardForth’s varint reader is designed for Awkward’s internal formats, not for Protobuf’s full varint range. The key line you found:

if (shift == 7 * 9) {
    err = util::ForthError::varint_too_big;
    return 0;
}

This means:

• AwkwardForth allows at most 9 bytes of varint payload.
• Protobuf negative int32 values require 10 bytes.
• Therefore AwkwardForth always errors on negative int32 fields encoded in Protobuf’s canonical way.

This is not a bug — it’s a design choice based on the fact that Awkward’s own varints never exceed 9 bytes.

Why does AwkwardForth stop at 9 Bytes?AwkwardForth’s ForthMachine32 uses 32‑bit stack integers. A 10‑byte varint can represent values up to:

2^64 - 1

But a 32‑bit stack cannot hold that. So the runtime enforces:

• Maximum shift = 7 * 9 = 63 bits
• Which corresponds to 9 bytes of payload
• Which fits in a 64‑bit intermediate but not in a 32‑bit stack slot

This is why the error message says:

variable-length integer is too big to represent as a fixed-width integer

It’s literally true: the Forth stack cannot hold a 64‑bit varint.

So your intuition is correct: AwkwardForth cannot decode Protobuf’s negative int32 varints because they exceed the supported varint width.

Could zigzag (sint32) avoid the issue? Yes — zigzag encoding avoids 10‑byte varints for negative numbers.

Example:

-1 → zigzag → 1 → varint = 0x01

This fits comfortably within AwkwardForth’s limits. But as you noted, that’s not how Protobuf encodes int32. Only sint32 uses zigzag. So zigzag is a workaround, not a fix.

What would need to change for full Protobuf compatibility? To support Protobuf’s full varint range, AwkwardForth would need:

1. A 64‑bit stack type

Right now ForthMachine32 uses 32‑bit stack integers.
A ForthMachine64 variant would solve this.

2. A varint reader that accepts 10‑byte varints

The shift == 7 * 9 guard would need to be relaxed.

3. Signed decoding logic for Protobuf’s int32

Protobuf’s rules are:

• Read varint as uint64
• Cast to int32
• Interpret as two’s complement

AwkwardForth currently has no instruction that performs this cast.

4. Possibly a dedicated Protobuf decoding dialect

Because Protobuf’s wire format has additional complexities (tags, wire types, zigzag, length‑delimited fields, etc.)

To summarize:

• Your analysis is correct: AwkwardForth’s varint reader intentionally rejects 10‑byte varints.
• Protobuf negative int32 values always encode to 10‑byte varints.
• Therefore AwkwardForth cannot decode negative int32 fields in Protobuf wire format.
• This is a limitation of the current implementation, not a misuse on your part.
• Supporting full Protobuf decoding would require extending AwkwardForth’s integer width and varint logic.

Using the ForthMachineOf<int64_t, int32_t> template is very much in the spirit of how a 64‑bit‑capable VM could evolve, even if the current implementation isn’t fully parameterized for it yet. Several parts of AwkwardForth still assume 32‑bit stack values (varint decoding, arithmetic ops, bounds checks, opcode semantics), so a ForthMachine64 would need deeper adjustments before it becomes a real, supported runtime.

That said, this is exactly the kind of exploration that helps the project grow. Awkward is an open‑source ecosystem, and we absolutely welcome contributors. If you’re interested in experimenting with a 64‑bit variant or sketching out what full Protobuf‑compatible varint decoding would require, we’d be very happy to review and discuss your ideas or patches.

And merry Christmas to you too — your message radiates exactly the kind of thoughtful curiosity that makes open‑source communities thrive.