Skip to content

Serialize Signed Ints with a compact binary representation #953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

stalkopat
Copy link

@stalkopat stalkopat commented May 20, 2025

Motivation

Ints are currently always serialized as strings, even for binary formats. This causes quite a bit of overhead depending on the downstream usage. (For example persisting to disk)
I256::MAX with the existing serialization format uses ~85 bytes with bincode, v.s. 40 bytes with this change

Solution

Use is_human_readable to serialize Ints as compact binary blobs instead.

PR Checklist

  • Added Tests
  • Added Documentation
  • Breaking changes

Considerations

This change isn't backwards compatible with already serialized data that has been f.x. persisted to disk with a non human readable format. Keeping the deserialize_any in the deserialize should technically work, but apparently some of the self describing formats just treat strings and bytes interchangeably, which means those will face compatibility issues anyway.

Making this change has the additional benefit of now allowing f.x. bincode to work with ints (because it gets rid of the deserialize_any in the non readable path).

Json and related readable formats (which I assume is the majority of downstream usage) shouldn't be impacted.

@stalkopat
Copy link
Author

stalkopat commented May 20, 2025

On a second thought, up until ~2^103 the string representation is actually smaller than the fixed 32 byte representation, this might do more harm than good for the vast majority of use cases. Definitely think deserialize_any should go for non human readable formats though. Just serializing those as string doesn't quite feel right either.

@stalkopat stalkopat closed this May 20, 2025
@github-project-automation github-project-automation bot moved this to Done in Alloy May 20, 2025
@stalkopat stalkopat reopened this May 21, 2025
@github-project-automation github-project-automation bot moved this from Done to In Progress in Alloy May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

1 participant