[Proposal] Mathematical Formalization of TOON's Efficiency vs JSON (Character-based Approach)

[mateolafalce]

Hey Team!

I have been following the development of TOON and recently published a preliminary article exploring the efficiency gains of this format compared to JSON. While my initial article touches on the subject, I believe TOON would benefit significantly from a rigorous, mathematically formal proof of its superiority in terms of information density and structural overhead.

The Proposal

I would like to propose (and contribute to) a formal mathematical framework that proves TOON is strictly more efficient than JSON for specific classes of data structures (particularly uniform arrays and nested objects).

Unlike current benchmarks that focus on token counts, which are highly dependent on the specific tokenizer, I propose strictly formalizing this comparison based on Character Count. Tokenization is variable; Character length is an absolute metric of data transport and storage efficiency. If we prove $L_{char}(TOON) < L_{char}(JSON)$, the token savings naturally follow as a corollary, but the proof remains tokenizer-agnostic.

Before I proceed with a full formal paper or a documentation contribution:

• Is this type of formalization something you would be interested in including in the official TOON repository (README or /docs)?

Looking forward to your thoughts !

[ksmelnad]

Interesting and sensible.

[johannschopplich]

@mateolafalce OMG, thanks for opening this and for thinking so deeply about TOON's efficiency. I'd absolutely be interested in including this in the official materials. Some ideas on placement:

• A focused write‑up under docs/ or a "Benchmarks & Theory" section (e.g. docs/efficiency-formalization.md), with:
  • Clear assumptions (canonical JSON vs canonical TOON per the spec, treatment of value lengths, etc.),
  • Closed‑form expressions for each format on a given structure class,
  • And the derived inequalities / asymptotics.
• A short, high‑level summary or figure in the README / benchmarks page that links to the full derivation.

If you're up for it, a PR would be very welcome. Even a first pass with a couple of well‑defined structure families (say, uniform arrays of objects and maybe one nested‑object pattern) would already be valuable, and we can iterate from there on scope and presentation.

Feel free to sketch the outline here first if you'd like early feedback, or just open a PR when you're ready. We'll find the right place in the project for sure.

[mateolafalce]

Perfect. I'll sketch out the proposal in the next few days and get it up for review as soon as possible

[johannschopplich]

@mateolafalce No rush, good things take time. I'm looking forward to it tho. :)