Cross-language conformance testing for all TOON implementations #164
Description
Hi everyone,
With 20+ language implementations of TOON now (which is amazing!), we should probably have some automated testing to make sure they all produce compatible output. Right now it's hard to know if the Python implementation and the Java one would handle the same data identically.
The problem
Each language implementation might:
- Handle edge cases differently
- Use different delimiter selection strategies
- Have subtle parsing differences
- Produce valid but incompatible TOON for the same input
What I'm proposing
A cross-language conformance test suite that:
- Defines canonical test cases with expected outputs
- Runs the same tests against all language implementations
- Reports compatibility issues and differences
- Can be integrated into CI for each language repo
Test case categories
Basic functionality
- Simple objects and arrays
- Nested structures
- Various data types (strings, numbers, booleans, null)
Edge cases
- Empty objects/arrays
- Special characters in strings
- Very large numbers
- Unicode handling
Optimization scenarios
- Different delimiter selection strategies
- Repeated key patterns
- Various nesting levels
Implementation approach
Could build on the existing benchmark infrastructure:
- JSON test data that all implementations should handle identically
- Expected TOON output for each test case
- Automated runner that tests each language
- Diff reporting for mismatches
Benefits
- Catch compatibility issues early
- Help new language implementations get it right
- Build confidence in the format consistency
- Make it easier to debug cross-language issues
The community has done amazing work getting TOON implemented in so many languages. This would help ensure they all work together seamlessly.
Thoughts on the approach? Happy to help coordinate the test case definitions.