Binary TOON typically achieves 50-70% size reduction compared to text TOON, read README #201
+1,991
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Understood how the existing synchronous API works with complete data in memory - Designed streaming API interface - Created `encodeStream()` and `decodeStream()` functions that return Transform streams - Added stream-related types - Extended options interfaces with streaming-specific configurations - Implemented encodeStream() function - Processes streams of JSON values into TOON format strings - Implemented decodeStream() function - Processes streams of TOON strings into JSON values - Updated main index.ts exports - Exported streaming functions and types - Added tests for streaming functionality - Comprehensive test suite covering encoding, decoding, and round-trip scenarios - Updated CLI to support streaming operations - Added `--stream` flag with streaming implementations for both encode and decode operations
- Complex nested objects with arrays and primitives - Key folding and path expansion options - All TOON formatting features (tabular arrays, inline arrays, etc.) - Proper handling of strings, numbers, booleans (null excluded from streams due to Node.js limitations) Implementation Quality - Memory efficient: Processes data incrementally without loading entire datasets - Backpressure handling: Proper Node.js stream flow control - Error handling: Graceful handling of malformed input - Type safety: Full TypeScript support with proper type definitions - API consistency: Follows existing TOON library patterns and conventions
This reverts commit 3ef97e7.
merge changes
…token efficiency for very large datasets and CLI also
- **Data Archives**: Storage where size matters more than readability - **API Communication**: Machine-to-machine data transfer - **Large Datasets**: Anything over 1MB where compression helps - **Streaming**: High-throughput data pipelines **Do NOT use for:** - Human-readable files - Web API responses (JSON/TOON text expected) - Small datasets (<10KB) - Debugging scenarios (binary not inspectable)
|
what about vs. cbor |
Author
|
@He-Pin TOON's binary format is designed specifically for its token-efficient text format and maintains round-trip compatibility with TOON data, rather than conforming to CBOR standards. for others, CBOR uses encoded major types (e.g., null is 0xf6, boolean false is 0xf4) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.