Skip to content

Commit c789ebe

Browse files
docs: merge notes and limitations section with when to not use TOON section
1 parent 9d260e4 commit c789ebe

File tree

1 file changed

+0
-9
lines changed

1 file changed

+0
-9
lines changed

README.md

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1168,15 +1168,6 @@ By default, the decoder validates input strictly:
11681168
- **Array length mismatches**: Throws when declared length doesn't match actual count.
11691169
- **Delimiter mismatches**: Throws when row delimiters don't match header.
11701170

1171-
## Notes and Limitations
1172-
1173-
- Format familiarity and structure matter as much as token count. TOON's tabular format requires arrays of objects with identical keys and primitive values only. When this doesn't hold (due to mixed types, non-uniform objects, or nested structures), TOON switches to list format where JSON can be more efficient at scale.
1174-
- **TOON excels at:** Uniform arrays of objects (same fields, primitive values), especially large datasets with consistent structure.
1175-
- **JSON is better for:** Non-uniform data, deeply nested structures, and objects with varying field sets.
1176-
- **CSV is more compact for:** Flat, uniform tables without nesting. TOON adds structure (`[N]` array lengths, delimiter scoping, deterministic quoting) that improves LLM reliability with minimal token overhead.
1177-
- **Token counts vary by tokenizer and model.** Benchmarks use a GPT-style tokenizer (cl100k/o200k); actual savings will differ with other models (e.g., [SentencePiece](https://github.com/google/sentencepiece)).
1178-
- **TOON is designed for LLM input** where human readability and token efficiency matter. It's **not** a drop-in replacement for JSON in APIs or storage.
1179-
11801171
## Using TOON in LLM Prompts
11811172

11821173
TOON works best when you show the format instead of describing it. The structure is self-documenting – models parse it naturally once they see the pattern.

0 commit comments

Comments
 (0)