Skip to content

Commit 23444b6

Browse files
docs: use automd for README updates
1 parent 3ee5afa commit 23444b6

File tree

6 files changed

+288
-27
lines changed

6 files changed

+288
-27
lines changed

README.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ For advanced use cases requiring precise token counts, please use a full tokeniz
1010

1111
The following table shows the accuracy of the token count approximation for different input texts:
1212

13-
<!-- START GENERATED TOKEN COUNT TABLE -->
13+
<!-- automd:file src="./docs/bench.md" -->
14+
1415
| Description | Actual GPT Token Count | Estimated Token Count | Token Count Deviation |
1516
| --- | --- | --- | --- |
1617
| Short English text | 10 | 11 | 10.00% |
@@ -19,7 +20,8 @@ The following table shows the accuracy of the token count approximation for diff
1920
| Die Verwandlung by Franz Kafka (German) | 35309 | 35069 | 0.68% |
2021
| 道德經 by Laozi (Chinese) | 11712 | 12059 | 2.96% |
2122
| TypeScript ES5 Type Declarations (~ 4000 loc) | 49304 | 52615 | 6.72% |
22-
<!-- END GENERATED TOKEN COUNT TABLE -->
23+
24+
<!-- /automd -->
2325

2426
## Features
2527

docs/bench.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
| Description | Actual GPT Token Count | Estimated Token Count | Token Count Deviation |
2+
| --- | --- | --- | --- |
3+
| Short English text | 10 | 11 | 10.00% |
4+
| German text with umlauts | 48 | 49 | 2.08% |
5+
| Metamorphosis by Franz Kafka (English) | 31796 | 35705 | 12.29% |
6+
| Die Verwandlung by Franz Kafka (German) | 35309 | 35069 | 0.68% |
7+
| 道德經 by Laozi (Chinese) | 11712 | 12059 | 2.96% |
8+
| TypeScript ES5 Type Declarations (~ 4000 loc) | 49304 | 52615 | 6.72% |

package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
],
3838
"scripts": {
3939
"build": "tsdown",
40-
"docs:generate": "tsx scripts/generateTable.ts",
40+
"automd": "tsx scripts/generateTable.ts && automd",
4141
"lint": "eslint .",
4242
"lint:fix": "eslint . --fix",
4343
"test": "vitest",
@@ -47,6 +47,7 @@
4747
"devDependencies": {
4848
"@antfu/eslint-config": "^6.0.0",
4949
"@types/node": "^24.7.2",
50+
"automd": "^0.4.2",
5051
"bumpp": "^10.3.1",
5152
"eslint": "^9.37.0",
5253
"gpt-tokenizer": "^3.2.0",

0 commit comments

Comments
 (0)