Skip to content

Commit 2b4230c

Browse files
committed
updated readme
1 parent 5846be2 commit 2b4230c

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,9 @@ const tokenizerConfig = await fetch(`https://huggingface.co/${modelId}/resolve/m
4444
const tokenizer = new Tokenizer(tokenizerJson, tokenizerConfig);
4545

4646
// Tokenize text
47-
const tokens = tokenizer.tokenize('Hello World'); // ['Hello', 'ĠWorld']
48-
const encoded = tokenizer.encode('Hello World'); // [9906, 4435]
49-
const decoded = tokenizer.decode(encoded); // 'Hello World'
47+
const tokens = tokenizer.tokenize('Hello World'); // ['Hello', 'ĠWorld']
48+
const encoded = tokenizer.encode('Hello World'); // { ids: [9906, 4435], tokens: ['Hello', 'ĠWorld'], attention_mask: [1, 1] }
49+
const decoded = tokenizer.decode(encoded.ids); // 'Hello World'
5050
```
5151

5252
## Requirements

0 commit comments

Comments
 (0)