tok/DESCRIPTION at main · mlverse/tok · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Package: tok
Title: Fast Text Tokenization
Version: 0.2.2.9000
Authors@R: c(
    person("Tomasz", "Kalinowski", , "tomasz@posit.co", c("ctb", "cre")),
    person("Daniel", "Falbel", , "dfalbel@gmail.com", c("aut")),
    person("Regouby", "Christophe", , "christophe.regouby@free.fr", c("ctb")),
    person(family = "Posit", role = c("cph"))
  )
Description:
  Interfaces with the 'Hugging Face' tokenizers library to provide implementations
  of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm
  <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both
  training new vocabularies and tokenizing texts.
License: MIT + file LICENSE
SystemRequirements: Cargo (Rust's package manager), rustc >= 1.77.2
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
Depends:
    R (>= 4.2.0)
Imports:
    R6,
    cli
Suggests:
    rmarkdown,
    testthat (>= 3.0.0),
    hfhub (>= 0.1.1),
    withr
Config/testthat/edition: 3
URL: https://github.com/mlverse/tok
BugReports: https://github.com/mlverse/tok/issues
Config/rextendr/version: 0.5.0
Config/roxygen2/version: 8.0.0