File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -19,7 +19,7 @@ knitr::opts_chunk$set(
1919[ ![ R build status] ( https://github.com/mlverse/tok/workflows/R-CMD-check/badge.svg )] ( https://github.com/mlverse/tok/actions )
2020[ ![ CRAN status] ( https://www.r-pkg.org/badges/version/tok )] ( https://CRAN.R-project.org/package=tok )
2121[ ![ ] ( https://cranlogs.r-pkg.org/badges/tok )] ( https://cran.r-project.org/package=tok )
22- [ ![ extendr] ( https://img.shields.io/badge/extendr-^0.8.1-276DC2 )] ( https://extendr.github.io /extendr/extendr_api/ )
22+ [ ![ extendr] ( https://img.shields.io/badge/extendr-^0.8.1-276DC2 )] ( https://extendr.rs /extendr/extendr_api/ )
2323<!-- badges: end -->
2424
2525tok provides bindings to the [ 🤗tokenizers] ( https://huggingface.co/docs/tokenizers/v0.13.3/en/index ) library. It uses the same Rust libraries that powers the Python implementation.
Original file line number Diff line number Diff line change @@ -10,7 +10,7 @@ status](https://github.com/mlverse/tok/workflows/R-CMD-check/badge.svg)](https:/
1010[ ![ CRAN
1111status] ( https://www.r-pkg.org/badges/version/tok )] ( https://CRAN.R-project.org/package=tok )
1212[ ![ ] ( https://cranlogs.r-pkg.org/badges/tok )] ( https://cran.r-project.org/package=tok )
13- [ ![ extendr] ( https://img.shields.io/badge/extendr-%5E0.8.1-276DC2 )] ( https://extendr.github.io /extendr/extendr_api/ )
13+ [ ![ extendr] ( https://img.shields.io/badge/extendr-%5E0.8.1-276DC2 )] ( https://extendr.rs /extendr/extendr_api/ )
1414<!-- badges: end -->
1515
1616tok provides bindings to the
Original file line number Diff line number Diff line change 1- CRAN re-submission with Windows fixes that caused it to be removed.
1+ Fixes:
22
3- Note:
4- installed size is 5.3Mb
5- sub-directories of 1Mb or more:
6- libs 4.5Mb
3+ For the HTML refman generated via tools::pkg2HTML() we now get
74
8- Is caused by compiling the included tokenizers Rust library and its dependencies.
5+ line 827 column 1 - Warning: <a > anchor "method-tok_normalizer_nfc-new" already defined
6+ line 843 column 1 - Warning: <a > anchor "method-tok_normalizer_nfc-clone" already defined
7+ line 1130 column 1 - Warning: <a > anchor "method-tok_pre_tokenizer_whitespace-new" already defined
8+ line 1146 column 1 - Warning: <a > anchor
9+ "method-tok_pre_tokenizer_whitespace-clone" already defined
10+
11+ the first apparently from
12+
13+ ./man/normalizer_nfkc.Rd:\item \href{#method-tok_normalizer_nfc-new}{\code{normalizer_nfkc$new()}}
14+ ./man/normalizer_nfc.Rd:\item \href{#method-tok_normalizer_nfc-new}{\code{normalizer_nfc$new()}}
You can’t perform that action at this time.
0 commit comments