Skip to content

NEWS.md

Latest commit

 

History

History
48 lines (30 loc) · 1.11 KB

NEWS.md

File metadata and controls

48 lines (30 loc) · 1.11 KB

tok (development version)

  • Lowered minimum supported rust version (MSRV) from 1.91 to 1.77.2, at request of CRAN (#29).

tok 0.2.2

  • Updated upstream tokenizers library to 0.22.2.
  • Update extendr-api to 0.9.0.
  • Changed maintainer to Tomasz Kalinowski.

tok 0.2.1

  • Fixed tiny documentation issue for normalizers.

tok 0.2.0

  • Updated upstream tokenizers to 0.20.3
  • Update extendr-api to 0.8.1

tok 0.1.5

  • add message translation in FR (#19, @cregouby)

tok 0.1.4

  • Updated libR-sys to fix mac oldrel notes. (#18)
  • Report rust version during compilation.

tok 0.1.3

tok 0.1.2

  • Added support for authentication. (#7)
  • Added support for training tokenizers from scratch. (#8)
  • Added support for serializing tokenizers. (#8)
  • Implemented support for configuring padding and truncation. (#9)
  • Added support for decoders, post_processors and normalizers. (#10)
  • Added support for get_vocab_size(). (#10)
  • Updated extendr version to avoid non-API function calls. (#11)

tok 0.1.1

  • Dependencies are now vendored to avoid versioning problems on CRAN. (#5)

tok 0.1.0

  • Initial CRAN submission.