Count dataset tokens#1055

Draft

plaguss wants to merge 38 commits intodevelopfrom

count-dataset-tokens

Commits on Oct 11, 2024

Commits on Oct 15, 2024

Commits on Oct 16, 2024

Commits on Oct 17, 2024

Fix async llms not returning properly the generations grouped by num_generations
plaguss
committed
Fix async llms not processing multiple generations
plaguss
committed

Commits on Oct 18, 2024

Fix vllm sorting mechanism and add mocked generate method to the test suite
plaguss
committed

Commits on Oct 22, 2024

Checkpoint
plaguss
committed

Commits on Oct 23, 2024

Commits on Oct 24, 2024

Commits on Oct 25, 2024

Fix dummy offline batch generation
plaguss
committed

Commits on Oct 28, 2024