Refactored modules/tokenizers to be a subdir of modules/transforms

[Ankur-singh]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (Refactored to convey that tokenizers are a subset of transforms.)

Please link to any issues this PR addresses. #1301

Changelog

What are the changes made in this PR?

• Moved modules/tokenizers to be a subdir of modules/transforms
• Updated corresponding imports and doc strings
• Updated Docs and links to source code

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
• include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

(tune) ➜  torchtune git:(move-tokenizers-to-transforms) ✗ tune run eleuther_eval --config recipes/configs/qwen2_5/evaluation.yaml         
Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Qwen2_5-0_5B-Instruct
  checkpoint_files:
  - model.safetensors
  model_type: QWEN2
  output_dir: ./
device: cuda
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.qwen2_5.qwen2_5_0_5b
output_dir: ./
quantizer: null
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.qwen2_5.qwen2_5_tokenizer
  max_seq_len: null
  merges_file: /tmp/Qwen2_5-0_5B-Instruct/merges.txt
  path: /tmp/Qwen2_5-0_5B-Instruct/vocab.json

2025-01-05:10:51:01,626 INFO     [_utils.py:28] Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Qwen2_5-0_5B-Instruct
  checkpoint_files:
  - model.safetensors
  model_type: QWEN2
  output_dir: ./
device: cuda
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.qwen2_5.qwen2_5_0_5b
output_dir: ./
quantizer: null
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.qwen2_5.qwen2_5_tokenizer
  max_seq_len: null
  merges_file: /tmp/Qwen2_5-0_5B-Instruct/merges.txt
  path: /tmp/Qwen2_5-0_5B-Instruct/vocab.json

Model is initialized with precision torch.bfloat16.
2025-01-05:10:51:02,219 INFO     [eleuther_eval.py:503] Model is initialized with precision torch.bfloat16.
2025-01-05:10:51:02,419 INFO     [huggingface.py:132] Using device 'cuda:0'
2025-01-05:10:51:02,886 INFO     [huggingface.py:369] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
Running evaluation on the following tasks: ['truthfulqa_mc2']
2025-01-05:10:51:13,798 INFO     [eleuther_eval.py:540] Running evaluation on the following tasks: ['truthfulqa_mc2']
2025-01-05:10:51:13,799 INFO     [task.py:415] Building contexts for truthfulqa_mc2 on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [00:00<00:00, 1745.50it/s]
2025-01-05:10:51:14,294 INFO     [evaluator.py:496] Running loglikelihood requests
Running loglikelihood requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5882/5882 [08:40<00:00, 11.31it/s]
Eval completed in 522.65 seconds.
2025-01-05:10:59:56,450 INFO     [eleuther_eval.py:549] Eval completed in 522.65 seconds.
Max memory allocated: 10.00 GB
2025-01-05:10:59:56,451 INFO     [eleuther_eval.py:550] Max memory allocated: 10.00 GB


|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.4178|±  |0.0146|


2025-01-05:10:59:56,536 INFO     [eleuther_eval.py:554] 

|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.4178|±  |0.0146|

UX

If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example

• I did not change any public API
• I have added an example to docs or docstrings
• Updated the docs

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2231

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f6a5b2c with merge base 779569e ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.22222% with 1 line in your changes missing coverage. Please review.

Project coverage is 64.11%. Comparing base (779569e) to head (f6a5b2c).

Files with missing lines	Patch %	Lines
recipes/eleuther_eval.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2231       +/-   ##
===========================================
+ Coverage   23.97%   64.11%   +40.14%     
===========================================
  Files         358      353        -5     
  Lines       21207    20680      -527     
===========================================
+ Hits         5084    13260     +8176     
+ Misses      16123     7420     -8703     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Ankur-singh]

@joecummings looks like there is a merge conflict but I am unable to see it. How do I fix it?

[ebsmothers]

@Ankur-singh sorry we missed this comment until now. In my Github UI I see the following just above the "Add a comment" box (might be slightly different depending on your permissions).

So seems like the merge conflict is in _sentencepiece.py