Skip to content

Commit 787dda2

Browse files
committed
fix the t5 docs
1 parent f3569dc commit 787dda2

1 file changed

Lines changed: 19 additions & 32 deletions

File tree

nunchaku/models/text_encoders/t5_encoder.py

Lines changed: 19 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,22 @@
11
"""
2-
Nunchaku T5 Encoder integration.
2+
Nunchaku T5 Encoder
33
4-
This module provides a wrapper for loading and quantizing T5EncoderModel weights from safetensors
5-
for use in Nunchaku's quantized transformer backend.
4+
This module provides the implementation of the Nunchaku T5 Encoder.
65
76
Classes
87
-------
9-
- NunchakuT5EncoderModel : T5 encoder with quantized linear layer injection and safetensors loading.
8+
.. autosummary::
9+
:toctree: generated/
10+
:nosignatures:
1011
11-
Functions
12-
---------
13-
- NunchakuT5EncoderModel.from_pretrained : Load a quantized T5 encoder from a safetensors file.
12+
NunchakuT5EncoderModel
1413
1514
Example
1615
-------
1716
.. code-block:: python
1817
19-
from nunchaku.models.text_encoders.t5_encoder import NunchakuT5EncoderModel
20-
model = NunchakuT5EncoderModel.from_pretrained("path/to/model.safetensors")
18+
from nunchaku import NunchakuT5EncoderModel
19+
model = NunchakuT5EncoderModel.from_pretrained("mit-han-lab/nunchaku-t5/awq-int4-flux.1-t5xxl.safetensors")
2120
"""
2221

2322
import json
@@ -49,17 +48,11 @@ class NunchakuT5EncoderModel(T5EncoderModel):
4948
from safetensors files and replacing linear layers with quantized ``W4Linear`` modules
5049
where appropriate.
5150
52-
Methods
53-
-------
54-
from_pretrained(pretrained_model_name_or_path, **kwargs)
55-
Load a quantized T5 encoder from a safetensors file, replacing supported linear layers
56-
with quantized versions.
57-
5851
Example
5952
-------
6053
.. code-block:: python
6154
62-
model = NunchakuT5EncoderModel.from_pretrained("path/to/model.safetensors")
55+
model = NunchakuT5EncoderModel.from_pretrained("mit-han-lab/nunchaku-t5/awq-int4-flux.1-t5xxl.safetensors")
6356
"""
6457

6558
@classmethod
@@ -71,27 +64,21 @@ def from_pretrained(cls, pretrained_model_name_or_path: str | os.PathLike[str],
7164
initializes the model on the 'meta' device (no memory allocation for weights),
7265
and replaces supported linear layers with quantized ``W4Linear`` modules.
7366
74-
Parameters
75-
----------
76-
pretrained_model_name_or_path : str or os.PathLike
77-
Path to the safetensors file containing the model weights and metadata.
78-
**kwargs
79-
Additional keyword arguments:
80-
- torch_dtype : torch.dtype, optional
81-
Data type for model initialization (default: torch.bfloat16).
82-
- device : str or torch.device, optional
83-
Device to load the model onto (default: "cuda").
84-
85-
Returns
86-
-------
87-
NunchakuT5EncoderModel
88-
The loaded and quantized T5 encoder model.
67+
:param pretrained_model_name_or_path: Path to the safetensors file containing the model weights and metadata.
68+
:type pretrained_model_name_or_path: str or os.PathLike
69+
:param torch_dtype: (optional) Data type for model initialization (default: torch.bfloat16). Need to set this to `torch.float16` for Turing GPUs.
70+
:type torch_dtype: torch.dtype, optional
71+
:param device: (optional) Device to load the model onto (default: "cuda"). Currently, only "cuda" is supported. If the model is loaded on CPU, we will automatically move it to GPU.
72+
:type device: str or torch.device, optional
73+
74+
:return: The loaded and quantized T5 encoder model.
75+
:rtype: NunchakuT5EncoderModel
8976
9077
Example
9178
-------
9279
.. code-block:: python
9380
94-
model = NunchakuT5EncoderModel.from_pretrained("path/to/model.safetensors", device="cuda")
81+
model = NunchakuT5EncoderModel.from_pretrained("mit-han-lab/nunchaku-t5/awq-int4-flux.1-t5xxl.safetensors")
9582
"""
9683
pretrained_model_name_or_path = Path(pretrained_model_name_or_path)
9784
state_dict, metadata = load_state_dict_in_safetensors(pretrained_model_name_or_path, return_metadata=True)

0 commit comments

Comments
 (0)