You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
initializes the model on the 'meta' device (no memory allocation for weights),
72
65
and replaces supported linear layers with quantized ``W4Linear`` modules.
73
66
74
-
Parameters
75
-
----------
76
-
pretrained_model_name_or_path : str or os.PathLike
77
-
Path to the safetensors file containing the model weights and metadata.
78
-
**kwargs
79
-
Additional keyword arguments:
80
-
- torch_dtype : torch.dtype, optional
81
-
Data type for model initialization (default: torch.bfloat16).
82
-
- device : str or torch.device, optional
83
-
Device to load the model onto (default: "cuda").
84
-
85
-
Returns
86
-
-------
87
-
NunchakuT5EncoderModel
88
-
The loaded and quantized T5 encoder model.
67
+
:param pretrained_model_name_or_path: Path to the safetensors file containing the model weights and metadata.
68
+
:type pretrained_model_name_or_path: str or os.PathLike
69
+
:param torch_dtype: (optional) Data type for model initialization (default: torch.bfloat16). Need to set this to `torch.float16` for Turing GPUs.
70
+
:type torch_dtype: torch.dtype, optional
71
+
:param device: (optional) Device to load the model onto (default: "cuda"). Currently, only "cuda" is supported. If the model is loaded on CPU, we will automatically move it to GPU.
72
+
:type device: str or torch.device, optional
73
+
74
+
:return: The loaded and quantized T5 encoder model.
75
+
:rtype: NunchakuT5EncoderModel
89
76
90
77
Example
91
78
-------
92
79
.. code-block:: python
93
80
94
-
model = NunchakuT5EncoderModel.from_pretrained("path/to/model.safetensors", device="cuda")
81
+
model = NunchakuT5EncoderModel.from_pretrained("mit-han-lab/nunchaku-t5/awq-int4-flux.1-t5xxl.safetensors")
0 commit comments