Can't use quantized models of schnell, dev, qwen or fibo. Only z-image-turbo works

[timaschew]

I'm on a MacBook Pro M1 with macOS 15.
I've installed pyenv, then python 3.11 and made it default globally and created a venv.
Finally I've installed mflux (0.13.3)
BTW: uv did not work for me, at least there are more instructions required in the README.md of mlux

I've downloaded the quantized models from hugging face before via their python module.
I've downloaded all the listed models:

• dhairyashil/FLUX.1-schnell-mflux-v0.6.2-4bit
• dhairyashil/FLUX.1-dev-mflux-4bit
• akx/FLUX.1-Kontext-dev-mflux-4bit
• filipstrand/FLUX.1-Krea-dev-mflux-4bit
• filipstrand/Qwen-Image-mflux-6bit
• filipstrand/Z-Image-Turbo-mflux-4bit
• briaai/Fibo-mlx-4bit
• briaai/Fibo-mlx-8bit

Then I've linked each model via ln, for example ln -s ~/.cache/huggingface/hub/models--filipstrand--Z-Image-Turbo-mflux-4bit/snapshots/b3a8f31115a11f2f9e2fa0bfbc8d78dcc3e6568b model-links/z-image-turbo-4bit to my project directory for easier references.

I've tried two ways of generate an image:

z-image-turbo ✅

mflux-generate --model model-links/z-image-turbo-4bit --steps 9  --prompt "Luxury food photograph"

That didn't work, I've also tried to set --base-model z-image-turbo but still the same error:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/bin/mflux-generate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/cli/flux_generate.py", line 28, in main
    flux = Flux1(
           ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/variants/txt2img/flux.py", line 38, in __init__
    FluxInitializer.init(
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 37, in init
    weights = FluxInitializer._load_weights(path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 164, in _load_weights
    return WeightLoader.load(
           ^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 57, in load
    weights, q_level, version = WeightLoader._load_component(root_path, component, raw_weights_cache)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 99, in _load_component
    raw_weights = WeightLoader._load_safetensors(component_path, component.loading_mode)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 203, in _load_safetensors
    return WeightLoader._load_mlx_native(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 221, in _load_mlx_native
    raise FileNotFoundError(f"No safetensors files found in {path}")
FileNotFoundError: No safetensors files found in model-links/z-image-turbo-4bit/text_encoder_2

Then I've tried this syntax:

mflux-generate-z-image-turbo --model filipstrand/Z-Image-Turbo-mflux-4bit --steps 9 --prompt "Luxury food photograph"

and it worked, yeah!

flux1.schnell ❌

Then I've tried to use flux.1-schnell

I've used this command:

mflux-generate --model model-links/flux.1-schnell-4bit --base-model schnell --steps 2 --seed 2 --prompt "Luxury food photograph"

but I got this error:

Could not extract SentencePiece model from model-links/flux.1-schnell-4bit/tokenizer_2/spiece.model using sentencepiece library due to
SentencePieceExtractor requires the protobuf library but it was not found in your environment. Check out the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.
. Falling back to TikToken extractor.
Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py", line 161, in convert_to_native_format
    local_kwargs = SentencePieceExtractor(vocab_file).extract(cls.model, **local_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 154, in __init__
    requires_backends(self, "protobuf")
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1844, in requires_backends
    raise ImportError("".join(failed))
ImportError:
SentencePieceExtractor requires the protobuf library but it was not found in your environment. Check out the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 1854, in extract_vocab_merges_from_model
    from tiktoken.load import load_tiktoken_bpe
ModuleNotFoundError: No module named 'tiktoken'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/bin/mflux-generate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/cli/flux_generate.py", line 28, in main
    flux = Flux1(
           ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/variants/txt2img/flux.py", line 38, in __init__
    FluxInitializer.init(
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 38, in init
    FluxInitializer._init_tokenizers(model, path, model_config)
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 171, in _init_tokenizers
    model.tokenizers = TokenizerLoader.load_all(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 50, in load_all
    tokenizer = TokenizerLoader.load(d, model_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 30, in load
    raw_tokenizer = TokenizerLoader._load_raw_tokenizer(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 118, in _load_raw_tokenizer
    tokenizer = cls.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1757, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2006, in _from_pretrained
    init_kwargs = cls.convert_to_native_format(**init_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py", line 184, in convert_to_native_format
    ).extract_vocab_merges_from_model(vocab_file)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 1856, in extract_vocab_merges_from_model
    raise ValueError(
ValueError: `tiktoken` is required to read a `tiktoken` file. Install it with `pip install tiktoken`.

I've installed tiktoken and tried to run again, but then I got antoher error:

Could not extract SentencePiece model from model-links/flux.1-schnell-4bit/tokenizer_2/spiece.model using sentencepiece library due to
SentencePieceExtractor requires the protobuf library but it was not found in your environment. Check out the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.
. Falling back to TikToken extractor.
Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py", line 161, in convert_to_native_format
    local_kwargs = SentencePieceExtractor(vocab_file).extract(cls.model, **local_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 154, in __init__
    requires_backends(self, "protobuf")
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1844, in requires_backends
    raise ImportError("".join(failed))
ImportError:
SentencePieceExtractor requires the protobuf library but it was not found in your environment. Check out the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/tiktoken/load.py", line 168, in load_tiktoken_bpe
    token, rank = line.split()
    ^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/bin/mflux-generate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/cli/flux_generate.py", line 28, in main
    flux = Flux1(
           ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/variants/txt2img/flux.py", line 38, in __init__
    FluxInitializer.init(
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 38, in init
    FluxInitializer._init_tokenizers(model, path, model_config)
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 171, in _init_tokenizers
    model.tokenizers = TokenizerLoader.load_all(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 50, in load_all
    tokenizer = TokenizerLoader.load(d, model_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 30, in load
    raw_tokenizer = TokenizerLoader._load_raw_tokenizer(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/tokenizer/tokenizer_loader.py", line 118, in _load_raw_tokenizer
    tokenizer = cls.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1757, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2006, in _from_pretrained
    init_kwargs = cls.convert_to_native_format(**init_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py", line 184, in convert_to_native_format
    ).extract_vocab_merges_from_model(vocab_file)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 1860, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/tiktoken/load.py", line 171, in load_tiktoken_bpe
    raise ValueError(f"Error parsing line {line!r} in {tiktoken_bpe_file}") from e
ValueError: Error parsing line b'\x0e' in model-links/flux.1-schnell-4bit/tokenizer_2/spiece.model

I've the same error with flux.1-dev

qwen ❌

This is my command:

mflux-generate-qwen --model model-links/qwen-6bit --base-model qwen --prompt "Luxury food photograph" --steps 20

and here I get this error:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/bin/mflux-generate-qwen", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/cli/qwen_image_generate.py", line 51, in main
    image = qwen.generate_image(
            ^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/variants/txt2img/qwen_image.py", line 85, in generate_image
    prompt_embeds, prompt_mask, negative_prompt_embeds, negative_prompt_mask = QwenPromptEncoder.encode_prompt(
                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/model/qwen_text_encoder/qwen_prompt_encoder.py", line 29, in encode_prompt
    prompt_embeds, prompt_mask = qwen_text_encoder(
                                 ^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/model/qwen_text_encoder/qwen_text_encoder.py", line 17, in __call__
    hidden_states = self.encoder(input_ids, attention_mask)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/model/qwen_text_encoder/qwen_encoder.py", line 110, in __call__
    hidden_states = layer(hidden_states, attention_mask_4d, position_embeddings)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/model/qwen_text_encoder/qwen_encoder_layer.py", line 43, in __call__
    hidden_states = self.input_layernorm(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/qwen/model/qwen_text_encoder/qwen_rms_norm.py", line 16, in __call__
    result = self.weight.astype(mx.float32) * hidden_states
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
ValueError: [broadcast_shapes] Shapes (3584) and (1,1024,672) cannot be broadcast.

fibo ❌

I've this command:

mflux-generate-fibo --model model-links/fibo-4bit --steps 20 --guidance 4.0 --seed 42 --output animal_bakers.png   --prompt "Three cartoon animal chefs in a colorful bakery kitchen"

And then it starts to download another model:

Downloading model from HuggingFace: briaai/FIBO-vlm...

When I use this syntax:

mflux-generate --model model-links/fibo-4bit --base-model fibo --steps 20 --guidance 4.0 --seed 42 --output animal_bakers.png   --prompt "Three cartoon animal chefs in a colorful bakery kitchen"

Then I get this error:

Traceback (most recent call last):
  File "/Users/johndoe/dev/mflux/.venv/bin/mflux-generate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/cli/flux_generate.py", line 28, in main
    flux = Flux1(
           ^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/variants/txt2img/flux.py", line 38, in __init__
    FluxInitializer.init(
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 37, in init
    weights = FluxInitializer._load_weights(path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 164, in _load_weights
    return WeightLoader.load(
           ^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 57, in load
    weights, q_level, version = WeightLoader._load_component(root_path, component, raw_weights_cache)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 99, in _load_component
    raw_weights = WeightLoader._load_safetensors(component_path, component.loading_mode)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 203, in _load_safetensors
    return WeightLoader._load_mlx_native(path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johndoe/dev/mflux/.venv/lib/python3.11/site-packages/mflux/models/common/weights/loading/weight_loader.py", line 221, in _load_mlx_native
    raise FileNotFoundError(f"No safetensors files found in {path}")
FileNotFoundError: No safetensors files found in model-links/fibo-4bit/text_encoder_2

It's true that the fibo-4bit model contains only these directories:

model-links/fibo-4bit
├── README.md -> ../../blobs/cd1bb9bb5dc69e7285cf0f57dc975fbdd5178a05
├── text_encoder
│   └── 0.safetensors -> ../../../blobs/97e7ec568880484c86d206af82b9b5be8c1af4928e37891af8cf9d5d56152022
├── tokenizer
│   ├── chat_template.jinja -> ../../../blobs/e01e3a1bca00ae47bca8326b38cc397729f87481
│   ├── special_tokens_map.json -> ../../../blobs/190d5624dbbc1ad56f2f34c9d58e03fef7e5328b
│   ├── tokenizer_config.json -> ../../../blobs/61910c2db5cbdc9e6a6f37e14aaf00584cc6ad47
│   └── tokenizer.json -> ../../../blobs/7b6a500b662a34eb3f0374db856ba4ad7de4c81040571d78dc0d357238930005
├── transformer
│   ├── 0.safetensors -> ../../../blobs/50ee8eaa604d4299cc1126d2ac0e54f426775cfb02fc0849d1700a63b1cc17ae
│   ├── 1.safetensors -> ../../../blobs/a1b206f77d5d566bed9692bf1df0af154eeea84f3159282e484b465e8c3aee1b
│   └── 2.safetensors -> ../../../blobs/1c51d429fec6c11233296e242c71f43fe286e8c06d50e05ed3bf631bccc97ecb
└── vae
    ├── 0.safetensors -> ../../../blobs/1307796b5605b8d2cf674f0ec2f404f740e58a9f67243b163cbe2c507afc4e66
    └── 1.safetensors -> ../../../blobs/e66dbac001481641a326bfe3f1ed8dffb1e4299b5f3b32b4bfccdf999fbbb01a

[filipstrand]

@timaschew Thanks for raising this issue, with v.13 I made a big change regarding how model weighs were loaded and I may have accidentally broken some parts here. I'll have a look at this shortly and see if I can reproduce this and add a fix

[timaschew]

Thank you for your fast response.
Then I will just try to install 0.12.1

[timaschew]

I've tested mflux 0.12.1.
schnell and dev works fine now.

qwen

But I was not able to get qwen working.

mflux-generate-qwen --path model-links/qwen-6bit --model qwen --prompt "Luxury food photograph" --steps 2

error:

Traceback (most recent call last):
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/bin/mflux-generate-qwen", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/generate_qwen.py", line 27, in main
    qwen = QwenImage(
           ^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/variants/txt2img/qwen_image.py", line 39, in __init__
    QwenImageInitializer.init(
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/qwen_initializer.py", line 30, in init
    weights = QwenWeightHandler.load_regular_weights(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/weights/qwen_weight_handler.py", line 43, in load_regular_weights
    qwen_text_encoder, _, _ = QwenWeightHandler._load_qwen_text_encoder(root_path=root_path, load_visual_weights=load_visual_weights)  # fmt: off
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/weights/qwen_weight_handler.py", line 71, in _load_qwen_text_encoder
    return QwenWeightHandler._load_component(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/weights/qwen_weight_handler.py", line 95, in _load_component
    weights = QwenWeightHandler._load_safetensors_shards(component_path, loading_mode=loading_mode)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/qwen/weights/qwen_weight_handler.py", line 124, in _load_safetensors_shards
    with open(index_path) as f:
         ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'model-links/qwen-6bit/text_encoder/model.safetensors.index.json'

model-links/qwen-6bit
├── README.md -> ../../blobs/9697e51677edd90d7c084b93695ff515bc74822e
├── text_encoder
│   ├── 0.safetensors -> ../../../blobs/6dca932b003eea2f35bdba367756202152c439a48da3a7fa5051ed4c2db1cd7b
│   ├── 1.safetensors -> ../../../blobs/4fa593721b7bcb4cf0b2fd8fd2a7a2ff814b66b0f232526ab529861e9c12b2d7
│   └── 2.safetensors -> ../../../blobs/b323f4d283cf17565c315d892d02a718b2b56efc029e8ae4aba27f79e4113b89
├── tokenizer
│   ├── added_tokens.json -> ../../../blobs/482ced4679301bf287ebb310bdd1790eb4514232
│   ├── chat_template.jinja -> ../../../blobs/28028c056af412405debd878cdda0171e35fa5d1
│   ├── merges.txt -> ../../../blobs/31349551d90c7606f325fe0f11bbb8bd5fa0d7c7
│   ├── special_tokens_map.json -> ../../../blobs/ac23c0aaa2434523c494330aeb79c58395378103
│   ├── tokenizer_config.json -> ../../../blobs/eaed590d62aaf0ba31e284b66ddcb18222f066c2
│   └── vocab.json -> ../../../blobs/6bce3a0a3866c4791a74d83d78f6824c3af64ec3
├── transformer
│   ├── 0.safetensors -> ../../../blobs/6b599131d7cad6e02e589d87bd420362e87fbb0865e0cf39029a93ac611c0072
│   ├── 1.safetensors -> ../../../blobs/4ef9184208caf7de672ee4cb59c2b167359d6cfebd74a28bf486cea350fd320a
│   ├── 2.safetensors -> ../../../blobs/f39b8d57a465bf4bde2dd9df47f725b2edb0622872ea172b9666f4bb291457d5
│   ├── 3.safetensors -> ../../../blobs/7f2823ba445636687e0f6a21023b016ef6aad035b3b718558d565e0acf19abbd
│   ├── 4.safetensors -> ../../../blobs/4eb0f0c84b985305a42109c2bba0ba14d92de0f2516f369d872a6539fa77a2e2
│   ├── 5.safetensors -> ../../../blobs/ed8227cc48a805890c6daa8ebfd6ddd7409812138a19d872028fff2c45437495
│   ├── 6.safetensors -> ../../../blobs/918a769c37ce25dbfeb86b4cf1d602b24d08e8af11c2c0d33693fc8c8798b2cd
│   └── 7.safetensors -> ../../../blobs/b08159c878d1eb0315137bf452aa7dc9c4a61b8f2665b05e0702df623408f1aa
└── vae
    └── 0.safetensors -> ../../../blobs/b0731cff3148133686b98fd2b0a7b1b58eaa56ab7855a24ab5d151cb951b85aa

5 directories, 19 files

and the other syntax:

 mflux-generate --path model-links/qwen-6bit --model qwen --prompt "Luxury food photograph" --steps 2

Leads to this error:

Traceback (most recent call last):
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/bin/mflux-generate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/generate.py", line 27, in main
    model = Flux1(
            ^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/flux/variants/txt2img/flux.py", line 39, in __init__
    FluxInitializer.init(
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/flux/flux_initializer.py", line 40, in init
    weights = WeightHandler.load_regular_weights(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/flux/weights/weight_handler.py", line 53, in load_regular_weights
    t5_encoder, _, _ = WeightHandler._load_t5_encoder(root_path=root_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adminton/dev/mflux/.venv-mflux-0-12/lib/python3.11/site-packages/mflux/models/flux/weights/weight_handler.py", line 89, in _load_t5_encoder
    weights["final_layer_norm"] = weights["encoder"]["final_layer_norm"]
                                  ~~~~~~~^^^^^^^^^^^
TypeError: list indices must be integers or slices, not str

fibo

It seems that fibo (quantized version) is not supported by the CLI.

Command:

mflux-generate --path model-links/fibo-4bit --model fibo --steps 20 --guidance 4.0 --seed 42 --output animal_bakers.png   --prompt "Three cartoon animal chefs in a colorful bakery kitchen"

Error:

mflux-generate: error: argument --model/-m: Value must be either dev schnell krea-dev dev-krea qwen or in format "org/model". Got: fibo

Same error when I'm using mflux-generate-fibo command.

[filipstrand]

I'd really like to take another stab at redesigning/updating the CLI since so much have changed over time. I feel like it is in a partially broken state atm, and could be simplified in various ways. We have concepts that could probably be unified like both --path and --model etc.

For example, we introduced the --base-model idea way back in order to pick the model config, since some custom flux variants didn't provide this #103, but I'm honestly not sure if it is compatible with non-flux versions (I have not tried this myself in a while).

Also mflux-generate sounds like a general command (but is actually FLUX1 specific). For Qwen Image, for example, the user should run mflux-generate-qwen instead and not mflux-generate --model qwen. But this is all very confusing at the moment (coding agents can figure these things out by themselves mostly) and a lot can be improved.

With v.013, I did put in some effort to improve the way we resolve models internally with --model flag. It is cleaner now, but I don't think it it was properly documented in the readme. I was thinking that it should mostly be enough to specify

1. The mflux command, like mflux-generate-fibo
2. Optionally, the --model, like --model briaai/Fibo-mlx-4bit (which can be a HF remote repo, or local folder)

For example, this should work for Z-image, using https://huggingface.co/filipstrand/Z-Image-Turbo-mflux-4bit

mflux-generate-z-image-turbo \
  --model filipstrand/Z-Image-Turbo-mflux-4bit \
  --prompt "t3chnic4lly vibrant 1960s close-up of a woman sitting under a tree in a blue skirt and white blouse, she has blonde wavy short hair and a smile with green eyes lake scene by a garden with flowers in the foreground 1960s style film She's holding her hand out there is a small smooth frog in her palm, she's making eye contact with the toad." \
  --width 1280 \
  --height 720 \
  --seed 456 \
  --steps 9 \
  --lora-paths renderartist/Technically-Color-Z-Image-Turbo \
  --lora-scales 0.5

and for FIBO, as another example, this command should work with https://huggingface.co/briaai/Fibo-mlx-4bit

mflux-generate-fibo \
    --model briaai/Fibo-mlx-4bit \
    --prompt-file ~/Desktop/bird.json \
    --width 1024 \
    --height 1024 \
    --steps 20 \
    --seed 42

After some testing these don't work because of old model weights stored on HF:

Qwen Image https://huggingface.co/filipstrand/Qwen-Image-mflux-6bit

mflux-generate-qwen \
  --model filipstrand/Qwen-Image-mflux-6bit \
  --prompt "Close-up portrait of a majestic tiger in its natural habitat, detailed fur texture, piercing eyes, natural forest background, soft natural lighting, wildlife photography, photorealistic, high detail, professional wildlife shot" \
  --negative-prompt "blurry, low quality, distorted, deformed, ugly, bad anatomy, bad proportions, extra limbs, duplicate, watermark, signature, text, letters, cartoon, anime, painting, drawing, illustration, 3d render, cgi, zoo, cage, artificial" \
  --width 1920 \
  --height 816 \
  --steps 30 \
  --seed 42 

Krea https://huggingface.co/filipstrand/FLUX.1-Krea-dev-mflux-4bit:

mflux-generate \
    --model filipstrand/FLUX.1-Krea-dev-mflux-4bit \
    --prompt "A photo of a dog" \
    --steps 25 \
    --seed 2674888 \
    --height 1024 \
    --width 1024

Update: OK, this seems to be due to old weights on HF... basically recent refactors have somehow broken the previously pre-quantized weights and the HF ones should be updated. It is a bit sad state to be in, since there is lot of weights that must be updated. But when quantising them locally, using the mflux-save command on the latest v.0.13, all seems to work well.

I'll make sure to update the HF weights soon! In the mean time, if you have the disk space allowed on your machines, you can simply run the full weights on with -q 4 or -q 8 flag or use the mflux-save command and save local pre-quantised copies and then remove the big original weights. Thanks again for raising the issue!

---

So in other words, in the meantime, when running v.013, a solution is the following:

Assuming you have temporarly enough disk space and have downloaded, say Qwen Image, save a local quantized copy like so:

mflux-save --model qwen --quantize 6 --path "/your/local/path/to/mflux-qwen-image-q6"

then throw away full precision model from HF directory, then only use the local one

mflux-generate-qwen \
  --model /your/local/path/to/mflux-qwen-image-q6 \
  --prompt "Close-up portrait of a majestic tiger in its natural habitat, detailed fur texture, piercing eyes, natural forest background, soft natural lighting, wildlife photography, photorealistic, high detail, professional wildlife shot" \
  --negative-prompt "blurry, low quality, distorted, deformed, ugly, bad anatomy, bad proportions, extra limbs, duplicate, watermark, signature, text, letters, cartoon, anime, painting, drawing, illustration, 3d render, cgi, zoo, cage, artificial" \
  --width 1920 \
  --height 816 \
  --steps 30 \
  --seed 42 

I'll begin by updating the readme a bit to clarify and point people here in case they get the same problems!