exir "missing out vars"

[antmikinka]

My Script I ran to cause this error
python -m examples.models.llama2.export_llama --checkpoint /Users/anthonymikinka/executorch/llama-2-7b-chat/consolidated.00.pth --params /Users/anthonymikinka/executorch/llama-2-7b-chat/params.json -kv --use_sdpa_with_kv_cache --coreml --group_size 128 -qmode 8da4w -d fp32 --verbose --max_seq_length 512 -o "/Volumes/NVME 3/ExecuTorch Models"

Above this is a lot of this EdgeOpOverload, but otherwise MIL backend and default pipelines built.
Lots of ops were removed earlier on before the MIL building.
Below is some terminal code and the traceback error.

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.dequantize_per_token.default>: schema = quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::dequantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.dequantize_per_token.default>: schema = quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::dequantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.choose_qparams_per_token_asymmetric.default>: schema = quantized_decomposed::choose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::choose_qparams_per_token_asymmetric can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::choose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.quantize_per_token.default>: schema = quantized_decomposed::quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::quantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensor']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.dequantize_per_token.default>: schema = quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::dequantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.choose_qparams_per_token_asymmetric.default>: schema = quantized_decomposed::choose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::choose_qparams_per_token_asymmetric can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::choose_qparams_per_token_asymmetric(Tensor input, ScalarType dtype) -> (Tensor, Tensor)']'`

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.quantize_per_token.default>: schema = quantized_decomposed::quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::quantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::quantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype) -> Tensor']'

INFO:root:Failed converting '<EdgeOpOverload: quantized_decomposed.dequantize_per_token.default>: schema = quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor' to its out variant with error: 'SchemaKind.out variant of operator quantized_decomposed::dequantize_per_token can't be found. We've found the schemas of all the overloads: ['quantized_decomposed::dequantize_per_token(Tensor input, Tensor scales, Tensor zero_points, int quant_min, int quant_max, ScalarType dtype, ScalarType output_dtype) -> Tensor']'


Traceback (most recent call last):
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/anthonymikinka/executorch/examples/models/llama2/export_llama.py", line 30, in <module>
    main()  # pragma: no cover
  File "/Users/anthonymikinka/executorch/examples/models/llama2/export_llama.py", line 26, in main
    export_llama(modelname, args)
  File "/Users/anthonymikinka/executorch/examples/models/llama2/export_llama_lib.py", line 545, in export_llama
    return _export_llama(modelname, args)
  File "/Users/anthonymikinka/executorch/examples/models/llama2/export_llama_lib.py", line 869, in _export_llama
    builder = builder_exported_to_edge.to_backend(partitioners).to_executorch()
  File "/Users/anthonymikinka/executorch/examples/models/llama2/builder.py", line 319, in to_executorch
    self.export_program = self.edge_manager.to_executorch(
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/program/_program.py", line 842, in to_executorch
    new_gm_res = p(new_gm)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/torch/fx/passes/infra/pass_base.py", line 40, in __call__
    res = self.call(graph_module)
  File "/opt/anaconda3/envs/executorch/lib/python3.10/site-packages/executorch/exir/passes/__init__.py", line 422, in call
    raise RuntimeError(f"Missing out variants: {missing_out_vars}")
RuntimeError: Missing out variants: {'quantized_decomposed::dequantize_per_token', 'quantized_decomposed::choose_qparams_per_token_asymmetric', 'quantized_decomposed::dequantize_per_channel_group', 'quantized_decomposed::quantize_per_token'}

cc @kimishpatel @YifanShenSZ @cymbalrush

[cccclai]

Are you trying to lower the model to CoreML by passing --coreml? We're still actively working on enabling llama2 7b with CoreML. The xnnpack backend is ready for llama2 7b model.

[antmikinka]

Are you trying to lower the model to CoreML by passing --coreml? We're still actively working on enabling llama2 7b with CoreML. The xnnpack backend is ready for llama2 7b model.

@cccclai ah ok sweet thank you for letting me know!! I would have still been trying haha

Is the XNNPACK a .mlpackage? I have to build the xnnpack stuff, I just did mps and coreml.

Do you have more info on what models you have ready for CoreML?

Is there any .mlmodel/.mlpackage model configs (or any end products of converting) for in executorch?

[cccclai]

xnnpack (https://github.com/google/XNNPACK) is a software library with a list highly optimized operators in CPU. It can work on iOS too.

Regarding CoreML questions, I'd defer to @cymbalrush and @YifanShenSZ to answer.

[SS-JIA]

Will also cc: @shoumikhin for iOS/MacOS related inquiries.

[YifanShenSZ]

Hey @antmikinka, would this simpler export work for you?

python -m examples.models.llama2.export_llama --checkpoint /Users/anthonymikinka/executorch/llama-2-7b-chat/consolidated.00.pth --params /Users/anthonymikinka/executorch/llama-2-7b-chat/params.json -kv --coreml

Concretely, this is a good start point that we have tested and made sure working. For all other arguments, could you please try to add them one by one until issue pops up? (so we can have more clarity on what went wrong)

[antmikinka]

@YifanShenSZ I kept running into Disk Memory Issues on my Macbook Pro Even upto 30GB free space.
I added the
--group_size 128 -qmode 8da4w -d fp32 --verbose --max_seq_length 512 -o "/Volumes/NVME 3/ExecuTorch Models"
I got the error above once again.

I started to work on the arguments.

• I did -kv --coreml --group_size 128 -d fp32 --verbose --max_seq_length 512 -o "/Volumes/NVME 3/ExecuTorch Models"
  ran out of space once again.

• I did -kv --coreml -qmode 8da4w -d fp32 --verbose -o "/Volumes/NVME 3/ExecuTorch Models"
  ran into the quantized issue. I am thinking that the -qmode 8da4w argument may be the issue.
  To possibly help narrow this down, I took the last couple hundred lines of my terminal and created a gist.
  PyTorch-executorch-issue 3443 terminal.txt

• I did -kv --coreml -qmode 8da4w
  I made a log file for this one. Here is the gist executorch.log
  I got the quantized error as well. Looks like I was right about the -qmode 8da4w

[antmikinka]

Just tried -kv --coreml --verbose --group_size 128 --max_seq_length 128
ran out of stroage, used 29gb trying to convert.

I may try again later today after trying to free up some more storage. let me know if that log file has helped.

[BodhiHu]

I still encountered the same issue with latest code, and had raised the issue:
#7775

And note that I followed the steps from the official docs for llama:

https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md#option-b-download-and-export-llama-3-8b-instruct-model

Another thing is, it requires lots of memory when converting the models, even for the 3B model.
I had to downgrade to a 1B model so that exporting won't OOM. (with 16G mem)

[kimishpatel]

I still encountered the same issue with latest code, and had raised the issue: #7775

And note that I followed the steps from the official docs for llama:

https://github.com/pytorch/executorch/blob/main/examples/models/llama/README.md#option-b-download-and-export-llama-3-8b-instruct-model

Another thing is, it requires lots of memory when converting the models, even for the 3B model. I had to downgrade to a 1B model so that exporting won't OOM. (with 16G mem)

Are you trying to use coreml backend?