Skip to content

[BUG] LLM QNN export ignores --chunk_size when generating prefill IO #4612

Description

@LudovicoYIN

Problem

In the LLM QNN export path, transformers/llm/export/npu/generate_llm_qnn.py exposes a --chunk_size argument, but prefill IO generation did not actually honor it.

Before the fix, convert_llm() always called:

makeIOJson(args, 128, hidden_size, mask_type)

which meant the prefill QNN graph could still be generated against 128-token IO even when --chunk_size 256/512 was requested.

Impact

This can make generated QNN artifacts inconsistent with runtime configuration:

  • prefill graph shape may not match the requested chunk size
  • runtime chunk_limits and generated graphs may diverge
  • ppl_eval results can become misleading
  • chat / prefill behavior may differ from expectation

Reproduction

Run QNN export with a non-128 chunk size, for example:

python transformers/llm/export/npu/generate_llm_qnn.py \
  --model <model_dir> \
  --soc_id 43 \
  --dsp_arch v73 \
  --mnn_path <mnn_build> \
  --chunk_size 512

Expected:

  • generated prefill IO and QNN graph use 512-token shape

Actual before fix:

  • prefill IO generation still used 128

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions