Qwen3.5-2B AutoRound BM1688 W4BF16 compile fails: invalid scale channel in A16MatMul weight reorder

[tianna1121]

Summary

Compiling the official Intel/Qwen3.5-2B-int4-AutoRound model for BM1688 W4BF16 fails in tpuc-opt weight reorder with:

invalid scale channel
UNREACHABLE executed at /__w/tpu-mlir/tpu-mlir/lib/Dialect/Tpu/Transforms/WeightReorder/BM1684X/A16MatMul.cpp:274!

This reproduces with both whole-model and pipeline-parallel export attempts. The prebuilt Sophgo BM1688 bmodel from LLM-TPU/models/Qwen3_5/README.md runs on our BM1688/SylixOS board, so this looks like a local compiler/toolchain compatibility issue rather than a board runtime issue.

Environment

• Host: Ubuntu 22.04, x86_64, NVIDIA 4090D
• Docker image: sophgo/tpuc_dev:latest
• LLM-TPU commit: 63d8ab34dab5fee381f3f492bf86c061b862ffa6
  • 63d8ab3 fix: overlapping memory copy bug in qwen3.5
• tpu_mlir pip package:
  • Version: 1.28.1
  • Summary: Machine learning compiler based on MLIR for TPU v1.28.1-g43676b3-20260429
• llm_convert.py v1.28.1-20260429
• tpuc-opt: LLVM 18.0.0git
• Target chip: bm1688
• Model: Intel/Qwen3.5-2B-int4-AutoRound

Reproduction

Download the official model:

hf download Intel/Qwen3.5-2B-int4-AutoRound \
  --local-dir /workspace/models/Qwen3.5-2B-int4-AutoRound \
  --max-workers 4

Whole-model export attempt:

llm_convert.py \
  -m /workspace/models/Qwen3.5-2B-int4-AutoRound \
  --max_input_length 1024 \
  -s 2048 \
  -c bm1688 \
  --max_pixels 768,768 \
  --out_dir /workspace/build/qwen35_2b_official_int4_bm1688

This generates MLIR successfully, then fails during bmodel deployment at:

tpuc-opt block_0_bm1688_w4bf16_tpu.mlir ...
invalid scale channel
UNREACHABLE executed at .../A16MatMul.cpp:274

Pipeline-parallel export attempt:

llm_convert.py \
  -m /workspace/models/Qwen3.5-2B-int4-AutoRound \
  --max_input_length 1024 \
  -s 2048 \
  -c bm1688 \
  --num_device 3 \
  --distribute_strategy pp \
  --max_pixels 768,768 \
  -o /workspace/build/qwen35_2b_official_int4_bm1688_pp3

This also generates MLIR successfully, then fails during bmodel deployment at:

tpuc-opt block_1_bm1688_w4bf16_tpu.mlir ...
invalid scale channel
UNREACHABLE executed at .../A16MatMul.cpp:274

The failing command in the PP case is:

tpuc-opt block_1_bm1688_w4bf16_tpu.mlir --mlir-disable-threading --strip-io-quant="quant_input=True quant_output=True quant_input_list= quant_output_list= quant_output_bf16=False quant_input_int8=False quant_output_int8=False " --processor-tpu-optimize --dev-parallel --weight-reorder --subnet-divide="dynamic=True" --op-reorder --topo-sort --layer-group="opt=2 group_by_cores=auto compress_mode=none debugger=0 disable_group_overlap=false lgcache=true config_filename= enable_lghash=False lghash_dir=" --core-parallel --after-layergroup-weight-reorder --address-assign -o block_1_bm1688_w4bf16_final.mlir --debug_cmd=

Full relevant log excerpt

invalid scale channel
UNREACHABLE executed at /__w/tpu-mlir/tpu-mlir/lib/Dialect/Tpu/Transforms/WeightReorder/BM1684X/A16MatMul.cpp:274!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0. Program arguments: tpuc-opt block_1_bm1688_w4bf16_tpu.mlir --init --mlir-disable-threading "--strip-io-quant=quant_input=True quant_output=True quant_input_list= quant_output_list= quant_output_bf16=False quant_input_int8=False quant_output_int8=False " --processor-tpu-optimize --dev-parallel --weight-reorder --subnet-divide=dynamic=True --op-reorder --topo-sort "--layer-group=opt=2 group_by_cores=auto compress_mode=none debugger=0 disable_group_overlap=false lgcache=true config_filename= enable_lghash=False lghash_dir=" --core-parallel --after-layergroup-weight-reorder --address-assign --deinit --mlir-print-debuginfo -o block_1_bm1688_w4bf16_final.mlir
Aborted (core dumped)
RuntimeError: [!Error]: tpuc-opt block_1_bm1688_w4bf16_tpu.mlir --mlir-disable-threading --strip-io-quant="quant_input=True quant_output=True quant_input_list= quant_output_list= quant_output_bf16=False quant_input_int8=False quant_output_int8=False " --processor-tpu-optimize --dev-parallel --weight-reorder --subnet-divide="dynamic=True" --op-reorder --topo-sort --layer-group="opt=2 group_by_cores=auto compress_mode=none debugger=0 disable_group_overlap=false lgcache=true config_filename= enable_lghash=False lghash_dir=" --core-parallel --after-layergroup-weight-reorder --address-assign -o block_1_bm1688_w4bf16_final.mlir --debug_cmd=
parallel: This job failed:
(pushd block_1 && model_deploy.py --mlir block_1.mlir --quantize w4bf16 --q_group_size 128 --quant_input --quant_output --chip bm1688 --num_core 2 --num_device 1 --model block_1.bmodel --addr_mode basic --high_precision --dynamic && popd ) > block_1.log

Expected behavior

The official Qwen3.5-2B AutoRound INT4 model should compile to BM1688 W4BF16 bmodel, or the compiler should report a recoverable unsupported-op/shape diagnostic instead of aborting in weight reorder.

Additional context

• The prebuilt Sophgo bmodel qwen3.5-2b-int4-autoround_w4bf16_seq2048_bm1688_2core_dynamic_20260415_212627.bmodel can be loaded on our BM1688/SylixOS board.
• Board smoke test against the running Qwen3.5 service passed:
  • model id: qwen3.5-bm1688
  • endpoint: /v1/chat/completions
  • FTL around 0.369s
  • TPS around 12.96 tok/s
• We are trying to compile a fine-tuned Qwen3.5-2B model. The same failure reproducing with the official Intel AutoRound baseline suggests the blocker is in the current local TPU-MLIR compile path.

[tianna1121]

Additional data point with the dfss beta build requested by the user:

I retried the official baseline Intel/Qwen3.5-2B-int4-AutoRound on BM1688 split-PP with:

python -m dfss --url=open@sophgo.com:/ext_model_information/LLM/mlir_club/tpu-mlir_v1.28.beta.0-37-gdf2b86866-20260522.tar.gz

source /workspace/tpumlir-venv/bin/activate
source /workspace/deps/tpu_mlir_dfss/tpu-mlir_v1.28.beta.0-37-gdf2b86866-20260522/tpu-mlir_v1.28.beta.0-37-gdf2b86866-20260522/envsetup.sh

llm_convert.py \
  -m /workspace/models/Qwen3.5-2B-int4-AutoRound \
  --max_input_length 1024 \
  -s 2048 \
  -c bm1688 \
  --num_device 3 \
  --distribute_strategy pp \
  --max_pixels 768,768 \
  -o /workspace/build/qwen35_2b_official_int4_bm1688_pp3_df2b86866

Version:

llm_convert.py v1.28.beta.0-37-gdf2b86866-20260522

Result:

• The previous invalid scale channel / A16MatMul.cpp:274 failure no longer appears.
• block_0/block_0.bmodel, block_1/block_1.bmodel, and block_2/block_2.bmodel are generated.
• The export still fails later at block_cache_0.bmodel codegen.

New failure excerpt from block_cache_0.log:

[0][ERR] !has_bias || bias_dtype == DT_FP32failed!
[0]/workspace/nntoolchain/TPU1686/firmware_core/src/local_layer/nodechip_batch_matmul_local.c:449:
nodechip_batch_matmul_float_local_v2: Assertion "!has_bias || bias_dtype == DT_FP32" failed.

[Running]: tpuc-opt block_cache_0_bm1688_w4bf16_final.mlir --codegen="model_file=block_cache_0.bmodel embed_debug_info=False model_version=latest bmodel_only=False gdma_check=False rvti=False" -o /dev/null

I also retried the same split-PP command with --num_core 1; it still fails at block_cache_0 with the same codegen assertion, so this does not look like only a 2-core scheduling issue.

Could you please advise whether this block_cache_0 BM1688 W4BF16 batch matmul bias dtype assertion is expected in this beta, and whether there is another recommended TPU-MLIR build or llm_convert.py/model_deploy.py option for Qwen3.5-2B INT4 split-PP export?