Skip to content

Invalid variant names in unit tests due to model variant change uplift #3303

New issue
New issue
Open
Open
Invalid variant names in unit tests due to model variant change uplift#3303
Assignees
kmabeeTTctr-lelanchelianTT
@ddilbazTT

Description

@ddilbazTT
ddilbazTT
opened on Feb 13, 2026

Due to tenstorrent/tt-forge-models@18d662b there are a lot of invalid variants in tests such as: tests/torch/graphs/test_mlp.py tests/torch/graphs/test_attention.py

It would be ideal if the variant names are fixed and a long-term solution is added so that such tests fail instead of silently being skipped. I.E. https://github.com/tenstorrent/tt-xla/actions/runs/21969404262/job/63467192426

tests/torch/graphs/test_attention.py::test_qwen3_attention_prefill[NOTSET-1024-single_device] SKIPPED [  8%]
tests/torch/graphs/test_attention.py::test_qwen3_attention_decode[NOTSET-single_device] SKIPPED [  8%]
tests/torch/graphs/test_attention.py::test_qwen3_concat_heads[NOTSET-1024] SKIPPED [  8%]
tests/torch/graphs/test_attention.py::test_qwen3_create_heads[NOTSET-1024] SKIPPED [  9%]
tests/torch/graphs/test_attention.py::test_qwen3_attention[NOTSET-1024-single_device] SKIPPED [  9%]
tests/torch/graphs/test_attention.py::test_bge_m3_attention_prefill[NOTSET-1024] SKIPPED [  9%]
tests/torch/graphs/test_attention.py::test_bge_m3_concat_heads[NOTSET-1024] SKIPPED [  9%]
tests/torch/graphs/test_attention.py::test_bge_m3_create_heads[NOTSET-1024] SKIPPED [ 10%]
tests/torch/graphs/test_attention.py::test_bert_create_heads[NOTSET-1024] SKIPPED [ 10%]
tests/torch/graphs/test_attention.py::test_qwen2_5_attention_prefill[NOTSET-1024-single_device] SKIPPED [ 10%]
tests/torch/graphs/test_attention.py::test_qwen2_5_attention_decode[NOTSET-single_device] SKIPPED [ 10%]
tests/torch/graphs/test_attention.py::test_qwen2_5_attention[NOTSET-1024-single_device] SKIPPED [ 10%]
tests/torch/graphs/test_attention.py::test_gemma_attention_prefill[NOTSET-1024-single_device] SKIPPED [ 11%]
tests/torch/graphs/test_attention.py::test_gemma_attention_decode[NOTSET-single_device] SKIPPED [ 11%]
tests/torch/graphs/test_attention.py::test_gemma_attention[NOTSET-1024-single_device] SKIPPED [ 11%]
tests/torch/graphs/test_attention.py::test_mistral_attention_prefill[NOTSET-1024-single_device] SKIPPED [ 11%]
tests/torch/graphs/test_attention.py::test_mistral_attention_decode[NOTSET-single_device] SKIPPED [ 11%]
tests/torch/graphs/test_attention.py::test_mistral_attention[NOTSET-1024-single_device] SKIPPED [ 12%]
tests/torch/graphs/test_attention.py::test_gpt_oss_attention[single_device-NOTSET] SKIPPED [ 12%]

All the following NOTSET variants are invalid:

(venv) /localdev/ddilbaz/tt-xla-2> pytest --collect-only tests/torch/graphs/test_attention.py
============================================================= test session starts =============================================================
platform linux -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0
rootdir: /localdev/ddilbaz/tt-xla-2
configfile: pytest.ini
plugins: forked-1.6.0, json-report-1.5.0, jaxtyping-0.3.7, split-0.11.0, metadata-3.1.1, anyio-4.12.1
collected 106 items                                                                                                                           

<Dir tt-xla-2>
  <Package tests>
    <Dir torch>
      <Dir graphs>
        <Module test_attention.py>
          <Function test_llama_attention_prefill[3.0_8B-1024-single_device]>
          <Function test_llama_attention_prefill[3.0_8B-1024-llmbox]>
          <Function test_llama_attention_prefill[3.1_8B-1024-single_device]>
          <Function test_llama_attention_prefill[3.1_8B-1024-llmbox]>
          <Function test_llama_attention_prefill[3.1_70B-1024-single_device]>
          <Function test_llama_attention_prefill[3.1_70B-1024-llmbox]>
          <Function test_llama_attention_prefill[3.2_1B-1024-single_device]>
          <Function test_llama_attention_prefill[3.2_1B-1024-llmbox]>
          <Function test_llama_attention_prefill[3.2_3B-1024-single_device]>
          <Function test_llama_attention_prefill[3.2_3B-1024-llmbox]>
          <Function test_llama_attention_prefill[3.3_70B_Instruct-1024-single_device]>
          <Function test_llama_attention_prefill[3.3_70B_Instruct-1024-llmbox]>
          <Function test_llama_attention_prefill[Huggyllama_7B-1024-single_device]>
          <Function test_llama_attention_prefill[Huggyllama_7B-1024-llmbox]>
          <Function test_llama_attention_prefill[Tinyllama_v1.1-1024-single_device]>
          <Function test_llama_attention_prefill[Tinyllama_v1.1-1024-llmbox]>
          <Function test_llama_attention_decode[3.0_8B-single_device]>
          <Function test_llama_attention_decode[3.0_8B-llmbox]>
          <Function test_llama_attention_decode[3.1_8B-single_device]>
          <Function test_llama_attention_decode[3.1_8B-llmbox]>
          <Function test_llama_attention_decode[3.1_70B-single_device]>
          <Function test_llama_attention_decode[3.1_70B-llmbox]>
          <Function test_llama_attention_decode[3.2_1B-single_device]>
          <Function test_llama_attention_decode[3.2_1B-llmbox]>
          <Function test_llama_attention_decode[3.2_3B-single_device]>
          <Function test_llama_attention_decode[3.2_3B-llmbox]>
          <Function test_llama_attention_decode[3.3_70B_Instruct-single_device]>
          <Function test_llama_attention_decode[3.3_70B_Instruct-llmbox]>
          <Function test_llama_attention_decode[Huggyllama_7B-single_device]>
          <Function test_llama_attention_decode[Huggyllama_7B-llmbox]>
          <Function test_llama_attention_decode[Tinyllama_v1.1-single_device]>
          <Function test_llama_attention_decode[Tinyllama_v1.1-llmbox]>
          <Function test_llama_concat_heads[3.0_8B-1024]>
          <Function test_llama_concat_heads[3.1_8B-1024]>
          <Function test_llama_concat_heads[3.1_70B-1024]>
          <Function test_llama_concat_heads[3.2_1B-1024]>
          <Function test_llama_concat_heads[3.2_3B-1024]>
          <Function test_llama_concat_heads[3.3_70B_Instruct-1024]>
          <Function test_llama_concat_heads[Huggyllama_7B-1024]>
          <Function test_llama_concat_heads[Tinyllama_v1.1-1024]>
          <Function test_llama_create_heads[3.0_8B-1024]>
          <Function test_llama_create_heads[3.1_8B-1024]>
          <Function test_llama_create_heads[3.1_70B-1024]>
          <Function test_llama_create_heads[3.2_1B-1024]>
          <Function test_llama_create_heads[3.2_3B-1024]>
          <Function test_llama_create_heads[3.3_70B_Instruct-1024]>
          <Function test_llama_create_heads[Huggyllama_7B-1024]>
          <Function test_llama_create_heads[Tinyllama_v1.1-1024]>
          <Function test_llama_attention[3.0_8B-1024-single_device]>
          <Function test_llama_attention[3.0_8B-1024-llmbox]>
          <Function test_llama_attention[3.1_8B-1024-single_device]>
          <Function test_llama_attention[3.1_8B-1024-llmbox]>
          <Function test_llama_attention[3.1_70B-1024-single_device]>
          <Function test_llama_attention[3.1_70B-1024-llmbox]>
          <Function test_llama_attention[3.2_1B-1024-single_device]>
          <Function test_llama_attention[3.2_1B-1024-llmbox]>
          <Function test_llama_attention[3.2_3B-1024-single_device]>
          <Function test_llama_attention[3.2_3B-1024-llmbox]>
          <Function test_llama_attention[3.3_70B_Instruct-1024-single_device]>
          <Function test_llama_attention[3.3_70B_Instruct-1024-llmbox]>
          <Function test_llama_attention[Huggyllama_7B-1024-single_device]>
          <Function test_llama_attention[Huggyllama_7B-1024-llmbox]>
          <Function test_llama_attention[Tinyllama_v1.1-1024-single_device]>
          <Function test_llama_attention[Tinyllama_v1.1-1024-llmbox]>
          <Function test_qwen3_attention_prefill[NOTSET-1024-single_device]>
          <Function test_qwen3_attention_prefill[NOTSET-1024-llmbox]>
          <Function test_qwen3_attention_prefill_push[8B-1024-single_device]>
          <Function test_qwen3_attention_prefill_push[8B-1024-llmbox]>
          <Function test_qwen3_attention_decode[NOTSET-single_device]>
          <Function test_qwen3_attention_decode[NOTSET-llmbox]>
          <Function test_qwen3_concat_heads[NOTSET-1024]>
          <Function test_qwen3_create_heads[NOTSET-1024]>
          <Function test_qwen3_attention[NOTSET-1024-single_device]>
          <Function test_qwen3_attention[NOTSET-1024-llmbox]>
          <Function test_bge_m3_attention_prefill[NOTSET-1024]>
          <Function test_bge_m3_concat_heads[NOTSET-1024]>
          <Function test_bge_m3_create_heads[NOTSET-1024]>
          <Function test_bert_create_heads[NOTSET-1024]>
          <Function test_qwen2_5_attention_prefill[NOTSET-1024-single_device]>
          <Function test_qwen2_5_attention_prefill[NOTSET-1024-llmbox]>
          <Function test_qwen2_5_attention_prefill_push[7B_Instruct-1024-single_device]>
          <Function test_qwen2_5_attention_prefill_push[7B_Instruct-1024-llmbox]>
          <Function test_qwen2_5_attention_decode[NOTSET-single_device]>
          <Function test_qwen2_5_attention_decode[NOTSET-llmbox]>
          <Function test_qwen2_5_attention[NOTSET-1024-single_device]>
          <Function test_qwen2_5_attention[NOTSET-1024-llmbox]>
          <Function test_gemma_attention_prefill[NOTSET-1024-single_device]>
          <Function test_gemma_attention_prefill[NOTSET-1024-llmbox]>
          <Function test_gemma_attention_prefill_push[2_9B_IT-1024-single_device]>
          <Function test_gemma_attention_prefill_push[2_9B_IT-1024-llmbox]>
          <Function test_gemma_attention_decode[NOTSET-single_device]>
          <Function test_gemma_attention_decode[NOTSET-llmbox]>
          <Function test_gemma_attention[NOTSET-1024-single_device]>
          <Function test_gemma_attention[NOTSET-1024-llmbox]>
          <Function test_mistral_attention_prefill[NOTSET-1024-single_device]>
          <Function test_mistral_attention_prefill[NOTSET-1024-llmbox]>
          <Function test_mistral_attention_prefill_push[7B-1024-single_device]>
          <Function test_mistral_attention_prefill_push[7B-1024-llmbox]>
          <Function test_mistral_attention_decode[NOTSET-single_device]>
          <Function test_mistral_attention_decode[NOTSET-llmbox]>
          <Function test_mistral_attention[NOTSET-1024-single_device]>
          <Function test_mistral_attention[NOTSET-1024-llmbox]>
          <Function test_compiled_batched_attention>
          <Function test_eager_batched_attention>
          <Function test_gpt_oss_attention[single_device-NOTSET]>
          <Function test_gpt_oss_attention[llmbox-NOTSET]>

======================================================== 106 tests collected in 0.42s =========================================================
(venv) /localdev/ddilbaz/tt-xla-2> pytest --collect-only tests/torch/graphs/test_mlp.py
============================================================= test session starts =============================================================
platform linux -- Python 3.11.14, pytest-9.0.2, pluggy-1.6.0
rootdir: /localdev/ddilbaz/tt-xla-2
configfile: pytest.ini
plugins: forked-1.6.0, json-report-1.5.0, jaxtyping-0.3.7, split-0.11.0, metadata-3.1.1, anyio-4.12.1
collected 28 items                                                                                                                            

<Dir tt-xla-2>
  <Package tests>
    <Dir torch>
      <Dir graphs>
        <Module test_mlp.py>
          <Function test_qwen3_mlp[NOTSET-1024-single_device]>
          <Function test_qwen3_mlp[NOTSET-1024-llmbox]>
          <Function test_llama_mlp[3.0_8B-1024-single_device]>
          <Function test_llama_mlp[3.0_8B-1024-llmbox]>
          <Function test_llama_mlp[3.1_8B-1024-single_device]>
          <Function test_llama_mlp[3.1_8B-1024-llmbox]>
          <Function test_llama_mlp[3.1_70B-1024-single_device]>
          <Function test_llama_mlp[3.1_70B-1024-llmbox]>
          <Function test_llama_mlp[3.2_1B-1024-single_device]>
          <Function test_llama_mlp[3.2_1B-1024-llmbox]>
          <Function test_llama_mlp[3.2_3B-1024-single_device]>
          <Function test_llama_mlp[3.2_3B-1024-llmbox]>
          <Function test_llama_mlp[3.3_70B_Instruct-1024-single_device]>
          <Function test_llama_mlp[3.3_70B_Instruct-1024-llmbox]>
          <Function test_llama_mlp[Huggyllama_7B-1024-single_device]>
          <Function test_llama_mlp[Huggyllama_7B-1024-llmbox]>
          <Function test_llama_mlp[Tinyllama_v1.1-1024-single_device]>
          <Function test_llama_mlp[Tinyllama_v1.1-1024-llmbox]>
          <Function test_gemma_mlp[NOTSET-1024-single_device]>
          <Function test_gemma_mlp[NOTSET-1024-llmbox]>
          <Function test_mistral_mlp[NOTSET-1024-single_device]>
          <Function test_mistral_mlp[NOTSET-1024-llmbox]>
          <Function test_qwen2_5_mlp[NOTSET-1024-single_device]>
          <Function test_qwen2_5_mlp[NOTSET-1024-llmbox]>
          <Function test_falcon_mlp[NOTSET-1024-single_device]>
          <Function test_falcon_mlp[NOTSET-1024-llmbox]>
          <Function test_gpt_oss_mlp[NOTSET-single_device]>
          <Function test_gpt_oss_mlp[NOTSET-llmbox]>

========================================================= 28 tests collected in 0.13s =========================================================

Assigning to @kmabeeTT for triage.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions