Fix SDPA TT_METAL_WATCHER issues

[pavlejosipovic]

Summary

• generate_reduce_scaler hardcoded 2048 bytes and 4 faces, assuming full 32x32 bf16 tiles. When circular buffers use half tiles (1024B, 2 faces), this overwrites adjacent L1 memory causing watcher-detected corruption.
• Restore the half_tile template parameter so the zero-fill size and face iteration adapt to the actual tile dimensions. Also fix idle core runtime args count mismatch in sdpa_decode_program_factory.
• Remove watcher skips in test_sdpa_prefill.py (now passing with the fix)
• Restore watcher skip in test_flash_mla.py for Blackhole (separate issue)

Fixes: #37631
Fixes: #29225

Replaces #37833 (closed due to bad rebase)

Test plan

• All post-commit tests
• Blackhole post-commit tests
• Nightly tt-metal L2 tests
• apc nightly debug run with watcher

🤖 Generated with Claude Code

[pavlejosipovic]

/codeowners ping

[tenstorrent-github-bot]

CodeOwners Group Analysis

This PR requires approval from one member of each of the following groups:

Summary: 2 pending groups, 0 approved groups

Group Information:

• ⏳ tenstorrent/metalium-developers-ttnn-core (Team) - Members: Pavlo Hilei, Brian Liu, Joseph Chu, Artem Yerofieiev, Diego Gomez | Pending approval

  📁 Files owned by this team (1 files)

  • ttnn/cpp/ttnn/kernel/dataflow/generate_reduce_scaler.hpp

---

• ⏳ tenstorrent/metallium-maintainers-llama-models (Team) - Members: Raymond Kim, Colman Glagovich, Evan Smal, Jonathan Su, Harry Andrews | Pending approval

  📁 Files owned by this team (3 files)

  • tests/ttnn/unit_tests/operations/sdpa/test_sdpa_prefill.py
  • ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/kernels/dataflow/writer_decode_all.cpp
  • ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/sdpa_decode_program_factory.cpp

Note: At least one approval from each group is sufficient.

[tenstorrent-github-bot]

Hi Evan Smal (@esmalTT), Raymond Kim (@tt-rkim), this PR Fix SDPA TT_METAL_WATCHER issues by Pavle Josipović (@pavlejosipovic) needs your approval/review to merge this.

[Copilot]

Pull request overview

This PR fixes TT_METAL_WATCHER-detected corruption/issues in SDPA decode by ensuring the reduce-scaler generation logic matches the actual circular buffer tile size (full vs half tiles) and by aligning idle-core runtime argument counts with what the decode reader kernel expects. It also updates SDPA prefill unit tests to remove watcher skips now that the underlying issue is addressed.

Changes:

• Restore half-tile awareness for generate_reduce_scaler and pass the correct half/full-tile mode from the SDPA decode writer kernel.
• Fix idle-core reader runtime-arg vector length in SdpaDecodeProgramFactory to match the reader kernel’s expected arg reads.
• Remove TT_METAL_WATCHER skip decorators from test_sdpa_prefill.py (per PR description: now passing with the fix).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/sdpa_decode_program_factory.cpp	Fix idle-core reader runtime-arg count to prevent watcher OOB runtime-arg access.
ttnn/cpp/ttnn/operations/transformer/sdpa_decode/device/kernels/dataflow/writer_decode_all.cpp	Detect half-tile scalar CBs and invoke generate_reduce_scaler with the correct template mode.
ttnn/cpp/ttnn/kernel/dataflow/generate_reduce_scaler.hpp	Reintroduce half_tile template parameter to size the zero-fill and face-looping correctly.
tests/ttnn/unit_tests/operations/sdpa/test_sdpa_prefill.py	Remove watcher-enabled skips now that corruption/OOM issues should be resolved by the kernel fix.