DRAM sharding config for matmuls in LLMs

[bmalesevicTT]

DRAM-sharded matmul (tt-mlir PR #9150) — benchmark analysis and root cause

Analysis of the DRAM-sharded (DS) matmul optimizer integration across four
platforms, the mechanism behind the Blackhole regressions, and why the obvious
fix to the in0_block_w divisor constraint is not viable.

1. What was compared

All eight runs are tt-xla f6a38d65. The only variable is the tt-mlir pin.

pin	what
99c621da	baseline (tt-xla main pin, 2026-07-29) — before
ae602833	head of bmalesevic/dram-sharding-matmul-optimizer-integration-rebased (PR #9150) — after

platform	before run	after run
n150 (Wormhole B0)	30806869145	30767975002
p150 (Blackhole)	30806889612	30768002414
qb2-blackhole (TP)	30806911269	31583231291
wh-galaxy-6u	30806940457	30806775666 (never ran)

All runs used skip device perf: true, so every number below is end-to-end
samples/sec. There is no kernel-level attribution anywhere in this analysis.

2. Headline results

Split by whether the model's TTNN artifacts actually contain
#ttnn.matmul_multi_core_reuse_multi_cast_dram_sharded_program_config. Models
with zero DS matmuls are the control group — same dispatch, same runner pool,
same tt-mlir swap, but the optimizer changed nothing about them.

platform	affected n	affected median	control n	control median	verdict
n150	11	+2.5% (11/11 up)	35	+0.4%	win
p150	11	−6.5% (9/11 down)	30	+0.4%	loss
qb2	7	+0.70% (5/7 up)	14	+0.04%	~neutral
galaxy	—	no data	—	—	—

Both control groups on n150/p150 centre on +0.4%, and 28 of 30 p150 controls sit
inside ±3%, so the affected-group signal is real and not machine drift. The one
control outlier is mnist (−12.2% p150, −3.5% n150), which runs at 25–28k
samples/sec and is host-bound.

qb2's +0.70% is barely outside its own control band (±1.6%), so the honest read
is neutral, not a win.

Detail — n150 (+2.5%)

qwen_2_5_1_5b_instruct   35.50 →  43.85   +23.5    (had NO program config before)
qwen_3_4b                22.29 →  23.55    +5.7
qwen_3_0_6b              48.88 →  51.55    +5.5
llama_3_2_3b_instruct    30.24 →  31.25    +3.3
tiiuae_falcon3-1b-base   53.19 →  54.87    +3.2
tiiuae_falcon3-3b-base   35.64 →  36.52    +2.5
qwen_2_5_0_5b_instruct   68.53 →  70.12    +2.3
qwen_2_5_3b_instruct     29.80 →  30.39    +2.0
llama_3_2_1b_instruct    64.00 →  65.21    +1.9
google_gemma-1.1-2b-it   38.40 →  39.10    +1.8
qwen_3_1_7b              35.68 →  36.22    +1.5

Detail — p150 (−6.5%)

qwen_2_5_3b_instruct     55.18 →  39.70   -28.1
tiiuae_falcon3-7b-base   34.06 →  29.76   -12.6
qwen_3_8b                30.50 →  26.86   -11.9
google_gemma-1.1-2b-it   74.06 →  67.55    -8.8
llama_3_2_1b_instruct   120.32 → 112.23    -6.7
tiiuae_falcon3-1b-base  100.57 →  94.03    -6.5
tiiuae_falcon3-3b-base   64.11 →  60.06    -6.3
llama_3_1_8b_instruct    36.64 →  35.03    -4.4
qwen_3_0_6b              97.64 →  95.46    -2.2
qwen_2_5_0_5b_instruct  125.41 → 130.29    +3.9   (only 24/121 converted)
qwen_2_5_1_5b_instruct   62.47 →  68.78   +10.1   (had NO config before)

Detail — qb2 (+0.70%)

qwen_2_5_coder_32b_instruct_tp   17.25 → 18.07   +4.7
llama_3_1_70b_tp                 12.18 → 12.67   +4.0
falcon3_7b_tp                    54.60 → 56.01   +2.6
falcon3_10b_tp                   40.32 → 40.60   +0.7
gpt_oss_20b_tp_batch_size_1      21.38 → 21.49   +0.5
llama_3_1_8b_instruct_tp         50.05 → 49.31   -1.5
mistral_small_24b_instruct_tp    28.63 → 27.62   -3.5

3. What DS actually changes in the IR

Verified across 25 model × platform combinations: every "before" artifact
contains zero DS program configs. In the "after" artifacts DS appears in
exactly one graph per model — g1, the decode graph (g2 for gemma, whose
graph order differs). Prefill (g0) config mix is unchanged, and the DS counts are
identical between n150 and p150, so both platforms execute the same decision.

model	graph	matmuls	before	after: DS
qwen_2_5_0_5b	g1	121	121 × 1d-mcast	24
qwen_2_5_1_5b	g1	141	none — default heuristic	112
qwen_2_5_3b	g1	181	181 × 1d-mcast	144
qwen_3_0_6b	g1	141	141 × 1d-mcast	140
qwen_3_8b	g1	181	181 × 1d-mcast	180
llama_3_2_1b	g1	81	81 × 1d-mcast	80
llama_3_1_8b	g1	161	161 × 1d-mcast	160
falcon3_7b	g1	141	141 × 1d-mcast	140
gemma_1_1_2b	g2	91	1d/2d-mcast mix	90

The vLLM path is untouched. All 34 vllm_* benchmarks show zero DS matmuls
in every graph, before and after. This analysis says nothing about vLLM serving.

Role of each core group

From tt-metal
matmul_multicore_reuse_mcast_dram_sharded_program_factory.cpp:

group	count	set by
in0 storage / mcast senders	kNumIn0Cores = 8	tt-mlir policy constant (MatmulRules.cpp:41)
compute cores	num_worker_cores = num_dram_banks	hardware — 12 on WH, 8 on BH
output storage cores	div_up(N_tiles, numOutCores) → grid	tt-mlir, numOutCores = worker grid volume

The output is not capped at 8: <1x48> / <1x56> on n150, <1x48> /
<1x94> on p150. So 8–12 compute cores scatter results across up to 94 storage
cores.

4. Two hard failures introduced

4a. n150 — DRAM OOM on the 7B–8B class

Four models produced a clean number before with zero OOM lines and die after;
a fifth already failed for another reason. Surfaces as
RuntimeError: Bad StatusOr access: INTERNAL: Error code: 13; artifacts stop
after g1.

model	before sps	requested	free	largest block
llama_3_1_8b_instruct	21.32	117.4 MB	24.7 MB	5.2 MB
mistral_7b	19.37	117.4 MB	28.1 MB	8.2 MB
tiiuae_falcon3-7b-base	18.45	141.6 MB	20.1 MB	7.3 MB
qwen_2_5_7b_instruct	15.89	135.8 MB	16.8 MB	7.5 MB
qwen_3_8b	(failed before too)	100.7 MB	20.2 MB	4.5 MB

TT_FATAL: Out of Memory: Not enough space to allocate 117440512 B DRAM buffer
across 12 banks ... (free: 24670720 B, largest free block: 5199552 B)
  @ tt-metal/tt_metal/impl/allocator/bank_manager.cpp:462

Note free ≫ largest block: fragmentation, not only capacity. All five run fine on
p150, which has more DRAM.

Wormhole also pays a padding penalty the 8-bank parts don't:
padToDRAMBanks(8960, 12) rounds N up to a multiple of 384 → 9216 (2.9% waste);
with 8 banks 8960 is already a multiple of 256.

4b. qb2 — L1 circular-buffer clash

qwen_3_32b_tp_qb2 scored 16.57 sps before and fails after (artifacts stop at
2 graphs):

TT_THROW: Statically allocated circular buffers in program 348024 clash with L1
buffers on core range [0-0 - 0-0]. L1 buffer allocated at 1506688 and static
circular buffer region ends at 1508608

A 1920-byte overshoot — with the conservative L1 reservation already in place.

The other five new qb2 failures are not DS-attributable (all have zero DS
matmuls): glm_image, hunyuan_image_2_1, vllm_llama_3_1_70b_qb2_tp timed
out; vllm_qwen2_5_coder_32b hit a vLLM engine-init error;
wan14b_vae_encoder_720p_sharded exited 3.

5. Root cause: in0_block_w collapse

computeShardParams (MatmulProgramConfig.cpp) starts in0BlockW at
kPerCore = kTiles / kNumIn0Cores and walks it down to the largest divisor
whose circular buffers fit L1. Each halving of in0_block_w doubles
num_blocks = kTiles / in0_block_w, i.e. the number of serialized K-blocks with
their mcast and CB-sync overhead.

Define ratio = achieved in0_block_w / kPerCore. 1.00 means never walked down.

plat	model	Δ%	min ratio	worst shape
n150	qwen_2_5_1_5b	+23.5	1.00	—
n150	qwen_3_4b	+5.7	1.00	—
n150	qwen_3_0_6b	+5.5	1.00	—
n150	llama_3_2_3b	+3.3	1.00	—
n150	qwen_2_5_3b	+2.0	1.00	—
p150	qwen_2_5_1_5b	+10.1	1.00	—
qb2	falcon3_7b	+2.6	1.00	—
qb2	llama_3_1_8b	−1.5	1.00	—
p150	llama_3_2_1b	−6.7	1.00	—
qb2	coder_32b	+4.7	0.33	K6912 9/27
qb2	llama_3_1_70b	+4.0	0.25	K7168 7/28
qb2	mistral_24b	−3.5	0.25	K5120 5/20
p150	llama_3_1_8b	−4.4	0.25	K14336 14/56
p150	gemma_1_1_2b	−8.8	0.25	K2048 2/8
p150	qwen_3_8b	−11.9	0.25	K4096 4/16
p150	falcon3_7b	−12.6	0.17	K23040 18/90
p150	qwen_2_5_3b	−28.1	0.02	K11008 1/43

group	n	median	pos/neg
in0_block_w at max	9	+3.30%	7 / 2
in0_block_w walked down	8	−6.60%	2 / 6

All five n150 models keep in0_block_w at max. 12 banks → narrower per-bank
weight shard → smaller in1 CB → the budget never binds. That is a mechanical
explanation of the n150/p150 split, and it supersedes the earlier "Blackhole has
worse per-bank bandwidth" hypothesis, which was speculation.

4 of 17 are misclassified (p150 llama_3_2_1b −6.7% and qb2 llama_3_1_8b
−1.5% lose without a walk-down; qb2 llama_3_1_70b and coder_32b win with one
— both very large, heavily DRAM-bound models). So a residual effect remains
unexplained.

The −28.1% case in full

p150 qwen_2_5_3b, decode graph:

BEFORE (99c621da)                          AFTER (ae602833)
70x 1d-mcast K2048  N11008  grid 11x8      72x DS  K2048  N11008  ibw=4   1x8 [64,43]
36x 1d-mcast K2048  N2560   grid 11x8      36x DS  K2048  N2048   ibw=8   1x8 [64,8]
36x 1d-mcast K2048  N2048   grid 11x6      36x DS  K11008 N2048   ibw=1   1x8 [344,8]  <--
36x 1d-mcast K11008 N2048   grid 11x6      36x 1d-mcast K2048 N2560 (unchanged)

kTiles = 11008/32 = 344, kPerCore = 344/8 = 43, and 43 is prime, so the
kPerCore % in0BlockW == 0 constraint admits only {43, 1}. Blackhole
l1Available = 0.95 × (1572864 − 98304) = 1,400,832:

cbBudget = 1,400,832 − in0TensorBuf 88,064 − outTensorBuf 16,384 = 1,296,384
ibw=43 → fixed 49,152 + in0CB 176,128 + in1CB 1,122,816 = 1,348,096  over by 51,712
ibw=1  → fixed 49,152 + in0CB   4,096 + in1CB    26,112 =    79,360  fits

So that projection runs 344 serialized K-blocks instead of 8. in1CB is 83%
of the requirement. This reproduces the emitted ibw=1 exactly.

The qb2 pair that disagrees at identical shard width

llama_3_1_70b_tp +4.0%:

n	M	K	N	config	weight/bank [Kt, Nt]	out
160	32	8192	7168	ibw=16, m=1, n=3	1x8 [256, 28]	1x75 [1,3]
80	32	8192	2560	ibw=32, m=1, n=1	1x8 [256, 10]	1x80 [1,1]
80	32	2048	8192	ibw=8, m=1, n=3	1x8 [64, 32]	1x86 [1,3]
80	32	7168	8192	ibw=7, m=1, n=3	1x8 [224, 32]	1x86 [1,3]
1	32	8192	32064	1d-mcast (LM head)	1x1 [256, 1002]	1x101 [1,10]

mistral_small_24b_tp −3.5%:

n	M	K	N	config	weight/bank	out
80	32	5120	8192	ibw=5, m=1, n=3	1x8 [160, 32]	1x86 [1,3]
40	32	5120	1536	ibw=20, m=1, n=1	1x8 [160, 6]	1x48 [1,1]
40	32	1024	5120	ibw=4, m=1, n=2	1x8 [32, 20]	1x80 [1,2]
40	32	8192	5120	ibw=16, m=1, n=2	1x8 [256, 20]	1x80 [1,2]
1	32	5120	131072	1d-mcast (LM head)	1x1 [160, 4096]	1x108 [1,38]

Why tensor parallelism rescues Blackhole

TP splits the weight across devices, shrinking the per-bank shard and therefore
in1CB. Same model, same 8 banks, decode graph:

llama_3_1_8b, N-tiles per bank
p150, single device	56, 24, 16, 16 → max 56
qb2, tensor-parallel	16, 16, 14, 6 → max 16

6. Why relaxing the in0_block_w divisor constraint is NOT viable

The tempting fix — let in0_block_w take values that don't divide kPerCore, so
K=11008 gets options between 43 and 1 — breaks tt-metal. Two independent
reasons.

6a. A hard TT_FATAL on Kt % in0_block_w

matmul_multicore_reuse_mcast_dram_sharded_program_factory.cpp:1012:

TT_FATAL(Kt % in0_block_w == 0, "Kt ({}) must be divisible by in0_block_w ({})", Kt, in0_block_w);

Uncatchable abort. Any in0_block_w not dividing Kt is rejected outright.

6b. The kernel assumes a uniform, exact block→sender partition

Host (same file):

uint32_t num_blocks = K / in0_block_w;                                  // line 181
uint32_t num_blocks_per_shard = num_blocks / input_all_storage_cores_vec.size();  // line 291

Both are integer division with no exactness check. The only TT_FATAL on
num_blocks_per_shard is guarded by if (per_core_M > 1), and DS requires
per_core_M == 1, so it never fires here.

num_blocks_per_shard is then a constexpr compile-time arg to
reader_bmm_tile_layout_in0_sender_dram_sharded.cpp, which derives:

constexpr uint32_t num_blocks_per_shard = get_compile_time_arg_val(14);
constexpr uint32_t num_storage_cores  = num_blocks / num_blocks_per_shard;   // line 36
...
tt_l1_ptr uint32_t* in0_mcast_sender_noc_x = (tt_l1_ptr uint32_t*)(get_arg_addr(3));
tt_l1_ptr uint32_t* in0_mcast_sender_noc_y = (tt_l1_ptr uint32_t*)(get_arg_addr(3 + num_storage_cores));
const uint32_t sender_block_id = sender_id * num_blocks_per_shard;           // line 50
...
for (uint32_t i = 0; i < num_blocks_per_shard; ++i)      // sender: blocks [sid*nbps, sid*nbps+nbps)
for (uint32_t block = 0; block < num_blocks; ++block)    // receiver
    const uint32_t block_id = block / num_blocks_per_shard;

If num_blocks is not an exact multiple of the sender count, three things go
wrong at once:

1. num_storage_cores is recomputed wrong, so the mcast y-coordinate array
   is read from the wrong get_arg_addr offset → mcast to garbage NoC coords.
2. Senders collectively cover only n_senders × nbps blocks; the tail blocks
   are never sent, so part of K is never accumulated.
3. The receive loop still iterates all num_blocks and computes
   block_id = block / nbps ≥ n_senders for the tail, indexing past the sender
   array and waiting on a semaphore no one will signal → hang.

Wrong numerics and/or deadlock — never a clean error.

6c. tt-mlir's constraint is exactly right

Let Kt = K in tiles, nc = kNumIn0Cores, q = Kt/nc (integer, guaranteed by
the eligibility gate Kt % nc == 0).

• tt-mlir requires ibw | q.
• tt-metal requires (a) ibw | Kt and (b) nc | (Kt/ibw).

⇒ If ibw | q: Kt = nc·q so Kt/ibw = nc·(q/ibw) — both (a) and (b) hold.
⇐ If (a) and (b): Kt/ibw = nc·m → Kt = nc·m·ibw → q = m·ibw → ibw | q.

The conditions are equivalent. kPerCore % in0BlockW == 0 is the exact
necessary and sufficient condition, not a conservative tt-mlir invention.
For Kt=344, nc=8 the legal set really is {1, 43}.

6d. in0_last_ktile_w is unrelated

uint32_t in0_last_ktile_w = a.logical_shape()[-1] % in0_tile_shape[1];   // line 1010

It is K_elements % 32 — the count of valid columns in the final K tile when
K is not tile-aligned, consumed by pad_last_ktile<>() to zero the invalid
columns. It has nothing to do with block granularity and offers no path to
partial blocks. (An earlier guess that it might enable a remainder block was
wrong.)

7. Recommendations

Ordered by value and confidence.

1. Decline DS when in0BlockW lands far below kPerCore (or when
   num_blocks exceeds a bound) and fall back to mcast. computeShardParams
   already returns std::nullopt for the no-fit case, so this is a guard in the
   same place, not a new mechanism and not a fitted threshold. On the measured
   set it removes every p150 regression while keeping all nine at-max wins.
2. Reconsider the in1 CB depth. tt-metal triple-buffers it
   (in1_CB_tiles *= 3 when B*num_blocks > 1); tt-mlir's model matches
   exactly, so the 1,122,816 B is real. At ×2 it becomes 748,544 and ibw=43
   fits within cbBudget outright. This is a tt-metal pipeline-depth change and
   risks throughput on cases that already work — measure before adopting.
3. If kNumIn0Cores becomes a lever, search it per-op — do not swap it for
   another constant. See §7.1 below.
4. Do not relax the divisor constraint — see §6.
5. File the qwen_2_5_1_5b gap separately. Its +23.5% / +10.1% is not a DS
   win: its decode graph previously had no matmul program config at all, so DS
   was beating TTNN's default heuristic. Any change that turns DS off for it
   re-exposes that gap. The durable fix is emitting a proper mcast config there.

7.1 Why kNumIn0Cores = 8 is not a grid number, and why 10/11 are wrong

The in0 activation is a single width-sharded L1 row — emitted as <1x8> at
core_range<(0,0),(7,0)> — so nc is bounded by the grid width: 8 on
Wormhole, 11 on Blackhole (confirmed by p150 output core_ranges
(0,0)-(10,3) + (0,4)-(3,4) = 48 cores). That makes "8 = one Wormhole row"
a tempting reading, and suggests 10 or 11 for Blackhole.

The measurements say otherwise. The binding constraint is the eligibility gate
kTiles % nc == 0 with kTiles = K/32, evaluated over 2750 DS matmul instances
and 20 distinct K values sampled across all three platforms:

nc	fits WH row (8)	fits BH row (11)	eligible K values	eligible ops	coverage
2	yes	yes	20/20	2750	100%
4	yes	yes	20/20	2750	100%
8	yes	yes	20/20	2750	100%
10	no	yes	5/20	504	18%
11	no	yes	0/20	0	0%
16	no	no	15/20	2466	90%

Observed kTiles: 24, 32, 40, 48, 64, 80, 96, 112, 128, 160, 216, 224, 256,
280, 304, 344, 384, 448, 512, 720.

8 works because transformer K dims are essentially always multiples of 256, so
kTiles is essentially always a multiple of 8. Its equality with Wormhole's
row width is a coincidence of two unrelated facts (tile width 32 combined with
model dims, versus a physical grid dimension). 11 divides none of the observed
kTiles; 10 divides only the five that are multiples of 320 (K = 1280, 2560,
5120, 8960, 23040). Setting nc = 10 or 11 on Blackhole would not retune DS —
it would disable it for 82% / 100% of matmuls, "fixing" the p150 regression by
switching the feature off.

As an L1 lever nc cuts both ways, since
in1CB = in0_block_w × shardWTiles × weightTile × 3 and
in0_block_w ≤ kTiles/nc:

• larger nc shrinks ibw_max and so shrinks in1CB;
• smaller nc gives the walk-down more divisors to land on.

For the K=11008 case (kTiles=344; q=43 is prime at nc=8):

nc	q = kTiles/nc	best fitting ibw	num_blocks	coverage
8 (today)	43 (prime)	1	344	100%
4	86 = 2·43	2	172	100%
2	172	4	86	100%

nc=2 is 4× better on the worst case at full coverage, but
in0TensorBuf = perCoreM·q·2048 grows 88 KB → 352 KB, consuming the budget it
frees, and 2 mcast senders will likely hurt the shapes that are currently fine.
nc=16 halves in1CB but needs the in0 row to wrap across grid rows, and it
does not rescue the bad cases: K=11008 becomes ineligible, and p150
falcon3_7b still needs 45 × 90 × 1088 × 3 ≈ 13 MB because its per-bank shard
is 90 tiles wide.

Conclusion: no single global constant is correct. The good nc depends on
the factorization of kTiles and on shardWTiles for that specific matmul. If
nc is made a lever it should be searched per-op jointly with in0_block_w to
minimize num_blocks subject to the L1 budget. The decline-on-collapse guard
(recommendation 1) remains the cheaper first move.

Minor modelling discrepancies found in computeShardParams

• outTensorBufPerCore uses numIn0Cores where the resident output shard is
  perCoreM × perCoreN (which already derives from numOutCores). For a
  1×8960 output on a 110-core grid that reserves 35 tiles against 3 resident
  (~66 KB of phantom L1). But relaxing it frees only ~14 KB on the −28.1%
  matmul, which needs ~52 KB, so it does not rescue that case — and given §4b's
  L1 CB clash, loosening the budget is the wrong direction.
• The budget omits tt-metal's out_reshard_CB_size
  (per_core_M × per_core_N_storage tiles), a small under-estimate in the
  opposite direction.

8. Status of the stashed patch

A candidate fix exists in git stash on the tt-mlir submodule (detached HEAD at
ae602833), 11 tracked files +136/−9 plus one new lit test in stash@{0}^3:

• per-arch gate declining DS on Blackhole (archPrefersDRAMSharded)
• force-dram-sharded-matmul pipeline option to bypass it
• the outTensorBufPerCore denominator change

It was verified green (75 lit tests: 72 pass / 2 unsupported / 1 XFAIL; 305
OptimizerTests; 13 DS gtests) but should not land as written:

• qb2 is Blackhole and is not harmed by DS, so an arch gate forfeits its wins.
• The arch axis was fitted to two data points; §5 shows the mechanism is
  in0_block_w collapse, which is arch-correlated but not arch-determined.
• The L1 budget change is the wrong direction given §4b.

Pre-existing and unrelated: OpModelStrategyTests aborts in
ScoreCandidateL1ShardedResult and ReshardCandidatesTest.WithTensorLayoutsMapDoesNotCrash
(deriveCanonicalL1CoreRangeSet: gridShape[1] == 1U failed, was 4 == 1).
Confirmed by stashing the diff, rebuilding, and reproducing on clean ae602833.

9. Caveats

• One sample per model per run. No repeats, so no variance estimate. The
  control spread (±3% on n150/p150, ±1.6% on qb2) is the practical noise floor;
  treat individual deltas under ~3% as directional only. Medians are the
  trustworthy figures.
• No device perf anywhere. All eight runs used skip device perf: true.
• p150's before run lost 13 models to infra (an
  action-download-artifact codeload timeout and
  TT_FATAL @ topology_mapper.cpp:546), so those have no p150 comparison.
• galaxy has no data at all. Both galaxy and the first qb2 attempt were
  dispatched with tt-mlir 626a734b, which was force-pushed out of the branch
  and is the head of no ref. docker-build / build-image died with
  fatal: reference is not a tree: 626a734b... (exit 128) → build skipped →
  the CI image tag was never pushed → every perf job failed container init with
  Docker pull failed / Value cannot be null. (Parameter 'ContainerId').
• TTFT moved only weakly (−1% to −5% both platforms) with no clean
  correlation, consistent with prefill being untouched.

10. Reproducing

# perf numbers: one ~3 MB zip per run, grep the benchmark blocks
gh api repos/tenstorrent/tt-xla/actions/runs/<id>/logs > run.zip
# "| Avg. samples per second:" / "| TTFT (ms):" per job log

# which models got DS, and their configs
gh api repos/tenstorrent/tt-xla/actions/runs/<id>/artifacts --paginate \
  --jq '.artifacts[] | select(.name|startswith("ttnn-mlir")) | "\(.id) \(.name)"'
gh api repos/tenstorrent/tt-xla/actions/artifacts/<aid>/zip > a.zip
grep -c matmul_multi_core_reuse_multi_cast_dram_sharded_program_config *_g1_*.mlir

# A/B a single platform from tt-xla: both options live on
# TTIRToTTNNCommonPipelineOptions, which module_builder.cc already builds
#   options.disableDRAMShardedMatmul = true;   // existing kill switch

[bmalesevicTT]

before adding guard to avoid transforming shape-inadequate matmul - DRAM-sharded matmul on p150: slower on every model measured: https://claude.ai/code/artifact/01907768-d057-400c-bb87-632afee36389

after adding guard - Declining the collapsed block width removes the Blackhole regression: https://claude.ai/code/artifact/c9c23343-c87d-4f75-a287-8496df6201f2
guard examined on n150 as well, should be a bit different to show gains according to device time measurements as the guard has false positives, but it actually helps some models not DRAM OOM as it is(because if we don't shard decode weights, there are no duplicate weights for dram interleaved prefill ones) so we keep it as-is: https://claude.ai/code/artifact/41edbd38-becc-4db8-96f3-9f20a1f9dfc3

lofi located as the compute-bound reason Blackhole doesn't progress: https://claude.ai/code/artifact/7772cbbf-c9f3-4221-adb6-1237cd86cb15

Forcing LoFi speeds up p150, costs no significant correctness measured locally with top1/top5 on n150/p150: https://claude.ai/code/artifact/cf9ca218-4477-4c47-9279-35d717c3bd1a
n150 LoFi effect analysis: https://claude.ai/code/artifact/56c454e9-9a75-4a52-a793-da4a868e5e5d