Shuffle partition recommendations can only increase, with no size-aware bound

[parthosa]

Problem

All four AutoTuner paths for spark.sql.shuffle.partitions raise or hold; none lowers. Three multiply by a fixed factor on a yes/no trigger without looking at data volume. The fourth, the ColumnarExchange bound, is size-aware — but once a doubling carries the value past it, it stops being the maximum and drops out. The count ratchets up with nothing to check it.

Evidence

Three successive runs of one query, same ~4.08 TB shuffle, 12 × 16-core cluster. Every raise came from spill ×2:

ran at	recommended	outcome
500	1900	cut off at 694s
1900	3800	cut off at 760s
3800	7600	succeeded in 303s

The third run succeeded and was still told to double, over 3.0 GB of residual spill. Nothing can bring 7600 back down. Meanwhile ceil(4.079 TB / 1 GiB) = 3800 — the configuration that worked — was computable from the first run.

What's needed

A pass sizing partitions from measured shuffle input, able to move the value either way, with the existing spill and OOM paths as floors. Sizing from consumer-stage totals, not the largest single exchange. See #2129.

Open questions

1. Should the target derive from spark.rapids.sql.batchSizeBytes? Raising the batch for kernel efficiency silently halves the partition count — two unrelated concerns on one knob.
2. May a measured target correct a spill-driven raise? A pass computes 3840 against that 7600, but only helps if existing raises aren't inviolable.

[parthosa]

Corrected the target-size section of the description.

The earlier version compared the 1 GiB target against a hand-tuned run using 500 partitions and concluded 1 GiB was too aggressive. That comparison was invalid — the hand-tuned run had 12 executors × 4 cores (48 slots) against the GPU cluster's 12 × 16 (192). Bytes-per-partition doesn't compare across a 4× difference in cluster width; normalized on execution waves the two are 10.4 vs 9.9, not 7.6× apart.

The 1 GiB target is corroborated by the run table in the description: ÷ 2 GiB gives 1900, which was cut off at 760s; ÷ 1 GiB gives 3800, which succeeded in 303s and then 187s.