`uint16_t nonce` in signing loop wraps around for ML-DSA-87, causing nonce reuse

[programsurf]

The rejection sampling counter nonce in ref/sign.c:97 is uint16_t:

uint16_t nonce = 0;                    // sign.c:97
...
nonce += L;                            // sign.c:123

For ML-DSA-87 (L=7), after 9362 iterations this wraps from 65534 back to 5 (65534 + 7 = 65541, truncated to 5 in uint16_t). After the wrap, poly_uniform_gamma1() gets a previously used counter value and produces identical SHAKE-256 output — nonce reuse in mask generation.

FIPS 204 defines κ as an "integer" without specifying width, so uint16_t isn't technically a spec violation. But the overflow clearly isn't intended behavior.

The probability of hitting 9362 iterations is ~2^{-23400} for ML-DSA-87, so this is theoretical. The fix is one line:

- uint16_t nonce = 0;
+ uint32_t nonce = 0;

Microsoft SymCrypt had the same issue (UINT16 k in mldsa.c:461) and has acknowledged it as a bug: microsoft/SymCrypt#55

I've also contacted NIST (pqc-comments@nist.gov) to suggest clarifying the minimum bit width of κ in the spec.

Sunwoo Lee and Seunghyun Yoon, Korea Institute of Energy Technology (KENTECH)

Edit: Sunwoo Lee, Woohyun Choi, and Seunghyun Yoon, Korea Institute of Energy Technology (KENTECH)
@yoonsh @woohyunchoi-kentech