nvidia-modeset: fix UB in DP DSC flatnessDetThresh calculation

[edenfunf]

Problem

EvoSetDpDscParams() (nvkms-evo3.c) and EvoSetDpDscParamsC9() (nvkms-evo4.c) compute the DSC flatness detection threshold as:

flatnessDetThresh = (2 << (pDscInfo->dp.bitsPerPixelX16 - 8));

bitsPerPixelX16 is the target bits-per-pixel multiplied by 16 (e.g. 128 for 8.0 bpp, 384 for 24.0 bpp). This produces a left shift of 120–376 bits, which is undefined behavior in C — the shift amount far exceeds the width of NvU32. The hardware register field FLATNESS_DET_THRESH is only 10 bits wide, so whatever garbage value results from the UB gets truncated and programmed into the DSC encoder.

The existing code even has an engineer comment acknowledging this:

// XXX: I'm pretty sure that this is wrong.
// BitsPerPixelx16 is something like (24 * 16) = 384, and 2 << (384 - 8) is
// an insanely large number.

The HDMI DSC path (EvoSetHdmiDscParams / EvoSetHdmiDscParamsC9) handles this correctly by deriving bpc from the pixelDepth parameter and computing 2 << (bpc - 8). However, the DP path does not receive a pixelDepth parameter.

Fix

Per VESA DSC 1.2a section 4.1, the correct formula is:

flatness_det_thresh = 2 << (bits_per_component - 8)

Since the DP path doesn't have pixelDepth, we extract bits_per_component directly from PPS byte 3 bits [7:4], which the DP library has already computed when generating the PPS.

This patch introduces two shared utility functions:

• nvDscPpsGetBitsPerComponent() — extracts bpc from PPS DWord 0 bits [31:28]
• nvDscComputeFlatnessDetThresh() — computes 2 << (bpc - 8) with validation/clamping for safety

Both the DP and HDMI paths now use nvDscComputeFlatnessDetThresh(), consolidating the duplicated inline validation that previously existed only in the HDMI path.

Impact

Affects all DisplayPort DSC displays on all GPU generations supported by the open kernel modules. The flatness detection threshold controls how the DSC encoder identifies flat (uniform) image regions for optimized bit allocation. An incorrect threshold may cause suboptimal compression and potential visual artifacts.

Files Changed

• src/nvidia-modeset/include/nvkms-utils.h — declare new helpers
• src/nvidia-modeset/src/nvkms-utils.c — implement nvDscPpsGetBitsPerComponent() and nvDscComputeFlatnessDetThresh()
• src/nvidia-modeset/src/nvkms-evo3.c — fix EvoSetDpDscParams(), refactor EvoSetHdmiDscParams()
• src/nvidia-modeset/src/nvkms-evo4.c — fix EvoSetDpDscParamsC9(), refactor EvoSetHdmiDscParamsC9()

Closes #1029