Fixes in SP int and DH

[embhorn]

Description

• Fix SP int truncation issue
• Check params to GeneratePrivateDh186

Thanks to Kr0emer for reporting these issues and testing the fixes.

Fixes:

• zd21821
• zd21822

Testing

Added regression test to dh_fips_generate_test

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10478

Scan targets checked: wolfcrypt-bugs, wolfcrypt-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes integer truncation issues in sp_int bit/digit shift APIs and adds an upper-bound check on *privSz in GeneratePrivateDh186 to prevent stack buffer overflow. Adds a regression test for the DH case.

Changes:

• Widen intermediate computations in sp_set_bit, sp_rshd, sp_rshb, and sp_mod_2d to int to avoid sp_size_t (potentially word16) wrap-around before bounds checks.
• Reject *privSz > DH_MAX_SIZE in GeneratePrivateDh186 before writing into the stack cBuf.
• Add a regression test in dh_fips_generate_test exercising the oversize privSz rejection path.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
wolfcrypt/src/sp_int.c	Avoid narrowing of shift/digit counts to sp_size_t before bounds checks.
wolfcrypt/src/dh.c	Add *privSz > DH_MAX_SIZE guard in GeneratePrivateDh186.
wolfcrypt/test/test.c	Regression test invoking wc_DhGenerateKeyPair with oversized *privSz.

Comments suppressed due to low confidence (1)

wolfcrypt/test/test.c:1

• The regression test only covers the case *privSz == DH_MAX_SIZE + 1. It would be worth also exercising a value comfortably above any plausible byte interpretation (e.g. DH_MAX_SIZE * 2 or 0xFFFFFFFF - 8) and a boundary value at exactly DH_MAX_SIZE to lock down the accepted/rejected boundary, especially given the bit-vs-byte ambiguity in the new bound.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[embhorn]

Will rebase after #10480 gets merged

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-baremetal

• FLASH: .text -64 B (-0.1%, 64,827 B / 262,144 B, total: 25% used)
  No memory changes detected for:
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-tls12

[embhorn]

Jenkins retest this please