Validate RSA public exponents before verification

[thieman]

What this PR changes

Closes #18.

This PR extends the RSA public-key policy used during signature verification. The provider already checks that an RSA modulus is in the rustls-webpki-compatible 2048-8192 bit range. This PR adds the missing check for the RSA public exponent.

The new policy rejects RSA public exponents that are:

• too small, such as e = 1 or e = 2;
• even;
• larger than the reference provider accepts.

Normal keys such as e = 65537 continue to be accepted.

What is an RSA public exponent?

An RSA public key has two main numbers:

• n, the modulus — the large number people usually mean when they say "2048-bit RSA key";
• e, the public exponent — a smaller number used as part of the verification operation.

For RSA to make cryptographic sense, e cannot be arbitrary. RFC 8017, the PKCS #1 RSA specification, says a valid RSA public exponent is an integer between 3 and n - 1 with the required number-theoretic relationship to the key: https://datatracker.ietf.org/doc/html/rfc8017#section-3.1

In practice, almost all modern RSA public keys use e = 65537. That value is odd, large enough, and efficient.

Why this matters

This crate is a rustls crypto provider. That means applications can swap it in place of the default providers. When they do, they should not silently accept RSA certificates/signatures that rustls' built-in providers would reject.

The reference providers are stricter than "whatever CNG accepts". ring documents its RSA verification algorithms as requiring the public exponent to be an odd integer of 2-33 bits: https://docs.rs/ring/latest/ring/signature/index.html

rustls-webpki wires its RSA algorithms around the same reference-provider constraints, including 2048-8192 bit RSA algorithms: https://docs.rs/rustls-webpki/latest/src/webpki/ring_algs.rs.html

If we do not check e ourselves, then behavior depends on CNG's key import policy. That is not the contract we want for a rustls provider. The provider should enforce the same public-key policy before handing the key to CNG.

Why this fix is the right thing to do

This PR adds exponent validation next to the existing modulus validation in src/verify.rs. That keeps the policy in one place: before RSA public keys are imported into CNG for verification.

The implementation checks:

• exponent bit length is between 2 and 33 bits;
• exponent is odd.

For positive integers, that means:

• e = 1 is rejected;
• e = 2 is rejected;
• even values are rejected;
• values above 2^33 - 1 are rejected;
• common values like 65537 are accepted.

This matches the ring/aws-lc-rs style policy we are trying to align with, rather than inventing a new policy for this crate.

How to read this if you are not a crypto expert

A bad RSA exponent is like a malformed lock component. The rest of the public key might look large enough, but if this small parameter is nonsensical, signature verification should not proceed.

CNG may reject many bad keys itself. But for security-sensitive provider behavior, we do not want to rely on undocumented or platform-specific acceptance rules. We make the policy explicit and test it.

Validation

This PR adds focused tests for the RSA public-key policy:

• accepts 65537;
• accepts the maximum 33-bit odd exponent;
• rejects 1;
• rejects 2;
• rejects an even exponent;
• rejects an oversized exponent.

The PR keeps the change surface to src/verify.rs.

[thieman]

Ensure that there is test coverage here covering all exact bounds we've added, e.g. RSA_MIN__MODULUS_BITS etc.

[thieman]

[GPT 5.5] Addressed in 0ec47f8. The modulus exact bounds were already covered (2047, 2048, 8192, 8193), and the exponent test covered the upper accepted bound plus rejected neighbors. The missing case was the lower accepted exponent bound, so I added explicit coverage for e = 3 alongside the existing e = 1 / e = 2 rejection cases.

[webern]

So we are being more strict with the exponent values than the libraries we are calling? LGTM/not-a-crypto-expert.

[thieman]

So we are being more strict with the exponent values than the libraries we are calling? LGTM/not-a-crypto-expert.

I don't think it's making a claim as to what CNG does, maybe it would reject. Instead this is mirroring some validation that the other rustls providers seem to be doing. The other PRs in this sequence are doing similar mirroring.