[release/9.0-staging] fix: in rsa signatures, configure digest before padding mode

[github-actions]

Backport of #114261 to release/9.0-staging

/cc @vcsjones @rcatolino

Customer Impact

• Customer reported
• Found internally

This was originally reported by a customer in #114260, and a request to back ported it was made in #115693.

Customers using RSA.SignHash with RSASSA-PSS on Linux which is locked-down to FIPS are unable to use it because the configuration was done in such a way that confused the FIPS-only validation logic in OpenSSL. Customers have no work-arounds for this, other than do disable the FIPS enforcement on Linux.

Regression

• Yes
• No

No. It's been like this since we moved off the managed implementation of PSS padding for Linux.

Testing

This was manually validated on an Ubuntu Pro installation that was FIPS enforced.

Risk

Low. The code that configures the signing context has only changed in the order in which it does the configuration. The area has strong unit test covered to ensure existing scenarios did not regress.

IMPORTANT: If this backport is for a servicing release, please verify that:

• The PR target branch is release/X.0-staging, not release/X.0.

Package authoring no longer needed in .NET 9

IMPORTANT: Starting with .NET 9, you no longer need to edit a NuGet package's csproj to enable building and bump the version.
Keep in mind that we still need package authoring in .NET 8 and older versions.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.