Support a customizable header exclusion list for transparent proxy scenarios

[sreth]

What should we add or change to make your life better?

Add support for customizable header exclusion lists in HttpTransformer
Currently, YARP's RequestUtilities uses a hardcoded list of 17 headers to exclude during header copying for reverse proxy scenarios. For transparent forward proxy implementations, we need to preserve most of these headers to maintain transparency while only filtering the minimal RFC 9110 hop-by-hop headers (6 headers).

Proposed Solutions:

• Introduce a deny-list counterpart to AllowedHeadersTransform (which YARP is already planning to add) so that consumers can explicitly define which headers should be excluded.
• This would enable consumers to specify their own exclusion list while leveraging YARP's battle-tested security checks (header injection prevention, content header handling, RFC compliance).

Why is this important to you?

Use Case: Transparent Forward Proxy
We're building a transparent forward proxy where the destination server should not detect the proxy's presence. YARP's current 17-header exclusion list is designed for reverse proxies. For forward proxies, we only need to filter the 6 RFC 9110 hop-by-hop headers: Connection, Transfer-Encoding, Keep-Alive, Upgrade, Proxy-Connection, TE.

Current Workaround:
We've duplicated ~200 lines from YARP's RequestUtilities including security validation, ContentHeaders set, EmptyHttpContent class, and smuggling prevention logic.

Problems with this approach:

1. Code duplication and maintenance burden
2. Manual security fix syncing required

Benefits:

1. Eliminate ~200 lines of duplicated code
2. Reuse battle-tested YARP security checks
3. Auto-benefit from YARP security updates
4. Enable flexible header policies (reverse/forward proxies)

Impact:
This feature would make YARP more flexible for scenarios beyond traditional reverse proxies (forward proxies, transparent proxies, API gateways with custom header policies) while reducing maintenance burden for teams who need slightly different header filtering behavior.