[ICACF-15] Restrict gateway test endpoint to approved hosts - SSRF protection

[MohanLaksh]

Summary

Implements SSRF protection for endpoint to prevent open proxy abuse and internal service access.

JIRA: https://jsw.ibm.com/browse/ICACF-15
GitHub Issue: Closes #4314

---

Problem

The /admin/gateways/test endpoint accepted arbitrary user-supplied URLs without proper validation, creating an SSRF vulnerability where attackers could:

• Use the IBM server as an open proxy to reach external hosts
• Access internal microservices (e.g., context-broker) that should not be publicly accessible
• Bypass URL filters using trailing-dot FQDN normalization (e.g., evil.com.)

---

Changes

1. Configuration (mcpgateway/config.py)

• Added gateway_test_allowed_hosts setting for allowlist enforcement
• Supports wildcards (*.example.com) and exact matches
• Empty list = backwards compatible (SSRF protection only)

2. Validator Enhancement (mcpgateway/common/validators.py)

• New method: SecurityValidator.validate_host_allowlist() with wildcard support
• Critical bug fix: _validate_ssrf() now uses normalized hostname (strips trailing dots) for IP parsing and DNS resolution
• Security Impact: Fixes CVE-level trailing-dot FQDN bypass vulnerability

3. Endpoint Hardening (mcpgateway/admin.py)

• Enforces allowlist validation after URL validation
• Returns generic 400 error for non-allowlisted hosts (no internal details)
• Maintains existing SSRF protection (private IPs, loopback, cloud metadata)

4. Test Coverage

• 8 unit tests in tests/security/test_input_validation.py::TestGatewayTestEndpointSecurity
• 3 E2E tests in tests/e2e/test_admin_apis.py::TestAdminGatewayAPIs
• Tests cover: allowlist matching, trailing-dot normalization, private IP blocking, allowlist enforcement in live endpoint

---

Security Impact

Before:

• ❌ Arbitrary URLs accepted without validation
• ❌ Trailing-dot FQDN bypass active (CVE-level)
• ❌ No allowlist enforcement
• ❌ Open proxy vulnerability

After:

• ✅ Allowlist enforcement restricts outbound requests
• ✅ Trailing-dot normalization prevents bypass
• ✅ Private IPs, loopback, and cloud metadata blocked unconditionally
• ✅ Generic error messages prevent information disclosure

---

Testing

Test Results:

✅ 17,685 passed, 544 skipped, 2 xfailed, 0 failures (3 new E2E tests added)
✅ 8/8 unit tests passing (security validation)
✅ 3/3 E2E tests passing (endpoint integration)
✅ Coverage ≥95% on modified code (lines 13827-13830 covered)
✅ make ruff - All checks passed
✅ All SSRF protection tests verified

Manual Verification:

✅ Trailing-dot URLs blocked (169.254.169.254.)
✅ Allowlist exact matching works
✅ Allowlist wildcard matching works
✅ Trailing-dot normalized in allowlist
✅ Non-allowlisted hosts blocked
✅ Private IPs blocked unconditionally

---

Acceptance Criteria

#	Criterion	Status
✅	Server-side allowlist enforced	VERIFIED
✅	URL canonicalization/normalization (trailing-dot FQDN)	VERIFIED
✅	Private IPs blocked unconditionally (RFC 1918, loopback, link-local)	VERIFIED
✅	Generic 400 error messages (no internal details)	VERIFIED
✅	Regression test for trailing-dot bypass	VERIFIED
✅	OOB callback blocking test	VERIFIED

---

Configuration

Development (Permissive):

GATEWAY_TEST_ALLOWED_HOSTS=[]  # Empty = SSRF blocking only
SSRF_PROTECTION_ENABLED=true
SSRF_ALLOW_LOCALHOST=true
SSRF_ALLOW_PRIVATE_NETWORKS=true

Production (Strict - Recommended):

GATEWAY_TEST_ALLOWED_HOSTS=["api.approved.com", "*.allowed.com"]
SSRF_PROTECTION_ENABLED=true
SSRF_ALLOW_LOCALHOST=false
SSRF_ALLOW_PRIVATE_NETWORKS=false
SSRF_DNS_FAIL_CLOSED=true

---

Deployment Notes

1. Backwards Compatible: Empty GATEWAY_TEST_ALLOWED_HOSTS preserves existing SSRF-only behavior
2. Recommended: Set explicit allowlist in production
3. Migration: Audit existing gateway URLs and add to allowlist before enforcing
4. Monitoring: Watch for 400 errors on /admin/gateways/test after deployment

---

Files Changed

• .env.example - Added GATEWAY_TEST_ALLOWED_HOSTS documentation
• mcpgateway/admin.py - Allowlist enforcement in admin_test_gateway()
• mcpgateway/common/validators.py - Added validate_host_allowlist() + fixed trailing-dot bug
• mcpgateway/config.py - Added gateway_test_allowed_hosts setting
• tests/security/test_input_validation.py - Added 8 comprehensive unit tests
• tests/e2e/test_admin_apis.py - Added 3 E2E integration tests for coverage

---

Checklist

• Code follows project style guidelines
• All tests passing (17,685 passed, 0 failures)
• Security tests added and verified (11 total: 8 unit + 3 E2E)
• Documentation updated (.env.example)
• Backwards compatible
• No new linting issues (make ruff passes)
• Commit message follows conventional commits
• All acceptance criteria met
• Coverage ≥95% on modified code