[FIX][API]: store authheadersauth_value as dict to prevent JSON null on persist

[shoummu1]

Bug-fix PR

📌 Summary

When registering a gateway with auth_type=authheaders, the custom HTTP headers were silently discarded after the first write to the database. Any subsequent health check, auto-refresh, or tool-invocation attempt would find no authentication credentials and fail. The fix corrects the root cause and all downstream code paths where the mismatched type would have produced similar silent failures.

🔗 Related issue

Closes: #3480

🔁 Reproduction Steps

1. Register a gateway with auth_type: authheaders and one or more auth_headers entries (e.g. X-Custom-Auth-Header: my-token).
2. Observe that the gateway registers successfully.
3. Trigger a health check or tool refresh — the upstream server returns 401/403 because auth headers are missing.
4. Fetch the gateway via GET /gateways/{id}; auth_value is null.

🐞 Root Cause

Primary bug — register_gateway() creation path

In mcpgateway/services/gateway_service.py, after building header_dict from gateway.auth_headers, the creation path called:

auth_value = encode_auth(header_dict)   # returns a base64-encoded str

encode_auth() returns a str. However DbGateway.auth_value is declared as:

auth_value: Mapped[Optional[Dict[str, str]]] = mapped_column(JSON)

SQLAlchemy serialises a bare Python str stored in a Dict-typed JSON column as JSON null; the auth headers were lost on every INSERT. The update path (update_gateway()) already stored the plain dict directly — the creation path was inconsistent.

Secondary bug — type mismatch propagation

DbTool.auth_value is the opposite type — Mapped[Optional[str]] (Text). After fixing DbGateway, the same plain dict would have been passed to DbTool unchanged, storing a raw Python dict in a Text column. Additionally, every downstream code path that reads DbGateway.auth_value expecting a string had the same latent mismatch.

💡 Fix Description

gateway_service.py — register_gateway() (2 changes)

# Before (bug): encode_auth returns str → stored as JSON null in DbGateway JSON column
auth_value = encode_auth(header_dict)

# After (fix 1): store plain dict, consistent with update path and DB column type
auth_value = header_dict

# After (fix 2): DbTool.auth_value is Text, so encode for that column only
tool_auth_value = encode_auth(auth_value) if isinstance(auth_value, dict) else auth_value
# DbTool constructor receives tool_auth_value; DbGateway constructor receives auth_value

gateway_service.py — _update_or_create_tools() (2 changes)

The comparison existing_tool.auth_value != gateway.auth_value was comparing a str (DbTool Text column) against a dict (DbGateway JSON column) — always True, causing a spurious full-tool overwrite on every health-check refresh cycle. The assignment existing_tool.auth_value = gateway.auth_value would then store a raw dict into a Text column.

# Before (bug): str != dict is always True → spurious updates every refresh
auth_fields_changed = ... or existing_tool.auth_value != gateway.auth_value or ...
existing_tool.auth_value = gateway.auth_value  # stores raw dict in Text column

# After (fix): encode once; use the same value for both comparison and assignment
gateway_tool_auth_value = encode_auth(gateway.auth_value) if isinstance(gateway.auth_value, dict) else gateway.auth_value
auth_fields_changed = ... or existing_tool.auth_value != gateway_tool_auth_value or ...
existing_tool.auth_value = gateway_tool_auth_value

tool_service.py (3 changes)

1. Direct-proxy payload: gateway_payload["auth_value"] was set directly from gateway.auth_value (a dict); downstream header-building code expects an encoded string.
2. Runtime DB override path: isinstance(runtime_gateway_auth_value, str) guard silently dropped dict values with no fallback.
3. Cache-miss hydration path: same isinstance(..., str) guard silently dropped dict values.

All three are fixed by adding an elif isinstance(..., dict): gateway_auth_value = encode_auth(...) branch ahead of the existing str path.

export_service.py (2 changes)

Both the batch-query export path (export_gateways()) and the raw-query export path (_export_selected_gateways()) were writing the raw dict into the export payload. An import round-trip would fail because decode_auth() expects an encoded string.

# Before (bug): raw dict in export output
gateway_data["auth_value"] = auth_value

# After (fix): encode if dict, pass through if already a string
gateway_data["auth_value"] = encode_auth(auth_value) if isinstance(auth_value, dict) else auth_value

🧪 Tests

tests/unit/mcpgateway/services/test_gateway_service_helpers.py — 2 regression tests added:

test_authheaders_auth_value_stored_as_dict (creation path):

• Documents that encode_auth() returns str (why storing it in a JSON dict column produces null).
• Runs a full register_gateway() call with two custom auth headers.
• Captures values at the exact moment db.add() is called, before _prepare_gateway_for_read() mutates the object for the API response.
• Asserts DbGateway.auth_value is a plain dict matching the original headers.
• Asserts DbTool.auth_value is an encoded str that round-trips via decode_auth() back to the original headers dict.

test_update_or_create_tools_authheaders_encodes_for_dbtool (update/refresh path):

• Asserts that when the existing tool already has the correct encoded auth_value, no spurious update is triggered (comparison correctly resolves to no change).
• Asserts DbTool.auth_value remains an encoded string after the path runs.

🧪 Verification

Check	Command	Status
Unit tests	make test	✅ 268 gateway service tests pass (9/9 helper tests)
Lint suite	make lint	Pass
Coverage ≥ 90 %	make coverage	Pass
Manual regression	Register gateway with authheaders, confirm auth_value persisted as JSON object in DB, confirm health check sends correct headers	Pass

📐 MCP Compliance

• Matches current MCP spec
• No breaking change to MCP clients

✅ Checklist

• Code formatted (make black isort pre-commit)
• No secrets/credentials committed

[crivetimihai]

Thanks @shoummu1. Excellent root cause analysis and thorough fix — the type mismatch between DbGateway.auth_value (JSON column) and encode_auth() returning a string is a real data loss bug, and you've correctly traced all downstream code paths where the mismatch propagated. The regression tests are solid. Two items: (1) CI checks are failing — please investigate, (2) the title has a minor typo: missing space in authheaders``auth_value`` — should be authheaders auth_value`.

[crivetimihai]

Review & Changes

Thanks @shoummu1 — excellent root cause analysis and thorough fix. I've rebased onto main, reviewed all code paths, tested end-to-end (curl + Playwright against a live deployment), and added a few commits on top:

Commits added

1. c92c4ba — fix: isort import order and add authheaders dict export tests

   • Fixed isort violation in export_service.py (the encode_auth import was placed between mcpgateway.db imports)
   • Applied black formatting to multi-line assert statements
   • Added 2 regression tests covering the dict→encode_auth branch in both export gateway paths (_export_gateways batch-fetched + _export_selected_gateways)

2. 0835c9f — fix: compare plaintext auth values to avoid spurious tool updates

   • Critical fix: encode_auth() uses os.urandom(12) for AES-GCM nonce, so each call produces different ciphertext for identical plaintext. The original comparison at _update_or_create_tools():4142 used encoded values, which would ALWAYS differ — triggering spurious full-tool overwrites on every health-check refresh cycle (the exact bug the PR description says it fixes)
   • Fix: decode existing tool auth_value and compare against plaintext gateway dict. Only encode on the actual write path when auth content truly changed
   • Strengthened the test to assert byte-for-byte identity of existing.auth_value (no spurious re-encryption)

3. 7c29197 — test: assert encode_auth is called in hydration path

   • The hydration test used integration_type="ABC" which exits at "Invalid tool type" before reaching decode_auth. Added a spy on encode_auth to verify the dict→string conversion actually happens during cache-miss hydration

Pre-existing bugs found (not in this PR's scope)

Filed two related issues discovered during testing:

• [BUG][API]: admin_test_gateway crashes with ValueError on decode_auth of masked auth_value #3539 — admin_test_gateway crashes with ValueError on decode_auth of masked auth_value (affects all gateways, pre-existing)
• [BUG][API]: _prepare_gateway_for_read mutates ORM object, causing auth_value dict→string writeback #3540 — _prepare_gateway_for_read mutates ORM object in active session, causing dict→string writeback (6 call sites still use the deprecated method instead of safe convert_gateway_to_read)

Testing performed

• All 279 unit tests pass
• Manual curl testing: register with authheaders → auth_value persists (not null) → update → tool invocation via MCP → export round-trip
• Playwright admin UI testing: gateway list, detail view, console logs clean
• Container logs: no auth-related errors

[crivetimihai]

LGTM — rebased, fixed the spurious update comparison bug (random nonce), added missing export tests, and verified end-to-end. All 279 tests pass.