[TESTING]: Plugin runtime management — global enable/disable, RateLimiterPlugin dynamic config, multi-instance validation

[gandhipratik203]

Summary

The dynamic plugin configuration and per-tool bindings added in #4068 / #4143 are the supported way to manage plugins at runtime. This issue covers two pillars of plugin runtime management:

1. Global plugin enable/disable — toggle any plugin on/off at runtime via the API, verify it takes effect across all gateway instances
2. RateLimiterPlugin dynamic configuration — change rate limiter settings at runtime (limits, mode, scope), verify the new config takes effect across all instances

Currently there are no tests that verify either capability works correctly across multiple gateway instances. In a 3-pod deployment, a change made via the API on pod A must be visible to pods B and C — and we don't currently test or guarantee that.

This issue covers auditing the propagation mechanism, writing tests that expose gaps, and fixing any bugs found.

Context

• The enable_plugins() function in mcpgateway/plugins/framework/__init__.py is a per-process in-memory toggle — it does not propagate across pods
• The plugin config loader (mcpgateway/plugins/framework/loader/config.py) supports Jinja2 env var resolution but reads from a static YAML file at startup
• Tool plugin bindings (/v1/tools/plugin_bindings) persist to Postgres — but it's unclear whether the plugin manager reads these per-request or caches them in-memory
• Rate limiter uses Redis as shared state for rate counting (works cross-pod), but the plugin enable/disable and config state may not be shared

Related: #4222 (SQL Sanitizer e2e tests), PII Filter e2e tests — both depend on the plugin runtime management infrastructure being reliable across instances.

Approach

Test-driven hardening — write the tests first, let failures expose bugs, fix the bugs, tests become the proof.

Deliverables

1. Audit — map the state flow

Trace the plugin runtime management path end-to-end:

• API call → DB write → plugin manager reload → request pipeline
• Identify where state is per-process (in-memory) vs shared (DB/Redis)
• Document the cache TTLs and invalidation mechanisms (if any)
• Specifically: when a binding is created/updated via /v1/tools/plugin_bindings, how does each pod learn about it?

2. Integration tests (tests/integration/test_plugin_runtime_management.py)

Against a live gateway (not mocked):

Global enable/disable:

• Enable a plugin via API, verify it takes effect on subsequent requests
• Disable a plugin via API, verify it stops running
• Per-tool binding: bind plugin to tool A only, call tool A (plugin runs), call tool B (plugin doesn't)
• Binding deletion: remove binding, verify plugin stops running

RateLimiterPlugin dynamic configuration:

• Change rate limit from 30/m to 100/m via API, verify the new limit is enforced
• Switch mode from permissive to enforce via API, verify enforcement on subsequent requests
• Change by_user vs by_tenant settings, verify the correct scope is applied

Multi-instance validation:

• Make API call to one pod, verify behaviour on a different pod (the core test)
• Pod restart survival: create binding + config change, delete a pod, verify both persist after recreate

3. Load test (tests/loadtest/locustfile_plugin_runtime_management.py)

Locust file for multi-instance environments:

• Admin user class: periodically enables/disables plugins and changes rate limits via the API
• Normal user class (100+): continuously calls tools and verifies plugins are active/inactive and rate limits match the current config
• Success criteria: 0% inconsistency across all pods under concurrent load

4. Bug fixes

When tests fail, fix the underlying code. Possible fixes:

• Add DB/Redis read for dynamic bindings on each request (if currently cached in-memory only)
• Add Redis pub/sub or cache invalidation for plugin state/config changes
• Add cache TTL configuration for plugin manager state
• Ensure dynamic config changes propagate to all Gunicorn workers within a pod (not just the one handling the API call)
• Ensure fail_on_plugin_error behaves correctly with dynamically enabled plugins

Each fix ships with the test that proves it works.

5. E2E validation

Run the full test suite on a multi-instance deployment:

• tests/integration/ via pytest against the gateway
• tests/loadtest/ via Locust with multiple workers
• Document results and any remaining limitations

Priority order

1. Single-instance global enable/disable (does the API actually change runtime behaviour?)
2. Single-instance RateLimiterPlugin dynamic config (does changing settings take effect?)
3. Multi-instance propagation (does pod B see pod A's change?)
4. Persistence across pod restarts
5. Concurrency under load
6. Security (can a non-admin user toggle plugins or change rate limits?)

Environments

• Colima — single and multi-instance (docker-compose with replicas: 3) for integration test development
• OCP — optional, for validation on a real cluster if available

[gandhipratik203]

Closing summary

All five deliverables in this issue are landed, plus a class of follow-on rate-limiter-specific gaps that surfaced during the testing phase and are being addressed in separate PRs. Work was framed strictly around the test-driven hardening approach this issue itself called out — tests first, failures expose bugs, fixes ship paired with their regression tests.

---

Approach — test-driven hardening, as prescribed

This issue explicitly asked for the TDD flow in its Approach section:

"Test-driven hardening — write the tests first, let failures expose bugs, fix the bugs, tests become the proof."

That's exactly how the work was shaped. The deliverables below should be read as the output of that loop, not as separate work streams:

    ┌──────────────────────────────────────────────────────────────────┐
    │  1. AUDIT                                                         │
    │     Map state flow: API → DB → plugin manager → request          │
    │     → Identified 14 multi-instance gaps in the framework          │
    └──────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │  2. TEST HARNESS (written first, deliberately FAILS on baseline) │
    │                                                                    │
    │     Unit:         test_admin_plugin_runtime  (561 lines)          │
    │                   test_plugin_runtime_management (unit)           │
    │                   test_main_extended (124 lines)                  │
    │                                                                    │
    │     Integration:  test_plugin_runtime_management (606 lines)      │
    │                   test_plugin_runtime_redis (523 lines)           │
    │                   test_runtime_state_cluster (295 lines)          │
    │                   test_plugin_dynamic_behavior (363 lines)        │
    │                   test_rate_limiter_dynamic_behavior (295 lines)  │
    │                   test_rate_limiter_multi_tenant (225 lines)      │
    │                                                                    │
    │     Load:         locustfile_plugin_dynamic_config (519 lines)    │
    └──────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │  3. TEST FAILURES → BUG SURFACE                                   │
    │     Each failing test = one specific multi-instance gap          │
    │     Ranked by priority: single-instance → multi-instance →       │
    │     persistence → concurrency → security                          │
    └──────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │  4. BUG FIXES (each paired with its regression test from step 2) │
    │     Every fix ships with the failing test that now passes.        │
    │     14-gap audit → 14 specific fixes in #4292                    │
    └──────────────────────────────┬───────────────────────────────────┘
                                   │
                                   ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │  5. E2E VALIDATION                                                │
    │     Docker-compose 3 replicas + OCP benchmark                    │
    │     Harness + fixes delivered together in #4292 (merged)         │
    │     Follow-on #4343 gaps discovered; addressed in #40 + #4380    │
    └──────────────────────────────────────────────────────────────────┘

The value of this shape: every bug fix has a corresponding test that proves it stays fixed. There are no undocumented assumptions, no "we fixed something and we think it works." Every claim has a test pinning it.

---

Deliverable coverage

1. Audit — state flow mapped ✅

The audit traced the runtime plugin management path end-to-end: API call → DB write → plugin manager reload → request pipeline. It identified where state is per-process vs shared, documented cache TTLs and invalidation mechanisms, and specifically investigated how each pod learns about ToolPluginBinding mutations.

Audit output → 14 multi-instance gaps, each called out in #4292's commit message. Sample:

• TTL semantics for runtime-state cache vs mode-override cache
• Redis pub/sub propagation on the plugin:invalidation channel
• MGET batching for efficient reads of mode overrides
• Wildcard binding eviction semantics (evict all (team, tool) contexts on a * binding change)
• DB graceful degradation — behaviour when Postgres is temporarily unavailable
• Stale local-override pruning via monotonic clock
• app.state.plugin_manager sync so freshly-toggled nodes don't serve stale
• Import-cycle break via _state.py leaf module (runtime state was reachable from both framework and manager)
• Factory-init failure visibility — surface one ERROR on first shared-toggle request instead of silent swallow
• Admin-view GET self-heal mirrors framework.get_plugin_manager() into admin caches
• update_plugin_mode validates against configured plugin set (not live manager) so pre-stage works on disabled node
• Admin plugin-view cache sync deny-path regressions
• Redis-synced local overrides expire at the cluster's 24h TTL
• Single-node fallback to in-process map with explicit redis_persisted signalling

---

2. Integration tests ✅ — 6 files, ~2.3K lines, plus ~1.7K lines of unit tests

File	Lines	Scenarios covered
test_plugin_runtime_management.py	606	Global enable/disable; per-tool binding create/update/delete; per-team override semantics; pod-restart persistence
test_plugin_runtime_redis.py	523	Redis-backed runtime mgmt; pub/sub invalidation; 24h TTL; MGET read batching; single-node fallback
test_runtime_state_cluster.py	295	Two-coordinator convergence via in-process Redis simulator — proves pub/sub actually propagates state
test_plugin_dynamic_behavior.py	363	Generic dynamic plugin behaviour via ReplaceBadWordsPlugin; full toggle cycle; burst-mode regression pin for cross-replica invalidation
test_rate_limiter_dynamic_behavior.py	295	RateLimiterPlugin-specific — burst enforce/disabled, mode persisted in Redis, mode visible via admin API, mode reverts after disable
test_rate_limiter_multi_tenant.py	225	Multi-tenant rate limiting E2E — tenant_id flows from gateway to plugin, Redis keys carry the tenant prefix; validates against a real deployment

Plus unit tests — test_admin_plugin_runtime.py (561 lines) exercises the admin API's deny paths, configured-plugin validation, expired-override pruning, shared-provider reset, and backing-dict identity; test_plugin_runtime_management.py unit layer (1691 lines) covers the framework and manager internals.

Coverage against the issue's Priority order:

Priority	Covered by
1. Single-instance global enable/disable	test_plugin_runtime_management (integration + unit)
2. Single-instance RateLimiterPlugin dynamic config	test_rate_limiter_dynamic_behavior, test_rate_limiter_multi_tenant
3. Multi-instance propagation	test_plugin_runtime_redis, test_runtime_state_cluster, test_plugin_dynamic_behavior::TestCrossReplicaBehavior
4. Persistence across pod restarts	test_plugin_runtime_redis (TTL + Redis state survival)
5. Concurrency under load	locustfile_plugin_dynamic_config.py (see next)
6. Security — non-admin can't toggle	test_admin_plugin_runtime (deny-path tests)

---

3. Load test ✅

tests/loadtest/locustfile_plugin_dynamic_config.py — 519 lines covering the four scenarios this issue asked for:

1. Global toggle — admin user class disables/enables plugins, normal users continuously call tools; assertion that tool behaviour switches in lockstep with the admin state
2. Per-plugin mode — switch RateLimiter between enforce/disabled, verify enforcement flips
3. Per-tool binding — bind to tool A only, verify tool B unaffected
4. Per-user isolation — verify rate limits apply independently per user

Runs headless against docker-compose (3 replicas) and against OCP (validated as part of #4053's OCP benchmark flow).

Success criteria met — 0% inconsistency across all pods under concurrent load.

---

4. Bug fixes ✅ — each paired with its regression test

The 14 audit findings each got a specific fix in #4292. Representative map of fix → test pairing:

Audit finding	Fix	Regression test
Runtime state lives in-memory per process, doesn't propagate	Redis pub/sub on plugin:invalidation + leaf _state.py module	test_runtime_state_cluster (two coordinators converge via pub/sub)
Wildcard binding change must evict every (team, tool) context	invalidate_team path that sweeps all contexts sharing a team id	test_plugin_runtime_management::test_wildcard_binding_invalidation
Pods with Redis unavailable silently skip toggles	redis_persisted: false field in admin response + one-time ERROR on degraded node	test_plugin_runtime_redis single-node fallback tests
Plugins disabled at boot can't pre-stage overrides	update_plugin_mode validates against configured plugin set, not live manager	test_admin_plugin_runtime::test_configured_name_validation
Freshly-enabled node serves stale cache on next GET	Admin GET self-heal mirrors framework.get_plugin_manager() into admin caches	test_admin_plugin_runtime::test_remote_disable_self_heal
Local overrides leak indefinitely if Redis goes away mid-flight	Overrides expire at cluster's 24h TTL; durable entries remain sticky until explicit release	test_admin_plugin_runtime::test_expired_override_pruning

And ten more similar pairings in the same PR.

---

5. E2E validation ✅

• Docker-compose, 3 replicas behind NGINX — full integration suite passes; load test sustains documented throughput with 0% cross-pod inconsistency
• OCP — validated via #4053's benchmark flow (669 RPS sustained, 0% failures under 750 users)

Both environments documented with config reproducers in their respective PR bodies.

---

Follow-on work discovered during testing

While running the integration tests against the merged runtime-management infrastructure, a separate class of gaps surfaced — rate-limiter-specific, not visible from the plugin-manager audit alone:

• Filed #4343 with 10 rate-limiter gaps, severity-mapped (2 HIGH, 3 MEDIUM, 5 LOW), each with impact + fix strategy
• IBM/cpex-plugins#40 (in review, all reviewer findings addressed) — closes 8 of the 10 gaps cpex-side; reviewed by @lucarlig
• #4380 (draft, awaits MCP Gateway 0.1.0 – Initial public release #40 merge) — closes the main-repo-side G1 gap (tenant_id populated in GlobalContext), adds multi-tenant integration tests validating G1 + G2 end-to-end through a real deployment, adds a framework mode-propagation regression pin (TestBadWordsToggleBurst)

These are being tracked and delivered on their own PRs rather than expanding this issue's scope.
Follow up issue: IBM/cpex-plugins#49

---

Closing criteria

• Audit documented (14 multi-instance gaps in #4292 commit message)
• Integration tests: 6 files, 2,307 lines, covering all 6 priority-ordered scenarios
• Load test: 4 scenarios, 0% inconsistency under load
• Bug fixes: 14 audit findings, each paired with its regression test
• E2E validation: docker-compose (3 replicas) + OCP
• Follow-on scope tracked separately: #4343, #40, #4380

Closing.