Skip to content

fix(observability): resolve A2A and plugin post-invoke silent-success traps#4238

Open
bogdanmariusc10 wants to merge 2 commits intomainfrom
4211-bug-silent-success-metric-traps-in-a2a-200-ok-path-and-plugin-tool_post_invoke-mutation
Open

fix(observability): resolve A2A and plugin post-invoke silent-success traps#4238
bogdanmariusc10 wants to merge 2 commits intomainfrom
4211-bug-silent-success-metric-traps-in-a2a-200-ok-path-and-plugin-tool_post_invoke-mutation

Conversation

@bogdanmariusc10
Copy link
Copy Markdown
Collaborator

🔗 Related Issue

Closes #4211


📝 Summary

Fixes two pre-existing silent-success traps that caused metrics_buffer.record_tool_metric(success=True) to be recorded for responses the caller sees as errors, inflating tool success rates and masking real failures in observability.

Trap 1 - A2A 200-OK branch ignores JSONRPC error envelopes:
The A2A integration path unconditionally set is_error=False and success=True for all 200-OK responses, regardless of whether the response body contained a JSONRPC {"error": {...}} envelope or application-level error indicators like {"isError": true}.

Trap 2 - Plugin TOOL_POST_INVOKE hook silently flips error state:
The plugin post-invoke reconstruction dropped is_error entirely (Pydantic defaults to False), and the success flag computed before plugin invocation was never re-synced with the plugin-modified state. This caused validation plugins that flip success→error to have their intent erased, and recovery plugins that flip error→success to still record success=False.

Both fixes are symmetric with the REST and direct-proxy paths corrected in #4202.


🏷️ Type of Change

  • Bug fix
  • Feature / Enhancement
  • Documentation
  • Refactor
  • Chore (deps, CI, tooling)
  • Other (describe below)

🧪 Verification

Check Command Status
Lint suite make lint ✅ Pass
Unit tests make test ✅ Pass
Coverage ≥ 80% make coverage ✅ Pass

✅ Checklist

  • Code formatted (make black isort pre-commit)
  • Tests added/updated for changes
  • Documentation updated (if applicable)
  • No secrets or credentials committed

📓 Notes

Implementation Details

Trap 1 Fix (A2A) - mcpgateway/services/tool_service.py lines 5536-5559:

  • Added detection for JSONRPC error envelopes ({"error": {...}})
  • Added detection for application-level error indicators ({"isError": true} or {"is_error": true})
  • Route A2A responses through _coerce_to_tool_result for uniform handling
  • Compute success = not tool_result.is_error after coercion
  • Explicitly set success = False in non-200 branch (no scope fall-through)

Trap 2 Fix (Plugin Post-Invoke) - mcpgateway/services/tool_service.py lines 5584-5599:

  • Preserve is_error during ToolResult reconstruction from plugin payload
  • Check both camelCase (isError) and snake_case (is_error) aliases
  • Re-sync success = not tool_result.is_error after plugin post-invoke completes
  • Ensures metrics reflect plugin intent (recovery plugins show success=True, validation plugins show success=False)

Regression Tests Added

Five new tests in tests/unit/mcpgateway/services/test_tool_service_coverage.py (lines 7231-7820):

A2A Error Handling:

  1. test_a2a_jsonrpc_error_envelope_records_success_false_in_metrics - JSONRPC error envelope detection
  2. test_a2a_application_level_is_error_true_records_success_false - Application-level error indicator detection
  3. test_a2a_non_200_response_explicitly_sets_success_false - Non-200 HTTP status handling

Plugin Post-Invoke Error State:
4. test_plugin_post_invoke_flips_error_to_success_records_success_true - Recovery plugin pattern
5. test_plugin_post_invoke_flips_success_to_error_records_success_false - Validation plugin pattern

Impact

  • Metrics Accuracy: Tool and server success rates will now correctly reflect actual failures
  • Observability: Real failures will no longer be masked in monitoring dashboards
  • Plugin Behavior: Plugin intent (recovery/validation) is now properly reflected in metrics
  • Consistency: A2A path now symmetric with REST and MCP direct-proxy paths fixed in [BUG]: MCP ToolError responses fail output schema validation #4202

Related PRs

Two pre-existing bugs caused metrics_buffer.record_tool_metric(success=True)
to be recorded for responses the caller sees as errors, inflating tool success
rates and masking real failures in observability.

Trap 1 - A2A 200-OK branch ignores JSONRPC error envelopes:
- Added detection for JSONRPC error envelopes ({"error": {...}})
- Added detection for application-level error indicators ({"isError": true})
- Route A2A responses through _coerce_to_tool_result for uniform handling
- Compute success = not tool_result.is_error after coercion
- Explicitly set success = False in non-200 branch (no scope fall-through)

Trap 2 - Plugin TOOL_POST_INVOKE hook silently flips error state:
- Preserve is_error during ToolResult reconstruction from plugin payload
- Check both camelCase (isError) and snake_case (is_error) aliases
- Re-sync success = not tool_result.is_error after plugin post-invoke
- Ensures metrics reflect plugin intent (recovery/validation plugins)

Regression tests added:
- test_a2a_jsonrpc_error_envelope_records_success_false_in_metrics
- test_a2a_application_level_is_error_true_records_success_false
- test_a2a_non_200_response_explicitly_sets_success_false
- test_plugin_post_invoke_flips_error_to_success_records_success_true
- test_plugin_post_invoke_flips_success_to_error_records_success_false

Fixes symmetric with REST and direct-proxy paths fixed in #4202.
Natural follow-ups to #4202, addressing A2A and plugin-hook paths.

Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
@bogdanmariusc10 bogdanmariusc10 added this to the Release 1.0.0 milestone Apr 16, 2026
@bogdanmariusc10 bogdanmariusc10 added the observability Observability, logging, monitoring label Apr 16, 2026
@bogdanmariusc10 bogdanmariusc10 added the SHOULD P2: Important but not vital; high-value items that are not crucial for the immediate release label Apr 16, 2026
Signed-off-by: Bogdan-Marius-Catanus <bogdan-marius.catanus@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

observability Observability, logging, monitoring SHOULD P2: Important but not vital; high-value items that are not crucial for the immediate release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Silent success-metric traps in A2A 200-OK path and plugin TOOL_POST_INVOKE mutation

1 participant