feat: add MCP server allowlist/blocklist and plugin trust tiers (#425, #426)

[imran-siddique]

Description

Two marketplace governance features:

MCP Server Allowlist/Blocklist (#425)

Marketplace operators can restrict which MCP servers plugins are allowed to use.

• \MarketplacePolicy\ YAML config with allowlist/blocklist mode
• Enforcement at plugin registration time (rejects non-compliant plugins)
• CLI: \�gent-marketplace evaluate --marketplace-policy --manifest \
• 35 tests

Plugin Trust Tiers (#426)

Progressive trust levels for plugins based on agentmesh trust scoring.

Tier	Score	Token Budget	Tool Calls	Access
Revoked	0-299	0	0	none
Probationary	300-499	2,000	5	read-only
Standard	500-699	8,000	20	read-write
Trusted	700-899	32,000	100	full
Verified	900-1000	100,000	500	full

• File-backed \PluginTrustStore\ with event recording
• CLI: \�gent-marketplace trust \
• 43 tests

Total: 135 tests passing

Closes #425, Closes #426

[github-actions]

🤖 AI Agent: breaking-change-detector — Summary

🔍 API Compatibility Report

Summary

This pull request introduces new features to the microsoft/agent-governance-toolkit repository, including MCP Server Allowlist/Blocklist and Plugin Trust Tiers. After analyzing the diff, no breaking changes were detected. The changes are primarily additive, introducing new classes, methods, and CLI commands.

Findings

Severity	Package	Change	Impact
🔵	agent-marketplace	Added MCPServerPolicy, MarketplacePolicy, and ComplianceResult classes	New public API, not breaking
🔵	agent-marketplace	Added evaluate_plugin_compliance function	New public API, not breaking
🔵	agent-marketplace	Added PluginTrustStore, get_trust_tier, get_tier_config, and related functions	New public API, not breaking
🔵	agent-marketplace	Added CLI commands evaluate and trust	New CLI commands, not breaking
🔵	agent-marketplace	Extended PluginRegistry constructor with optional marketplace_policy parameter	New optional parameter, not breaking
🔵	agent-marketplace	Modified PluginRegistry.register to accept mcp_servers parameter	New optional parameter, not breaking

Migration Guide

No migration steps are required as no breaking changes were introduced.

Notes

• The new APIs and CLI commands should be documented to ensure downstream users can leverage the new features effectively.
• The changes are well-contained and backward-compatible, ensuring smooth integration for existing users.

✅ No breaking changes detected.

[github-actions]

🤖 AI Agent: docs-sync-checker — Issues Found

📝 Documentation Sync Report

Issues Found

• ❌ evaluate_plugin(manifest_path: str, marketplace_policy: str) -> None in agent_marketplace/cli_commands.py — missing docstring.

• ❌ trust_plugin(plugin_name: str, store_path: str) -> None in agent_marketplace/cli_commands.py — missing docstring.

• ❌ PluginRegistry.register(manifest: PluginManifest, mcp_servers: list[str] | None = None) -> None in agent_marketplace/registry.py — missing docstring for updated parameters.

• ❌ PluginTrustStore.get_score(plugin_name: str) -> int in agent_marketplace/trust_tiers.py — missing docstring.

• ❌ PluginTrustStore.set_score(plugin_name: str, score: int) -> None in agent_marketplace/trust_tiers.py — missing docstring.

• ❌ PluginTrustStore.record_event(plugin_name: str, event: str, delta: int) -> None in agent_marketplace/trust_tiers.py — missing docstring.

• ❌ PluginTrustStore.get_tier(plugin_name: str) -> str in agent_marketplace/trust_tiers.py — missing docstring.

• ⚠️ packages/agent-marketplace/README.md — does not mention the new CLI commands evaluate and trust, nor does it explain the new MCP server allowlist/blocklist or plugin trust tiers.

• ⚠️ CHANGELOG.md — no entry for the addition of MCP server allowlist/blocklist or plugin trust tiers.

Suggestions

• 💡 Add docstring for evaluate_plugin(manifest_path: str, marketplace_policy: str) -> None explaining its purpose, parameters, and behavior.
• 💡 Add docstring for trust_plugin(plugin_name: str, store_path: str) -> None explaining its purpose, parameters, and behavior.
• 💡 Update the docstring for PluginRegistry.register(manifest: PluginManifest, mcp_servers: list[str] | None = None) -> None to include the new mcp_servers parameter and its role in marketplace policy enforcement.
• 💡 Add docstrings for the following methods in PluginTrustStore:
  • get_score(plugin_name: str) -> int
  • set_score(plugin_name: str, score: int) -> None
  • record_event(plugin_name: str, event: str, delta: int) -> None
  • get_tier(plugin_name: str) -> str
• 💡 Update packages/agent-marketplace/README.md to:
  • Document the new CLI commands evaluate and trust.
  • Explain the MCP server allowlist/blocklist feature and how to configure it.
  • Describe the plugin trust tiers and their implications.
• 💡 Add a new entry in CHANGELOG.md summarizing the addition of:
  • MCP server allowlist/blocklist.
  • Plugin trust tiers.
  • New CLI commands (evaluate, trust).

Type Hints

All new public APIs appear to have complete type annotations. ✅

Example Code

No updates to examples/ were found in the diff. If example code exists for the agent-marketplace package, ensure it reflects the new CLI commands and features.

---

Please address the above issues and suggestions to ensure documentation remains in sync with the codebase. Let me know if you need further clarification!

[github-actions]

🤖 AI Agent: test-generator — `packages/agent-marketplace/src/agent_marketplace/__init__.py`

🧪 Test Coverage Analysis

packages/agent-marketplace/src/agent_marketplace/__init__.py

• ✅ Existing coverage: This file primarily imports and exposes functions and classes from other modules. Coverage depends on tests for the imported modules.
• ❌ Missing coverage: No direct functionality to test in this file.
• 💡 Suggested test cases: None specific to this file; ensure comprehensive testing of imported modules.

---

packages/agent-marketplace/src/agent_marketplace/cli_commands.py

• ✅ Existing coverage:
  • CLI commands for publishing plugins (publish_plugin) are covered.
  • CLI argument parsing and basic error handling are tested.
• ❌ Missing coverage:
  • New evaluate_plugin command:
    • Validation of marketplace-policy file format and existence.
    • Extraction of MCP servers from plugin manifests.
    • Handling of compliance results (compliant vs. non-compliant plugins).
  • New trust_plugin command:
    • Correct retrieval of trust scores and tiers.
    • Proper rendering of trust tier properties in the CLI output.
• 💡 Suggested test cases:
  1. test_evaluate_plugin_compliance — Validate that compliant plugins pass and non-compliant plugins fail with appropriate violations.
  2. test_evaluate_plugin_invalid_policy — Test behavior when the marketplace policy file is malformed or missing.
  3. test_trust_plugin_tier_display — Ensure trust tier properties (e.g., token budget, tool calls) are displayed correctly for different tiers.
  4. test_trust_plugin_missing_plugin — Verify behavior when querying trust information for a non-existent plugin.

---

packages/agent-marketplace/src/agent_marketplace/marketplace_policy.py

• ✅ Existing coverage:
  • Policy model validation (e.g., mode validation in MCPServerPolicy).
  • Loading marketplace policies from YAML files.
• ❌ Missing coverage:
  • Edge cases for evaluate_plugin_compliance:
    • Plugins with no MCP server declaration when require_declaration is enabled.
    • MCP servers violating allowlist/blocklist policies.
    • Plugins with unsupported types or missing signatures.
    • Combination of multiple violations.
• 💡 Suggested test cases:
  1. test_policy_allowlist_violation — Test plugins using MCP servers not in the allowlist.
  2. test_policy_blocklist_violation — Test plugins using MCP servers explicitly blocked.
  3. test_policy_require_declaration_violation — Test plugins failing to declare MCP servers when required.
  4. test_policy_multiple_violations — Validate behavior when a plugin violates multiple policy rules simultaneously.
  5. test_policy_signature_requirement — Ensure unsigned plugins are rejected when require_signature is enabled.

---

packages/agent-marketplace/src/agent_marketplace/registry.py

• ✅ Existing coverage:
  • Basic plugin registration and storage functionality.
• ❌ Missing coverage:
  • Enforcement of marketplace policies during plugin registration:
    • Plugins violating MCP server policies.
    • Plugins violating type restrictions or signature requirements.
    • Handling of compliant vs. non-compliant plugins.
• 💡 Suggested test cases:
  1. test_registry_policy_compliance — Ensure compliant plugins are registered successfully.
  2. test_registry_policy_violation — Verify that non-compliant plugins are rejected with appropriate errors.
  3. test_registry_policy_no_policy — Test behavior when no marketplace policy is provided during registration.

---

packages/agent-marketplace/src/agent_marketplace/trust_tiers.py

• ✅ Existing coverage:
  • Trust tier definitions (TRUST_TIERS, DEFAULT_TIER_CONFIGS).
  • Basic trust score computation (compute_initial_score).
• ❌ Missing coverage:
  • Edge cases for trust scoring:
    • Scores at boundaries (e.g., 0, 299, 300, 499, etc.).
    • Plugins with missing descriptions or capabilities.
    • Plugins with invalid or expired signatures.
  • Trust store functionality:
    • Persistence of scores and events.
    • Handling of missing or corrupted trust store files.
    • Event recording and score adjustment.
  • Capability filtering:
    • Filtering capabilities based on tier requirements.
• 💡 Suggested test cases:
  1. test_compute_initial_score_boundaries — Validate initial scores for plugins at boundary conditions (e.g., no description, no capabilities, etc.).
  2. test_get_trust_tier_boundaries — Ensure correct tier assignment for scores at tier boundaries (e.g., 299 → revoked, 300 → probationary).
  3. test_filter_capabilities_by_tier — Test capability filtering for plugins at different tiers.
  4. test_trust_store_persistence — Verify that scores and events persist correctly across store reloads.
  5. test_trust_store_event_recording — Ensure trust events adjust scores correctly and are logged properly.

---

Summary of Suggested Tests

• CLI Commands: Add tests for evaluate_plugin and trust_plugin commands.
• Marketplace Policy: Focus on edge cases for policy violations and combinations.
• Registry: Ensure policy enforcement during plugin registration.
• Trust Tiers: Test boundary conditions, capability filtering, and trust store persistence.

By addressing these areas, the test coverage will be significantly improved, particularly for edge cases and new functionality introduced in this pull request.

[github-actions]

🤖 AI Agent: code-reviewer

Review Summary

This pull request introduces two major features to the agent-marketplace package: an MCP Server Allowlist/Blocklist mechanism and Plugin Trust Tiers. These features are critical for enhancing the security and governance of the marketplace. The implementation appears to be well-structured, with clear documentation, comprehensive test coverage, and adherence to the existing codebase's style and conventions. However, there are a few areas that require attention to ensure robustness, security, and maintainability.

---

🔴 CRITICAL

1. Insufficient Validation for MCP Server Policy:

   • The MCPServerPolicy class does not validate whether the allowed and blocked lists contain valid MCP server names (e.g., valid domain names or IP addresses). This could lead to misconfigurations or potential security issues.
   • Action: Add validation logic to ensure that the allowed and blocked lists contain valid MCP server names. Consider using a regex or a domain validation library.

2. Potential Sandbox Escape via Plugin Capabilities:

   • The filter_capabilities function uses a static _TIER_CAPABILITY_REQUIREMENTS dictionary to determine which capabilities are allowed for a given tier. However, this mechanism does not account for dynamic or user-defined capabilities, which could lead to a sandbox escape if a malicious plugin declares an unexpected capability.
   • Action: Implement stricter validation for capabilities, ensuring that only predefined, explicitly allowed capabilities are processed. Consider raising an error for any unrecognized capabilities.

3. Lack of Cryptographic Signature Verification:

   • While the evaluate_plugin_compliance function checks for the presence of a signature if require_signature is enabled, it does not verify the validity of the signature. This could allow plugins with invalid or tampered signatures to bypass the policy.
   • Action: Integrate cryptographic signature verification into the compliance evaluation process. Ensure that the signature is validated against a trusted public key.

4. Trust Store Integrity:

   • The PluginTrustStore class relies on a JSON file for persistence but does not implement any integrity checks (e.g., cryptographic hashing or signing) to detect tampering with the file.
   • Action: Add integrity checks to the trust store file. For example, compute a hash of the file's contents and store it separately, or use a digital signature to verify the file's authenticity.

---

🟡 WARNING

1. Breaking Change in PluginRegistry.register:

   • The register method in PluginRegistry now requires an additional mcp_servers argument. This is a breaking change for any existing code that calls this method without the new argument.
   • Action: Make the mcp_servers argument optional by providing a default value of None. Update the method's docstring to reflect this change.

2. Backward Compatibility for Trust Store:

   • The PluginTrustStore class introduces a new JSON-based storage format. If this replaces an existing storage mechanism, it could break compatibility with older versions of the library.
   • Action: Provide a migration path or a compatibility layer for users upgrading from older versions of the library.

---

💡 SUGGESTIONS

1. Improved Error Messages:

   • The error messages in evaluate_plugin_compliance and other functions are clear but could include more actionable details. For example, when a plugin violates the MCP server policy, the error message could suggest how to fix the issue (e.g., "Add the following MCP servers to the allowlist: ...").
   • Action: Enhance error messages to provide actionable guidance for resolving policy violations.

2. Thread Safety in PluginTrustStore:

   • The PluginTrustStore class is not thread-safe, which could lead to race conditions if multiple threads attempt to read/write to the trust store simultaneously.
   • Action: Add thread safety mechanisms, such as file locks or a threading lock, to ensure safe concurrent access to the trust store.

3. Unit Test Coverage for Edge Cases:

   • While the PR mentions 135 tests, it is unclear if edge cases (e.g., malformed MCP server names, invalid YAML policies, etc.) are adequately covered.
   • Action: Review the test suite to ensure that edge cases are tested, particularly for the load_marketplace_policy, evaluate_plugin_compliance, and PluginTrustStore methods.

4. Dynamic Configuration for Trust Tiers:

   • The trust tier configurations are currently hardcoded in DEFAULT_TIER_CONFIGS. This could limit flexibility for marketplace operators who may want to customize these settings.
   • Action: Allow the trust tier configurations to be loaded from an external configuration file or database, with the hardcoded values serving as defaults.

5. Logging Improvements:

   • The logging in PluginTrustStore.record_event is useful but could be extended to include more context, such as the plugin's current tier and the reason for the score adjustment.
   • Action: Enhance logging to include additional context, such as the plugin's current tier and the reason for the score adjustment.

6. Documentation:

   • While the docstrings are generally well-written, there is no high-level documentation explaining how to use the new features.
   • Action: Update the project's documentation to include a detailed guide on configuring and using the MCP Server Allowlist/Blocklist and Plugin Trust Tiers features.

7. Performance Considerations:

   • The evaluate_plugin_compliance function performs multiple list operations (e.g., not in, append, etc.) that could become a bottleneck for large policies or manifests with many MCP servers.
   • Action: Optimize the function by using sets for membership checks and other operations that can benefit from constant-time lookups.

---

Summary of Actionable Feedback

Critical Issues

• Add validation for MCP server names in MCPServerPolicy.
• Implement stricter validation for plugin capabilities to prevent sandbox escapes.
• Add cryptographic signature verification in evaluate_plugin_compliance.
• Implement integrity checks for the PluginTrustStore file.

Warnings

• Make mcp_servers argument in PluginRegistry.register optional to avoid breaking changes.
• Provide a migration path for the new trust store format if it replaces an existing mechanism.

Suggestions

• Improve error messages to provide actionable guidance.
• Add thread safety mechanisms to PluginTrustStore.
• Ensure edge cases are covered in the test suite.
• Allow dynamic configuration of trust tier settings.
• Enhance logging for better debugging and auditability.
• Optimize evaluate_plugin_compliance for better performance with large datasets.

---

Overall Assessment

The PR introduces valuable features that enhance the security and governance capabilities of the agent-marketplace package. However, addressing the critical issues and warnings is essential to ensure the robustness and backward compatibility of the implementation. The suggestions, while not mandatory, would further improve the quality and usability of the code.

[github-actions]

🤖 AI Agent: security-scanner — Security Review for PR #425 and #426

Security Review for PR #425 and #426

This pull request introduces two new features to the agent-marketplace package: MCP Server Allowlist/Blocklist and Plugin Trust Tiers. Both features are critical for enforcing marketplace governance and plugin security. Below is a detailed security review based on the provided diff.

---

Findings

1. Prompt Injection Defense Bypass

• Risk: 🔵 LOW
• Details: The changes do not directly involve user-facing prompts or natural language processing components. However, the evaluate_plugin_compliance function checks plugin manifests for compliance with policies. If a malicious actor can manipulate the manifest content (e.g., by injecting unexpected or malformed data), it could potentially bypass the compliance checks.
• Recommendation: Add strict validation for all inputs, especially the manifest and mcp_servers parameters in evaluate_plugin_compliance. Ensure that all fields in the manifest are sanitized and validated against expected formats and values.

---

2. Policy Engine Circumvention

• Risk: 🟠 HIGH
• Details: The evaluate_plugin_compliance function enforces the marketplace policy, but there are potential gaps:
  • The require_declaration flag in MCPServerPolicy only checks if mcp_servers is None. If an attacker provides an empty list, the check will pass, potentially bypassing the requirement.
  • The allowed_plugin_types check only verifies the plugin_type.value field in the manifest. If the manifest is tampered with or malformed, this check could be bypassed.
• Recommendation:
  • Update the require_declaration check to ensure that mcp_servers is not empty when require_declaration is True.
  • Add stricter validation for the plugin_type field in the manifest to ensure it matches the expected schema and values.

---

3. Trust Chain Weaknesses

• Risk: 🟠 HIGH
• Details: The require_signature policy mandates Ed25519 signatures for plugins. However:
  • There is no validation of the signature's authenticity or its association with a trusted key.
  • The evaluate_plugin_compliance function only checks for the presence of a signature but does not verify its validity.
• Recommendation:
  • Implement signature verification using a trusted key or certificate authority.
  • Ensure that the signature verification process is robust and resistant to tampering or replay attacks.

---

4. Credential Exposure

• Risk: 🔵 LOW
• Details: There is no evidence of sensitive credentials being logged or exposed in the provided diff. However, the PluginTrustStore logs trust events, including plugin names and score changes.
• Recommendation: Ensure that logging is appropriately secured and does not expose sensitive information. Consider adding a configuration option to disable logging in production environments.

---

5. Sandbox Escape

• Risk: 🔵 LOW
• Details: The changes do not introduce any new functionality that interacts with the underlying system or executes untrusted code. No evidence of sandbox escape vulnerabilities was found.
• Recommendation: None.

---

6. Deserialization Attacks

• Risk: 🟠 HIGH
• Details: The load_marketplace_policy function uses yaml.safe_load to parse YAML files, which is a safer alternative to yaml.load. However, there is no validation of the parsed data beyond checking if it is a dictionary. This could allow an attacker to craft a malicious YAML file that bypasses policy enforcement.
• Recommendation:
  • Perform strict schema validation on the parsed YAML data before instantiating the MarketplacePolicy object.
  • Consider using a library like Cerberus or jsonschema for schema validation.

---

7. Race Conditions

• Risk: 🟡 MEDIUM
• Details: The PluginTrustStore class uses a file-backed JSON store for persistence. Concurrent access to the same file could lead to race conditions, resulting in data corruption or loss.
• Recommendation:
  • Implement file locking to prevent concurrent writes to the trust store.
  • Consider using a thread-safe database (e.g., SQLite) instead of a JSON file for persistence.

---

8. Supply Chain Risks

• Risk: 🟠 HIGH
• Details:
  • The yaml library is used for loading marketplace policy files. If an attacker can replace the yaml library with a malicious package (e.g., through dependency confusion or typosquatting), it could lead to code execution.
  • The pydantic library is used for data validation. If this library is compromised, it could introduce vulnerabilities.
• Recommendation:
  • Use dependency pinning to specific versions in your requirements.txt or pyproject.toml file.
  • Regularly audit dependencies for known vulnerabilities using tools like Dependabot or Snyk.

---

Summary of Findings

Category	Risk	Details	Recommendation
Prompt Injection Defense	🔵 LOW	Potential for crafted manifest input to bypass compliance checks.	Add strict validation for all manifest fields and inputs.
Policy Engine Circumvention	🟠 HIGH	Weak checks for require_declaration and plugin_type.	Strengthen validation logic for these fields.
Trust Chain Weaknesses	🟠 HIGH	Signature presence is checked, but not its validity or authenticity.	Implement robust signature verification using trusted keys or certificates.
Credential Exposure	🔵 LOW	No sensitive data exposure, but logging of trust events could be sensitive.	Ensure secure logging practices.
Sandbox Escape	🔵 LOW	No evidence of sandbox escape vulnerabilities.	None.
Deserialization Attacks	🟠 HIGH	Insufficient validation of YAML data in load_marketplace_policy.	Add strict schema validation for parsed YAML data.
Race Conditions	🟡 MEDIUM	Potential race conditions in PluginTrustStore file-backed storage.	Implement file locking or use a thread-safe database for persistence.
Supply Chain	🟠 HIGH	Dependency on yaml and pydantic without version pinning.	Pin dependencies and regularly audit for vulnerabilities.

---

Overall Risk Assessment: 🟠 HIGH

The proposed changes introduce critical governance features, but there are several areas where security can be improved to ensure robust protection against potential attacks. Addressing the identified issues will significantly enhance the security posture of the agent-marketplace package.

Let me know if you'd like me to review specific sections in more detail or provide additional recommendations.