docs: sync audit redaction status and framing with current code

[imran-siddique]

Summary

Syncs documentation with current code to address doc/code inconsistencies identified in external review.

Changes (6 files)

SOC2 mapping (docs/compliance/soc2-mapping.md):

• Updated executive summary, coverage table, confidentiality gaps, privacy gaps, and critical gaps summary to reflect that CredentialRedactor now redacts credential-like secrets (API keys, tokens, JWTs, connection strings, PEM keys, etc.) before audit persistence
• Remaining gap accurately scoped: non-credential PII (email, phone, addresses) is not yet redacted in audit entries
• Replaced stale line-number references with symbol references
• Removed incorrect claim that logger.info() exposes raw parameters

Framing consistency (4 files):

• README.md: kernel-level enforcement -> policy-layer enforcement
• docs/OWASP-COMPLIANCE.md: kernel level -> application layer
• docs/modern-agent-architecture-overview.md: same fix
• Product/API names (StatelessKernel, agent-os-kernel) intentionally preserved

OWASP precision (2 files):

• docs/COMPARISON.md: added footnote clarifying 10/10 means mitigation components exist per risk category
• docs/compliance/owasp-llm-top10-mapping.md: updated LLM06 row for credential redaction

Motivation

External review identified SOC2 docs still claimed audit logs store unredacted parameters, while current mcp_gateway.py applies CredentialRedactor.redact_data_structure(params) before creating AuditEntry records. This PR brings docs in line with code and tightens claims to match application-layer governance framing.

[github-actions]

Welcome to the Agent Governance Toolkit! Thanks for your first pull request.
Please ensure tests pass, code follows style (ruff check), and you have signed the CLA.
See our Contributing Guide.

[github-actions]

🤖 AI Agent: breaking-change-detector — Summary

🔍 API Compatibility Report

Summary

This pull request primarily updates documentation to align with the current state of the codebase. It does not introduce any changes to the code itself. Therefore, there are no breaking changes to the public API.

Findings

Severity	Package	Change	Impact
✅	N/A	No changes to public API	No impact on downstream users

Migration Guide

No migration steps are necessary as this PR does not modify the public API.

Conclusion

✅ No breaking changes detected. This PR only updates documentation and does not affect the functionality or compatibility of the agent-governance-toolkit package.

[github-actions]

🤖 AI Agent: security-scanner — Security Analysis of Pull Request

Security Analysis of Pull Request

This pull request primarily updates documentation to align with the current codebase. While no direct code changes are made, documentation updates can still reveal potential security risks or gaps in the implementation. Below is the security analysis based on the provided diff:

---

🔵 Finding 1: Prompt Injection Defense Bypass

Analysis:
The documentation explicitly states that AGT does not govern what agents "think" or "say," meaning it does not detect indirect prompt injection or reasoning corruption. This is a known limitation, as outlined in the newly added LIMITATIONS.md. While this is transparently disclosed, it highlights a potential attack vector where malicious inputs could manipulate agent reasoning to bypass policy enforcement indirectly.

Attack Vector:
An attacker could craft a prompt that causes the agent to generate malicious outputs or take unintended actions, even if the individual actions are allowed by policy. For example, if the agent is allowed to call read_database and send_slack_message, a prompt injection could lead to leaking sensitive data via Slack.

Recommendation:

• Implement prompt injection detection as part of the governance layer, or integrate with external tools like Azure AI Content Safety for input/output filtering.
• Add workflow-level policies to evaluate sequences of actions rather than individual actions.
• Consider adding intent declaration where agents must declare their intended goal before executing actions, allowing the policy engine to validate the overall plan.

---

🔵 Finding 2: Policy Engine Circumvention

Analysis:
The documentation clarifies that the policy engine operates at the application layer, not the kernel level. While this is a deliberate design choice, it introduces a potential risk of policy circumvention if the application layer is compromised or bypassed.

Attack Vector:
If an attacker gains control of the application layer, they could bypass the policy engine entirely, allowing unauthorized actions to be executed.

Recommendation:

• Strengthen container-level isolation for agents as recommended in the documentation (docs/ARCHITECTURE.md).
• Consider integrating OS-level syscall interception for critical actions (e.g., file deletion, network access) to provide an additional layer of enforcement.
• Regularly audit the application layer for vulnerabilities that could lead to policy circumvention.

---

🔵 Finding 3: Trust Chain Weaknesses

Analysis:
The documentation mentions cryptographic verification (Ed25519 signatures, trust score lookups, IATP handshakes) as part of the governance layer. While no specific weaknesses are identified, the performance section (docs/LIMITATIONS.md) highlights latency in distributed deployments, which could be exploited in timing-based attacks.

Attack Vector:
An attacker could exploit latency in cryptographic verification or network round-trips to perform timing-based attacks, such as replay attacks or denial-of-service (DoS) attacks targeting inter-agent communication.

Recommendation:

• Implement nonce-based replay protection for inter-agent messages.
• Use rate limiting and token bucket mechanisms to mitigate DoS attacks.
• Optimize cryptographic verification to reduce latency, especially in high-throughput environments.

---

🔵 Finding 4: Credential Exposure

Analysis:
The updated SOC2 mapping (docs/compliance/soc2-mapping.md) confirms that CredentialRedactor now redacts credential-like secrets before audit persistence. However, the documentation also states that non-credential PII (e.g., email, phone, addresses) is not yet redacted.

Attack Vector:
If non-credential PII is stored in audit logs without redaction, it could be exposed in the event of a data breach or unauthorized access to audit logs.

Recommendation:

• Extend the CredentialRedactor functionality to include non-credential PII redaction.
• Use regex-based content policies to identify and redact sensitive information in audit logs.
• Encrypt audit logs at rest and enforce strict access controls.

---

🔵 Finding 5: Sandbox Escape

Analysis:
The documentation recommends running each agent in a separate container for OS-level isolation. While this is a best practice, no specific mechanisms are mentioned to detect or prevent sandbox escapes.

Attack Vector:
An attacker could exploit vulnerabilities in the container runtime or misconfigured permissions to escape the sandbox and gain access to the host system.

Recommendation:

• Use hardened container runtimes (e.g., gVisor, Kata Containers) to reduce the risk of sandbox escapes.
• Regularly scan container images for vulnerabilities using tools like Trivy or Clair.
• Implement host-level monitoring to detect suspicious activity originating from containers.

---

🔵 Finding 6: Deserialization Attacks

Analysis:
The documentation mentions YAML/JSON policy documents but does not specify whether deserialization is performed safely. This could be an attack vector if untrusted input is deserialized without validation.

Attack Vector:
An attacker could craft malicious YAML/JSON payloads to exploit deserialization vulnerabilities, potentially leading to remote code execution or data corruption.

Recommendation:

• Use safe deserialization libraries that enforce strict schema validation (e.g., PyYAML's safe_load).
• Reject or sanitize untrusted input before deserialization.
• Implement unit tests to validate deserialization behavior against malicious payloads.

---

🔵 Finding 7: Race Conditions

Analysis:
The documentation does not mention mechanisms to prevent race conditions in policy checks or concurrent trust evaluations.

Attack Vector:
An attacker could exploit race conditions to bypass policy enforcement, especially in high-concurrency environments with multiple agents.

Recommendation:

• Use locks or atomic operations to ensure thread-safe policy evaluation and trust score updates.
• Implement stress tests to identify and fix race conditions in multi-agent deployments.

---

🔵 Finding 8: Supply Chain Risks

Analysis:
The diff includes updates to dependency names (e.g., @agentmesh/sdk → @microsoft/agentmesh-sdk). While this change appears legitimate, supply chain risks such as dependency confusion or typosquatting remain a concern.

Attack Vector:
An attacker could publish a malicious package with a similar name (e.g., @microsoft-agentmesh-sdk) to exploit dependency confusion.

Recommendation:

• Use dependency pinning to ensure only trusted versions are installed.
• Regularly audit dependencies for vulnerabilities using tools like Snyk or Dependabot.
• Verify package authenticity by checking publisher details and signatures.

---

Overall Risk Assessment

This pull request primarily updates documentation and does not introduce new code vulnerabilities. However, the documented limitations highlight several areas where security could be improved. None of the findings are immediate vulnerabilities, but they represent potential risks in specific scenarios.

Findings Summary

1. Prompt injection defense bypass — 🔵 LOW
2. Policy engine circumvention — 🔵 LOW
3. Trust chain weaknesses — 🔵 LOW
4. Credential exposure — 🔵 LOW
5. Sandbox escape — 🔵 LOW
6. Deserialization attacks — 🔵 LOW
7. Race conditions — 🔵 LOW
8. Supply chain risks — 🔵 LOW

Recommendations

• Address known limitations (e.g., prompt injection, non-credential PII redaction, workflow-level policies).
• Strengthen container isolation and cryptographic verification mechanisms.
• Regularly audit dependencies and implement layered defense strategies.

This pull request is safe to merge but should be accompanied by follow-up tasks to address the identified risks.

[github-actions]

🤖 AI Agent: code-reviewer

Review Summary

This pull request primarily updates documentation to align with the current state of the codebase, particularly focusing on audit redaction, OWASP compliance, and framing consistency. While the changes are mostly documentation-related, they touch on critical areas such as security claims, policy enforcement, and audit logging. Below is a detailed review of the changes.

---

🔴 CRITICAL Issues

1. Audit Redaction Scope

   • The SOC2 mapping documentation highlights that CredentialRedactor redacts credential-like secrets (API keys, tokens, JWTs, etc.) before audit persistence. However, non-credential PII (e.g., email, phone numbers, addresses) is explicitly excluded from redaction.
   • Risk: This creates a potential privacy violation, especially in jurisdictions with strict data protection laws (e.g., GDPR, CCPA). Audit logs containing unredacted PII could be exploited if compromised.
   • Actionable Fix: Extend the CredentialRedactor functionality to include non-credential PII redaction. Update the documentation to reflect this enhancement.

2. Policy Layer Enforcement

   • The documentation changes from "kernel-level enforcement" to "policy-layer enforcement," which is more accurate given the application-layer nature of the toolkit. However, this clarification highlights that the toolkit does not provide OS-level isolation or sandboxing.
   • Risk: Without OS-level isolation, agents running in shared environments (e.g., containers) may be vulnerable to sandbox escape attacks.
   • Actionable Fix: Recommend running agents in isolated containers or VMs in the documentation. Additionally, consider integrating OS-level security mechanisms (e.g., seccomp, AppArmor, SELinux) into the toolkit.

---

🟡 WARNING: Potential Breaking Changes

1. TypeScript SDK Renaming
   • The TypeScript SDK has been renamed from @agentmesh/sdk to @microsoft/agentmesh-sdk. While this change is reflected in the documentation, it is a breaking change for users who have already integrated the previous package name.
   • Actionable Fix: Provide clear migration instructions in the documentation and release notes. Consider publishing a deprecation notice for the old package to guide users to the new one.

---

💡 Suggestions for Improvement

1. Known Limitations Documentation

   • The new docs/LIMITATIONS.md file is an excellent addition for transparency. However, it could benefit from more detailed examples and links to specific code or modules that address the mentioned gaps (e.g., linking to PromptDefenseEvaluator for prompt injection testing).
   • Suggestion: Add code snippets or references to relevant modules to make the document more actionable for developers.

2. OWASP Compliance Precision

   • The updated OWASP compliance documentation clarifies that "10/10 means mitigation components exist for each risk category, not that each risk is fully eliminated." While this is a good clarification, it would be helpful to include a table or section that explicitly lists the gaps for each OWASP category.
   • Suggestion: Add a detailed breakdown of OWASP compliance gaps and potential mitigations.

3. Audit Log Outcomes

   • The docs/LIMITATIONS.md file mentions that audit logs record attempts but not outcomes. This is a significant limitation for organizations relying on audit logs for forensic analysis.
   • Suggestion: Prioritize implementing post-action verification hooks and outcome attestation in audit logs. Document this as a roadmap item.

4. Thread Safety

   • While this PR does not directly touch on thread safety, the documentation should explicitly state whether the policy engine and audit logging components are thread-safe. This is critical for concurrent agent execution.
   • Suggestion: Add a section in the documentation that explicitly addresses thread safety and provides guidelines for using the toolkit in multi-threaded environments.

5. Backward Compatibility

   • The documentation changes reflect the current state of the codebase but do not address backward compatibility for users relying on older versions of the toolkit.
   • Suggestion: Include a compatibility matrix in the documentation to help users understand which versions of the toolkit are compatible with their existing deployments.

---

Final Notes

• The documentation updates are well-written and improve clarity, especially regarding the scope of the toolkit's capabilities and limitations.
• Addressing the critical issues and suggestions above will enhance the toolkit's security posture and usability.
• Ensure that future updates to the codebase are accompanied by corresponding updates to the LIMITATIONS.md file to maintain transparency.

Let me know if you need further assistance or clarification!

[github-actions]

✅ PR Review Summary

Check	Status	Details
🔍 Code Review	⏳ Pending	Awaiting results
🛡️ Security Scan	❌ Failed	Security Analysis of Pull Request
🔄 Breaking Changes	⏳ Pending	Awaiting results
📝 Docs Sync	⏳ Pending	Awaiting results
🧪 Test Coverage	❌ Failed	packages/agent-compliance/src/agent_compliance/__init__.py

Verdict: ❌ Changes needed — see failures above

💡 Individual agent reports are collapsed below for reference.

[github-actions]

🤖 AI Agent: docs-sync-checker — Issues Found

📝 Documentation Sync Report

Issues Found

• ⚠️ README.md — Several sections updated to reflect changes in terminology (e.g., "kernel-level enforcement" → "policy-layer enforcement") and package names (e.g., @agentmesh/sdk → @microsoft/agentmesh-sdk). Ensure all references are consistent across the README.
• ⚠️ CHANGELOG.md — No explicit entry for the terminology updates or the new "Known Limitations" document. While these are documentation changes, they represent significant clarifications and should be noted.
• ⚠️ docs/ — A new LIMITATIONS.md file was added, but its inclusion is not mentioned in the CHANGELOG.
• ⚠️ examples/ — Example code snippets in QUICKSTART.md and QUICKSTART.ja.md were updated to reflect the new TypeScript package name. Ensure all other examples in the repository are consistent with this change.

Suggestions

• 💡 Add a CHANGELOG entry summarizing the documentation updates, including the addition of LIMITATIONS.md and the terminology changes.
• 💡 Verify that all example code across the repository reflects the updated TypeScript package name (@microsoft/agentmesh-sdk).
• 💡 Ensure that the new LIMITATIONS.md document is linked in all relevant documentation sections for visibility.

Additional Notes

• The PR primarily focuses on syncing documentation with the current codebase and addressing inconsistencies. No new public APIs were introduced, so there are no issues related to missing docstrings or type hints.
• The updates to the README and other documentation files appear to align with the stated changes in the PR description.

✅ Documentation is mostly in sync, but the above issues should be addressed for full consistency.

[github-actions]

🤖 AI Agent: test-generator — `packages/agent-compliance/src/agent_compliance/__init__.py`

🧪 Test Coverage Analysis

packages/agent-compliance/src/agent_compliance/__init__.py

• ✅ Existing coverage: Basic initialization and module imports are likely covered by existing tests for the agent-compliance package.
• ❌ Missing coverage: Specific edge cases for compliance-related logic (if any) are not evident from the file name alone.
• 💡 Suggested test cases:
  1. test_compliance_policy_edge_cases — Validate compliance policy evaluation for edge cases like conflicting rules or missing data.
  2. test_compliance_policy_bypass_attempt — Simulate attempts to bypass compliance checks and ensure they are logged or blocked.

---

packages/agent-hypervisor/src/hypervisor/__init__.py

• ✅ Existing coverage: Module initialization and imports are likely covered by hypervisor-related tests.
• ❌ Missing coverage: Specific hypervisor functionality (e.g., execution rings, isolation mechanisms) may lack edge case testing.
• 💡 Suggested test cases:
  1. test_execution_ring_privilege_escalation — Simulate an agent attempting to escalate privileges across execution rings.
  2. test_hypervisor_concurrent_access — Test for race conditions when multiple agents interact with the hypervisor simultaneously.

---

packages/agent-mesh/src/agentmesh/__init__.py

• ✅ Existing coverage: Basic initialization and imports are likely covered by agent mesh tests.
• ❌ Missing coverage: Specific mesh-related functionality (e.g., trust scoring, inter-agent communication) may lack edge case testing.
• 💡 Suggested test cases:
  1. test_mesh_trust_score_expiration — Validate behavior when trust scores expire or are revoked.
  2. test_mesh_handshake_timeout — Simulate a timeout during the initial inter-agent handshake.

---

packages/agent-mesh/src/agentmesh/cli/main.py

• ✅ Existing coverage: CLI initialization and basic commands are likely covered by existing tests.
• ❌ Missing coverage: Edge cases for CLI commands, such as malformed inputs or invalid configurations.
• 💡 Suggested test cases:
  1. test_cli_invalid_command — Ensure the CLI gracefully handles unknown or malformed commands.
  2. test_cli_policy_load_failure — Simulate a failure when loading a policy file and verify error handling.

---

packages/agent-os/src/agent_os/__init__.py

• ✅ Existing coverage: Basic initialization and imports are likely covered by agent OS tests.
• ❌ Missing coverage: Specific agent OS functionality (e.g., policy enforcement, audit logging) may lack edge case testing.
• 💡 Suggested test cases:
  1. test_policy_enforcement_conflicting_rules — Test how the policy engine handles conflicting rules.
  2. test_audit_log_injection_attempt — Validate that audit logs are secure against injection attacks.

---

packages/agent-os/src/agent_os/_mcp_metrics.py

• ✅ Existing coverage: Metrics collection and basic functionality are likely covered by existing tests.
• ❌ Missing coverage: Edge cases for metrics reporting, such as high-frequency updates or invalid metric names.
• 💡 Suggested test cases:
  1. test_high_frequency_metric_updates — Ensure the system handles rapid metric updates without performance degradation.
  2. test_invalid_metric_name — Validate behavior when an invalid or malformed metric name is reported.

---

packages/agent-os/src/agent_os/escalation.py

• ✅ Existing coverage: Basic escalation logic is likely covered by existing tests.
• ❌ Missing coverage: Edge cases for escalation scenarios, such as circular escalation paths or invalid escalation requests.
• 💡 Suggested test cases:
  1. test_circular_escalation_path — Simulate a circular escalation path and ensure it is detected and handled.
  2. test_invalid_escalation_request — Validate behavior when an invalid escalation request is made.

---

packages/agent-os/src/agent_os/lite.py

• ✅ Existing coverage: Basic functionality for the "lite" version of agent OS is likely covered.
• ❌ Missing coverage: Edge cases for lightweight deployments, such as limited resources or missing dependencies.
• 💡 Suggested test cases:
  1. test_lite_mode_low_memory — Simulate running in a low-memory environment and validate behavior.
  2. test_lite_mode_missing_dependency — Ensure the system handles missing dependencies gracefully.

---

packages/agent-os/src/agent_os/policies/data_classification.py

• ✅ Existing coverage: Basic policy evaluation for data classification is likely covered.
• ❌ Missing coverage: Edge cases for data classification, such as ambiguous classifications or conflicting rules.
• 💡 Suggested test cases:
  1. test_ambiguous_data_classification — Validate behavior when data matches multiple classification rules.
  2. test_data_classification_policy_bypass — Simulate attempts to bypass data classification policies.

---

packages/agent-os/src/agent_os/reversibility.py

• ✅ Existing coverage: Basic functionality for reversibility is likely covered.
• ❌ Missing coverage: Edge cases for action reversibility, such as partial reversals or irreversible actions.
• 💡 Suggested test cases:
  1. test_partial_action_reversal — Validate behavior when only part of an action can be reversed.
  2. test_irreversible_action — Ensure the system correctly identifies and logs irreversible actions.

---

packages/agent-os/src/agent_os/server/__main__.py

• ✅ Existing coverage: Server initialization and basic functionality are likely covered.
• ❌ Missing coverage: Edge cases for server operation, such as malformed requests or high-concurrency scenarios.
• 💡 Suggested test cases:
  1. test_server_malformed_request — Validate behavior when the server receives a malformed request.
  2. test_server_high_concurrency — Simulate high-concurrency scenarios and ensure stability.

---

packages/agent-sre/src/agent_sre/__init__.py

• ✅ Existing coverage: Basic initialization and imports are likely covered by SRE-related tests.
• ❌ Missing coverage: Specific SRE functionality, such as chaos experiments or error budget tracking, may lack edge case testing.
• 💡 Suggested test cases:
  1. test_chaos_experiment_partial_failure — Simulate a partial failure during a chaos experiment and validate behavior.
  2. test_error_budget_exceeded — Ensure the system correctly handles scenarios where the error budget is exceeded.

---

Summary

The existing test coverage appears to handle basic functionality across modules, but there are significant gaps in edge case testing, particularly for policy evaluation, trust scoring, chaos experiments, concurrency, and input validation. The suggested test cases aim to address these gaps and improve overall robustness.