feat: add event bus, task outcomes, diff policy, sandbox provider (#398, #396, #395, #394)

[imran-siddique]

4 features with 22 tests:

#398 — \GovernanceEventBus: pub/sub for cross-gate composition (policy→trust→circuit breaker)
#396 — \TaskOutcomeRecorder: severity scoring, diminishing returns, time-based recovery
#395 — \DiffPolicy: file/line count limits + path glob restrictions for agent-authored changes
#394 — \SandboxProvider: pluggable ABC + subprocess implementation + NoOp for testing

5 files, +672 lines, 22 tests.

[github-actions]

🤖 AI Agent: breaking-change-detector — Summary

🔍 API Compatibility Report

Summary

This pull request introduces several new features and modules, including GovernanceEventBus, TaskOutcomeRecorder, DiffPolicy, and SandboxProvider. The changes are primarily additive, with no evidence of removed or modified existing APIs that would cause breaking changes. All new APIs are well-defined and documented in the code.

Findings

Severity	Package	Change	Impact
🔵	agent-os	Added GovernanceEventBus	New API, not breaking
🔵	agent-os	Added TaskOutcomeRecorder	New API, not breaking
🔵	agent-os	Added DiffPolicy	New API, not breaking
🔵	agent-os	Added SandboxProvider abstraction	New API, not breaking
🔵	agent-os	Added BudgetPolicy and BudgetTracker	New API, not breaking
🔵	agent-os	Added GovernanceAuditLogger	New API, not breaking
🔵	agent-os	Added compatibility helpers (NoOpPolicyEvaluator, etc.)	New API, not breaking

Migration Guide

No migration steps are necessary as no breaking changes were identified.

Additional Notes

• The new APIs should be documented in the project's official documentation to ensure downstream users can leverage these features effectively.
• The compatibility helpers (NoOpPolicyEvaluator, etc.) provide graceful degradation, which is a thoughtful addition for optional dependencies.

✅ No breaking changes detected.

[github-actions]

🤖 AI Agent: docs-sync-checker — Issues Found

📝 Documentation Sync Report

Issues Found

1. New Public APIs Without Docstrings

   • ❌ JsonlFileBackend.write() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ JsonlFileBackend.flush() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ JsonlFileBackend.close() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ InMemoryBackend.write() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ InMemoryBackend.flush() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ LoggingBackend.write() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ LoggingBackend.flush() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ GovernanceAuditLogger.add_backend() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ GovernanceAuditLogger.log() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ GovernanceAuditLogger.flush() in agent-os/src/agent_os/audit_logger.py — missing docstring
   • ❌ GovernanceEventBus.subscribe() in agent-os/src/agent_os/event_bus.py — missing docstring
   • ❌ GovernanceEventBus.unsubscribe() in agent-os/src/agent_os/event_bus.py — missing docstring
   • ❌ GovernanceEventBus.publish() in agent-os/src/agent_os/event_bus.py — missing docstring
   • ❌ GovernanceEventBus.get_history() in agent-os/src/agent_os/event_bus.py — missing docstring
   • ❌ GovernanceEventBus.clear_history() in agent-os/src/agent_os/event_bus.py — missing docstring
   • ❌ SandboxProvider.run() in agent-os/src/agent_os/sandbox_provider.py — missing docstring
   • ❌ SandboxProvider.is_available() in agent-os/src/agent_os/sandbox_provider.py — missing docstring
   • ❌ SubprocessSandboxProvider.run() in agent-os/src/agent_os/sandbox_provider.py — missing docstring
   • ❌ SubprocessSandboxProvider.is_available() in agent-os/src/agent_os/sandbox_provider.py — missing docstring

2. README Sections Out of Date

   • ⚠️ packages/agent-os/README.md — Missing documentation for new features:
     • GovernanceEventBus
     • TaskOutcomeRecorder
     • DiffPolicy
     • SandboxProvider

3. CHANGELOG Missing Entries

   • ⚠️ CHANGELOG.md — No entry for the following new features:
     • GovernanceEventBus
     • TaskOutcomeRecorder
     • DiffPolicy
     • SandboxProvider

4. Example Code Outdated

   • ⚠️ examples/ — No examples provided for the new features:
     • GovernanceEventBus
     • TaskOutcomeRecorder
     • DiffPolicy
     • SandboxProvider

5. Type Hints

   • ✅ All new public APIs have complete type annotations.

---

Suggestions

• 💡 Add docstrings for the following methods:

  • JsonlFileBackend.write(entry: AuditEntry) -> None
  • JsonlFileBackend.flush() -> None
  • JsonlFileBackend.close() -> None
  • InMemoryBackend.write(entry: AuditEntry) -> None
  • InMemoryBackend.flush() -> None
  • LoggingBackend.write(entry: AuditEntry) -> None
  • LoggingBackend.flush() -> None
  • GovernanceAuditLogger.add_backend(backend: Any) -> None
  • GovernanceAuditLogger.log(entry: AuditEntry) -> None
  • GovernanceAuditLogger.flush() -> None
  • GovernanceEventBus.subscribe(event_type: str, handler: EventHandler) -> None
  • GovernanceEventBus.unsubscribe(event_type: str, handler: EventHandler) -> None
  • GovernanceEventBus.publish(event_type: str, source: str = "", agent_id: str = "", **data: Any) -> GovernanceEvent
  • GovernanceEventBus.get_history(event_type: str | None = None, limit: int = 100) -> list[GovernanceEvent]
  • GovernanceEventBus.clear_history() -> None
  • SandboxProvider.run(agent_id: str, command: list[str], config: SandboxConfig | None = None) -> SandboxResult
  • SandboxProvider.is_available() -> bool
  • SubprocessSandboxProvider.run(agent_id: str, command: list[str], config: SandboxConfig | None = None) -> SandboxResult
  • SubprocessSandboxProvider.is_available() -> bool

• 💡 Update packages/agent-os/README.md to include sections for:

  • GovernanceEventBus (purpose, usage, examples)
  • TaskOutcomeRecorder (purpose, usage, examples)
  • DiffPolicy (purpose, usage, examples)
  • SandboxProvider (purpose, usage, examples)

• 💡 Add entries to CHANGELOG.md for the new features:

  • GovernanceEventBus
  • TaskOutcomeRecorder
  • DiffPolicy
  • SandboxProvider

• 💡 Add example code in examples/ for:

  • GovernanceEventBus
  • TaskOutcomeRecorder
  • DiffPolicy
  • SandboxProvider

---

Summary

The PR introduces several new features but lacks sufficient documentation and examples. Additionally, the README and CHANGELOG need updates to reflect the changes. Once these issues are addressed, the documentation will be in sync.

[github-actions]

🤖 AI Agent: code-reviewer

Code Review for PR: feat: add event bus, task outcomes, diff policy, sandbox provider (#398, #396, #395, #394)

---

General Feedback

This PR introduces four significant features: GovernanceEventBus, TaskOutcomeRecorder, DiffPolicy, and SandboxProvider. Each feature is well-documented, and the code is generally clean and adheres to the repository's coding standards. However, there are several areas where improvements can be made, particularly around security, thread safety, and type safety.

---

🔴 CRITICAL Issues

1. Sandbox Escape Risk in SubprocessSandboxProvider

• The SubprocessSandboxProvider uses Python's subprocess.run without any additional security measures to isolate the execution environment.
• Risk: This implementation does not provide true sandboxing. A malicious agent could execute arbitrary commands, access sensitive files, or compromise the host system.
• Recommendation:
  • Clearly document that SubprocessSandboxProvider is not secure and should only be used for testing or non-sensitive tasks.
  • Implement a secure sandboxing mechanism, such as using Docker or a virtualized environment, to isolate agent execution.
  • Consider integrating with existing containerization tools like Docker or Firecracker for production-grade isolation.

---

2. Lack of Input Validation in DiffPolicy

• The DiffPolicy.evaluate method does not validate the structure of the files input. If a malformed DiffFile object is passed, it could cause runtime errors or unexpected behavior.
• Risk: This could lead to policy bypass or runtime crashes.
• Recommendation:
  • Use Pydantic models to validate the structure of DiffFile and DiffPolicyResult.
  • Add type checks and raise appropriate exceptions for invalid inputs.

---

3. Potential Denial of Service in GovernanceEventBus

• The _history attribute in GovernanceEventBus stores up to _max_history events. However, there is no mechanism to prevent a malicious actor from flooding the event bus with events, which could lead to memory exhaustion.
• Risk: This could be exploited to cause a denial-of-service (DoS) attack.
• Recommendation:
  • Implement rate-limiting for event publishing.
  • Add a mechanism to discard old events when the history size exceeds _max_history.

---

4. Unrestricted Wildcard Handlers in GovernanceEventBus

• The GovernanceEventBus allows wildcard (*) handlers to process all events. However, there is no restriction on what these handlers can do.
• Risk: A malicious or poorly implemented wildcard handler could disrupt the entire event bus.
• Recommendation:
  • Add a mechanism to validate or restrict the behavior of wildcard handlers.
  • Consider logging or monitoring the execution of wildcard handlers to detect anomalies.

---

🟡 WARNING Issues

1. Potential Breaking Change in pyproject.toml

• The addition of pydantic>=2.4.0 to pyproject.toml introduces a new dependency.
• Risk: If existing users are using a different version of Pydantic, this could cause compatibility issues.
• Recommendation:
  • Clearly document this dependency in the release notes.
  • Consider using a more flexible version range if possible (e.g., pydantic>=2.4,<3.0).

---

💡 Suggestions for Improvement

1. Thread Safety in GovernanceEventBus

• The GovernanceEventBus uses a defaultdict for _handlers and a list for _history, but there are no locks or synchronization mechanisms.
• Suggestion:
  • Use a thread-safe data structure like queue.Queue for _history.
  • Use a threading lock (threading.Lock) to synchronize access to _handlers.

2. Limit Output Size in SubprocessSandboxProvider

• The SubprocessSandboxProvider.run method captures up to 10,000 characters of stdout and stderr. While this is a good start, it might not be sufficient for all use cases.
• Suggestion:
  • Make the output size limit configurable via SandboxConfig.
  • Add truncation warnings to the captured output.

3. Improve Logging in GovernanceEventBus

• The GovernanceEventBus logs exceptions when handlers fail but does not provide detailed context about the event or the handler.
• Suggestion:
  • Include the event type, handler name, and event data in the log message.

4. Add Unit Tests for Edge Cases

• While the PR includes 22 tests, there are no tests for edge cases such as:
  • Publishing events with invalid data.
  • Handling malformed DiffFile objects in DiffPolicy.
  • Simulating timeouts and exceptions in SubprocessSandboxProvider.
• Suggestion:
  • Add unit tests to cover these edge cases.

5. Document Security Considerations

• The PR introduces several features with security implications (e.g., SandboxProvider, DiffPolicy, GovernanceEventBus).
• Suggestion:
  • Add a "Security Considerations" section to the documentation, outlining potential risks and recommended mitigations.

---

Summary of Feedback

Category	Count
🔴 CRITICAL	4
🟡 WARNING	1
💡 SUGGESTION	5

This PR introduces valuable features but has critical security and thread safety issues that must be addressed before merging. Additionally, there is a potential breaking change due to the new Pydantic dependency. Addressing these issues and incorporating the suggested improvements will enhance the robustness and security of the codebase.

[github-actions]

🤖 AI Agent: test-generator — `audit_logger.py`

🧪 Test Coverage Analysis

audit_logger.py

• ✅ Existing coverage: The GovernanceAuditLogger class and its methods (add_backend, log, log_decision, flush) are covered by tests in tests/agent_os/test_audit_logger.py. The JsonlFileBackend, InMemoryBackend, and LoggingBackend classes are also tested for basic functionality.
• ❌ Missing coverage:
  • Edge cases for AuditEntry serialization (e.g., invalid or unexpected metadata types).
  • Error handling in LoggingBackend when the logger fails.
  • File handling edge cases in JsonlFileBackend (e.g., file write errors, permission issues).
• 💡 Suggested test cases:
  1. test_audit_entry_invalid_metadata — Test AuditEntry.to_json with non-serializable metadata.
  2. test_logging_backend_error_handling — Simulate logger failure in LoggingBackend.write and verify graceful handling.
  3. test_jsonl_backend_file_error — Simulate file write errors in JsonlFileBackend and verify error handling.

---

compat.py

• ✅ Existing coverage: Basic functionality of NoOpPolicyEvaluator, NoOpGovernanceMiddleware, and get_evaluator is covered in tests/agent_os/test_compat.py.
• ❌ Missing coverage:
  • Behavior when agent_os.policies.evaluator is installed and PolicyEvaluator is used.
  • Edge cases for NoOpPolicyEvaluator.evaluate and NoOpGovernanceMiddleware.
• 💡 Suggested test cases:
  1. test_policy_evaluator_with_toolkit_installed — Mock agent_os.policies.evaluator to test behavior when the real PolicyEvaluator is available.
  2. test_noop_policy_evaluator_edge_cases — Test NoOpPolicyEvaluator.evaluate with unexpected arguments or malformed input.
  3. test_noop_governance_middleware_wrap — Test NoOpGovernanceMiddleware.wrap with various callable types.

---

diff_policy.py

• ✅ Existing coverage: The DiffPolicy class and its evaluate method are covered in tests/agent_os/test_diff_policy.py. Basic scenarios for file count, line count, and path restrictions are tested.
• ❌ Missing coverage:
  • Edge cases for allowed_paths and blocked_paths (e.g., overlapping patterns, edge glob patterns).
  • Large diffs with thousands of files or lines.
• 💡 Suggested test cases:
  1. test_diff_policy_overlapping_paths — Test behavior when a file matches both allowed_paths and blocked_paths.
  2. test_diff_policy_large_diff — Test performance and correctness with a large number of files and lines.
  3. test_diff_policy_edge_glob_patterns — Test edge cases for glob patterns (e.g., **/*, *.py, **/dir/*).

---

event_bus.py

• ✅ Existing coverage: The GovernanceEventBus class and its methods (subscribe, unsubscribe, publish, get_history, clear_history) are covered in tests/agent_os/test_event_bus.py. Basic pub/sub functionality and event history are tested.
• ❌ Missing coverage:
  • Wildcard event handlers (*) and their interaction with specific event handlers.
  • Error handling when event handlers raise exceptions.
  • Performance with a large number of events and handlers.
• 💡 Suggested test cases:
  1. test_event_bus_wildcard_handlers — Test that wildcard handlers receive all events, regardless of type.
  2. test_event_bus_handler_error_handling — Simulate exceptions in event handlers and verify that other handlers are not affected.
  3. test_event_bus_large_history — Test performance and correctness with a large number of events and handlers.

---

budget.py

• ✅ Existing coverage: The BudgetPolicy and BudgetTracker classes are covered in tests/agent_os/policies/test_budget.py. Basic functionality for tracking resource usage and checking policy violations is tested.
• ❌ Missing coverage:
  • Edge cases for remaining and utilization methods (e.g., division by zero, negative values).
  • Behavior when all policy limits are None.
• 💡 Suggested test cases:
  1. test_budget_tracker_remaining_edge_cases — Test remaining with None limits and edge values.
  2. test_budget_tracker_utilization_edge_cases — Test utilization with None limits and edge values.
  3. test_budget_tracker_negative_values — Test behavior when negative values are recorded (e.g., negative tokens or cost).

---

sandbox_provider.py

• ✅ Existing coverage: The SubprocessSandboxProvider class and its run method are covered in tests/agent_os/test_sandbox_provider.py. Basic subprocess execution and timeout handling are tested.
• ❌ Missing coverage:
  • Edge cases for SandboxConfig (e.g., invalid configurations, extreme values).
  • Error handling in SubprocessSandboxProvider.run (e.g., invalid commands, permission errors).
  • Behavior of NoOpSandboxProvider.
• 💡 Suggested test cases:
  1. test_sandbox_config_invalid_values — Test SandboxConfig with invalid or extreme values (e.g., negative memory, zero timeout).
  2. test_subprocess_provider_invalid_command — Test SubprocessSandboxProvider.run with an invalid command and verify error handling.
  3. test_noop_sandbox_provider — Verify that NoOpSandboxProvider implements the SandboxProvider interface and returns expected results.

---

task_outcome.py

• ❌ Existing coverage: No corresponding test file found in tests/.
• ❌ Missing coverage:
  • The TaskOutcomeRecorder class and its methods (record_outcome, get_outcomes, clear_outcomes) are not covered.
  • Edge cases for severity scoring and time-based recovery.
• 💡 Suggested test cases:
  1. test_task_outcome_recording — Test record_outcome with various severities and timestamps.
  2. test_task_outcome_time_based_recovery — Test recovery behavior over time for different severities.
  3. test_task_outcome_clear — Verify that clear_outcomes correctly clears all recorded outcomes.

---

Summary

• Files with missing coverage: audit_logger.py, compat.py, diff_policy.py, event_bus.py, budget.py, sandbox_provider.py, task_outcome.py
• New test cases suggested: 18

Adding the suggested test cases will improve coverage, particularly for edge cases and error handling.

[github-actions]

🤖 AI Agent: security-scanner — Security Analysis of the Pull Request

Security Analysis of the Pull Request

1. Prompt Injection Defense Bypass

• Finding: No direct evidence of prompt injection vulnerabilities was found in the changes. However, the DiffPolicy class does not sanitize or validate the path attribute of DiffFile objects before applying fnmatch.fnmatch. This could potentially allow maliciously crafted file paths to bypass the policy checks.
• Rating: 🟠 HIGH
• Attack Vector: An attacker could craft file paths that exploit edge cases in fnmatch pattern matching to bypass the blocked_paths or allowed_paths rules.
• Recommendation: Implement strict validation and normalization of file paths before applying fnmatch. For example, ensure paths are resolved to their canonical form using os.path.realpath() or similar methods to prevent directory traversal attacks.

---

2. Policy Engine Circumvention

• Finding: The NoOpPolicyEvaluator and NoOpGovernanceMiddleware classes allow all actions and bypass governance policies when the agent-os-kernel dependency is not installed. While this behavior is documented, it creates a significant risk if the dependency is accidentally omitted or removed.
• Rating: 🔴 CRITICAL
• Attack Vector: If the agent-os-kernel dependency is missing, the toolkit will silently fall back to the no-op implementations, effectively disabling all policy enforcement and governance controls.
• Recommendation: Raise a critical error or log a high-severity warning if the agent-os-kernel dependency is not installed. Provide an explicit configuration option to enable the no-op mode, ensuring that it cannot be triggered accidentally.

---

3. Trust Chain Weaknesses

• Finding: No changes in this PR directly impact SPIFFE/SVID validation or certificate pinning mechanisms. However, the GovernanceEventBus does not enforce authentication or integrity checks for published events.
• Rating: 🟠 HIGH
• Attack Vector: An attacker with access to the event bus could publish malicious events (e.g., fake policy.violation or trust.penalty events) to manipulate the governance system.
• Recommendation: Implement authentication and integrity checks for event publishing. For example, require event publishers to sign events with a private key and verify the signature before processing.

---

4. Credential Exposure

• Finding: No hardcoded secrets or sensitive credentials were found in the code. However, the SandboxConfig class allows environment variables to be passed to sandboxed processes via the env_vars attribute, which could inadvertently expose sensitive information.
• Rating: 🟡 MEDIUM
• Attack Vector: If sensitive environment variables (e.g., API keys, tokens) are passed to the sandboxed process, they could be leaked through logs or exploited by malicious code running in the sandbox.
• Recommendation: Add a mechanism to explicitly whitelist environment variables that can be passed to the sandbox. Log a warning if sensitive variables are detected in the env_vars dictionary.

---

5. Sandbox Escape

• Finding: The SubprocessSandboxProvider does not provide any real isolation, as it relies on the subprocess.run method without additional security measures. This is explicitly documented, but it still represents a significant risk.
• Rating: 🔴 CRITICAL
• Attack Vector: Malicious code executed within the subprocess could exploit the lack of isolation to access the host system, exfiltrate data, or perform other malicious actions.
• Recommendation: Implement a more secure sandboxing mechanism, such as using Docker or a similar containerization technology, to provide proper isolation. If SubprocessSandboxProvider is intended only for testing, ensure it cannot be used in production by adding runtime checks or explicit configuration flags.

---

6. Deserialization Attacks

• Finding: No unsafe deserialization methods (e.g., pickle.loads, yaml.load) were found in the changes. The use of json.loads and json.dumps appears safe.
• Rating: 🔵 LOW
• Recommendation: No immediate action required. Continue to avoid unsafe deserialization methods.

---

7. Race Conditions

• Finding: The GovernanceEventBus class uses a shared _history list to store events without any locking mechanism. This could lead to race conditions in concurrent environments.
• Rating: 🟠 HIGH
• Attack Vector: In a multithreaded or multiprocess environment, concurrent access to the _history list could result in data corruption or loss of events.
• Recommendation: Use thread-safe data structures (e.g., queue.Queue or collections.deque) or implement locking mechanisms to ensure thread safety.

---

8. Supply Chain

• Finding: A new dependency, pydantic>=2.4.0, was added to pyproject.toml. This is a legitimate and widely-used library, but it introduces a new supply chain risk.
• Rating: 🟡 MEDIUM
• Attack Vector: If the pydantic package is compromised on PyPI, it could introduce malicious code into the project.
• Recommendation: Use dependency pinning to a specific version or hash to mitigate supply chain risks. Regularly audit dependencies for vulnerabilities using tools like pip-audit or safety.

---

Summary of Findings

Category	Rating	Finding	Recommendation
Prompt Injection Defense Bypass	🟠 HIGH	Potential bypass of DiffPolicy path restrictions via unsanitized paths.	Normalize and validate file paths before applying fnmatch.
Policy Engine Circumvention	🔴 CRITICAL	No-op policy evaluator disables governance if dependency is missing.	Raise critical error or require explicit configuration for no-op mode.
Trust Chain Weaknesses	🟠 HIGH	No authentication or integrity checks for event publishing.	Require signed events and verify signatures before processing.
Credential Exposure	🟡 MEDIUM	Potential leakage of sensitive environment variables in SandboxConfig.	Whitelist environment variables and log warnings for sensitive variables.
Sandbox Escape	🔴 CRITICAL	SubprocessSandboxProvider lacks security isolation.	Use a secure sandboxing mechanism like Docker for production environments.
Deserialization Attacks	🔵 LOW	No unsafe deserialization methods found.	No immediate action required.
Race Conditions	🟠 HIGH	Potential race conditions in GovernanceEventBus due to shared state.	Use thread-safe data structures or implement locking mechanisms.
Supply Chain	🟡 MEDIUM	New dependency pydantic introduces supply chain risk.	Pin dependency versions and audit dependencies regularly.

Final Recommendation

This PR introduces several critical and high-severity security risks that must be addressed before merging. Specifically:

1. Address the potential for prompt injection bypass in DiffPolicy.
2. Prevent silent fallback to no-op governance components by requiring explicit configuration or raising critical errors.
3. Implement proper sandboxing for SubprocessSandboxProvider to prevent sandbox escapes.
4. Add thread safety to GovernanceEventBus.

Once these issues are resolved, the PR can be re-evaluated for merging.