feat: add governance policy linter CLI command (#404)

[imran-siddique]

Description

Adds a \lint-policy\ subcommand to the \�gent-compliance\ CLI that validates YAML policy files for common mistakes.

Lint checks

Check	Severity
Missing \�ersion, \
ame, \
ules\	error
Invalid YAML / non-mapping root	error
Unknown operator	error
Unknown action	error
Invalid priority (non-integer)	error
Empty rules list	warning
Deprecated fields (\ ype\→\�ction, \op\→\operator)	warning
Conflicting rules (same condition with allow + deny)	warning

Usage

\\�ash
agent-compliance lint-policy policies/
agent-compliance lint-policy policy.yaml --json
agent-compliance lint-policy policies/ --strict # treat warnings as errors
\\

Tests

34 new tests — all passing.

Closes #404

[github-actions]

🤖 AI Agent: breaking-change-detector — Summary

🔍 API Compatibility Report

Summary

The pull request introduces a new CLI subcommand lint-policy for validating YAML policy files. This addition is additive and does not introduce any breaking changes to the existing API. The new functionality is encapsulated in a new module lint_policy.py and integrates seamlessly with the existing CLI framework.

Findings

Severity	Package	Change	Impact
🔵	agent-compliance	Added lint-policy CLI subcommand	New functionality, no impact on existing code
🔵	agent-compliance	Added lint_path() and lint_file()	New public API for linting policy files
🔵	agent-compliance	Added LintMessage and LintResult classes	New public API for structured lint results

Migration Guide

No migration steps are required as this change is additive and does not affect existing functionality.

Notes

• The new lint-policy subcommand and associated API (lint_path, lint_file, LintMessage, LintResult) should be documented to ensure downstream users can leverage the new functionality effectively.
• The CLI integration and comprehensive tests (34 new tests) ensure robustness and reliability of the new feature.

✅ No breaking changes detected.

[github-actions]

🤖 AI Agent: test-generator — `packages/agent-compliance/src/agent_compliance/cli/main.py`

🧪 Test Coverage Analysis

packages/agent-compliance/src/agent_compliance/cli/main.py

• ✅ Existing coverage: The cmd_lint_policy function and its integration into the CLI are covered by the tests in test_lint_policy.py. Specifically:
  • CLI argument parsing for lint-policy is tested.
  • JSON output and strict mode behavior are validated.
  • Various policy linting scenarios (e.g., valid policies, missing fields, conflicting rules) are tested indirectly through the lint-policy command.
• ❌ Missing coverage:
  • Error handling in cmd_lint_policy when an exception is raised (e.g., invalid path, unexpected errors).
  • Edge cases for invalid CLI arguments (e.g., missing path, invalid --json or --strict combinations).
• 💡 Suggested test cases:
  1. test_cmd_lint_policy_invalid_path — Ensure the CLI gracefully handles invalid file paths and outputs an appropriate error message.
  2. test_cmd_lint_policy_unexpected_exception — Simulate an unexpected exception in cmd_lint_policy and verify the error message is printed to stderr.
  3. test_cmd_lint_policy_missing_path_argument — Verify the CLI handles missing required arguments (e.g., path) with a clear error message.

---

packages/agent-compliance/src/agent_compliance/lint_policy.py

• ✅ Existing coverage: The lint_file and lint_path functions are extensively tested in test_lint_policy.py. Covered scenarios include:
  • Valid policies with no issues.
  • Missing required fields (version, name, rules).
  • Empty rules list.
  • Unknown operators and actions.
  • Conflicting rules (e.g., allow and deny for the same condition).
  • Deprecated fields and invalid priorities.
  • Malformed YAML files.
• ❌ Missing coverage:
  • Concurrency: The behavior of lint_path when multiple files are being processed concurrently is not tested. Potential race conditions or file access conflicts could arise.
  • Large payloads: The performance and behavior of lint_file and lint_path with large YAML files or directories containing many files are not tested.
  • Edge cases:
    • Policies with deeply nested conditions or rules.
    • Policies with non-standard YAML structures (e.g., comments, anchors, or aliases).
    • Policies with non-UTF-8 encodings.
• 💡 Suggested test cases:
  1. test_lint_path_concurrent_access — Simulate multiple threads or processes accessing the same directory or file to test for race conditions.
  2. test_lint_file_large_policy — Test the linter's behavior with a very large YAML policy file to ensure it handles large inputs gracefully.
  3. test_lint_path_large_directory — Test the linter's performance and correctness when processing a directory with a large number of YAML files.
  4. test_lint_file_deeply_nested_conditions — Validate the linter's ability to handle policies with deeply nested conditions or rules.
  5. test_lint_file_non_utf8_encoding — Ensure the linter handles non-UTF-8 encoded files gracefully and provides a clear error message.
  6. test_lint_file_yaml_aliases — Test the linter's handling of YAML anchors and aliases to ensure they are processed correctly.

---

Summary

The existing test coverage for the new lint-policy functionality is robust, covering most common scenarios and errors. However, additional tests for error handling, concurrency, and edge cases (e.g., large inputs, non-standard YAML structures) would further strengthen the coverage and ensure robustness in production environments.

[github-actions]

🤖 AI Agent: code-reviewer

Pull Request Review: feat: add governance policy linter CLI command (#404)

Summary

This PR introduces a new lint-policy subcommand to the agent-compliance CLI, which validates YAML policy files for common mistakes. The linter checks for missing fields, invalid YAML, unknown operators/actions, deprecated fields, conflicting rules, and other issues. It supports JSON output and a strict mode to treat warnings as errors.

---

🔴 CRITICAL Issues

1. Unvalidated YAML Parsing:

   • The linter uses yaml.safe_load to parse YAML files. While safe_load mitigates some risks (e.g., arbitrary code execution), it does not validate the schema of the YAML file. This could lead to false negatives if the file structure deviates from expectations but still parses successfully.
   • Action: Integrate schema validation using pydantic models or cerberus to ensure the YAML structure adheres strictly to the expected format.

2. Thread Safety:

   • The lint_path function processes multiple files in a directory sequentially. If this function is used in a concurrent context (e.g., multi-threaded CLI execution), the shared LintResult object could lead to race conditions.
   • Action: Refactor lint_path to use thread-local storage or ensure results aggregation is thread-safe.

3. Sandbox Escape via File Paths:

   • The lint_path function accepts arbitrary file paths without sanitization. This could allow malicious users to supply paths that escape the intended directory scope or access sensitive files.
   • Action: Validate and sanitize file paths to ensure they are within the expected directory scope. Reject paths with traversal patterns like ../.

---

🟡 WARNING: Potential Breaking Changes

1. CLI Behavior Change:

   • Adding the --strict flag changes the behavior of the CLI by treating warnings as errors. This could break existing workflows that rely on warnings being non-blocking.
   • Action: Clearly document this change in the release notes and provide guidance for users who may be affected.

2. Deprecation of Fields:

   • The linter flags deprecated fields (type, op, etc.) with warnings. If these fields are removed in future versions, it will break backward compatibility.
   • Action: Provide a migration guide for users and ensure deprecation warnings are prominently documented.

---

💡 Suggestions for Improvement

1. Enhanced Conflict Detection:

   • The current conflict detection logic only checks for allow and deny actions with identical conditions. Consider extending this to detect conflicts across other actions (e.g., audit vs. block).
   • Action: Expand the seen dictionary logic to track all actions and their interactions.

2. Improved Error Reporting:

   • The error messages for invalid YAML files (Invalid YAML: {exc}) could be more user-friendly by including specific suggestions for fixing the issue.
   • Action: Parse the yaml.YAMLError object to extract detailed information about the error (e.g., expected token, line number).

3. Unit Test Coverage:

   • While the tests cover a wide range of scenarios, edge cases like deeply nested rules, large YAML files, and invalid file encodings are not explicitly tested.
   • Action: Add tests for these edge cases to ensure robustness.

4. Performance Optimization:

   • The _find_line function performs linear scans of the file's lines, which could be slow for large YAML files.
   • Action: Optimize _find_line using precomputed line indices or a more efficient search algorithm.

5. CLI Output Formatting:

   • The CLI output for lint-policy could be improved by adding color-coded messages (e.g., red for errors, yellow for warnings) for better readability.
   • Action: Use a library like colorama or rich to enhance the CLI output.

---

Additional Observations

1. Use of Frozen Sets:

   • The use of frozenset for KNOWN_OPERATORS and KNOWN_ACTIONS is appropriate for immutability and performance. This is a good design choice.

2. Strict Mode Implementation:

   • The --strict flag is implemented correctly, ensuring warnings are treated as errors when enabled. This is a useful feature for enforcing stricter policy compliance.

3. Documentation:

   • The PR includes clear documentation for the new CLI command, its options, and expected behavior. This is commendable.

---

Final Recommendation

• Address the CRITICAL issues related to YAML validation, thread safety, and sandbox escape.
• Document the WARNING changes to ensure users are aware of potential impacts.
• Consider implementing the SUGGESTIONS to improve functionality, performance, and user experience.

Once the critical issues are resolved, this PR will be ready for merging.

[github-actions]

🤖 AI Agent: docs-sync-checker — Issues Found

📝 Documentation Sync Report

Issues Found

• ❌ cmd_lint_policy(args: argparse.Namespace) -> int in packages/agent-compliance/src/agent_compliance/cli/main.py — missing docstring

• ❌ lint_file(path: str | Path) -> LintResult in packages/agent-compliance/src/agent_compliance/lint_policy.py — missing detailed docstring (parameters, return values, exceptions)

• ❌ lint_path(path: str | Path) -> LintResult in packages/agent-compliance/src/agent_compliance/lint_policy.py — missing detailed docstring (parameters, return values, exceptions)

• ❌ _lint_rules(rules: list[Any], lines: list[str], filepath: str, result: LintResult) -> None in packages/agent-compliance/src/agent_compliance/lint_policy.py — missing docstring

• ❌ _find_line(lines: list[str], needle: str, start: int = 0) -> int in packages/agent-compliance/src/agent_compliance/lint_policy.py — missing docstring

• ⚠️ packages/agent-compliance/README.md — The new lint-policy command is not documented in the README. The usage examples and description should be added to the README.

• ⚠️ CHANGELOG.md — No entry for the new lint-policy command addition.

Suggestions

• 💡 Add a docstring for cmd_lint_policy(args: argparse.Namespace) -> int explaining its purpose, parameters, return values, and exceptions.
• 💡 Update the docstring for lint_file(path: str | Path) -> LintResult to include detailed explanations of parameters, return values, and exceptions.
• 💡 Update the docstring for lint_path(path: str | Path) -> LintResult to include detailed explanations of parameters, return values, and exceptions.
• 💡 Add a docstring for _lint_rules(rules: list[Any], lines: list[str], filepath: str, result: LintResult) -> None explaining its purpose and parameters.
• 💡 Add a docstring for _find_line(lines: list[str], needle: str, start: int = 0) -> int explaining its purpose and parameters.
• 💡 Update packages/agent-compliance/README.md to include a new section for the lint-policy command, with usage examples and a description of its functionality.
• 💡 Add an entry to CHANGELOG.md under a new version section, summarizing the addition of the lint-policy command and its features.

Action Items

1. Add missing docstrings to the identified functions.
2. Update the packages/agent-compliance/README.md to document the new lint-policy command.
3. Add a new entry in CHANGELOG.md to reflect the addition of the lint-policy command.

Please address these issues to ensure documentation is in sync with the new feature.

[github-actions]

🤖 AI Agent: security-scanner — Security Findings for Pull Request #404

Security Findings for Pull Request #404

1. Prompt Injection Defense Bypass: 🔵 LOW

   • Attack Vector: The linter does not directly handle user input that could be used to craft malicious YAML files. However, if the linter is used in a context where user input is processed without validation, it could lead to prompt injection.
   • Fix: Ensure that any user-provided YAML files are validated against a strict schema before being processed by the linter.

2. Policy Engine Circumvention: 🔵 LOW

   • Attack Vector: The linter checks for common mistakes but does not enforce policy execution. If a user can bypass the linter, they might introduce invalid policies that could be executed later.
   • Fix: Integrate the linter into the policy enforcement pipeline to ensure that only validated policies are executed.

3. Trust Chain Weaknesses: 🔵 LOW

   • Attack Vector: The linter does not handle trust chain validation for the YAML files it processes. If a malicious actor can provide a YAML file, they could potentially introduce harmful rules.
   • Fix: Implement a mechanism to verify the source of the YAML files, ensuring they come from a trusted origin.

4. Credential Exposure: 🔵 LOW

   • Attack Vector: The linter outputs error messages that could potentially expose sensitive information if the YAML files contain secrets or sensitive data.
   • Fix: Sanitize error messages to ensure they do not leak sensitive information. Implement logging practices that avoid exposing sensitive data.

5. Sandbox Escape: 🔵 LOW

   • Attack Vector: The linter operates in a controlled environment but does not explicitly prevent execution of arbitrary code through YAML. If the YAML parsing library is misconfigured, it could lead to code execution.
   • Fix: Use a secure YAML parser that restricts execution of arbitrary code and validate the input structure strictly.

6. Deserialization Attacks: 🔴 CRITICAL

   • Attack Vector: The linter uses yaml.safe_load, which is generally safe, but if any part of the code were to use yaml.load, it could lead to deserialization vulnerabilities.
   • Fix: Ensure that only yaml.safe_load is used throughout the codebase and review all YAML processing points to prevent unsafe deserialization.

7. Race Conditions: 🔵 LOW

   • Attack Vector: The linter processes files sequentially, but if it were to be modified to handle concurrent processing, there could be a risk of race conditions affecting the linting results.
   • Fix: Implement locking mechanisms or ensure that concurrent executions do not interfere with shared resources.

8. Supply Chain: 🟠 HIGH

   • Attack Vector: The linter imports pyyaml without version constraints, which could lead to dependency confusion or typosquatting if a malicious version is published.
   • Fix: Specify exact versions of dependencies in the requirements file and regularly audit dependencies for vulnerabilities.

Summary

The changes introduced in this pull request enhance the functionality of the governance toolkit by adding a linter for policy files. However, several security concerns need to be addressed, particularly around deserialization attacks and supply chain vulnerabilities. Implementing the suggested fixes will help mitigate these risks and strengthen the overall security posture of the toolkit.