Skip to content

docs: add copilot-instructions.md with PR review checklist#413

Merged
imran-siddique merged 2 commits intomicrosoft:mainfrom
imran-siddique:main
Mar 24, 2026
Merged

docs: add copilot-instructions.md with PR review checklist#413
imran-siddique merged 2 commits intomicrosoft:mainfrom
imran-siddique:main

Conversation

@imran-siddique
Copy link
Member

Adds mandatory review rules that Copilot must follow before merging any PR. Prevents the issues we hit with empty PRs, dependency confusion, and missing init files.

imran-siddique and others added 2 commits March 24, 2026 09:32
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mandatory review rules before merging any PR:
- Read actual diff (CI green is not sufficient)
- Dependency confusion scan on all install commands
- Verify __init__.py for new modules
- Verify dependencies declared in pyproject.toml
- No hardcoded secrets or plaintext config in pipelines
- Verify PR has actual changes (additions > 0)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file ci/cd CI/CD and workflows size/S Small PR (< 50 lines) labels Mar 24, 2026
@imran-siddique imran-siddique merged commit 094379e into microsoft:main Mar 24, 2026
53 checks passed
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Agent: code-reviewer

Review Feedback for PR: docs: add copilot-instructions.md with PR review checklist


🔴 CRITICAL: Security Concerns

  1. Dependency Confusion Risk
    The PR introduces a mandatory review checklist that includes scanning for dependency confusion. While this is a good practice, the list of registered package names (agent-os-kernel, agentmesh-platform, etc.) must be validated against an authoritative source (e.g., PyPI or an internal registry). Relying on hardcoded names in the checklist is insufficient and prone to human error.
    Action Required: Implement automated dependency validation in CI/CD pipelines to ensure no unregistered packages are installed.

  2. SHA-Pinning for GitHub Actions
    The checklist mandates SHA-pinning for GitHub Actions but does not enforce it programmatically. This leaves room for human error during reviews.
    Action Required: Add a CI check to validate that all GitHub Actions in workflows are SHA-pinned.

  3. Hardcoded Secrets
    While the checklist prohibits hardcoded secrets, no automated mechanism is mentioned to enforce this.
    Action Required: Integrate a secrets scanning tool (e.g., GitHub Advanced Security, TruffleHog) into the CI pipeline to detect hardcoded secrets.

  4. Unsafe YAML Parsing
    The checklist mandates the use of yaml.safe_load() but does not enforce it programmatically.
    Action Required: Add a static analysis rule (e.g., via ruff or bandit) to flag unsafe YAML parsing (yaml.load()).

  5. Dangerous Functions
    The checklist prohibits pickle.loads, eval(), exec(), and shell=True, but no automated enforcement is mentioned.
    Action Required: Extend static analysis rules to flag these patterns in code reviews.


🟡 WARNING: Potential Breaking Changes

  1. Dependency Addition (pydantic>=2.4.0)
    The addition of pydantic>=2.4.0 in agent-compliance/pyproject.toml could introduce breaking changes if the library's API differs significantly from earlier versions.
    Action Required: Verify compatibility with existing code and run regression tests to ensure no breaking changes.

  2. Package Name Change in README
    The README for agentmesh-integrations/openclaw-skill changes the package name from agentmesh-governance to agent-governance-toolkit. If users have already installed the old package, this change could cause confusion or break existing workflows.
    Action Required: Provide clear migration instructions in the README and ensure backward compatibility where feasible.


💡 Suggestions for Improvement

  1. Automated PR Review Checklist
    While the checklist is comprehensive, manual enforcement is error-prone. Consider implementing automated checks for each rule using GitHub Actions or pre-commit hooks.

  2. OWASP Agentic Top 10 Compliance
    Expand the checklist to include specific OWASP Agentic Top 10 rules, such as preventing sandbox escapes, securing inter-agent communication, and validating trust/identity mechanisms.

  3. Thread Safety
    Add a rule to verify thread safety in concurrent agent execution, especially for shared resources.

  4. Type Safety Enforcement
    Include a rule to ensure all Pydantic models are validated and type-safe. This can be enforced using mypy or similar tools.

  5. Backward Compatibility Testing
    Add a CI step to test backward compatibility for public APIs across supported Python versions (3.9–3.12).


Summary of Actions

  • 🔴 CRITICAL: Automate dependency validation, SHA-pinning, secrets scanning, and unsafe code detection.
  • 🟡 WARNING: Verify compatibility for pydantic>=2.4.0 and address potential confusion from package name changes.
  • 💡 SUGGESTION: Enhance the checklist with automated enforcement, OWASP compliance, thread safety, type safety, and backward compatibility testing.

This PR introduces valuable documentation, but additional automation and validation are required to ensure security and compliance.

@github-actions
Copy link

🤖 AI Agent: security-scanner — Security Review of Pull Request

Security Review of Pull Request

This PR primarily introduces a new documentation file (copilot-instructions.md) that outlines mandatory PR review rules and security guidelines. It also includes minor changes to a pyproject.toml file and a README file. Below is the security analysis based on the provided diff.


🔵 LOW: Potential for Misinterpretation in Dependency Confusion Instructions

Issue:
The instructions for dependency confusion checks in copilot-instructions.md are helpful but could lead to human error due to their reliance on manual verification. The list of registered package names is hardcoded, which may become outdated if new packages are added to the project but not updated in this document.

Attack Vector:
If a new package is added to the project but not included in the hardcoded list, a malicious actor could publish a package with the same name to PyPI, leading to a dependency confusion attack.

Recommendation:

  • Automate dependency validation by implementing a script in the CI pipeline to verify that all dependencies in pip install commands or pyproject.toml are registered and match the intended source (e.g., PyPI or an internal registry).
  • Maintain the list of registered package names in a single source of truth (e.g., a configuration file) that is programmatically checked, rather than relying on manual updates to documentation.

🔴 CRITICAL: Missing SHA Pinning for GitHub Actions

Issue:
The copilot-instructions.md file specifies that "All GitHub Actions must be SHA-pinned," but this PR does not include any enforcement mechanism or validation script to ensure compliance.

Attack Vector:
Without SHA-pinning, malicious actors could compromise a GitHub Action by modifying its code in the upstream repository. This could lead to arbitrary code execution in the CI/CD pipeline.

Recommendation:

  • Implement a CI check to validate that all GitHub Actions used in workflows are SHA-pinned.
  • Add a script to scan .github/workflows/*.yml files for unpinned actions and fail the build if any are found.
  • Update the documentation to include instructions for using SHA-pinned actions.

🟠 HIGH: Lack of Enforcement for yaml.safe_load()

Issue:
The copilot-instructions.md file mandates the use of yaml.safe_load() instead of yaml.load(), but there is no automated enforcement or CI check to ensure compliance.

Attack Vector:
If developers inadvertently use yaml.load() in the codebase, it could lead to deserialization attacks, allowing attackers to execute arbitrary code by crafting malicious YAML payloads.

Recommendation:

  • Add a static analysis tool (e.g., bandit) to the CI pipeline to detect unsafe YAML loading practices.
  • Write a custom linter rule or pre-commit hook to flag the use of yaml.load() in the codebase.
  • Update the CI pipeline to fail builds if unsafe YAML loading is detected.

🟠 HIGH: Potential for Supply Chain Attacks in pyproject.toml

Issue:
The pyproject.toml file adds a new dependency on pydantic>=2.4.0. While pydantic is a well-known library, there is no evidence in the PR that its integrity or compatibility with the project has been verified.

Attack Vector:
If the pydantic package or one of its dependencies is compromised (e.g., via a supply chain attack), it could introduce malicious code into the project.

Recommendation:

  • Verify the integrity of the pydantic package by checking its source repository, maintainers, and recent changes.
  • Use a dependency scanner (e.g., pip-audit, safety, or dependabot) to check for known vulnerabilities in pydantic and its dependencies.
  • Consider pinning the pydantic version to a specific, verified release to reduce the risk of introducing vulnerabilities through future updates.

🟡 MEDIUM: Potential for Hardcoded Secrets in Code

Issue:
The copilot-instructions.md file advises against hardcoding secrets but does not include any automated mechanism to detect or prevent this.

Attack Vector:
Developers might inadvertently commit sensitive information (e.g., API keys, tokens) to the repository, leading to potential credential exposure.

Recommendation:

  • Integrate a secret-scanning tool (e.g., truffleHog, git-secrets, or GitHub's secret scanning) into the CI pipeline to detect and block commits containing sensitive information.
  • Provide developers with pre-commit hooks to scan for secrets before committing code.

🔵 LOW: Documentation-Only Changes in README

Issue:
The change in the README.md file updates a pip install command to use the correct package name (agent-governance-toolkit). This is a documentation-only change and does not introduce any security risks.

Recommendation:
No action required.


Summary of Findings

  1. 🔵 LOW: Potential for misinterpretation in dependency confusion instructions.
  2. 🔴 CRITICAL: Missing SHA pinning for GitHub Actions.
  3. 🟠 HIGH: Lack of enforcement for yaml.safe_load().
  4. 🟠 HIGH: Potential for supply chain attacks in pyproject.toml.
  5. 🟡 MEDIUM: Potential for hardcoded secrets in code.
  6. 🔵 LOW: Documentation-only changes in README.md.

General Recommendations

  • Implement automated CI checks for all security rules outlined in copilot-instructions.md.
  • Use tools like pre-commit hooks, static analysis, and dependency scanners to enforce compliance and detect vulnerabilities.
  • Regularly review and update the copilot-instructions.md file to ensure it reflects the current state of the project and its dependencies.

This PR introduces important documentation but lacks the necessary automation to enforce the rules it describes. Addressing the identified issues is critical to ensure the security of the agent-governance-toolkit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd CI/CD and workflows dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation size/S Small PR (< 50 lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant