experiment (tools): tools/gate CI/CD policy gate#2535
Draft
rh-jfuller wants to merge 1 commit into
Draft
Conversation
Python CLI that evaluates SBOMs against configurable security and compliance policies using running Trustify instance. Exits non-zero when policy violations are found, designed to run in CI/CD pipelines as a quality gate. Key features: - Plugin-based checker architecture (Checker protocol + registry) - Built-in checker: severity/score thresholds, CVE deny/ignore lists, license allow/deny with glob patterns - Conforma checker: OPA/Rego policy evaluation via ec CLI or HTTP server mode - Four SBOM input modes: local file, upload, by-name, by-ID - Four output formats: table, JSON, SARIF, JUnit XML
Contributor
Reviewer's GuideIntroduces a new Sequence diagram for gate check command with upload and multiple checkerssequenceDiagram
actor CI as CI_pipeline
participant Gate as gate.py
participant Client as client.py
participant Trustify as Trustify_API
participant Builtin as BuiltinChecker
participant Conforma as ConformaChecker
participant EC as Conforma_ec
CI->>Gate: main() / _cmd_check(check --sbom file --upload)
Gate->>Gate: load_gate_config() or default_gate_config()
Gate->>Gate: _build_context()
Gate->>Gate: _context_from_upload(sbom_path)
Gate->>Client: upload_sbom(sbom_path)
Client->>Trustify: POST /api/v3/sbom
Trustify-->>Client: SBOM id
Client-->>Gate: upload result
Gate->>Gate: _context_from_id(sbom_id)
Gate->>Client: get_sbom_packages(sbom_id)
Client->>Trustify: GET /api/v3/sbom/{id}/packages
Gate->>Client: analyze_purls(purls)
Client->>Trustify: POST /api/v3/vulnerability/analyze
Gate->>Client: get_sbom_advisories(sbom_id)
Client->>Trustify: GET /api/v3/sbom/{id}/advisory
Gate->>Client: get_sbom_license_ids(sbom_id)
Client->>Trustify: GET /api/v3/sbom/{id}/all-license-ids
Client-->>Gate: CheckContext populated
loop for each checker
Gate->>Builtin: check(ctx, config)
Builtin-->>Gate: list[Violation]
Gate->>Conforma: check(ctx, config)
Conforma->>EC: ec validate input / HTTP POST
EC-->>Conforma: validation result
Conforma-->>Gate: list[Violation]
end
Gate->>Gate: GateResult(passed, violations)
Gate->>Gate: format_result(result, format)
Gate-->>CI: exit code (0 or 1)
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 1 security issue, 2 other issues, and left some high level feedback:
Security issues:
- Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)
General comments:
- Several non-CLI modules (e.g.
client.py,policy.load_gate_config,checkers/conforma.py) callsys.exit, which makes them hard to reuse; consider raising domain-specific exceptions instead and letgate.pytranslate those into exit codes. - The pyproject exposes a
gateconsole script while the README and CI examples invokepython gate.py; aligning usage examples with the installed entrypoint (or vice versa) would reduce confusion for users.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Several non-CLI modules (e.g. `client.py`, `policy.load_gate_config`, `checkers/conforma.py`) call `sys.exit`, which makes them hard to reuse; consider raising domain-specific exceptions instead and let `gate.py` translate those into exit codes.
- The pyproject exposes a `gate` console script while the README and CI examples invoke `python gate.py`; aligning usage examples with the installed entrypoint (or vice versa) would reduce confusion for users.
## Individual Comments
### Comment 1
<location path="tools/gate/client.py" line_range="146-153" />
<code_context>
+ return _extract_purls_via_api(sbom_path)
+
+
+def _extract_purls_via_api(sbom_path: Path) -> list[str]:
+ """Use Trustify's extract-sbom-purls endpoint as a fallback."""
+ content = sbom_path.read_bytes()
+ headers = _headers()
+ headers["Content-Type"] = "application/json"
+
+ with httpx.Client(base_url=TRUSTIFY_URL, headers=headers, timeout=TIMEOUT) as c:
+ r = c.post("/api/v3/ui/extract-sbom-purls", content=content)
+ r.raise_for_status()
+ result = r.json()
</code_context>
<issue_to_address>
**issue (bug_risk):** Fallback PURL extraction assumes API returns a list of dicts; list-of-strings responses will break.
In `_extract_purls_via_api`, the loop `for item in result if isinstance(result, list) else result.get("items", [])` calls `item.get(...)` unconditionally when `result` is a list. If `result` is `['pkg:...']` rather than a list of dicts, this will raise `AttributeError`. Please add type-aware handling (e.g., branch on `isinstance(item, dict)` vs `str`) and extract the PURL accordingly, similar to the defensive logic used elsewhere.
</issue_to_address>
### Comment 2
<location path="tools/gate/tests/test_builtin.py" line_range="88-106" />
<code_context>
+ })
+ assert result == []
+
+ def test_ignore_unfixed(self):
+ ctx = _make_ctx(vuln_analysis=[{
+ "purl": "pkg:npm/baz@3.0",
+ "vulnerabilities": [
+ {
+ "identifier": "CVE-2024-0010",
+ "severity": "critical",
+ "score": 9.8,
+ "status": "not_affected",
+ },
+ ],
+ }])
+ checker = BuiltinChecker()
+ result = checker.check(ctx, {
+ "vulnerabilities": {"max_severity": "low", "ignore_unfixed": True},
+ })
+ assert result == []
+
+ def test_severity_inferred_from_score(self):
</code_context>
<issue_to_address>
**suggestion (testing):** Add a complementary test where `ignore_unfixed` is false to ensure those statuses are still treated as violations
Please also add a test where the same statuses (`not_affected` / `under_investigation`) are checked with `ignore_unfixed` omitted or set to `False`, asserting that a violation is reported. This ensures the default behavior doesn’t silently change and guards against future regressions.
```suggestion
def test_ignore_unfixed(self):
ctx = _make_ctx(vuln_analysis=[{
"purl": "pkg:npm/baz@3.0",
"vulnerabilities": [
{
"identifier": "CVE-2024-0010",
"severity": "critical",
"score": 9.8,
"status": "not_affected",
},
],
}])
checker = BuiltinChecker()
result = checker.check(ctx, {
"vulnerabilities": {"max_severity": "low", "ignore_unfixed": True},
})
assert result == []
def test_unfixed_statuses_violate_by_default(self):
ctx = _make_ctx(vuln_analysis=[{
"purl": "pkg:npm/baz@3.0",
"vulnerabilities": [
{
"identifier": "CVE-2024-0011",
"severity": "critical",
"score": 9.8,
"status": "not_affected",
},
{
"identifier": "CVE-2024-0012",
"severity": "critical",
"score": 9.0,
"status": "under_investigation",
},
],
}])
checker = BuiltinChecker()
result = checker.check(ctx, {
"vulnerabilities": {"max_severity": "low"},
})
assert len(result) == 2
def test_unfixed_statuses_violate_when_ignore_unfixed_false(self):
ctx = _make_ctx(vuln_analysis=[{
"purl": "pkg:npm/baz@3.0",
"vulnerabilities": [
{
"identifier": "CVE-2024-0013",
"severity": "critical",
"score": 9.8,
"status": "not_affected",
},
{
"identifier": "CVE-2024-0014",
"severity": "critical",
"score": 9.0,
"status": "under_investigation",
},
],
}])
checker = BuiltinChecker()
result = checker.check(ctx, {
"vulnerabilities": {"max_severity": "low", "ignore_unfixed": False},
})
assert len(result) == 2
def test_severity_inferred_from_score(self):
```
</issue_to_address>
### Comment 3
<location path="tools/gate/checkers/conforma.py" line_range="127-132" />
<code_context>
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=120,
)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
*Source: opengrep*
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+146
to
+153
| def _extract_purls_via_api(sbom_path: Path) -> list[str]: | ||
| """Use Trustify's extract-sbom-purls endpoint as a fallback.""" | ||
| content = sbom_path.read_bytes() | ||
| headers = _headers() | ||
| headers["Content-Type"] = "application/json" | ||
|
|
||
| with httpx.Client(base_url=TRUSTIFY_URL, headers=headers, timeout=TIMEOUT) as c: | ||
| r = c.post("/api/v3/ui/extract-sbom-purls", content=content) |
Contributor
There was a problem hiding this comment.
issue (bug_risk): Fallback PURL extraction assumes API returns a list of dicts; list-of-strings responses will break.
In _extract_purls_via_api, the loop for item in result if isinstance(result, list) else result.get("items", []) calls item.get(...) unconditionally when result is a list. If result is ['pkg:...'] rather than a list of dicts, this will raise AttributeError. Please add type-aware handling (e.g., branch on isinstance(item, dict) vs str) and extract the PURL accordingly, similar to the defensive logic used elsewhere.
Comment on lines
+88
to
+106
| def test_ignore_unfixed(self): | ||
| ctx = _make_ctx(vuln_analysis=[{ | ||
| "purl": "pkg:npm/baz@3.0", | ||
| "vulnerabilities": [ | ||
| { | ||
| "identifier": "CVE-2024-0010", | ||
| "severity": "critical", | ||
| "score": 9.8, | ||
| "status": "not_affected", | ||
| }, | ||
| ], | ||
| }]) | ||
| checker = BuiltinChecker() | ||
| result = checker.check(ctx, { | ||
| "vulnerabilities": {"max_severity": "low", "ignore_unfixed": True}, | ||
| }) | ||
| assert result == [] | ||
|
|
||
| def test_severity_inferred_from_score(self): |
Contributor
There was a problem hiding this comment.
suggestion (testing): Add a complementary test where ignore_unfixed is false to ensure those statuses are still treated as violations
Please also add a test where the same statuses (not_affected / under_investigation) are checked with ignore_unfixed omitted or set to False, asserting that a violation is reported. This ensures the default behavior doesn’t silently change and guards against future regressions.
Suggested change
| def test_ignore_unfixed(self): | |
| ctx = _make_ctx(vuln_analysis=[{ | |
| "purl": "pkg:npm/baz@3.0", | |
| "vulnerabilities": [ | |
| { | |
| "identifier": "CVE-2024-0010", | |
| "severity": "critical", | |
| "score": 9.8, | |
| "status": "not_affected", | |
| }, | |
| ], | |
| }]) | |
| checker = BuiltinChecker() | |
| result = checker.check(ctx, { | |
| "vulnerabilities": {"max_severity": "low", "ignore_unfixed": True}, | |
| }) | |
| assert result == [] | |
| def test_severity_inferred_from_score(self): | |
| def test_ignore_unfixed(self): | |
| ctx = _make_ctx(vuln_analysis=[{ | |
| "purl": "pkg:npm/baz@3.0", | |
| "vulnerabilities": [ | |
| { | |
| "identifier": "CVE-2024-0010", | |
| "severity": "critical", | |
| "score": 9.8, | |
| "status": "not_affected", | |
| }, | |
| ], | |
| }]) | |
| checker = BuiltinChecker() | |
| result = checker.check(ctx, { | |
| "vulnerabilities": {"max_severity": "low", "ignore_unfixed": True}, | |
| }) | |
| assert result == [] | |
| def test_unfixed_statuses_violate_by_default(self): | |
| ctx = _make_ctx(vuln_analysis=[{ | |
| "purl": "pkg:npm/baz@3.0", | |
| "vulnerabilities": [ | |
| { | |
| "identifier": "CVE-2024-0011", | |
| "severity": "critical", | |
| "score": 9.8, | |
| "status": "not_affected", | |
| }, | |
| { | |
| "identifier": "CVE-2024-0012", | |
| "severity": "critical", | |
| "score": 9.0, | |
| "status": "under_investigation", | |
| }, | |
| ], | |
| }]) | |
| checker = BuiltinChecker() | |
| result = checker.check(ctx, { | |
| "vulnerabilities": {"max_severity": "low"}, | |
| }) | |
| assert len(result) == 2 | |
| def test_unfixed_statuses_violate_when_ignore_unfixed_false(self): | |
| ctx = _make_ctx(vuln_analysis=[{ | |
| "purl": "pkg:npm/baz@3.0", | |
| "vulnerabilities": [ | |
| { | |
| "identifier": "CVE-2024-0013", | |
| "severity": "critical", | |
| "score": 9.8, | |
| "status": "not_affected", | |
| }, | |
| { | |
| "identifier": "CVE-2024-0014", | |
| "severity": "critical", | |
| "score": 9.0, | |
| "status": "under_investigation", | |
| }, | |
| ], | |
| }]) | |
| checker = BuiltinChecker() | |
| result = checker.check(ctx, { | |
| "vulnerabilities": {"max_severity": "low", "ignore_unfixed": False}, | |
| }) | |
| assert len(result) == 2 | |
| def test_severity_inferred_from_score(self): |
Comment on lines
+127
to
+132
| result = subprocess.run( | ||
| cmd, | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=120, | ||
| ) |
Contributor
There was a problem hiding this comment.
security (python.lang.security.audit.dangerous-subprocess-use-audit): Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.
Source: opengrep
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2535 +/- ##
==========================================
+ Coverage 72.17% 72.19% +0.01%
==========================================
Files 465 465
Lines 28702 28702
Branches 28702 28702
==========================================
+ Hits 20715 20720 +5
+ Misses 6812 6802 -10
- Partials 1175 1180 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
rh-jfuller
marked this pull request as draft
July 23, 2026 17:06
Contributor
Author
|
I think maybe we can emulate the way https://github.com/EmbarkStudios/cargo-deny works as well |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Illustrates a CLI tool that answers "should this build be allowed to ship?" by evaluating SBOMs against configurable policies using data from a running Trustify instance. This can be used either via the command line or in CI job.
Policy evaluation uses plugin system. Each checker implements a Checker protocol, registers itself at import time, and receives a CheckContext populated with Trustify data (vulnerabilities, advisories, licenses, PURLs). The gate config file specifies which checkers to run.
Checkers
Adding a new checker is four steps: create a module, implement the protocol, call register_checker(), add the import.
SBOM input modes
Example command line usage
Local check with default policy (fail on critical vulns)
Upload + check with custom config, SARIF output
Local check with conforma policy (needs ec installed).
ensure a local EnterpriseContractPolicy spec exists
now run gate with that policy file
Example CI
Summary by Sourcery
Introduce a new
tools/gateCLI-based CI/CD policy gate that evaluates SBOMs against Trustify-backed security and compliance policies and fails builds on violations.New Features:
trustify-gateCLI with commands to check SBOMs (local, uploaded, or pre-ingested) and list available checker plugins.Enhancements:
Build:
pyproject.tomlfor thetrustify-gateproject, including dependencies, test extras, scripts entry point, and tooling configuration.Documentation:
Tests:
Chores:
tools/gateincluding examples, Makefile, pytest configuration, and .gitignore for the new gate tool.