feat(mindev): add --junit-file flag to test command and add Starlark testing docs - #6690
feat(mindev): add --junit-file flag to test command and add Starlark testing docs#6690krrish175-byte wants to merge 1 commit into
Conversation
|
coverage: 61.671%. remained the same — krrish175-byte:feat/mindev-test-junit-file into mindersec:main |
evankanderson
left a comment
evankanderson
left a comment
There was a problem hiding this comment.
I'm probably going to hold off on cutting a release until we figure out if there are any other mindev changes you'll want for tests.
|
|
||
| ## Testing with Starlark | ||
|
|
||
| In addition to evaluating rules against a single entity, `mindev test` allows you to run comprehensive Starlark-based tests against your rule types. This is the recommended way to verify rule behavior, as it lets you mock external systems like GitHub REST/GraphQL APIs and Git filesystems. |
There was a problem hiding this comment.
Two comments:
- It's probably worth linking to the starlark language documentation, so people can understand what's supported.
- I'd make a stronger statement than "recommended way" -- something like "
mindev testis the supported mechanism for verifying rule behavior, and is designed for integration into CI/CD pipelines as well as standalone usage".
There was a problem hiding this comment.
I think it's fine to add this usage here, but we should think about a standalone page describing "how to test rules".
This would cover both manual and automated (CI) testing, including referencing the action and examples of setup, as well as some hints on effective testing usage, and documentation of the custom functions added to the starlark runtime.
| ) | ||
|
|
||
| # 4. Assert the result | ||
| assert result["status"] == "success", "Expected rule to pass" |
There was a problem hiding this comment.
In the expanded page, we should include information about what entries are in the "result" object, as well as the input arguments to "eval".
| assert result["status"] == "error", "Expected rule to fail" | ||
| ``` | ||
|
|
||
| The `eval` builtin takes several optional arguments: |
There was a problem hiding this comment.
Some of these arguments are not optional (at a minimum, the "rule" argument, and maybe the "entity" argument).
|
Feel free to either merge this, or address some of the PR feedback and have me do another review (I'm fine with either, holding off on hitting merge to give you the choice). |
3 participants
Addresses feedback to support writing JUnit XML to a specific file while preserving standard terminal output. Also adds Starlark testing documentation to the mindev how-to docs.