Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions cmd/dev/app/test/test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import (
"encoding/xml"
"errors"
"fmt"
"io"
"os"

"github.com/spf13/cobra"

Expand All @@ -17,6 +19,7 @@ import (
// CmdTest returns the test cobra command
func CmdTest() *cobra.Command {
var outputFormat string
var junitFile string

cmd := &cobra.Command{
Use: "test [paths...]",
Expand Down Expand Up @@ -46,20 +49,25 @@ func CmdTest() *cobra.Command {
case "text":
formatFailuresHuman(cmd, results)
case "junit":
suites := ruletest.AsJUnit(results)
_, err := fmt.Fprint(cmd.OutOrStdout(), xml.Header)
if err != nil {
return fmt.Errorf("failed to write XML header: %w", err)
}
encoder := xml.NewEncoder(cmd.OutOrStdout())
encoder.Indent("", " ")
if err := encoder.Encode(suites); err != nil {
return fmt.Errorf("failed to encode JUnit XML: %w", err)
if err := writeJUnit(cmd.OutOrStdout(), results); err != nil {
return err
}
default:
return fmt.Errorf("unsupported output format %q: must be \"text\" or \"junit\"", outputFormat)
}

if junitFile != "" {
//nolint:gosec // path is provided by the user via a flag
f, err := os.Create(junitFile)
if err != nil {
return fmt.Errorf("failed to create junit file: %w", err)
}
defer f.Close()
if err := writeJUnit(f, results); err != nil {
return err
}
}

for _, res := range results {
if len(res.Failures) > 0 {
finalErr = errors.New("one or more tests failed")
Expand All @@ -71,6 +79,7 @@ func CmdTest() *cobra.Command {
}

cmd.Flags().StringVarP(&outputFormat, "output", "o", "text", "Output format (text, junit)")
cmd.Flags().StringVar(&junitFile, "junit-file", "", "File to write JUnit report to (in addition to standard output)")

return cmd
}
Expand All @@ -90,3 +99,16 @@ func formatFailuresHuman(cmd *cobra.Command, results []ruletest.TestResult) {
}
}
}

func writeJUnit(w io.Writer, results []ruletest.TestResult) error {
suites := ruletest.AsJUnit(results)
if _, err := fmt.Fprint(w, xml.Header); err != nil {
return fmt.Errorf("failed to write XML header: %w", err)
}
encoder := xml.NewEncoder(w)
encoder.Indent("", " ")
if err := encoder.Encode(suites); err != nil {
return fmt.Errorf("failed to encode JUnit XML: %w", err)
}
return nil
}
67 changes: 67 additions & 0 deletions docs/docs/how-to/mindev.md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine to add this usage here, but we should think about a standalone page describing "how to test rules".

This would cover both manual and automated (CI) testing, including referencing the action and examples of setup, as well as some hints on effective testing usage, and documentation of the custom functions added to the starlark runtime.

Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,73 @@ For more information on the rego print statement, the following blog post is a
good resource:
[Introducing the OPA print function](https://blog.openpolicyagent.org/introducing-the-opa-print-function-809da6a13aee)

## Testing with Starlark

In addition to evaluating rules against a single entity, `mindev test` allows you to run comprehensive Starlark-based tests against your rule types. This is the recommended way to verify rule behavior, as it lets you mock external systems like GitHub REST/GraphQL APIs and Git filesystems.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments:

  1. It's probably worth linking to the starlark language documentation, so people can understand what's supported.
  2. I'd make a stronger statement than "recommended way" -- something like "mindev test is the supported mechanism for verifying rule behavior, and is designed for integration into CI/CD pipelines as well as standalone usage".


You can run your Starlark tests using the following command:

```bash
mindev test /path/to/test.star
```

By default, `mindev test .` will recursively find and run all `*_test.star` files in the current directory and its subdirectories.

### Writing Starlark Tests

Test files use Starlark (a Python-like configuration language). A basic test file looks like this:

```python
# test_rule.star
def test_valid_case():
# 1. Define the input entity
entity = {
"repo_name": "example",
"repo_owner": "org"
}

# 2. Mock external calls if your rule uses the `rest` or `git` ingest types
def mock_http(content):
return {
"/repos/org/example/branches/main/protection": {
"enforce_admins": {"enabled": True}
}
}

# 3. Call the `eval` builtin to run the rule
result = eval(
rule="rule-types/github/branch_protection.yaml",
entity=entity,
mock_http=mock_http
)

# 4. Assert the result
assert result["status"] == "success", "Expected rule to pass"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the expanded page, we should include information about what entries are in the "result" object, as well as the input arguments to "eval".


def test_failure_case():
# ... mock the API to return invalid configuration ...
result = eval(...)
assert result["status"] == "error", "Expected rule to fail"
```

The `eval` builtin takes several optional arguments:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these arguments are not optional (at a minimum, the "rule" argument, and maybe the "entity" argument).

- `rule`: The path to the rule type YAML file
- `entity`: The entity object to evaluate (defaults to an empty entity if omitted)
- `profile`: A profile dictionary or path to evaluate the rule in the context of a profile
- `params`: Rule parameters dictionary (matches the `param_schema`)
- `mock_http`: A function that receives the request content and returns a dictionary of URL paths to mocked JSON responses
- `mock_fs`: A function or dictionary to mock filesystem contents for `git` ingest

### JUnit Output

If you want to use the test output in CI/CD pipelines, you can generate a JUnit XML report:

```bash
mindev test . --junit-file report.xml
```

This will write the standard text output to your terminal, and output a structured `report.xml` file containing the test results.

## Conclusion

Mindev is a powerful tool that helps you develop and debug rule types for
Expand Down