Skip to content

fix: warn and error on unmatched catalog IDs in Mobilize#185

Merged
jmeridth merged 1 commit intomainfrom
fix/mobilize-warn-unmatched-catalogs
Mar 30, 2026
Merged

fix: warn and error on unmatched catalog IDs in Mobilize#185
jmeridth merged 1 commit intomainfrom
fix/mobilize-warn-unmatched-catalogs

Conversation

@jmeridth
Copy link
Copy Markdown
Member

@jmeridth jmeridth commented Mar 30, 2026

Related privateerproj/privateer#214

What

Add warning logs for each requested catalog that doesn't match any available evaluation suite, return an error when no requested catalogs match at all, and clear Evaluation_Suites at the start of Mobilize() to ensure idempotent behavior on repeated calls.

Why

Mobilize() silently skipped unmatched catalog IDs, causing zero suites to execute with no indication of misconfiguration. This made it difficult to diagnose when a plugin's catalog IDs changed (e.g., osps-baselineosps-baseline-2026-02) and downstream consumers hadn't updated their configs. This caused a silent integration test failure in privateerproj/privateer.

Notes

  • Partial matches (some catalogs found, some not) log warnings but still succeed — this avoids breaking configs that intentionally list catalogs not supported by every plugin
  • Total misses (zero suites executed) return a new NO_MATCHING_CATALOGS error that includes both the requested and available catalog IDs, making the fix obvious without inspecting plugin internals
  • Evaluation_Suites is now cleared at the top of Mobilize() to prevent stale results from a prior call masking a zero-match on subsequent calls
  • The warning includes the full list of available catalog IDs so the user can see valid options

Testing

  • Added TestEvaluationOrchestrator_Mobilize_UnmatchedCatalogs with three subtests:
    • Error When No Requested Catalogs Match: verifies Mobilize returns NO_MATCHING_CATALOGS error when zero suites match
    • Repeated Calls Clear Prior Results: verifies a second Mobilize() with unmatched catalogs correctly errors instead of silently succeeding with stale results
    • Partial Match Succeeds With Warning: verifies Mobilize succeeds and executes the matching suite when at least one catalog matches
  • Full test suite passes (go vet ./..., go test ./..., golangci-lint run ./...)

@jmeridth jmeridth self-assigned this Mar 30, 2026
@github-actions github-actions bot added the fix label Mar 30, 2026
## What

Add warning logs for each requested catalog that doesn't match any
available evaluation suite, and return an error when no requested
catalogs match at all.

## Why

Mobilize() silently skipped unmatched catalog IDs, causing zero suites
to execute with no indication of misconfiguration. This made it very
difficult to diagnose when a plugin's catalog IDs changed (e.g.,
osps-baseline → osps-baseline-2026-02) and downstream consumers
hadn't updated their configs.

## Notes

- Partial matches (some catalogs found, some not) log warnings but
  still succeed — this avoids breaking configs that intentionally
  list catalogs not supported by every plugin
- Total misses (zero suites executed) return a new NO_MATCHING_CATALOGS
  error that includes both the requested and available catalog IDs
- The error message format is designed to make the fix obvious to
  the user without needing to inspect plugin internals

Signed-off-by: jmeridth <jmeridth@gmail.com>
@jmeridth jmeridth force-pushed the fix/mobilize-warn-unmatched-catalogs branch from e588614 to 376b3a1 Compare March 30, 2026 22:04
@jmeridth jmeridth marked this pull request as ready for review March 30, 2026 22:05
@jmeridth jmeridth requested a review from a team as a code owner March 30, 2026 22:05
@jmeridth jmeridth merged commit 6d20e08 into main Mar 30, 2026
12 checks passed
@jmeridth jmeridth deleted the fix/mobilize-warn-unmatched-catalogs branch March 30, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants