feat: add aggressive kueue gating and enhanced verbose output#184
Merged
lburgazzoli merged 1 commit intored-hat-data-services:rhoai-3.3from Mar 18, 2026
Merged
Conversation
Replace 6 per-resource-type kueue label checks (Notebook, ISVC,
LLMISVC, RayCluster, RayJob, PyTorchJob) with a single cluster-wide
DataIntegrityCheck that validates three invariants:
1. Every workload in a kueue-managed namespace has the queue-name label
2. Every workload with the queue-name label is in a kueue-managed namespace
3. Within each top-level CR's ownership tree, all resources agree on the
queue-name label value
Key design decisions:
- Top-down traversal from monitored CRs (Notebook, InferenceService,
LLMInferenceService, RayCluster, RayJob, PyTorchJob) through an
ownership graph of intermediate types (Deployment, StatefulSet,
ReplicaSet, DaemonSet, Job, CronJob, Pod)
- Ownership graph built once per namespace and reused across all CRs
- Pre-filter to only process kueue-relevant namespaces (union of
kueue-managed namespaces and namespaces with kueue-labeled workloads)
- Single KueueConsistency condition per run with first-violation-wins
per CR to avoid redundant diagnostics
Introduces ImpactProhibited as the highest severity level ("prohibited"),
representing conditions where an upgrade MUST NOT proceed:
- Kueue DataIntegrityCheck emits prohibited impact on label violations
- Kueue OperatorInstalledCheck escalated to prohibited when Kueue
managementState is Managed — there is no known, reliable upgrade
path from embedded Kueue; migration to the Red Hat build of Kueue
operator is required before upgrading regardless of whether RHBoK
is currently installed. The previous behavior distinguished between
Managed+operator-present (blocking) and Managed+operator-absent (pass);
both cases now emit prohibited unconditionally
Prohibited severity support across the output layer:
- SeverityLevelProhibited filter level for --severity flag
- Prohibited banner (box-drawn border) rendered above the summary
table when any prohibiteconditions are detected
- Double exclamation mark (‼) status icon for prohibited conditions
- PROHIBITED verdict in output (replaces FAIL for this severity)
- Summary line now includes Prohibited count
- Exit error message distinguishes prohibited from blocking
Refactors NotebookVerboseFormatter into EnhancedVerboseFormatter
in pkg/lint/check/:
- Handles mixed resource types by deriving per-object CRD FQN from
TypeMeta via per-object AnnotationObjectCRDName, with result-level
annotation preference for single-kind results
- Exports CRDFullyQualifiedName and DeriveCRDFQNFromTypeMeta helpers
- Supports optional per-object context via AnnotationObjectContext,
rendered as a sub-bullet beneath each impacted object reference
- All notebook checks updated to embed check.EnhancedVerboseFormatter
(drop-in replacement, no behavioral change)
Adds resource type definitions for StatefulSet, ReplicaSet, DaemonSet,
Job, and CronJob. Updates ToPartialObjectMetadata helper to preserve
UID and OwnerReferences need for ownership graph construction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Andy Stoneberg <astonebe@redhat.com>
lburgazzoli
approved these changes
Mar 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of
95906e89792cdb3e4650c405b0595755f70c5e52frommain.Original-PR: opendatahub-io#47
Replace 6 per-resource-type kueue label checks (Notebook, ISVC, LLMISVC, RayCluster, RayJob, PyTorchJob) with a single cluster-wide DataIntegrityCheck that validates three invariants:
Key design decisions:
Introduces ImpactProhibited as the highest severity level ("prohibited"), representing conditions where an upgrade MUST NOT proceed:
Prohibited severity support across the output layer:
Refactors NotebookVerboseFormatter into EnhancedVerboseFormatter in pkg/lint/check/:
Adds resource type definitions for StatefulSet, ReplicaSet, DaemonSet, Job, and CronJob. Updates ToPartialObjectMetadata helper to preserve UID and OwnerReferences need for ownership graph construction.