refactor(test): build-tag OCP scheme registrations and drop legacy SetupEnvTest (opendatahub-io#1329)

* refactor: guard OCP scheme registrations with distro build tags

Prometheus Operator monitoring APIs, OpenShift Route, and Istio v1
scheme registrations were wired unconditionally, pulling OCP-only
dependencies into upstream builds. This makes the upstream binary carry
dead scheme registrations and pulls in imports that don't belong there.

Uses the established companion file pattern (//go:build distro /
!distro) to gate these registrations, keeping the upstream build surface
clean while preserving full functionality in distro builds.

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

* refactor(test): migrate credentials suite to NewEnvTest, drop SetupEnvTest

The credentials suite was the last caller of the legacy SetupEnvTest
path. Migrate it to NewEnvTest().BuildEnvironment(), which uses a fresh
per-suite scheme (t.Scheme) instead of mutating the global scheme.Scheme.

With no callers left, remove SetupEnvTest and its build-tag companion
files (envtest_setup_default.go / envtest_setup_ocp.go), collapsing the
two test setup paths into one.

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

* simplify: drop manual capacity hints in scheme slice construction

Replace make+append pairs with the append([]T{...}, rest...) idiom.
The magic number capacity hints (5+len(distroSchemes), 8+len(...), etc.)
silently drift whenever entries are added or removed. The composite
literal approach allocates exactly once with no number to maintain.

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

* fix: wire GOTAGS through Dockerfile and docker-build for main controller

routev1.AddToScheme was moved behind //go:build distro in
main_schemes_ocp.go but the main Dockerfile had no ARG GOTAGS and the
docker-build Makefile target was not passing --build-arg GOTAGS.
This meant the binary was always compiled without the distro tag,
main_schemes_default.go (no-op) was linked, and v1.Route was never
registered - causing the InferenceGraph controller to crash on startup.

Follow the pattern already in place for llmisvc-controller.Dockerfile
and docker-build-llmisvc.

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

* docs(rules): extend build-tags rule to cover Dockerfiles and Makefile targets

Adds rules 7 and 8 to catch the class of bug where a Dockerfile builds
a Go binary that uses //go:build distro code but does not declare
ARG GOTAGS or pass -tags to go build - and the corresponding Makefile
target does not pass --build-arg GOTAGS.

The canonical references are llmisvc-controller.Dockerfile (correct)
and the main Dockerfile before this fix (incorrect).

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

---------

Signed-off-by: Bartosz Majsak <bartosz.majsak@gmail.com>

Author: bartoszmajsak

.rules/build-tags.md

@@ -36,6 +36,38 @@ lives in `*_ocp.go` files compiled only with `//go:build distro`; the upstream f
    full repository tree (diff-only mode), flag tentatively and ask the author to confirm whether a
    `*_default.go` companion exists in the same package.
 
+## Build system propagation rules
+
+These rules apply when the diff touches **Dockerfiles**, **Makefile targets**, **Tekton PipelineRuns**,
+or any other CI/build system file that builds Go binaries importing `pkg/scheme` or any
+`//go:build distro` gated code.
+
+The core principle: `GOTAGS=distro` must flow unbroken from the build system into `go build`.
+Every layer in the chain is a potential break point.
+
+7. **Dockerfile missing GOTAGS support** - Any Dockerfile that compiles a Go binary which imports
+   `pkg/scheme` (or any package with `//go:build distro` companion files) must declare `ARG GOTAGS`
+   in the builder stage and pass `-tags "${GOTAGS}"` to `go build`. Without this, `*_ocp.go` files
+   are silently skipped - causing missing scheme registrations, CRD watch failures, or runtime
+   crashes. Two valid patterns:
+   - `ARG GOTAGS=""` when the caller always supplies the value (Makefile / generic CI).
+   - `ARG GOTAGS="distro"` when the Dockerfile is exclusively used for distro builds (Konflux).
+   Canonical references: `llmisvc-controller.Dockerfile` (Makefile pattern),
+   `Dockerfiles/llmisvc-controller.Dockerfile.konflux` (Konflux pattern).
+
+8. **Build system invocation not passing GOTAGS** - Every invocation of a Dockerfile covered by
+   rule 7 must pass `GOTAGS=distro` through that build system's mechanism for Docker build
+   arguments. The mechanism varies by system - check each caller type in the diff:
+   - **Makefile**: `--build-arg GOTAGS=${GOTAGS}` on the `buildx build` call.
+     Canonical references: `docker-build` and `docker-build-llmisvc` in `Makefile`.
+   - **Tekton PipelineRun** (`.tekton/*.yaml`): `build-args: ["GOTAGS=distro"]` in `spec.params`.
+     Canonical reference: `odh-kserve-llmisvc-controller-pull-request.yaml` in `.tekton/`.
+   - **Konflux Dockerfiles** (`Dockerfiles/*.Dockerfile.konflux`): prefer `ARG GOTAGS="distro"`
+     as the default so the pipeline does not need to pass it explicitly.
+     Canonical reference: `Dockerfiles/llmisvc-controller.Dockerfile.konflux`.
+   If a new build system is introduced, apply the same principle: locate the equivalent of
+   `--build-arg` for that system and verify GOTAGS reaches the compiler.
+
 ## Exemptions - do not flag
 
 - Files under a `distro/` sub-package (e.g. `pkg/controller/.../distro/controller_rbac_ocp.go`)

.tekton/kserve-controller-pull.yaml

@@ -32,6 +32,9 @@ spec:
   - name: additional-tags
     value:
     - 'master-pr-{{revision}}'
+  - name: build-args
+    value:
+    - "GOTAGS=distro"
   - name: pipeline-type
     value: pull-request
   - name: enable-group-testing

Dockerfile

@@ -13,11 +13,12 @@ RUN --mount=type=cache,target=/go/pkg/mod \
 FROM deps AS builder
 
 ARG CMD=manager
+ARG GOTAGS=""
 COPY cmd/${CMD}/ cmd/${CMD}/
 COPY pkg/    pkg/
 RUN --mount=type=cache,target=/go/pkg/mod \
     --mount=type=cache,target=/root/.cache/go-build \
-    CGO_ENABLED=0 GOOS=linux GOFLAGS=-mod=readonly go build -a -o manager ./cmd/${CMD}
+    CGO_ENABLED=0 GOOS=linux GOFLAGS=-mod=readonly go build -a -tags "${GOTAGS}" -o manager ./cmd/${CMD}
 
 # ---- License stage (parallel with build on BuildKit) ----
 FROM deps AS license

Makefile

@@ -469,7 +469,7 @@ bump-version:
 
 # Build the docker image
 docker-build:
-	${ENGINE} buildx build ${ARCH} . -t ${KO_DOCKER_REPO}/${CONTROLLER_IMG}
+	${ENGINE} buildx build ${ARCH} --build-arg GOTAGS=${GOTAGS} . -t ${KO_DOCKER_REPO}/${CONTROLLER_IMG}
 	@echo "updating kustomize image patch file for manager resource"
 
 	# Use perl instead of sed to avoid OSX/Linux compatibility issue:

cmd/manager/main.go

@@ -37,8 +37,6 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/webhook"
 	"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
 
-	routev1 "github.com/openshift/api/route/v1"
-
 	"github.com/kserve/kserve/pkg/apis/serving/v1alpha1"
 	"github.com/kserve/kserve/pkg/apis/serving/v1beta1"
 	"github.com/kserve/kserve/pkg/constants"
@@ -149,8 +147,8 @@ func main() {
 		setupLog.Error(err, "unable to register API schemes")
 		os.Exit(1)
 	}
-	if err := routev1.AddToScheme(mgr.GetScheme()); err != nil {
-		setupLog.Error(err, "unable to add routev1 APIs to scheme")
+	if err := registerDistroSchemes(mgr.GetScheme()); err != nil {
+		setupLog.Error(err, "unable to register distro-specific API schemes")
 		os.Exit(1)
 	}
 

cmd/manager/main_schemes_default.go

@@ -0,0 +1,26 @@
+//go:build !distro
+
+/*
+Copyright 2026 The KServe Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import "k8s.io/apimachinery/pkg/runtime"
+
+// registerDistroSchemes is a no-op in upstream builds.
+func registerDistroSchemes(_ *runtime.Scheme) error {
+	return nil
+}

cmd/manager/main_schemes_ocp.go

@@ -0,0 +1,29 @@
+//go:build distro
+
+/*
+Copyright 2026 The KServe Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	routev1 "github.com/openshift/api/route/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+// registerDistroSchemes registers OCP-specific API schemes (e.g. OpenShift Routes).
+func registerDistroSchemes(s *runtime.Scheme) error {
+	return routev1.AddToScheme(s)
+}

pkg/credentials/credentials_suite_test.go

@@ -18,11 +18,9 @@ package credentials
 
 import (
 	"os"
-	"path/filepath"
 	"testing"
 
 	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/kubernetes/scheme"
 	"k8s.io/client-go/rest"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
@@ -36,16 +34,13 @@ var (
 )
 
 func TestMain(m *testing.M) {
-	crdDirectoryPaths := []string{
-		filepath.Join(pkgtest.ProjectRoot(), "test", "crds"),
-	}
-	t := pkgtest.SetupEnvTest(crdDirectoryPaths)
+	t := pkgtest.NewEnvTest().BuildEnvironment()
 	var err error
 	if cfg, err = t.Start(); err != nil {
 		log.Error(err, "Failed to start testing panel")
 	}
 
-	if c, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}); err != nil {
+	if c, err = client.New(cfg, client.Options{Scheme: t.Scheme}); err != nil {
 		log.Error(err, "Failed to start client")
 	}
 	if clientset, err = kubernetes.NewForConfig(cfg); err != nil {

pkg/scheme/register.go

@@ -21,7 +21,6 @@ import (
 	wvav1alpha1 "github.com/llm-d/llm-d-workload-variant-autoscaler/api/v1alpha1"
 	otelv1beta1 "github.com/open-telemetry/opentelemetry-operator/apis/v1beta1"
 	"github.com/pkg/errors"
-	monitoringv1 "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
 	istioclientv1beta1 "istio.io/client-go/pkg/apis/networking/v1beta1"
 	appsv1 "k8s.io/api/apps/v1"
 	autoscalingv2 "k8s.io/api/autoscaling/v2"
@@ -45,6 +44,10 @@ import (
 
 type addToSchemeFunc func(scheme *runtime.Scheme) error
 
+// distroSchemes collects distribution-specific scheme registration functions.
+// Populated via init() in build-tagged companion files (register_ocp.go).
+var distroSchemes []addToSchemeFunc
+
 // AddKServeAPIs registers all KServe APIs.
 func AddKServeAPIs(s *runtime.Scheme) error {
 	return addAll(s,
@@ -106,12 +109,6 @@ func AddOpenTelemetryAPIs(s *runtime.Scheme) error {
 	return addAll(s, otelv1beta1.AddToScheme)
 }
 
-// AddMonitoringAPIs registers Prometheus Operator monitoring APIs (PodMonitor, ServiceMonitor).
-// The scheme registration is unconditional; actual CRD availability is checked at watch setup time.
-func AddMonitoringAPIs(s *runtime.Scheme) error {
-	return addAll(s, monitoringv1.AddToScheme)
-}
-
 // AddControllerAPIs registers the baseline controller APIs used by production and tests.
 func AddControllerAPIs(s *runtime.Scheme) error {
 	return addAll(s,
@@ -122,19 +119,18 @@ func AddControllerAPIs(s *runtime.Scheme) error {
 
 // AddLLMISVCAPIs registers API groups required by the llmisvc manager.
 func AddLLMISVCAPIs(s *runtime.Scheme) error {
-	return addAll(s,
+	return addAll(s, append([]addToSchemeFunc{
 		AddControllerAPIs,
 		AddGatewayAPIs,
 		AddLeaderWorkerSetAPIs,
-		AddMonitoringAPIs,
 		AddKedaAPIs,
 		AddWVAAPIs,
-	)
+	}, distroSchemes...)...)
 }
 
 // AddAll registers all API groups supported by KServe managers and envtest suites.
 func AddAll(s *runtime.Scheme) error {
-	return addAll(s,
+	return addAll(s, append([]addToSchemeFunc{
 		AddControllerAPIs,
 		AddGatewayAPIs,
 		AddLeaderWorkerSetAPIs,
@@ -143,8 +139,7 @@ func AddAll(s *runtime.Scheme) error {
 		AddKedaAPIs,
 		AddWVAAPIs,
 		AddOpenTelemetryAPIs,
-		AddMonitoringAPIs,
-	)
+	}, distroSchemes...)...)
 }
 
 func addAll(s *runtime.Scheme, fns ...addToSchemeFunc) error {

pkg/scheme/register_ocp.go

@@ -0,0 +1,33 @@
+//go:build distro
+
+/*
+Copyright 2026 The KServe Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scheme
+
+import (
+	monitoringv1 "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+func init() {
+	distroSchemes = append(distroSchemes, AddMonitoringAPIs)
+}
+
+// AddMonitoringAPIs registers Prometheus Operator monitoring APIs (PodMonitor, ServiceMonitor).
+func AddMonitoringAPIs(s *runtime.Scheme) error {
+	return addAll(s, monitoringv1.AddToScheme)
+}