[Bug]: aicr validate --kubeconfig can target artifacts and validator Jobs at different clusters

[tjrasche]

Prerequisites

• I searched existing issues and found no duplicates
• I can reproduce this issue consistently
• This is not a security vulnerability (use Security Advisories instead)

Bug Description

aicr validate --kubeconfig <path> does not apply the selected kubeconfig consistently across the validation workflow.

The flag is currently honored for some operations:

• reading recipes and snapshots from cm:// URIs;
• deploying the snapshot-capture agent;
• CNCF submission collection;
• writing reports to a cm:// destination.

However, the validator engine creates the validation namespace, RBAC, input ConfigMaps, validator Jobs, watches, result ConfigMaps, and cleanup operations using the process-default Kubernetes client instead of the kubeconfig supplied on the command line.

If the default client targets cluster A and --kubeconfig targets cluster B, one aicr validate invocation can operate across both clusters. In the live-snapshot path, AICR can capture cluster B, run the actual validator Jobs against cluster A, and optionally write the resulting report back to cluster B.

This can produce a validation report that combines a snapshot from one cluster with live checks executed against another. It may also create an ephemeral cluster-admin binding on a cluster the user did not intend to validate.

This issue is intended to agree on the desired CLI contract before implementation because making --kubeconfig select the validation cluster changes its currently documented behavior.

Impact

High (major functionality broken)

Component

CLI (aicr)

Regression?

Unknown / first time using this feature

Steps to Reproduce

Use two distinct test clusters:

• cluster A: selected through the process-default KUBECONFIG;
• cluster B: selected explicitly through --kubeconfig.

Use valid local recipe and snapshot files:

export KUBECONFIG=/path/to/cluster-a.kubeconfig

aicr validate \
  --kubeconfig /path/to/cluster-b.kubeconfig \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase deployment

Watch both clusters while validation runs:

kubectl --kubeconfig /path/to/cluster-a.kubeconfig \
  get jobs -n aicr-validation --watch

kubectl --kubeconfig /path/to/cluster-b.kubeconfig \
  get jobs -n aicr-validation --watch

The validator Jobs appear on cluster A, despite cluster B being supplied through --kubeconfig.

A simpler diagnostic is to pass a nonexistent kubeconfig while using local input and output files:

export KUBECONFIG=/path/to/valid-cluster-a.kubeconfig

aicr validate \
  --kubeconfig /path/that/does/not/exist \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase deployment

The invalid explicit kubeconfig may never be loaded, and validation can proceed against cluster A.

The live-snapshot path exposes an additional inconsistency:

export KUBECONFIG=/path/to/cluster-a.kubeconfig

aicr validate \
  --kubeconfig /path/to/cluster-b.kubeconfig \
  --recipe recipe.yaml \
  --phase deployment

In that path:

1. The CLI initially creates the validation namespace using cluster A.
2. The snapshot-agent Job is deployed to cluster B.
3. The captured snapshot represents cluster B.
4. Validator Jobs are deployed to cluster A.

Expected Behavior

Proposed behavior for maintainer discussion:

When supplied to aicr validate, --kubeconfig should select the target cluster for every Kubernetes API operation performed by that invocation, including:

• snapshot-agent deployment;
• validation namespace creation;
• validation RBAC;
• input and result ConfigMaps;
• validator Job creation;
• Job watches and log/result collection;
• cleanup;
• cm:// recipe, snapshot, and output operations;
• CNCF submission collection.

The kubeconfig does not need to be mounted into validator pods. The outer AICR process should use it to create the Jobs on the selected cluster; the Jobs can continue using their in-cluster ServiceAccount credentials.

When the flag is omitted, existing default discovery through KUBECONFIG, ~/.kube/config, and in-cluster configuration should remain unchanged.

Actual Behavior

Assuming cluster A is selected by default and cluster B by the explicit flag:

Operation	Current cluster
Recipe or snapshot read from cm://	B
Preliminary namespace creation for live capture	A
Snapshot-agent Job and snapshot result	B
Validation namespace and RBAC	A
Validator Jobs, watches, results, and cleanup	A
Final report written to cm://	B
CNCF submission	B

Consequences include:

• a report can combine cluster B's snapshot with checks executed against cluster A;
• validation RBAC, including the temporary cluster-admin binding, can be created on the unintended cluster;
• a valid explicit kubeconfig can fail to help when the default cluster is unavailable;
• an invalid explicit kubeconfig can be silently ignored when recipe, snapshot, and output are local files;
• exporting KUBECONFIG often masks the problem because both paths then happen to resolve to the same cluster.

Environment

• AICR version (CLI aicr --version, API image tag, or commit SHA): main at 2b1be85
• Install method (release binary / build from source / container image): build from source
• Platform (eks/gke/aks/oke/ocp/kind/lke/bcm/other): any two distinct reachable Kubernetes clusters
• Kubernetes version: not platform-specific
• OS (ubuntu/cos/other) + version: not platform-specific
• Kernel version: not platform-specific
• GPU type (h100/h200/gb200/b200/a100/l40/l40s/rtx-pro-6000/other): not GPU-specific
• Workload intent (training/inference): not intent-specific

Command / Request Used

export KUBECONFIG=/path/to/cluster-a.kubeconfig

aicr validate \
  --kubeconfig /path/to/cluster-b.kubeconfig \
  --recipe recipe.yaml \
  --snapshot snapshot.yaml \
  --phase deployment

Logs / Error Output

Additional Context

Relevant code paths:

• The CLI passes kubeconfig to recipe and snapshot loading, but the ValidateOption list does not contain a validation kubeconfig option: pkg/cli/validate.go.
• The validator obtains its client exclusively through GetKubeClient(): pkg/validator/validator.go.
• The live-capture helper creates the namespace with the default client before passing the explicit kubeconfig to the snapshot agent: pkg/cli/validate.go.
• The current CLI reference describes --kubeconfig only in terms of ConfigMap input/output: docs/user/cli-reference.md.
• Related but not duplicate: refactor(serializer): add kubeconfig-aware NewFileWriterOrStdoutWithKubeconfig and update all CLI callers #826 addressed kubeconfig propagation for ConfigMap output.

Proposed implementation direction

Subject to maintainer agreement:

• Introduce a run-scoped validation kubeconfig option through pkg/client/v1 and pkg/validator.
• Pass the CLI's resolved kubeconfig into that option.
• Construct one client for the validation run and reuse it for namespace, RBAC, ConfigMaps, Jobs, watches, results, and cleanup.
• Remove the redundant preliminary namespace creation in the live-snapshot path, or make it use the same explicit client.
• Add regression coverage with distinct explicit and default clients, proving all validation operations use the explicit client.
• Update the CLI reference to describe the agreed end-to-end semantics.

Compatibility question

This changes behavior for a caller that currently supplies --kubeconfig only for ConfigMap artifact I/O while intentionally relying on the default client for validator Jobs.

That split-cluster behavior seems surprising and unsafe as an implicit default, but maintainers should decide whether:

1. --kubeconfig is an invocation-wide target-cluster selector; or
2. split-cluster workflows are supported explicitly through separate options such as --artifact-kubeconfig and --validation-kubeconfig.

The proposal favors option 1: one aicr validate invocation should use one cluster unless a future workflow explicitly models multiple clusters.

[github-actions]

Welcome to AICR, @tjrasche! Thanks for opening your first issue.

A maintainer will triage this shortly. In the meantime:

• Check the Contributing Guide for project conventions
• Browse existing labels for related topics
• Join the conversation in Discussions for questions

[tjrasche]

I can do the fix for this.

Just wanted to discuss before as it would be a change to current CLI behaviour.

prerequisite PRs would be:

• PR #1743 — Fix kubeconfig error classification
• PR #1735 — Add custom kubeconfig support for validation Jobs to the SDK