fix(evidence): restore --cncf-submission behavioral evidence collection

[yuanchen8911]

Summary

Restore the --cncf-submission behavioral evidence collection feature that was inadvertently removed by PR #290 (container-per-validator execution engine), plus fix several pre-existing bugs in the evidence collection script.

Motivation / Context

PR #290 refactored the validation engine and deleted the --cncf-submission flag, --feature flag, runCNCFSubmission() function, and pkg/evidence/collector.go that were originally added in PR #214. The docs still reference these flags but the implementation was gone.

Fixes: N/A
Related: #290, #214

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/api, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: pkg/evidence

Implementation Notes

Restored files (from latest pre-PR#290 state, not the original PR #214):

• pkg/evidence/collector.go — behavioral evidence collector (shell script orchestrator)
• pkg/evidence/collector_test.go — unit tests for feature resolution, script sections, collector options
• pkg/evidence/scripts/collect-evidence.sh — 1237-line evidence collection script

Bug fixes in the script:

Fix	Issue	Solution
DCGM metrics empty	kubectl run curl pod raced against DNS; DCGM container too minimal for kubectl exec	Port-forward to DCGM service with retry loop
DCGM result false FAIL	Stale dcgm_pod variable reference after refactor to dcgm_svc	Fixed variable name
ASG details empty	Custom ASGs lack eks:nodegroup-name tag; multi-line None broke string comparison	Strip whitespace + instance ID fallback via describe-auto-scaling-instances
ELB hostname exposed	Public endpoint in evidence docs	Post-processing sed redaction
NO_CLEANUP broken	Skipped both pre-run and post-run cleanup	cleanup_ns takes pre/post phase; pre-run always cleans stale resources

CLI additions:

• --cncf-submission flag triggers behavioral evidence collection (bypasses normal validation)
• --feature/-f flag for selective feature collection
• --kubeconfig propagated to evidence script via KUBECONFIG env var
• Flag validation: --cncf-submission requires --evidence-dir, --feature requires --cncf-submission

Testing

go test ./pkg/evidence/ ./pkg/cli/ -race -count=1
# ok  github.com/NVIDIA/aicr/pkg/evidence  1.628s
# ok  github.com/NVIDIA/aicr/pkg/cli       1.902s

Verified end-to-end on EKS cluster with H100 GPUs — all 8 evidence features collected successfully with all bug fixes confirmed.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: N/A — restores previously existing functionality

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)