Bump Karpenter from 1.10.0 to 1.12.0 to fix B200 capacity reservation panic (#503)

**Impact:** All clusters using Karpenter — B200 GPU node provisioning is
currently broken in production
**Risk:** medium

## What
Upgrades the Karpenter Helm chart from v1.10.0 to v1.12.0 and adds the
`ec2:DescribeInstanceStatus` IAM permission required by the new version.

## Why
Karpenter v1.10.0 has a nil pointer dereference bug in
`CapacityReservationFromEC2`
([aws/karpenter-provider-aws#9019](aws/karpenter-provider-aws#9019))
that panics when processing capacity-block reservations where the
`Interruptible` field is nil — which is standard EC2 API behavior. This
causes the controller to crash every ~30s (132 panics/hour observed),
making it impossible to provision B200 nodes. Three B200 runner pods
were stuck Pending for up to 10 hours on `pytorch-arc-cbr-production`.
The fix
([aws/karpenter-provider-aws#9080](aws/karpenter-provider-aws#9080))
shipped only in v1.12.0 — no patch exists for v1.10.x or v1.11.x.
v1.11.0 is skipped due to a known CPU regression.

## How
- Bumped the Helm chart version directly from v1.10.0 to v1.12.0,
skipping v1.11.x (CPU regression in v1.11.0, no bugfix backport to
v1.11.1)
- Added `ec2:DescribeInstanceStatus` to the `AllowRegionalReadActions`
IAM policy statement — v1.12.0's interruption controller requires this
for instance health checks

## Changes
- `modules/karpenter/deploy.sh`: Chart version `1.10.0` → `1.12.0`
- `modules/karpenter/terraform/main.tf`: Added
`ec2:DescribeInstanceStatus` to `AllowRegionalReadActions` IAM policy

## Notes
- **Deploy order matters**: Terraform (IAM) must be applied before the
Helm upgrade, otherwise the controller will fail
`DescribeInstanceStatus` calls with AccessDenied
- **Expected node drift**: v1.12.0 introduces CA bundle drift detection.
Existing nodes will be marked as drifted and gradually replaced per
disruption budgets — this is expected behavior, not a failure
- **Post-upgrade**: B200 pods should schedule within minutes once the
CapacityReservation reconciler stops panicking and provisions a node
from `cr-0c366fb8339a10f69`

## Testing
```
============================================================================================================================================ test session starts ============================================================================================================================================
platform darwin -- Python 3.13.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/jschmidt/meta/ci-infra-code-review/osdc
configfile: pyproject.toml
plugins: anyio-4.12.1, xdist-3.8.0, cov-7.0.0
16 workers [211 items]
...........................................................................................................................s.............................................s.....................................s.s.                                                                                   [100%]
========================================================================================================================================== short test summary info ==========================================================================================================================================
SKIPPED [1] modules/cache-enforcer/tests/smoke/test_cache_enforcer.py:107: No cache-enforcer pods desired (no runner nodes in cluster)
SKIPPED [1] modules/monitoring/tests/smoke/test_monitoring.py:173: No dcgm-exporter pods found (no GPU nodes)
SKIPPED [1] modules/cache-enforcer/tests/smoke/test_cache_enforcer.py:346: No cache-enforcer pods found (no runner nodes in cluster)
SKIPPED [1] modules/cache-enforcer/tests/smoke/test_cache_enforcer.py:306: No cache-enforcer pods found (no runner nodes in cluster)
================================================================================================================================ 207 passed, 4 skipped in 120.41s (0:02:00) =================================================================================================================================

Smoke tests completed in 2m2s
```
```
============================================================
  OSDC Integration Test Results
============================================================
  Cluster: arc-staging (pytorch-arc-staging)
  Date:    2026-04-26 05:26 UTC

  PR Workflow Jobs:
    ✓ test-pypi-cache-defaults       success
    ✓ test-git-cache                 success
    ✓ test-cpu-x86-amx               success
    ✓ test-pypi-cache-action-cpu     success
    ✓ test-cpu-x86-avx512            success
    ✓ test-pypi-cache-action-cuda    success
    ✓ test-cpu-arm64                 success
    ✓ test-gpu-t4                    success
    ✓ test-gpu-t4-multi              success
    ✓ test-harbor                    success
    ✓ test-cache-enforcer            success
    ✓ test-release-arm64             success
    ✓ build-amd64 / build            success
    ✓ build-arm64 / build            success

  Smoke            ⊘ SKIPPED
  Compactor        ⊘ SKIPPED

  Overall: PASSED
============================================================
```

---------

Signed-off-by: Jean Schmidt <contato@jschmidt.me>

Author: jeanschmidt

osdc/justfile

@@ -333,7 +333,7 @@ deploy-base cluster:
     trap 'deploy_log_finish cmd "$CLUSTER" "deploy-base" "$DEPLOY_LOG_START" failed' ERR
 
     echo ""
-    echo "━━━ BASE: Terraform ━━━"
+    echo "━━━ modules/eks/terraform ━━━"
     cd {{UPSTREAM}}/modules/eks/terraform
     tofu init -reconfigure \
         -backend-config="bucket=${BUCKET}" \

osdc/modules/eks/terraform/modules/harbor/outputs.tf

@@ -5,7 +5,7 @@ output "s3_bucket_name" {
 
 output "s3_bucket_region" {
   description = "Region of the S3 bucket for Harbor registry storage"
-  value       = aws_s3_bucket.harbor_registry.region
+  value       = aws_s3_bucket.harbor_registry.bucket_region
 }
 
 output "role_arn" {

osdc/modules/karpenter/deploy.sh

@@ -52,6 +52,30 @@ KARPENTER_LOG_LEVEL=$(uv run "$CFG" "$CLUSTER" karpenter.log_level info)
 KARPENTER_PDB_ENABLED=$(uv run "$CFG" "$CLUSTER" karpenter.pdb_enabled true)
 KARPENTER_PDB_MIN=$(uv run "$CFG" "$CLUSTER" karpenter.pdb_min_available 1)
 
+KARPENTER_VERSION="1.12.0"
+
+# Helm does not update CRDs on `helm upgrade` — only on initial install.
+# CRDs are in a separate chart (karpenter-crd), not the main karpenter chart.
+# On first migration, existing CRDs owned by the `karpenter` release must be
+# relabeled so `karpenter-crd` can adopt them.
+KARPENTER_CRDS=$(kubectl get crds -o name 2>/dev/null \
+  | grep -E "karpenter\.(sh|k8s\.aws)" || true)
+if [ -n "$KARPENTER_CRDS" ]; then
+  for crd in $KARPENTER_CRDS; do
+    kubectl label "$crd" app.kubernetes.io/managed-by=Helm --overwrite
+    kubectl annotate "$crd" \
+      meta.helm.sh/release-name=karpenter-crd \
+      meta.helm.sh/release-namespace=karpenter --overwrite
+  done
+fi
+
+echo "Updating Karpenter CRDs to v${KARPENTER_VERSION}..."
+helm upgrade --install karpenter-crd \
+  "oci://public.ecr.aws/karpenter/karpenter-crd" \
+  --version "${KARPENTER_VERSION}" \
+  --namespace karpenter --create-namespace \
+  --timeout 5m --wait
+
 echo "Installing Karpenter..."
 helm_upgrade_if_changed karpenter karpenter \
   --create-namespace \
@@ -68,5 +92,5 @@ helm_upgrade_if_changed karpenter karpenter \
   --timeout 10m \
   --wait \
   oci://public.ecr.aws/karpenter/karpenter \
-  --version 1.10.0
+  --version "${KARPENTER_VERSION}"
 echo "Karpenter installed."

osdc/modules/karpenter/terraform/main.tf

@@ -187,6 +187,7 @@ resource "aws_iam_policy" "karpenter_controller" {
           "ec2:DescribeCapacityReservations",
           "ec2:DescribeImages",
           "ec2:DescribeInstances",
+          "ec2:DescribeInstanceStatus",
           "ec2:DescribeInstanceTypeOfferings",
           "ec2:DescribeInstanceTypes",
           "ec2:DescribeLaunchTemplates",