add concise design document for ADR

Author: cadenmarchese

hive_removal_design.md

@@ -106,9 +106,10 @@ kubernetescli, err := kubernetes.NewForConfig(restConfig)
 
 **AKS Cluster Connection:**
 - Uses existing `getAksShardKubeconfig()` function
-- Connects to `aro-aks-cluster-{shard:03d}` (same clusters where Hive runs)
+- **Connects to the same AKS clusters where Hive currently runs** (`aro-aks-cluster-{shard:03d}`)
 - Retrieves admin credentials via `ListClusterAdminCredentials()`
 - Caches credentials in liveconfig with RWMutex
+- **No new infrastructure required** - reuses existing AKS clusters and client setup from Hive
 
 ### 4. Monitoring and Error Handling
 
@@ -137,6 +138,9 @@ Adapt `pkg/hive/failure/handler.go` patterns to parse installer logs and map to
 
 Return structured CloudError for user-facing error messages.
 
+**Log Parsing Migration:**
+The RP already parses install logs from Hive's output via regex matching. This logic needs to be reimplemented directly in `pkg/aksinstall` to parse logs from the provision pod instead of from Hive's nested output. Since we'll be reading installer logs directly (not through Hive's wrapper), this change will **simplify the parsing logic** - no need to extract installer output from Hive's formatting, just read the pod logs directly.
+
 **Cleanup Strategy:**
 - **On success:** Delete pod, delete all secrets, delete namespace
 - **On failure:** Delete pod, delete secrets, **keep namespace** for 24h debugging
@@ -179,11 +183,19 @@ func (m *manager) runAKSInstaller(ctx context.Context) error {
 **Modify `bootstrap()` method** (around line 440-470):
 - **Remove:** `hiveCreateNamespace`, `runHiveInstaller`, `hiveClusterInstallationComplete` steps
 - **Remove:** `hiveEnsureResources`, `hiveClusterDeploymentReady` steps (adoption flow)
-- **Replace with:** Direct call to `runAKSInstaller` with same pattern as `runPodmanInstaller`
+- **Replace with:** Environment-based routing to enable gradual rollout
 
-**Before:**
+**Implementation with Environment Toggle:**
 ```go
-if m.installViaHive {
+// Check environment variable to enable AKS installer mode
+// This allows testing in local dev, INT, and canary before full rollout
+if m.env.DeploymentMode() == deployment.Development || m.env.FeatureIsSet(env.FeatureUseAKSInstaller) {
+    s = append(s,
+        steps.Action(m.runAKSInstaller),
+        steps.Action(m.generateKubeconfigs),
+    )
+} else if m.installViaHive {
+    // Legacy Hive path (to be removed after validation)
     s = append(s,
         steps.Action(m.hiveCreateNamespace),
         steps.Action(m.runHiveInstaller),
@@ -193,14 +205,13 @@ if m.installViaHive {
 }
 ```
 
-**After:**
-```go
-// All cluster installations now use AKS direct
-s = append(s,
-    steps.Action(m.runAKSInstaller),
-    steps.Action(m.generateKubeconfigs),
-)
-```
+**Feature Flag:**
+- Add `FeatureUseAKSInstaller` to `pkg/env/env.go`
+- Enable in local development mode by default
+- Enable in INT/canary via environment variable for testing
+- Once validated, remove Hive path and feature flag entirely
+
+This approach allows safe testing in controlled environments before full rollout.
 
 ### 6. Complete Hive Removal
 
@@ -315,6 +326,34 @@ m.emitGauge("aksinstall.pod.phase", 1, map[string]string{"phase": podPhase})
 
 This ensures we have equal or better observability compared to Hive monitoring, but during installation rather than as periodic monitoring.
 
+### 9. Alerting and Incident Automation
+
+**Alerting Updates:**
+Our existing Hive alerting monitors pods in the AKS clusters where Hive runs. Since the new installer pods will run in the **same AKS clusters**, the alerting infrastructure requires minimal changes:
+- **Alert queries need updated namespace filters**: Change from monitoring Hive-managed namespaces to monitoring `aro-*` namespaces
+- **Alert names may need updates**: Rename from "Hive installer failure" to "ARO installer failure" for clarity
+- No infrastructure changes required - already monitoring non-Hive namespaced pods in these AKS clusters
+
+**Incident Automation Updates:**
+The automation between alert firing and incident creation currently assumes:
+- Installer logs are nested within Hive's output format
+- Specific Hive metadata is available in logs
+- ClusterDeployment status fields exist
+
+**Changes Required:**
+1. **Log query simplification**: Queries become simpler since installer output is direct (not nested in Hive formatting)
+2. **Remove Hive-specific fields**: Stop querying ClusterDeployment status, Hive conditions, etc.
+3. **Update log parsing**: Read pod logs directly from `aro-{docID}` namespaces instead of from Hive controller logs
+4. **Preserve correlation data**: Ensure pod labels/annotations include request IDs, client principal, subscription ID for incident routing
+
+**Example Query Change:**
+```
+Before: kubectl logs -n hive hive-clustersync-0 | grep "cluster-{uuid}" | extract installer output
+After:  kubectl logs -n aro-{uuid} installer-{uuid} | direct installer output
+```
+
+The automation becomes **simpler and more reliable** since we eliminate the indirection through Hive's log aggregation.
+
 ## Critical Files
 
 ### Files to Create:
@@ -327,14 +366,17 @@ This ensures we have equal or better observability compared to Hive monitoring,
 7. `pkg/aksinstall/metrics.go` - Metrics helper methods (emitGauge, emitFloat)
 
 ### Files to Modify:
-7. `pkg/cluster/install.go` - Add `runAKSInstaller()`, modify `bootstrap()` to remove Hive paths
+7. `pkg/cluster/install.go` - Add `runAKSInstaller()`, modify `bootstrap()` to add environment-based routing
 8. `pkg/cluster/cluster.go` - Remove `hiveClusterManager` field, add `liveConfig` if not present
 9. `pkg/backend/openshiftcluster.go` - Remove Hive client creation
 10. `pkg/monitor/monitor.go` - Remove Hive monitor builder
 11. `pkg/frontend/frontend.go` - Remove SyncSet admin API routes and hiveSyncSetManager field
 12. `pkg/util/liveconfig/hive.go` - Remove Hive feature flags, optionally rename `HiveRestConfig`
-13. `go.mod` - Remove Hive dependencies
-14. `pkg/api/go.mod` - Remove Hive dependencies if present
+13. `pkg/env/env.go` - Add `FeatureUseAKSInstaller` feature flag
+14. `go.mod` - Remove Hive dependencies (after full rollout)
+15. `pkg/api/go.mod` - Remove Hive dependencies if present (after full rollout)
+16. **Alerting configurations** - Update namespace filters from Hive namespaces to `aro-*` namespaces
+17. **Incident automation queries** - Simplify log extraction to read directly from installer pods
 
 ### Files to Delete:
 15. `pkg/hive/*.go` - All files in hive package
@@ -408,7 +450,12 @@ After implementation, verify the changes work correctly:
 
 **Risk 3: Direct Replacement (No Gradual Rollout)**
 - Impact: All new clusters immediately use new code path, higher risk if bugs exist
-- Mitigation: Thorough testing in dev/int, fast rollback capability, comprehensive error handling
+- Mitigation: **Environment-based feature flag** (`FeatureUseAKSInstaller`) enables gradual rollout:
+  - Enabled in local development by default for testing
+  - Enabled in INT environment for integration testing
+  - Enabled in canary/production regions for controlled rollout
+  - Fast rollback by disabling feature flag if issues arise
+  - Thorough testing at each stage before broader rollout
 
 ## Out of Scope
 

hive_removal_design_concise.md

@@ -0,0 +1,130 @@
+# Hive Removal - Implementation Overview
+
+## Current State
+
+Today, ARO-RP uses Hive (an external OpenShift controller) as an intermediary to manage cluster installations:
+- ARO-RP creates a `ClusterDeployment` custom resource in AKS
+- Hive watches this resource and creates/manages the installer pod
+- ARO-RP polls Hive for installation status
+
+## Proposed Solution
+
+Rather than Hive, ARO-RP itself will create and manage OpenShift installer pods directly in AKS. We already have a working implementation for local development (`pkg/containerinstall`) that runs the installer in podman. This proposal adapts that same pattern to use Kubernetes clients for production AKS clusters.
+
+## Implementation Overview
+
+### 1. New ARO-RP Package: `pkg/aksinstall`
+
+We will create installer pods directly in AKS using the Kubernetes API using the same `ContainerInstaller` interface as the existing podman-based installer. There will be shared logic between this new package and the existing `pkg/containerinstall`, but with different clients. Re-use of the existing `pkg/containerinstall` package with environment-specific configuration is also possible, if preferred.
+
+**Installation flow** (same steps used today in Hive):
+```
+1. Create namespace (aro-{clusterID})
+2. Create secrets (cluster config, credentials, manifests)
+3. Create installer pod
+4. Monitor pod until completion (60min timeout)
+5. Cleanup (delete pod/secrets, keep namespace on failure for debugging)
+```
+
+### 2. Infrastructure Reuse
+
+- We will re-use the same AKS clusters where Hive currently runs (`aro-aks-cluster-{shard}`) as well as AKS client code from `pkg/util/liveconfig/hive.go`
+- We will continue the same namespace pattern, same credential retrieval, and same admin access
+
+At first, the installer pods will be neighbors to the current Hive pods, then Hive will be removed.
+
+### 3. Pod Specification
+
+There are no changes required here. The installer pod runs the standard OpenShift installer with mounted configuration. It uses an OpenShift installer image from version catalog and runs `openshift-install create manifests && openshift-install create cluster`.
+
+**Installer Pod Contents**:
+- Azure credentials and cluster config from Kubernetes secrets
+- Custom manifests (workload identity, disabled samples)
+- Temporary volumes for installer working directories
+
+**Installer Pod Labels** for identification and monitoring:
+- `aro-cluster-id: {uuid}`
+- `aro-installer: true`
+
+### 4. Operability and Observability
+
+There will be a small amount of work required to transfer over our exiting Hive observability to the new ARO-RP flow:
+
+**Pod monitoring** (`pkg/aksinstall/pod.go`):
+- Implement `podFinished()` using pattern from `pkg/containerinstall/install.go:170-186` (containerFinished)
+- Replace podman API (`containers.Inspect`) with K8s API (`pods.Get().Status.Phase`)
+- On failure, replace `getContainerLogs()` with `kubernetescli.CoreV1().Pods().GetLogs()`
+- Keep 60-minute timeout from existing implementation
+
+**Error parsing / regex changes** (`pkg/aksinstall/failure.go`):
+- Copy regex patterns and error mapping from `pkg/hive/failure/handler.go`
+- Remove Hive log unwrapping logic (installer output is direct, not nested in Hive's JSON)
+- Keep CloudError type mappings: `AzureRequestDisallowedByPolicy`, `AzureZonalAllocationFailed`, etc.
+
+**New Metrics Required** (`pkg/aksinstall/install.go`):
+- Add metrics emission similar to Hive's current metrics but with `aksinstall.*` prefix
+- Installation lifecycle: `started`, `succeeded`, `failed`, `duration`, `failed.reason`
+
+**Alerting and IcM automation changes**:
+- Update namespace filters: `namespace=~"hive.*"` → `namespace=~"aro-.*"`
+- Rename alert titles for clarity
+- Simplify log queries: remove Hive controller log parsing, read pod logs directly
+- Remove ClusterDeployment CRD queries
+- Update correlation data source: pod labels instead of ClusterDeployment annotations
+
+### 5. Rollout Strategy
+
+**Environment-based rollout using RP_FEATURES / RP-Config**:
+
+The changes can be fully rolled out before we go live using RP-Config / RP features. We will keep Hive intact during the rollout in case there are issues. We can easily disable the RP feature flag to revert to Hive path if issues arise.
+
+```go
+if m.env.DeploymentMode() == development || m.env.FeatureIsSet(FeatureUseAKSInstaller) {
+    // New path: direct AKS installer
+    s = append(s,
+        steps.Action(m.runAKSInstaller),
+        steps.Action(m.generateKubeconfigs),
+    )
+} else if m.installViaHive {
+    // Legacy path (to be removed after validation)
+    s = append(s, hive installation steps...)
+}
+```
+
+**Phase 1: Development and Testing**
+- Implement `pkg/aksinstall` package
+- Enable feature flag in local development
+- Validate successful installations in dev environment
+
+**Phase 2: Staging Environment**
+- Enable feature flag in STG
+- Run integration tests (successful install, workload identity clusters, error handling)
+- Validate metrics and alerting
+
+**Phase 3: Canary Rollout**
+- Enable in canary region(s)
+- Monitor installation success rate
+- Validate incident automation with real failures
+
+**Phase 4: Production and Cleanup**
+- Enable in all production regions
+- Monitor for issues
+- Once stable, proceed to Hive cleanup
+
+### 6. Hive Cleanup Scope
+
+- Remove `pkg/hive/` integration code
+- Remove `pkg/monitor/hive/` monitoring (redundant)
+- Remove Admin APIs for Hive SyncSets (unused read-only endpoints)
+- Cluster manager: Remove `hiveClusterManager` field and wrapper methods
+- Backend: Remove Hive client creation
+- Monitor: Remove Hive monitor from monitoring chain
+- Frontend: Remove SyncSet admin API routes
+- Remove Hive imports from all files
+- Remove Hive dependencies from `go.mod` (both modules)
+- Remove Hive-related environment variables
+
+**Why complete removal is safe**:
+- Hive monitoring is redundant - ARO already monitors cluster health directly
+- SyncSet functionality is unused - ARO has read-only APIs but never creates SyncSets
+- No ongoing cluster management - Hive just maintains status that duplicates existing monitoring