Skip to content

Commit a9f9abf

Browse files
eshulman2eshulman2
committed
Enhancement: Add drift detection and automatic reconciliation
Proposal for drift detection feature.
1 parent eeb37af commit a9f9abf

File tree

1 file changed

+229
-0
lines changed

1 file changed

+229
-0
lines changed

‎enhancements/drift-detection.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Enhancement: Drift Detection and Automatic Reconciliation
2+
3+
| Field | Value |
4+
|-------|-------|
5+
| **Status** | implementable |
6+
| **Author(s)** | @eshulman |
7+
| **Created** | 2026-02-03 |
8+
| **Last Updated** | 2026-02-03 |
9+
| **Tracking Issue** | TBD |
10+
11+
## Summary
12+
13+
This enhancement introduces drift detection and automatic reconciliation for ORC managed resources. The feature enables ORC to periodically check OpenStack resources for changes made outside of ORC (via CLI, dashboard, or other tools) and automatically restore them to match the desired state defined in the Kubernetes specification.
14+
15+
Additionally, managed resources that are deleted externally from OpenStack will be automatically recreated by ORC, ensuring the declared state is maintained.
16+
17+
## Motivation
18+
19+
In production environments, OpenStack resources may be modified outside of ORC through various means:
20+
21+
- Direct OpenStack CLI/SDK operations
22+
- OpenStack Horizon dashboard
23+
- Other automation tools or controllers
24+
- Manual emergency interventions
25+
- Third-party integrations
26+
27+
Without drift detection, these changes go unnoticed until they cause issues, leading to configuration drift between the declared Kubernetes state and the actual OpenStack state. This undermines the declarative model that ORC provides.
28+
29+
Similar Kubernetes controllers for cloud resources have implemented drift detection:
30+
31+
- **AWS Controllers for Kubernetes (ACK)**: Drift detection is **enabled by default** with a 10-hour resync period. Uses a detect-then-correct approach: periodically describes the AWS resource and only updates if drift is found. Configuration is set per-controller by authors, not configurable per-resource by users. No per-resource opt-out mechanism documented. ([ACK Drift Recovery docs](https://aws-controllers-k8s.github.io/community/docs/user-docs/drift-recovery/))
32+
33+
- **Azure Service Operator (ASO)**: Drift detection is **enabled by default** with a 1-hour resync period. Uses a PUT-on-every-reconcile approach rather than detect-then-correct. Provides **per-resource opt-out** via `reconcile-policy` annotation for adopted resources users don't want fully managed. **Global configuration** via `AZURE_SYNC_PERIOD` environment variable. Rate limiting via token-bucket algorithm and `MAX_CONCURRENT_RECONCILES` for parallelism control. ([ASO Controller Settings](https://azure.github.io/azure-service-operator/guide/aso-controller-settings-options/), [ASO Change Detection ADR](https://azure.github.io/azure-service-operator/design/adr-2022-11-change-detection/))
34+
35+
**Key design observations:**
36+
- Both projects enable drift detection by default
37+
- ASO provides more user-facing configuration options (global and per-resource)
38+
- Neither project documents behavior for externally-deleted resources
39+
40+
## Goals
41+
42+
- **Ensure state consistency**: Managed resources in OpenStack should match the desired state declared in Kubernetes
43+
- **Detect external modifications**: Identify when OpenStack resources are modified outside of ORC
44+
- **Automatic correction**: Restore drifted resources to their desired state without manual intervention
45+
- **Resource recreation**: Recreate managed resources that are deleted externally from OpenStack
46+
- **Configurable frequency**: Allow operators to tune the resync interval based on their requirements
47+
- **Hierarchical configuration**: Support configuration at ORC-wide, resource-type, and per-resource levels
48+
- **Minimal API impact**: Avoid excessive OpenStack API calls that could trigger rate limiting
49+
50+
## Non-Goals
51+
52+
- **Real-time drift detection**: Event-driven detection of changes (would require OpenStack webhooks or very short polling intervals)
53+
- **Drift reporting without correction**: Alerting on drift without taking corrective action. This applies to both mutable fields (which are corrected, not just reported) and immutable fields (which are ignored, not reported). May be considered as a future enhancement.
54+
- **Selective field reconciliation**: Allowing some fields to drift while correcting others
55+
- **Conflict resolution with merge semantics**: Merging external changes with desired state
56+
- **Drift detection for unmanaged resources**: Unmanaged resources are explicitly not modified by ORC
57+
58+
## Proposal
59+
60+
### Periodic Resync Mechanism
61+
62+
The drift detection mechanism works by periodically triggering a full reconciliation of managed resources. Unlike event-driven reconciles (triggered by Kubernetes spec/status changes), drift detection uses a time-based trigger to catch changes made directly in OpenStack:
63+
64+
1. **Trigger**: After a resource reaches a stable state (Progressing=False), ORC schedules a resync after `resyncPeriod` duration
65+
2. **Fetch**: On resync, ORC fetches the current state of the OpenStack resource
66+
3. **Compare**: The current state is compared against the desired state in the Kubernetes spec
67+
4. **Update**: If drift is detected, ORC updates the OpenStack resource to match the desired state
68+
5. **Reschedule**: After successful reconciliation, the next resync is scheduled
69+
70+
### API Changes
71+
72+
The `ManagedOptions` struct is extended with a `resyncPeriod` field:
73+
74+
```yaml
75+
apiVersion: openstack.k-orc.cloud/v1alpha1
76+
kind: Network
77+
metadata:
78+
name: critical-network
79+
spec:
80+
cloudCredentialsRef:
81+
secretName: openstack-clouds
82+
cloudName: openstack
83+
managementPolicy: managed
84+
managedOptions:
85+
resyncPeriod: 1h # Check for drift every hour
86+
resource:
87+
description: Critical application network
88+
```
89+
90+
**Default value**: Disabled (`0`) - drift detection is opt-in initially until the design and implementation are proven. Users can enable it by setting a positive value (e.g., `10h` aligns with ACK's default).
91+
92+
**Enable resync**: Set `resyncPeriod` to a positive duration (e.g., `1h`, `10h`) to enable periodic drift detection for a resource.
93+
94+
### Configuration Hierarchy
95+
96+
Drift detection supports a three-level configuration hierarchy, with more specific configurations taking precedence:
97+
98+
| Level | Scope | Configuration Location | Precedence |
99+
|-------|-------|----------------------|------------|
100+
| ORC-wide | All resources across all types | See options below | Lowest |
101+
| Resource-type | All resources of a specific type (e.g., all Networks) | Strategy CR `resourceOverrides` section | Medium |
102+
| Per-resource | Individual resource instance | `spec.managedOptions.resyncPeriod` on the CR | Highest |
103+
104+
**Resolution order**: Per-resource → Strategy CR resourceOverrides → ORC-wide → Built-in default (disabled)
105+
106+
#### ORC-wide Configuration Options
107+
108+
The following are possible implementations to consider for global configuration:
109+
110+
1. **CLI flag**: `--default-resync-period=10h`
111+
2. **Environment variable**: `DEFAULT_RESYNC_PERIOD=10h` (similar to ASO's `AZURE_SYNC_PERIOD`)
112+
3. **Strategy CR**: A dedicated CR for organizational policies (see below)
113+
114+
#### Strategy CR (Future Consideration)
115+
116+
For sophisticated deployments, a Strategy CR could define both global defaults and per-resource-type overrides in a Kubernetes-native, auditable format:
117+
118+
```yaml
119+
apiVersion: openstack.k-orc.cloud/v1alpha1
120+
kind: ORCStrategy
121+
metadata:
122+
name: production
123+
namespace: orc-system
124+
spec:
125+
# Global defaults
126+
defaults:
127+
drift:
128+
resyncPeriod: 10h
129+
subResources:
130+
waitForReady: false # See PR #673 for SubResourcesReady condition
131+
132+
# Per-resource-type overrides
133+
resourceOverrides:
134+
Network:
135+
drift:
136+
resyncPeriod: 1h
137+
SecurityGroup:
138+
subResources:
139+
waitForReady: true
140+
```
141+
142+
Benefits of Strategy CR:
143+
- Kubernetes-native, versioned, auditable, RBAC-controlled
144+
- Supports multiple strategies (e.g., `production`, `development`)
145+
- Can be extended to include other configurable behaviors (e.g., sub-resource policies from [PR #673](https://github.com/k-orc/openstack-resource-controller/pull/673))
146+
147+
Alternatively, platform teams can use [kro (Kube Resource Orchestrator)](https://kro.run/) to wrap ORC resources with organizational defaults without changes to ORC itself.
148+
149+
### Resource Recreation on External Deletion
150+
151+
When a managed resource is deleted from OpenStack but the ORC object still exists:
152+
153+
1. On the next reconciliation, ORC attempts to fetch the resource by the ID stored in `status.id`
154+
2. If not found and the resource was originally created by ORC (not imported), ORC recreates it
155+
3. The new resource ID is stored in `status.id`
156+
157+
**Behavior when drift detection is disabled** (`resyncPeriod: 0`): External deletion remains a terminal error (current behavior preserved). Resource recreation only occurs when drift detection is enabled and a periodic resync discovers the missing resource. This maintains backwards compatibility.
158+
159+
For **imported resources** that are deleted externally, this is always a terminal error regardless of drift detection settings, because the resource was not created by ORC and recreating it would not restore the original resource.
160+
161+
### Field Coverage
162+
163+
Drift detection covers all **mutable fields** that ORC actuators implement update operations for. Before this feature is considered stable, all actuator implementations must be audited to ensure they cover all mutable fields.
164+
165+
## Risks and Edge Cases
166+
167+
### Split-Brain Scenarios
168+
169+
**Risk**: Multiple controllers or systems may be managing the same OpenStack resources, leading to conflicts where changes are repeatedly overwritten.
170+
171+
**Mitigation**:
172+
- Document that ORC should be the sole manager of resources it creates
173+
- Report conflicts in resource conditions for observability
174+
175+
### API Rate Limiting
176+
177+
**Risk**: Frequent resync across many resources could trigger OpenStack API rate limiting.
178+
179+
**Mitigation**:
180+
- Disabled by default; when enabled, recommend conservative intervals (e.g., 10 hours)
181+
- Add random jitter to resync times to avoid thundering herd: since reconciliation already uses "requeue after X duration", jitter simply adds a random offset (e.g., ±10%) to the resync period, spreading resyncs over time rather than having them fire simultaneously
182+
- Allow operators to disable or lengthen resync for stable resources
183+
184+
### Controller Resource Consumption
185+
186+
**Risk**: Frequent reconciliation increases CPU and memory usage on the ORC controller.
187+
188+
**Mitigation**:
189+
- Implement hash-based comparison as an optimization: on resync, compute a hash of the fetched OpenStack resource state and compare it to `status.observedStateHash` from the last reconciliation. If identical, skip the full field-by-field comparison (no drift detected). If different, proceed with full reconciliation. This avoids expensive value comparisons when nothing has changed.
190+
- Disabled by default; when enabled, conservative intervals limit reconciliation frequency
191+
192+
### Conflicts with External Systems
193+
194+
**Risk**: If resources are intentionally managed by external systems (e.g., autoscalers, other controllers), drift correction can cause unexpected behavior.
195+
196+
**Mitigation**:
197+
- Allow `resyncPeriod: 0` to disable drift detection
198+
- Use `managementPolicy: unmanaged` for externally managed resources
199+
- Document the implications clearly in the user guide
200+
201+
### Upgrade/Downgrade Considerations
202+
203+
**Risk**: Users upgrading to a version with drift detection may experience unexpected reconciliations.
204+
205+
**Mitigation**: Drift detection is disabled by default (opt-in), so users upgrading will not experience any behavior change unless they explicitly enable it. Document the new feature in release notes.
206+
207+
## Alternatives Considered
208+
209+
### Event-Driven Drift Detection
210+
211+
Use OpenStack notifications (Oslo messaging) to detect changes in real-time.
212+
213+
**Rejected because**: Requires OpenStack notification infrastructure, complex to implement, not all deployments have notifications enabled.
214+
215+
### Drift Detection Without Correction
216+
217+
Detect and report drift without automatically correcting it.
218+
219+
**Out of scope for this enhancement**: While drift notification has value for observability, it is better addressed as a separate alerting effort. This enhancement focuses on drift correction; reporting-only mode could be added as a future management policy option.
220+
221+
### Watch-Based Detection
222+
223+
Implement a watcher that periodically lists all resources from OpenStack and compares.
224+
225+
**Rejected because**: List operations can be expensive, harder to implement with proper filtering, and per-resource reconciliation integrates naturally with controller-runtime.
226+
227+
## Implementation History
228+
229+
- 2026-02-03: Enhancement proposed

0 commit comments

Comments
 (0)