Feature Request: Migrate Fleet to Helm v4

[thardeck]

Is your feature request related to a problem?

Fleet currently uses the Helm v3 SDK for deploying and managing Kubernetes applications through GitOps. Helm v4 was released with improvements and breaking API changes that require Fleet to migrate from v3 to v4. Staying on Helm v3 means:

• Missing out on Helm v4's improved client-side apply algorithm that better handles drift correction for complex resources
• Potential security vulnerabilities as Helm v3 will eventually reach end-of-life
• Inability to leverage new Helm v4 features and performance improvements
• Growing technical debt as the ecosystem moves to v4

Solution you'd like

Upgrade Fleet's Helm SDK dependency from v3 to v4 and adapt all Fleet code to work with Helm v4's breaking API changes.

Alternatives you've considered

1. Stay on Helm v3: Not viable long-term due to eventual EOL and missing improvements
2. Gradual migration with feature flags: Overly complex and would require maintaining both v3 and v4 code paths
3. Fork Helm v3: Would require significant maintenance burden and miss community improvements

Anything else?

PR Implementation: #4326

Follow-up Questions:

1. Should Fleet maintain the Atomic option name or adopt Helm's RollbackOnFailure naming? (Currently auto-mapped)
2. Should Fleet maintain the Force option name or adopt Helm's ForceReplace naming? (Currently auto-mapped)
3. Should Fleet support OCI digest-based chart installations (e.g., oci://registry.example.com/charts/app@sha256:abc123...)?

[thardeck]

QA Template

Testing

Engineering Testing

Manual Testing

• Tested drift correction with force mode: service port changes from external modification (4242 → 80) are correctly reverted
• Verified dry-run modes (client, server, none) work correctly
• Tested template mode rendering
• Verified TakeOwnership functionality doesn't conflict with ServerSideApply

Automated Testing

Unit Tests:

• internal/helmdeployer/*_test.go: Field mapping and configuration logic
• Covers DryRunStrategy enum conversion
• Covers ServerSideApply conditional logic

Integration Tests:

• integrationtests/agent/bundle_deployment_drift_test.go: 7 drift correction scenarios

  • Standard drift correction (no force)
  • Drift correction with force mode (resource replacement)
  • Drift correction on service ports (improved in Helm v4)
  • Drift detection without correction
  • Multiple drift scenarios

• integrationtests/helmops/: Helm operations with v4 SDK

  • Chart installation from OCI, HTTP, Git sources
  • Upgrade operations
  • Rollback operations
  • TLS certificate handling

Test Coverage:

• ✅ Covers: DryRunStrategy configuration, ServerSideApply logic, ForceReplace scenarios, drift correction
• ⚠️ Does not cover: All possible Helm chart types, all authentication methods in live environments, upgrade path from Fleet with Helm v3 to Fleet with Helm v4

QA Testing Considerations

Fresh Installation Testing

• Install Fleet with Helm v4 on a fresh cluster
• Create GitRepo resources with various chart sources:
  • Helm chart from OCI registry
  • Helm chart from HTTP/HTTPS URL
  • Helm chart from Git repository
  • Kustomize configuration
  • Raw Kubernetes YAML
• Verify Bundle creation and deployment
• Test drift correction:
  • Modify deployed resources manually (change labels, annotations)
  • Verify auto-correction without force mode
  • Modify immutable fields (service ports, container ports)
  • Verify force mode correction (resource replacement)
• Test dry-run modes:
  • DryRun: None (normal execution)
  • DryRun: Client (client-side validation only)
  • DryRun: Server (server-side validation)
• Test template mode (render templates without deployment)
• Verify TakeOwnership feature works correctly

Upgrade Scenario Testing

• Start with Fleet installation using Helm v3 SDK (previous version)
• Deploy multiple GitRepos with active BundleDeployments
• Upgrade Fleet to version with Helm v4
• Verify existing deployments continue working
• Verify existing releases are readable and manageable
• Test new deployments post-upgrade
• Verify drift correction works on pre-upgrade deployments
• Test upgrade/rollback operations on existing releases

Authentication and Registry Testing

• Test with various Git authentication methods:
  • SSH keys
  • Personal Access Tokens (PAT)
  • HTTPS with username/password
  • GitHub App authentication
• Test with various registry types:
  • Public OCI registries
  • Private OCI registries with authentication
  • Self-signed certificate registries
  • Harbor registry
  • Artifactory registry

Multi-Cluster Testing

• Deploy GitRepo targeting multiple clusters
• Verify Bundle distribution to all targets
• Test drift correction across multiple clusters
• Verify cluster-specific overrides work
• Test ClusterGroup targeting

Error Handling and Edge Cases

• Test with invalid ServerSideApply values (should fail gracefully)
• Test ForceReplace with ServerSideApply=true (should error appropriately)
• Test with malformed Helm charts
• Test with charts requiring specific Kubernetes versions
• Test with very large charts (memory/performance)
• Test with charts containing CRDs
• Test with charts containing hooks

Performance Testing

• Benchmark deployment time vs Helm v3 version
• Test with 50+ GitRepos
• Test with 100+ BundleDeployments
• Monitor memory usage during large deployments
• Verify no memory leaks in long-running controllers
• Compare the same scenarios with Helm v3 deployments

Logging and Observability

• Verify proper logging with context-based logging (no more Log method)
• Check for appropriate log levels
• Verify error messages are actionable
• Test metrics collection still works

Regression Considerations

HIGH RISK Areas:

1. Drift Correction (Probability: Medium)

   • The ServerSideApply logic changed significantly
   • Risk: Drift correction may behave differently than v3, especially with force mode
   • Mitigation: Comprehensive integration tests cover main scenarios
   • Test Focus: Various resource types (Services, Deployments, ConfigMaps) with external modifications

2. Helm Release Compatibility (Probability: Medium)

   • Helm v4 reads releases created by v3, but field mappings changed
   • Risk: Existing releases may not upgrade/rollback correctly after Fleet upgrade
   • Mitigation: Helm v4 maintains backward compatibility for release storage
   • Test Focus: Upgrade scenario testing with pre-existing v3 releases

3. DryRun Behavior (Probability: Low-Medium)

   • Migration from boolean to enum could cause unexpected behavior
   • Risk: Template mode or dry-run validation may work differently
   • Mitigation: Tests verify all DryRunStrategy values
   • Test Focus: Template rendering, client-side validation, server-side validation

MEDIUM RISK Areas:

4. Chart Source Handling (Probability: Low)

   • ContentCache field addition to downloader.Manager
   • Risk: Chart downloads may behave differently
   • Mitigation: Integration tests cover OCI, HTTP, Git sources
   • Test Focus: Various chart repositories and authentication methods

5. Resource Ownership (Probability: Low)

   • TakeOwnership behavior changed to set ServerSideApply=false
   • Risk: managedFields handling may differ from v3
   • Mitigation: Explicit ServerSideApply=false prevents conflicts
   • Test Focus: Resources with existing managedFields, ownership transfer scenarios

LOW RISK Areas:

6. Logging Changes (Probability: Very Low)

   • Migration from Log method to context-based logging
   • Risk: Some log messages may be formatted differently
   • Mitigation: Standard controller-runtime logging patterns
   • Test Focus: Log output review during testing

7. WaitStrategy Configuration (Probability: Very Low)

   • New required field in Helm v4
   • Risk: Wait behavior may differ slightly
   • Mitigation: Uses sensible defaults (StatusWatcherStrategy)
   • Test Focus: Deployment readiness detection

Recommended Testing Priority

Priority 1 (Must Test):

• Drift correction with and without force mode
• Fresh installation with various chart sources
• Upgrade from Helm v3 to v4 version with active deployments
• Multi-cluster deployments

Priority 2 (Should Test):

• All authentication methods (SSH, PAT, HTTPS)
• Private registries
• DryRun modes and template rendering
• Error handling for invalid configurations

Priority 3 (Nice to Test):

• Performance benchmarks
• Very large scale deployments (100+ resources)
• All edge cases and exotic chart types
• Long-running stability tests