feat: rollout total duration tracking

[agaudreault]

Requires #4785

[github-actions]

Published E2E Test Results

  4 files    4 suites   4h 11m 30s ⏱️
148 tests 134 ✅  7 💤 7 ❌
600 runs  564 ✅ 28 💤 8 ❌

For more details on these failures, see this check.

Results for commit 2851889.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 553 tests   2 553 ✅  3m 24s ⏱️
  131 suites      0 💤
    1 files        0 ❌

Results for commit 2851889.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 96.53465% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.03%. Comparing base (d1c2967) to head (2851889).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
rollout/sync.go	96.24%	3 Missing and 2 partials ⚠️
controller/metrics/metrics.go	95.65%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4774      +/-   ##
==========================================
- Coverage   85.09%   85.03%   -0.06%     
==========================================
  Files         166      166              
  Lines       19136    19315     +179     
==========================================
+ Hits        16283    16424     +141     
- Misses       2005     2024      +19     
- Partials      848      867      +19     

Flag	Coverage Δ
e2e	53.21% <94.05%> (+0.24%)	⬆️
unit-tests	81.47% <92.07%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zachaller]

EmitRolloutDuration runs before the status patch in persistRolloutStatus / patchCondition. On conflict/retry, FinishedAt is still unset and the same completion is observed again.

[agaudreault]

I am not sure there is a way around this. The 2 operations cannot be atomic. I prefer doing the status patch at the absolute end then emitting metrics.

In an ideal scenario, the controller would eventually RetryOnConflict on the status update, but that seems out of scope for this PR since it could have side effects.

Maybe what this code could do is return a callback function to do the metric(s) emission, and this callback is performed after the Patch? It does add a little complexity, but wouldn't affect the logic too much. Wdyt?

[zachaller]

Progression buckets stop at 3600s; total goes to 28800s. A long non-paused rollout still has useful total resolution but only hits +Inf for progression.

[agaudreault]

I did set progression much lower since it does not contains the manually paused steps, which can be very long.

I feel that there are not much difference investigating rollouts that took more than 1h compared to rollouts that took more than 2h+ of active progression time.

Maybe in the future, we can track the active/automated paused time outside this metric or with a label.

[zachaller]

rollbacked / fast-rollbacked are non-standard English. Renaming after release breaks the metrics label API.

[agaudreault]

What would be the proper terms for Rollout?

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
17.0% Duplication on New Code

See analysis details on SonarQube Cloud