Skip to content

ci: parallelize the repeated test job; add concurrency cancellation#4309

Open
ReneWerner87 wants to merge 1 commit into
mainfrom
ci/speed-up-test-and-benchmark
Open

ci: parallelize the repeated test job; add concurrency cancellation#4309
ReneWerner87 wants to merge 1 commit into
mainfrom
ci/speed-up-test-and-benchmark

Conversation

@ReneWerner87
Copy link
Copy Markdown
Member

@ReneWerner87 ReneWerner87 commented May 22, 2026

Motivation

Reduce CI wall-clock per commit without changing results, output or coverage.

Key finding: the Test workflow's long pole is the repeated job, and it is dominated by fixed test latency (133 time.Sleep calls: 14x 3s, 17x 1s, ...), not CPU. Removing -race alone saved only ~33s. The effective lever is parallelism.

Changes

test.yml

  • Concurrency group that cancels superseded in-progress PR runs (never cancels main).
  • Coverage computed only on the ubuntu/1.25.x job (the only one that uploads it); the other 5 matrix combinations skip -coverprofile -covermode=atomic.
  • repeated job sharded into 3 parallel matrix shards of -count=5 each = still 15 shuffled runs total (3 distinct shuffle seeds), -race kept.

benchmark.yml

  • Same PR-only concurrency cancellation. Otherwise unchanged (single job, default benchtime).

Measured impact (real run, not estimate)

Job Before After
Test repeated ~9m41s ~3m49s (slowest of 3 shards)
Test workflow overall ~10 min ~4 min

Functionality preserved

  • Coverage: identical, still uploaded only from ubuntu/1.25.x.
  • Race detection: unit matrix unchanged; repeated keeps -race.
  • Flaky/order detection: 15 shuffled runs total, unchanged.

Why the benchmark job is NOT sharded

An attempt to shard benchmarks across 4 runners cut wall-clock to ~4min but failed the comparison: a no-code-change commit reported a uniform ~1.5-2.5x ns/op "regression" with identical B/op/allocs/op, i.e. pure CPU variance. The benchmark comparison relies on a cpu-model cache key so it only compares same-CPU results; spreading work across GitHub's heterogeneous runners breaks that guarantee. Benchmark therefore stays a single job (correct over fast).

🤖 Generated with Claude Code

@ReneWerner87 ReneWerner87 requested a review from a team as a code owner May 22, 2026 10:01
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 22, 2026

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (2)
  • .github/workflows/benchmark.yml is excluded by !**/*.yml
  • .github/workflows/test.yml is excluded by !**/*.yml

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 12a54adf-0fa8-44f2-93b7-88e90c7c7b03

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/speed-up-test-and-benchmark

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes CI wall-clock time for Fiber’s test and benchmark workflows while keeping coverage upload behavior and existing validation intent intact.

Changes:

  • Add PR-only workflow concurrency cancellation to avoid wasted CI on superseded commits.
  • Pin gotestsum to v1.13.0 and reduce unnecessary coverage instrumentation to the single job that uploads coverage.
  • Speed up benchmark runs by reducing go test benchtime.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.github/workflows/test.yml Cancels superseded PR runs, pins gotestsum, computes coverage only on the upload job, and removes -race from the repeated/flakiness job.
.github/workflows/benchmark.yml Cancels superseded PR runs and reduces benchmark runtime via -benchtime=100ms.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 22, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.33%. Comparing base (86397b0) to head (a55fb1f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4309   +/-   ##
=======================================
  Coverage   91.33%   91.33%           
=======================================
  Files         132      132           
  Lines       13089    13089           
=======================================
  Hits        11955    11955           
  Misses        716      716           
  Partials      418      418           
Flag Coverage Δ
unittests 91.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite Current: 5fad032 Previous: 86397b0 Ratio
Benchmark_NewError (github.com/gofiber/fiber/v3) 85.77 ns/op 24 B/op 1 allocs/op 33.9 ns/op 24 B/op 1 allocs/op 2.53
Benchmark_NewError (github.com/gofiber/fiber/v3) - ns/op 85.77 ns/op 33.9 ns/op 2.53
Benchmark_Ctx_AutoFormat_JSON (github.com/gofiber/fiber/v3) 428.9 ns/op 16 B/op 1 allocs/op 285.3 ns/op 16 B/op 1 allocs/op 1.50
Benchmark_Ctx_AutoFormat_JSON (github.com/gofiber/fiber/v3) - ns/op 428.9 ns/op 285.3 ns/op 1.50
Benchmark_Ctx_Write (github.com/gofiber/fiber/v3) 19.14 ns/op 70 B/op 0 allocs/op 12.74 ns/op 76 B/op 0 allocs/op 1.50
Benchmark_Ctx_Write (github.com/gofiber/fiber/v3) - ns/op 19.14 ns/op 12.74 ns/op 1.50
Benchmark_App_RebuildTree (github.com/gofiber/fiber/v3) 25290 ns/op 15960 B/op 48 allocs/op 16340 ns/op 15960 B/op 48 allocs/op 1.55
Benchmark_App_RebuildTree (github.com/gofiber/fiber/v3) - ns/op 25290 ns/op 16340 ns/op 1.55
Benchmark_Router_WithCompression (github.com/gofiber/fiber/v3) 320.6 ns/op 8 B/op 1 allocs/op 209.5 ns/op 8 B/op 1 allocs/op 1.53
Benchmark_Router_WithCompression (github.com/gofiber/fiber/v3) - ns/op 320.6 ns/op 209.5 ns/op 1.53
Benchmark_PaginateMiddlewareWithCustomConfig (github.com/gofiber/fiber/v3/middleware/paginate) 21657 ns/op 11492 B/op 43 allocs/op 14378 ns/op 11474 B/op 43 allocs/op 1.51
Benchmark_PaginateMiddlewareWithCustomConfig (github.com/gofiber/fiber/v3/middleware/paginate) - ns/op 21657 ns/op 14378 ns/op 1.51
Benchmark_Rewrite/Next_always_true (github.com/gofiber/fiber/v3/middleware/rewrite) 556.1 ns/op 24 B/op 2 allocs/op 360.7 ns/op 24 B/op 2 allocs/op 1.54
Benchmark_Rewrite/Next_always_true (github.com/gofiber/fiber/v3/middleware/rewrite) - ns/op 556.1 ns/op 360.7 ns/op 1.54
Benchmark_Bind_URI (github.com/gofiber/fiber/v3) 3593 ns/op 768 B/op 32 allocs/op 2349 ns/op 768 B/op 32 allocs/op 1.53
Benchmark_Bind_URI (github.com/gofiber/fiber/v3) - ns/op 3593 ns/op 2349 ns/op 1.53
Benchmark_Ctx_Cookie (github.com/gofiber/fiber/v3) 205.9 ns/op 0 B/op 0 allocs/op 135.7 ns/op 0 B/op 0 allocs/op 1.52
Benchmark_Ctx_Cookie (github.com/gofiber/fiber/v3) - ns/op 205.9 ns/op 135.7 ns/op 1.52
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckSimple (github.com/gofiber/fiber/v3) 68.96 ns/op 8 B/op 1 allocs/op 44.71 ns/op 8 B/op 1 allocs/op 1.54
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckSimple (github.com/gofiber/fiber/v3) - ns/op 68.96 ns/op 44.71 ns/op 1.54
Benchmark_Ctx_IsProxyTrusted/WithProxyCheck (github.com/gofiber/fiber/v3) 76.16 ns/op 8 B/op 1 allocs/op 50.32 ns/op 8 B/op 1 allocs/op 1.51
Benchmark_Ctx_IsProxyTrusted/WithProxyCheck (github.com/gofiber/fiber/v3) - ns/op 76.16 ns/op 50.32 ns/op 1.51
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowPrivate (github.com/gofiber/fiber/v3) 81.49 ns/op 8 B/op 1 allocs/op 52.33 ns/op 8 B/op 1 allocs/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowPrivate (github.com/gofiber/fiber/v3) - ns/op 81.49 ns/op 52.33 ns/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowAll (github.com/gofiber/fiber/v3) 103.8 ns/op 8 B/op 1 allocs/op 66.56 ns/op 8 B/op 1 allocs/op 1.56
Benchmark_Ctx_IsProxyTrusted/WithProxyCheckAllowAll (github.com/gofiber/fiber/v3) - ns/op 103.8 ns/op 66.56 ns/op 1.56
Benchmark_Router_Handler_Unescape (github.com/gofiber/fiber/v3) 181.6 ns/op 0 B/op 0 allocs/op 116 ns/op 0 B/op 0 allocs/op 1.57
Benchmark_Router_Handler_Unescape (github.com/gofiber/fiber/v3) - ns/op 181.6 ns/op 116 ns/op 1.57
Benchmark_Compress_Levels/Brotli_LevelBestCompression (github.com/gofiber/fiber/v3/middleware/compress) - B/op 17 B/op 11 B/op 1.55
Benchmark_Communication_Flow (github.com/gofiber/fiber/v3) 81.98 ns/op 0 B/op 0 allocs/op 49.1 ns/op 0 B/op 0 allocs/op 1.67
Benchmark_Communication_Flow (github.com/gofiber/fiber/v3) - ns/op 81.98 ns/op 49.1 ns/op 1.67

This comment was automatically generated by workflow using github-action-benchmark.

@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 93c1fcb to 706ae86 Compare May 22, 2026 10:08
@ReneWerner87 ReneWerner87 changed the title ci: speed up test and benchmark workflows without changing coverage ci: speed up test workflow without changing coverage May 22, 2026
@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 706ae86 to 5fad032 Compare May 22, 2026 10:33
@ReneWerner87 ReneWerner87 changed the title ci: speed up test workflow without changing coverage ci: parallelize test and benchmark workflows via sharding May 22, 2026
Reduce CI wall-clock per commit without changing results, output or coverage.

test.yml:
- add a concurrency group that cancels superseded in-progress PR runs while
  never cancelling pushes to main
- compute coverage only on the ubuntu/1.25.x job (the only one that uploads it);
  the other 5 matrix combinations skip the coverage instrumentation
- shard the repeated job into 3 parallel matrix shards of -count=5 each (15
  shuffled runs total, with -race). Measured: ~9m41s -> ~3m49s. The job is
  dominated by fixed test latency (sleeps/timeouts), not CPU, so trimming -race
  alone did not help; splitting the repetitions does.

benchmark.yml:
- add the same PR-only concurrency cancellation

The benchmark job stays a single job on purpose. It cannot be sharded on
GitHub-hosted runners: the regression comparison relies on a cpu-model cache key
so it only compares same-CPU results. Spreading benchmarks across multiple
runners with heterogeneous CPUs breaks that guarantee and produces false alerts
(a no-code-change PR showed a uniform ~1.5-2.5x ns/op "regression" with
identical B/op and allocs/op, i.e. pure CPU variance).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ReneWerner87 ReneWerner87 force-pushed the ci/speed-up-test-and-benchmark branch from 5fad032 to a55fb1f Compare May 22, 2026 10:40
@ReneWerner87 ReneWerner87 changed the title ci: parallelize test and benchmark workflows via sharding ci: parallelize the repeated test job; add concurrency cancellation May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants