feat(#2182): mitigate cache stampede with single-flight

[demming]

Description

This fixes the cache stampede I reported in #2182 by using singleflight.

I'll follow up with an extra option to serve stale responses for freshly expired entries, like often with TOTP during failover.

It has a documentation impat - the Config now includes SingleFlight (and maybe StaleWhileRevalidate - either this in PR or in a follow-up PR).

I'll follow up on the tasks below in due time.

This PR isn't ready yet.

Changes introduced

List the new features or adjustments introduced in this pull request. Provide details on benchmarks, documentation updates, changelog entries, and if applicable, the migration guide.

• Benchmarks: Describe any performance benchmarks and improvements related to the changes.
• Documentation Update: Detail the updates made to the documentation and links to the changed files.
• Changelog/What's New: Include a summary of the additions for the upcoming release notes.
• Migration Guide: If necessary, provide a guide or steps for users to migrate their existing code to accommodate these changes.
• API Alignment with Express: Explain how the changes align with the Express API.
• API Longevity: Discuss the steps taken to ensure that the new or updated APIs are consistent and not prone to breaking changes.
• Examples: Provide examples demonstrating the new features or changes in action.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• [ x Enhancement (improvement to existing features and functionality)
• [ x Documentation update (changes to documentation)
• Performance improvement (non-breaking change which improves efficiency)
• Code consistency (non-breaking change which improves code reliability and robustness)

Checklist

Before you submit your pull request, please make sure you meet these requirements:

• Followed the inspiration of the Express.js framework for new functionalities, making them similar in usage.
• Conducted a self-review of the code and provided comments for complex or critical parts.
• Updated the documentation in the /docs/ directory for Fiber's documentation.
• Added or updated unit tests to validate the effectiveness of the changes or new features.
• Ensured that new and existing unit tests pass locally with the changes.
• Verified that any new dependencies are essential and have been agreed upon by the maintainers/community.
• Aimed for optimal performance with minimal allocations in the new code.
• Provided benchmarks for the new code to analyze and improve upon.

Commit formatting

Please use emojis in commit messages for an easy way to identify the purpose or intention of a commit. Check out the emoji cheatsheet here: CONTRIBUTING.md

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 955d8005-8b00-4979-86dd-484a338fbdfd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can approve the review once all CodeRabbit's comments are resolved.

Enable the reviews.request_changes_workflow setting to automatically approve the review once all CodeRabbit's comments are resolved.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to the cache middleware by integrating a single-flight mechanism. This feature is designed to mitigate the "cache stampede" problem, where multiple concurrent requests for the same uncached resource can overwhelm the backend. By enabling the new SingleFlight option, only one request will proceed to generate the response and populate the cache, while other concurrent requests for the same key will wait and share the result, thereby improving performance and reducing server load under high concurrency. Additionally, a StaleWhileRevalidate option has been added for future expansion.

Highlights

• Cache Stampede Mitigation: Implemented a single-flight mechanism in the cache middleware to prevent cache stampede by coalescing concurrent cache misses for the same key.
• New Configuration Options: Introduced SingleFlight and StaleWhileRevalidate fields to the cache middleware configuration, allowing users to enable and configure these behaviors.
• Dependency Update: Added golang.org/x/sync as a new dependency to support the single-flight functionality.

Changelog

• go.mod
  • Added golang.org/x/sync module.
• go.sum
  • Updated module checksums.
• middleware/cache/cache.go
  • Implemented single-flight logic to coalesce concurrent cache misses.
  • Introduced loadResult struct for shared responses.
  • Refined comments.
• middleware/cache/cache_test.go
  • Added Test_Cache_SingleFlight to verify cache stampede prevention.
  • Added Test_Cache_DefaultConfig_BackwardsCompatible to ensure existing behavior is maintained.
• middleware/cache/config.go
  • Added SingleFlight and StaleWhileRevalidate fields to the Config struct.
  • Updated ConfigDefault.

Activity

• The author indicated that the pull request is not yet ready and that further tasks will be addressed in due time.
• The author performed a self-review of the code and included comments for complex parts.
• Unit tests were added or updated to validate the new features and changes.
• The author confirmed that all unit tests pass locally.
• The author focused on optimal performance and minimal memory allocations.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a single-flight mechanism to mitigate cache stampedes, which is a valuable addition. The implementation is well-structured and includes relevant tests to verify the new behavior and ensure backward compatibility. I have a couple of suggestions to improve maintainability by refactoring some duplicated code and to keep the configuration API clean by deferring the addition of an unused option.

[demming]

I need to tackle proper panic handling from inside singleflight - the waiters must be notified to cancel.

I'm aware of the duplication as noted in the commit message. Will clean up before marking it ready for review.

This draft is intended to gather first feedback. FYI @ReneWerner87