Skip to content

[TT-12827] A gateway using a redis rate limiter panics if any Gateway sharing the same Redis is using the DRL#7558

Merged
lghiur merged 4 commits intomasterfrom
TT-12827-a-gateway-using-a-redis-rate-limiter-panics-if-any-gateway-sharing-the-same-redis-is-using-the-drl
Nov 25, 2025
Merged

[TT-12827] A gateway using a redis rate limiter panics if any Gateway sharing the same Redis is using the DRL#7558
lghiur merged 4 commits intomasterfrom
TT-12827-a-gateway-using-a-redis-rate-limiter-panics-if-any-gateway-sharing-the-same-redis-is-using-the-drl

Conversation

@MaciekMis
Copy link
Copy Markdown
Contributor

@MaciekMis MaciekMis commented Nov 21, 2025

Description

This pull request fixes an issue where a gateway that uses enable_redis_rolling_limiter, enable_sentinel_rate_limiter or enable_fixed_window_rate_limiter shares redis with a gateway which is sending DRL updates panics as soon as it receives a DRL update

Related Issue

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

Ticket Details

TT-12827
Status In Code Review
Summary A gateway using a redis rate limiter panics if any Gateway sharing the same Redis is using the DRL

Generated at: 2025-11-25 16:31:44

@github-actions
Copy link
Copy Markdown
Contributor

API Changes

no api changes detected

@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Nov 21, 2025

This pull request resolves a panic in the Tyk Gateway that occurs when gateways with different Redis-based rate-limiting configurations share the same Redis instance. Specifically, a gateway using a non-DRL (Distributed Rate Limiter) Redis rate limiter (e.g., Sentinel, rolling window) would panic upon receiving a DRL update notification from another gateway.

The fix introduces a check within the Redis event handler to verify if DRL is disabled on the receiving gateway. If it is, the DRL notification is safely ignored, preventing the system from accessing an uninitialized DRL manager and thus averting the panic.

Files Changed Analysis

  • gateway/server.go: The logic to determine if DRL is disabled has been refactored into a new helper function, isDRLDisabled(). This centralizes the check and improves code clarity.
  • gateway/redis_signals.go: The core of the fix resides here. The handleRedisEvent function now calls isDRLDisabled() before processing a NoticeGatewayDRLNotification. If DRL is disabled, it returns early, which is the key change that prevents the panic.
  • gateway/gateway_test.go: A new test suite, TestDistributedRateLimiterDisabledRedisEvents, has been added. It explicitly verifies that gateways configured with other Redis-based rate limiters correctly ignore DRL notifications, ensuring the fix is effective and preventing regressions.

Architecture & Impact Assessment

  • What this PR accomplishes: It fixes a critical panic, enabling stable operation of Tyk Gateway clusters where different Redis-based rate-limiting strategies coexist on a shared Redis instance.
  • Key technical changes: The primary change is the introduction of a guard clause in the Redis event handler to filter irrelevant DRL notifications on non-DRL nodes. This is controlled by the new isDRLDisabled() function.
  • Affected system components: The changes are localized to the Gateway's rate limiting and Redis event handling subsystems.
graph TD
    A[Redis Receives DRL Notification] --> B{isDRLDisabled?};
    B -->|Yes| C[Ignore Notification & Prevent Panic];
    B -->|No| D[Process DRL Update as Normal];
Loading

Scope Discovery & Context Expansion

  • The changes are well-contained within the gateway package, specifically targeting the interaction between the rate limiter configuration and the Redis pub/sub event handler.
  • The impact is limited to gateway nodes that have DRL disabled but are part of a cluster where other nodes use DRL. The fix ensures these nodes can coexist safely without crashing.
  • The added tests in gateway_test.go sufficiently cover the scenarios that caused the panic, confirming the fix is effective and isolated.
Metadata
  • Review Effort: 2 / 5
  • Primary Label: bug

Powered by Visor from Probelabs

Last updated: 2025-11-25T16:32:50.132Z | Triggered by: pr_updated | Commit: 6214b39

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Nov 21, 2025

✅ Security Check Passed

No security issues found – changes LGTM.

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Performance Check Passed

No performance issues found – changes LGTM.

Quality Issues (1)

Severity Location Issue
🟡 Warning gateway/gateway_test.go:642-671
The test cases for different rate limiters are nearly identical, leading to code duplication. This makes the test harder to maintain and extend. Additionally, the test state (gateway configuration) is manually reset at the end of each sub-test, which is fragile; if a test panics, the cleanup code will not run, potentially affecting subsequent tests.
💡 SuggestionRefactor these sub-tests into a single table-driven test. This will eliminate duplication and make it easier to add new cases. Use `t.Cleanup` within each sub-test's loop to ensure the configuration is reliably restored, even if the test panics.

Powered by Visor from Probelabs

Last updated: 2025-11-25T16:32:55.455Z | Triggered by: pr_updated | Commit: 6214b39

💡 TIP: You can chat with Visor using /visor ask <your question>

@MaciekMis
Copy link
Copy Markdown
Contributor Author

/visor ask handleRedisEvent function calls a callback once the event processing logic is done. In case of NoticeGatewayDRLNotification the callback is not called specifically for this event but after processing - handled(notif.Command). Due to this implementation callback is called when NoticeGatewayDRLNotification is processed but it is skipped when NoticeGatewayDRLNotification event occurs in non DRL Gateway. Please reevaluate your comment about tests.

@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Nov 24, 2025

You are correct. Thank you for the clarification.

In handleRedisEvent, the handled callback is intentionally not called for a NoticeGatewayDRLNotification when DRL is disabled. The function returns early to prevent a non-DRL gateway from processing the event.

The new test, TestDistributedRateLimiterDisabledRedisEvents, validates exactly this behavior. It provides a callback that fails the test if it's invoked, confirming that the event is correctly skipped.

Therefore, the implementation and the tests are aligned.

References:


Powered by Visor from Probelabs

Last updated: 2025-11-24T12:25:33.479Z | Triggered by: MaciekMis | Commit: 4618f45

💡 TIP: You can chat with Visor using /visor ask <your question>

…ter-panics-if-any-gateway-sharing-the-same-redis-is-using-the-drl
Copy link
Copy Markdown
Contributor

@buraksezer buraksezer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modifying the global config can lead to flaky tests, but this is already a known issue in the codebase.

…ter-panics-if-any-gateway-sharing-the-same-redis-is-using-the-drl
@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
E Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@MaciekMis
Copy link
Copy Markdown
Contributor Author

SonarCube scanned every file that had any change in this PR - those errors are not related to introduced code

@lghiur lghiur merged commit 26926b2 into master Nov 25, 2025
43 of 49 checks passed
@lghiur lghiur deleted the TT-12827-a-gateway-using-a-redis-rate-limiter-panics-if-any-gateway-sharing-the-same-redis-is-using-the-drl branch November 25, 2025 17:14
@MaciekMis
Copy link
Copy Markdown
Contributor Author

/release to release-5.8

probelabs Bot pushed a commit that referenced this pull request Nov 25, 2025
… sharing the same Redis is using the DRL (#7558)

<!-- Provide a general summary of your changes in the Title above -->

## Description

This pull request fixes an issue where a gateway that uses
enable_redis_rolling_limiter, enable_sentinel_rate_limiter or
enable_fixed_window_rate_limiter shares redis with a gateway which is
sending DRL updates panics as soon as it receives a DRL update

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why

<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-12827" title="TT-12827"
target="_blank">TT-12827</a>
</summary>

|         |    |
|---------|----|
| Status  | In Code Review |
| Summary | A gateway using a redis rate limiter panics if any Gateway
sharing the same Redis is using the DRL |

Generated at: 2025-11-25 16:31:44

</details>

<!---TykTechnologies/jira-linter ends here-->

(cherry picked from commit 26926b2)
@probelabs
Copy link
Copy Markdown
Contributor

probelabs Bot commented Nov 25, 2025

✅ Cherry-pick successful. A PR was created: #7565

MaciekMis added a commit that referenced this pull request Nov 26, 2025
…er panics if any Gateway sharing the same Redis is using the DRL (#7558) (#7565)

### **User description**
[TT-12827] A gateway using a redis rate limiter panics if any Gateway
sharing the same Redis is using the DRL (#7558)

<!-- Provide a general summary of your changes in the Title above -->

## Description

This pull request fixes an issue where a gateway that uses
enable_redis_rolling_limiter, enable_sentinel_rate_limiter or
enable_fixed_window_rate_limiter shares redis with a gateway which is
sending DRL updates panics as soon as it receives a DRL update

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why





<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-12827" title="TT-12827"
target="_blank">TT-12827</a>
</summary>

|         |    |
|---------|----|
| Status  | In Code Review |
| Summary | A gateway using a redis rate limiter panics if any Gateway
sharing the same Redis is using the DRL |

Generated at: 2025-11-25 16:31:44

</details>

<!---TykTechnologies/jira-linter ends here-->

[TT-12827]:
https://tyktech.atlassian.net/browse/TT-12827?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


___

### **PR Type**
Bug fix, Tests


___

### **Description**
- Skip DRL events when DRL disabled

- Centralize DRL disable check in helper

- Add tests for DRL skip behavior


___

### Diagram Walkthrough


```mermaid
flowchart LR
  handleRedisEvent["handleRedisEvent: DRL notice"] -- "isDRLDisabled() true" --> skip["return (skip)"]
  handleRedisEvent -- "isDRLDisabled() false" --> onStatus["onServerStatusReceivedHandler"]

  startDRL["startDRL()"] -- "isDRLDisabled() true" --> noInit["Don't init DRL"]
  startDRL -- "isDRLDisabled() false" --> initDRL["Init DRL + notifications"]
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>gateway_test.go</strong><dd><code>Tests for skipping
DRL notifications under other limiters</code></dd></summary>
<hr>

gateway/gateway_test.go

<ul><li>Add test to ensure DRL notices are skipped.<br> <li> Cover
Sentinel, Redis Rolling, and Fixed Window limiters.<br> <li> Validate no
handler invocation when alternate limiter enabled.</ul>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/7565/files#diff-d34c7069ce5e81d45082b19eb3e869ee1a086e185dcd6630e75e3ed0d368b546">+45/-0</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Bug fix</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>redis_signals.go</strong><dd><code>Guard DRL redis
event handling with disable check</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

gateway/redis_signals.go

<ul><li>Use centralized <code>isDRLDisabled()</code> gate for DRL notice
handling.<br> <li> Prevent processing DRL notifications when DRL is
disabled.</ul>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/7565/files#diff-18cb136722c238e19b02741a85510dfd464343f85365482f0873aa60a37718af">+2/-4</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>server.go</strong><dd><code>Centralize DRL disable
logic and apply in startDRL</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

gateway/server.go

<ul><li>Introduce <code>isDRLDisabled()</code> helper.<br> <li> Replace
inline condition in <code>startDRL</code> with helper.<br> <li> Ensure
DRL not initialized when disabled conditions met.</ul>


</details>


  </td>
<td><a
href="https://github.com/TykTechnologies/tyk/pull/7565/files#diff-4652d1bf175a0be8f5e61ef7177c9666f23e077d8626b73ac9d13358fa8b525b">+7/-3</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

</details>

___

Co-authored-by: Maciej Miś <maciej.mis@codilime.com>
NurayAhmadova pushed a commit that referenced this pull request Dec 1, 2025
… sharing the same Redis is using the DRL (#7558)

<!-- Provide a general summary of your changes in the Title above -->

## Description

This pull request fixes an issue where a gateway that uses
enable_redis_rolling_limiter, enable_sentinel_rate_limiter or
enable_fixed_window_rate_limiter shares redis with a gateway which is
sending DRL updates panics as soon as it receives a DRL update

## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->

## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->

## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->

## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why





<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-12827" title="TT-12827"
target="_blank">TT-12827</a>
</summary>

|         |    |
|---------|----|
| Status  | In Code Review |
| Summary | A gateway using a redis rate limiter panics if any Gateway
sharing the same Redis is using the DRL |

Generated at: 2025-11-25 16:31:44

</details>

<!---TykTechnologies/jira-linter ends here-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants