Skip to content

[TT-16950] Backport test fixes for nonroot gateway and RPC storage#8036

Merged
buger merged 2 commits intorelease-5.12.1from
backport-16950-5.12.1
Apr 15, 2026
Merged

[TT-16950] Backport test fixes for nonroot gateway and RPC storage#8036
buger merged 2 commits intorelease-5.12.1from
backport-16950-5.12.1

Conversation

@probelabs
Copy link
Copy Markdown
Contributor

@probelabs probelabs Bot commented Apr 15, 2026

Problem / Task

Backport test fixes for nonroot gateway and RPC storage to release-5.12.1.

Changes

  • Cherry-picked 7892fa3 (TT-16693: Update RPC storage handler tests)
  • Cherry-picked 461478b (fix: create writable directories for nonroot gateway in tests)

Testing

  • CI tests should pass

vladzabolotnyi and others added 2 commits April 15, 2026 19:20
@vladzabolotnyi @buger
TT-16693: Update RPC storage handler tests (#7828)
326ed29 
<!-- Provide a general summary of your changes in the Title above -->

[TT-16693](https://tyktech.atlassian.net/browse/TT-16693)
[TT-16695](https://tyktech.atlassian.net/browse/TT-16695)
[TT-16696](https://tyktech.atlassian.net/browse/TT-16696)

## Description
When running tests that use the StartTest helper with
`config.SlaveOptions.UseRPC` enabled, the gateway enters an "emergency
mode" and the gateway falls back to loading API Definitions and Policies
from a Redis backup instead of the master nodes.

Because the entire test suite runs against a single shared Redis
instance, tests running in parallel would often write to the same Redis
keys. This created a race condition where a test could read data written
by another test.

The data, while often valid JSON, would have an incorrect structure for
the context in which it was being read. The unmarshaling process would
succeed but produce a slice containing nil values
([]model.MergedAPI{nil}). Subsequent code that did not anticipate these
nil values would then panic, causing the test to fail unpredictably.

Subsequently, the `prepareSpecs` function iterates over this slice and
calls methods on the APIDefinition. This leads to a nil pointer
dereference panic when it encounters one of the nil entries.

## List of change
1. Update dispatcher setup at tests to assign default func handlers and
make code cleaner
2. Update the expected response values to provide better documentation
3. Add unique tags for RPC based tests to avoid fetching invalid values
populated by other tests

## How This Has Been Tested
This change is targeting flaky tests not the production code in
particular

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [x] I ensured that the documentation is up to date
- [x] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why



























<!---TykTechnologies/jira-linter starts here-->

### Ticket Details

<details>
<summary>
<a href="https://tyktech.atlassian.net/browse/TT-16693" title="TT-16693"
target="_blank">TT-16693</a>
</summary>

|         |    |
|---------|----|
| Status  | In Code Review |
| Summary | [CI GW] Failing Unit Test:
TestProcessKeySpaceChanges_UserKeyReset |

Generated at: 2026-03-10 06:40:24

</details>

<!---TykTechnologies/jira-linter ends here-->




























[TT-16693]:
https://tyktech.atlassian.net/browse/TT-16693?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[TT-16695]:
https://tyktech.atlassian.net/browse/TT-16695?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
[TT-16696]:
https://tyktech.atlassian.net/browse/TT-16696?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Vlad Zabolotnyi <vlad.z@tyk.io>
@buger @claude
fix: create writable directories for nonroot gateway in tests (#8022)
42387fa 
- The gateway base image now runs as `nonroot` (uid 65532) instead of
root
- Test docker-compose files mount host directories (`apps`, `policies`)
into `/opt/tyk-gateway/`, but Docker creates these as root when they
don't exist, causing permission denied errors
- Adds `mkdir -p && chmod 777` for mounted directories before `docker
compose up` in three test suites: specs, tracing, and metrics

- [ ] Verify specs contract tests pass in CI (apps/policies dirs created
writable)
- [ ] Verify tracing e2e tests pass in CI
- [ ] Verify metrics e2e tests pass in CI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@probelabs probelabs Bot requested a review from a team as a code owner April 15, 2026 19:20
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 15, 2026

🚨 Jira Linter Failed

Commit: 42387fa
Failed at: 2026-04-15 19:21:25 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details 
failed to validate branch and PR title rules: branch name 'backport-16950-5.12.1' must contain a valid Jira ticket ID (e.g., ABC-123)

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@probelabs
Copy link
Copy Markdown
Contributor Author

probelabs Bot commented Apr 15, 2026 •
edited
Loading

This pull request backports two test-related fixes to the release-5.12.1 branch, addressing issues with tests for non-root gateway execution and RPC storage handling.

Files Changed Analysis

  • ci/tests/specs/test.sh and ci/tests/tracing/Taskfile.yml: These files are updated to create apps and policies directories and set their permissions to be world-writable (777). This ensures that the gateway, when running as a non-root user in test environments, can write API definitions and policies without permission errors.
  • gateway/rpc_storage_handler_test.go: This file undergoes a significant refactoring to improve the reliability and maintainability of the RPC storage handler tests.
    • A new helper function, newDispatcher, is introduced to create mock RPC dispatchers with default handlers, reducing boilerplate code across multiple tests.
    • A generateUniqueTestTag function is added to create isolated test data namespaces, likely in Redis, preventing test flakiness from concurrent test runs.
    • Existing tests are updated to use these new helpers, resulting in cleaner and more robust test implementations.

Architecture & Impact Assessment

  • What this PR accomplishes: It fixes failing CI tests by ensuring the test environment is compatible with a non-root gateway user and improves the stability of RPC storage tests by refactoring them for better isolation and setup.
  • Key technical changes introduced:
    1. Modification of file permissions in CI scripts to allow writes by non-root users.
    2. Refactoring of Go tests to use helper functions for creating mock RPC dispatchers and generating unique test tags for data isolation.
  • Affected system components: The changes are confined to the testing infrastructure and do not impact the core gateway runtime logic. The primary components affected are the CI test execution environment and the unit/integration tests for the RPC storage handler.

Scope Discovery & Context Expansion

  • The changes in the CI scripts indicate a broader effort to run services, including the Tyk Gateway, as non-root users for better security posture, and these tests were failing under that new context.
  • The refactoring in gateway/rpc_storage_handler_test.go suggests previous issues with test flakiness or interference. The introduction of unique tags per test run is a common pattern to solve problems where tests share a common resource (like a Redis database) and overwrite each other's data. This change makes the test suite more reliable.
  • Further investigation could involve checking other test setup scripts for similar permission issues and identifying other test files that could benefit from the new newDispatcher and generateUniqueTestTag helpers to improve the overall quality of the test suite.
Metadata
  • Review Effort: 2 / 5
  • Primary Label: chore

Powered by Visor from Probelabs

Last updated: 2026-04-15T19:22:49.414Z | Triggered by: pr_opened | Commit: 42387fa

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs
Copy link
Copy Markdown
Contributor Author

probelabs Bot commented Apr 15, 2026 •
edited
Loading

Security Issues (2)

Severity Location Issue
🟡 Warning ci/tests/specs/test.sh:23
The test setup script uses `chmod 777` to create world-writable directories (`apps`, `policies`). While this is for a non-root user in a test environment, it is a bad practice and creates an insecure configuration. World-writable directories can be modified by any process or user on the system, potentially leading to test interference, data corruption, or exploitation if other vulnerabilities exist.
💡 SuggestionInstead of using `chmod 777`, use more restrictive permissions. If the user ID of the non-root gateway is known (e.g., 65532 as per the comment), consider changing the ownership of the directories with `chown 65532:65532 apps policies` after creating them. Alternatively, if group information is available, use `chmod 775` and ensure the gateway user is in the correct group.
🟡 Warning ci/tests/tracing/Taskfile.yml:30
The test setup task uses `chmod 777` to create world-writable directories (`apps`, `policies`). While this is for a non-root user in a test environment, it is a bad practice and creates an insecure configuration. World-writable directories can be modified by any process or user on the system, potentially leading to test interference, data corruption, or exploitation if other vulnerabilities exist.
💡 SuggestionInstead of using `chmod 777`, use more restrictive permissions. If the user ID of the non-root gateway is known (e.g., 65532 as per the comment), consider changing the ownership of the directories with `chown 65532:65532 apps policies` after creating them. Alternatively, if group information is available, use `chmod 775` and ensure the gateway user is in the correct group. The command could be run inside the container where the user exists.

Security Issues (2)

Severity Location Issue
🟡 Warning ci/tests/specs/test.sh:23
The test setup script uses `chmod 777` to create world-writable directories (`apps`, `policies`). While this is for a non-root user in a test environment, it is a bad practice and creates an insecure configuration. World-writable directories can be modified by any process or user on the system, potentially leading to test interference, data corruption, or exploitation if other vulnerabilities exist.
💡 SuggestionInstead of using `chmod 777`, use more restrictive permissions. If the user ID of the non-root gateway is known (e.g., 65532 as per the comment), consider changing the ownership of the directories with `chown 65532:65532 apps policies` after creating them. Alternatively, if group information is available, use `chmod 775` and ensure the gateway user is in the correct group.
🟡 Warning ci/tests/tracing/Taskfile.yml:30
The test setup task uses `chmod 777` to create world-writable directories (`apps`, `policies`). While this is for a non-root user in a test environment, it is a bad practice and creates an insecure configuration. World-writable directories can be modified by any process or user on the system, potentially leading to test interference, data corruption, or exploitation if other vulnerabilities exist.
💡 SuggestionInstead of using `chmod 777`, use more restrictive permissions. If the user ID of the non-root gateway is known (e.g., 65532 as per the comment), consider changing the ownership of the directories with `chown 65532:65532 apps policies` after creating them. Alternatively, if group information is available, use `chmod 775` and ensure the gateway user is in the correct group. The command could be run inside the container where the user exists.
\n\n ### ✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Performance Check Passed

No performance issues found – changes LGTM.

Powered by Visor from Probelabs

Last updated: 2026-04-15T19:22:41.144Z | Triggered by: pr_opened | Commit: 42387fa

💡 TIP: You can chat with Visor using /visor ask <your question>

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 15, 2026

API Changes

no api changes detected

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Apr 15, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@buger buger merged commit a515a61 into release-5.12.1 Apr 15, 2026
16 of 17 checks passed
@buger buger deleted the backport-16950-5.12.1 branch April 15, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@buger @vladzabolotnyi