Skip to content

Should we use a fixed seed for random in tests? #684

Open
@allrob23

Description

@allrob23

Checklist

  • I have looked into the Readme and Examples, and have not found a suitable solution or answer.
  • I have looked into the API documentation and have not found a suitable solution or answer.
  • I have searched the issues and have not found a suitable solution or answer.
  • I have searched the Auth0 Community forums and have not found a suitable solution or answer.
  • I agree to the terms within the Auth0 Code of Conduct.

Description

I was checking out the code in auth0/rest.py and noticed that _calculate_wait uses randint to add jitter to retry delays

auth0-python/auth0/rest.py

Lines 243 to 251 in 10555cb

def _calculate_wait(self, attempt: int) -> int:
# Retry the request. Apply a exponential backoff for subsequent attempts, using this formula:
# max(MIN_REQUEST_RETRY_DELAY, min(MAX_REQUEST_RETRY_DELAY, (100ms * (2 ** attempt - 1)) + random_between(1, MAX_REQUEST_RETRY_JITTER)))
# Increases base delay by (100ms * (2 ** attempt - 1))
wait = 100 * 2 ** (attempt - 1)
# Introduces jitter to the base delay; increases delay between 1ms to MAX_REQUEST_RETRY_JITTER (100ms)
wait += randint(1, self.MAX_REQUEST_RETRY_JITTER())

This randomness could make tests flaky, especially if they depend on timing or retry behavior, since the wait time varies between runs.

Fixing a seed (e.g., random.seed(42) in a test fixture) would make test outcomes consistent and easier to debug. Without it, we risk intermittent failures in CI or locally, which could slow down development. (Im not saying this problems is happening, but it can happen). I didn't see any seeds being set into the test code...

What do you think about adding this to the test setup? I’d be happy to submit a PR if you’re open to it!

One quick question: I saw pytest-randomly in poetry.lock under test-randomorder—does that already handle seeds in some way I might have missed?

Thanks for your input!

Reproduction

.

Additional context

No response

auth0-python version

master branch

Python version

3.12.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis points to a verified bug in the code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions