Fix more test flakiness #467
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fajpunk
force-pushed
the
tickets/DM-53251/more-flakiness
branch
9 times, most recently
from
68ee6d3 to
0bb258e
Compare
rra
approved these changes
Dec 1, 2025
Member
rra
left a comment
rra
left a comment
There was a problem hiding this comment.
Looks good as a workaround. I guess ideally we'd explicitly close the clients and catch the exception, but I don't know how to do that.
The app metrics resiliency tests intentionally break `aiokafka` clients
in an unclean way. When the aiokafka objects get garbage collected,
Python asyncio logs warning messages that task exceptions were never
retrieved, like this:
```
asyncio:base_events.py:1879 Future exception was never retrieved
future: <Future finished exception=NodeNotReadyError('Attempt to send a request to node which is not ready (node id 1).')>
aiokafka.errors.NodeNotReadyError: NodeNotReadyError: Attempt to send a request to node which is not ready (node id 1).```
The time that this logging happens is not deterministic. It can mess up
other tests, like tests of the Sentry functionality because the log
messages get picked up by the mock Sentry transport if they happen after
the Sentry fixtures have initialized:
https://github.com/lsst-sqre/safir/actions/runs/19583537252/job/56087067160
There may be a way to deterministically and cleanly cancel these failed
tasks, or it may be a bug in the aiokafka library, but until we figure
it out, run these last to make sure they don't infect other tests.
fajpunk
force-pushed
the
tickets/DM-53251/more-flakiness
branch
from
0bb258e to
b899551
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The app metrics resiliency tests intentionally break
aiokafkaclientsin an unclean way. When the aiokafka objects get garbage collected,
Python asyncio logs warning messages that task exceptions were never
retrieved, like this:
The time that this logging happens is not deterministic. It can mess up
other tests, like tests of the Sentry functionality because the log
messages get picked up by the mock Sentry transport if they happen after
the Sentry fixtures have initialized:
https://github.com/lsst-sqre/safir/actions/runs/19583537252/job/56087067160
There may be a way to deterministically and cleanly cancel these failed
tasks, or it may be a bug in the aiokafka library, but until we figure
it out, run these last to make sure they don't infect other tests.