Client not sending issues Python 3.6, Gunicorn, Gevent on 7th generation AWS instances in US-EAST-1 region

[jonhuber]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.26.1

Steps to Reproduce

The Sentry background thread deadlocks on 7th generation x86-64 AWS EC2 instance types in the us-east-1 region when running Django 1.11 on Gunicorn with Gevent workers on Python 3.6. It seems to be when the Sentry client tries to grab a piece of work off of it's internal queue.

While this is very specific, it is the only way I was able to reliably reproduce the problem. I have not encountered it on using the sync worker in Gunicorn or in other regions of AWS.

Here is a sample project with instructions on how to reproduce: https://github.com/jonhuber/sentryissue

requirements.txt

certifi==2025.1.31
Django==1.11.29
gevent==22.10.2
greenlet==2.0.2
gunicorn==21.2.0
importlib-metadata==4.8.3
packaging==21.3
pyparsing==3.1.4
pytz==2025.2
sentry-sdk==2.26.1
typing-extensions==4.1.1
urllib3==1.26.20
zipp==3.6.0
zope.event==4.6
zope.interface==5.5.2

Expected Result

I expect to see the Sentry client send events like so...

[sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (transaction)] project:1234 host:myhost.ingest.us.sentry.io
 [sentry] DEBUG: Sending envelope [envelope with 1 items (profile_chunk)] project:1234 host:myhost.ingest.us.sentry.io

Actual Result

I see the Sentry client indefinitely pend sending...

[sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 1
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 2
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 3
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 4
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 5
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 6
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 7
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 8
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 9
 [sentry] DEBUG: [Monitor] health check negative, downsampling with a factor of 10

[szokeasaurusrex]

@jonhuber thank you for the issue report and for the detailed reproduction, this information is extremely helpful.

So that I can understand how urgently we need to address this problem: is this problem blocking you from using Sentry in the way you need to? Or, have you been able to find a satisfactory workaround?

[jonhuber]

@jonhuber thank you for the issue report and for the detailed reproduction, this information is extremely helpful.

So that I can understand how urgently we need to address this problem: is this problem blocking you from using Sentry in the way you need to? Or, have you been able to find a satisfactory workaround?

@szokeasaurusrex I'd say it's probably not urgent. It's not affecting a production environment. It's blocking a migration from another application monitoring provider to Sentry. All of the other Python codebases (different dependency combinations) in our stack have worked great with the Sentry SDK. Assuming this issue could be resolved, we'd move forward with our migration to Sentry.

[sl0thentr0py]

@jonhuber that's the backpressure monitor downsampling your transactions. If you don't want this behavior, you can set enable_backpressure_handling=False.

As to why that happens, we consider the system 'unhealthy' when we're either rate limited or the background worker queue is full and downsample new transactions till we return to a healthy state. This save some useless CPU work but you can just turn that whole subsystem off.

    def is_healthy(self):
        # type: (Self) -> bool
        return not (self._is_worker_full() or self._is_rate_limited())