Django/Channels on Google Cloud Run results in ConnectionError

[alexanderjulmer]

Hi y'all,

I am struggling with a persistent Connection reset by peer error when running Django Channels on Google Cloud Run, using GCP Redis as a message broker:

ConnectionError: Error while reading from 10.189.0.4:6379 : (104, 'Connection reset by peer')

Task exception was never retrieved
future: <Task finished name='Task-1343' coro=<MyConsumer._listen_for_updates() ...> exception=ConnectionError("Error while reading from 10.189.0.4:6379 : (104, 'Connection reset by peer')")>

The error is triggered on this line:

message: IpcEvent = await self.channel_layer.receive(self.channel_name)

I've taken quite some steps to ensure my environment is configured for persistent, long-lived connections (see also https://docs.cloud.google.com/run/docs/triggering/websockets).

• Deployed with --min-instances=2 so it never scales to 0
• CPU is always allocated, not throttled
• Memory usage never above 50%
• 3600 seconds request timeout
• Session affinity activated

I set CHANNEL_LAYERS to use application-level health checks and network-level TCP keep-alives (see also django/channels_redis#407)

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [
                {
                    "address": os.environ.get("REDIS_URL", "redis://127.0.0.1:6379"),
                    "health_check_interval": 15,
                    "retry_on_timeout": True,
                    "socket_keepalive": True,
                }
            ],
        },
    }
}

I am out of ideas what I could do to fix this. Any help is greatly appreciated.

Thank you!

Versions

• django 5.1.14
• channels 4.3.1
• channels-redis 4.3.0
• redis 5.1.0b4
• asgiref 3.10.0