Error connecting celery-exporter to Google PubSub

[ssoto]

Missing Google Cloud Pub/Sub transport support

Problem Description

The celery-exporter fails to connect to Google Cloud Pub/Sub brokers due to missing dependencies for the gcpubsub:// transport. When attempting to use Pub/Sub as a broker, the application crashes with a KeyError.

How to Reproduce

1. Run celery-exporter with a Pub/Sub broker URL:

   docker run -p 9808:9808 danihodovic/celery-exporter --broker-url=gcpubsub://projects/my-gcp-project

2. Or using Docker Compose:

   celery-exporter:
     image: danihodovic/celery-exporter
     command: ["--broker-url", "gcpubsub://projects/my-gcp-project"]

Error Message

KeyError: gcpubsub

-------------------------------------------------------------------------------
__init__.py 63 resolve_transport
transport = TRANSPORT_ALIASES[transport]

KeyError:
gcpubsub

-------------------------------------------------------------------------------
__init__.py 72 resolve_transport
raise KeyError(f'No such transport: {transport}')

KeyError:
No such transport: gcpubsub

Root Cause

The issue occurs because:

1. Kombu (Celery's messaging library) doesn't include Pub/Sub transport by default
2. The required Google Cloud dependencies are not installed
3. The gcpubsub transport is not registered in Kombu's TRANSPORT_ALIASES

Proposed Solution

Add Google Cloud Pub/Sub support by including the necessary dependencies in pyproject.toml:

[tool.poetry.dependencies]
# ... existing dependencies
google-cloud-pubsub = "^2.18.0"
kombu = {extras = ["gcpubsub"], version = "^5.0.0"}

Why This Works

• kombu[gcpubsub] installs the Pub/Sub transport plugin
• google-cloud-pubsub provides the Google Cloud client library
• This registers the gcpubsub:// transport in Kombu's available transports

Testing

After applying this fix:

1. The exporter successfully connects: Connected to the broker gcpubsub://projects/my-gcp-project
2. Health check works: curl http://localhost:9808/health
3. Metrics are exposed: curl http://localhost:9808/metrics

Environment

• Celery version: 5.5.3
• Kombu version: 5.0.0+
• Python version: 3.11+
• Docker: Yes

Additional Notes

This enhancement would enable celery-exporter to work with:

• Google Cloud Pub/Sub in production
• Pub/Sub emulator for local development
• Any Celery setup using Pub/Sub as message broker