Add sender queue that expires oldest first

[StephenWakely]

Requirements for Contributing to this repository

• Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
• The pull request must only fix one issue, or add one feature, at the time.
• The pull request must update the test suite to demonstrate the changed functionality.
• After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

Changes the sender queue from a simple queue.Queue to a queue with the following properties:

• On adding to a full queue (after the configured timeout) the oldest item in the queue is dropped and the new item is added.
• Un-timestamped metrics added to the queue will expire if they are in the queue for longer than 10 seconds.

Description of the Change

With the recent addition of a socket connect timeout, metrics can be retried whilst facing network outages or agent downtimes. As it is retrying the sender queue can fill up, and eventually will overflow. When overflowing we want to make sure the most relevant metrics are kept.

Metrics sent without a timestamp have a limited time during which they are relevant. If we queue metrics for 2 minutes, and then send them all in one go, this would lead to 2 minutes worth of metrics being aggregated in a single time window, causing a huge spike. Metric expiry ensures this cannot happen.

Metrics sent with timestamps cannot expire as they are sent through to the Datadog backend as is.

Alternate Designs

Possible Drawbacks

There are performance implications with this change. Benchmarking:

• agent restart with 60 second restart time
• clients sending 28,000 timestamped metrics/second

During the restart the client queue grows to 218k. (Each item in the queue is a payload which can consist of more than one metric).

Memory usage is ~10mb more at the peak, settling down to approximately 26mb more afterwards - I presume due to greater memory fragmentation.

I don't get accurate cpu measurements during the agent downtime since the agent is down, but during typical usage CPU seems largely unaffected.

Verification Process

Additional Notes

I have some followup PRs coming:

• Make the socket_connect_timeout a binary option. With the queue now dropping oldest metrics, we don't need to limit the time we wait for a reconnect. We continue attempting to reconnect until success.
• Add timeout jjjjjejhxuctbibukjbpyxjtthdjbinjhenyeutipgkd
• add timeout bounds to wait_for_pending and stop functions.

Release Notes

Review checklist (to be filled by reviewers)

• Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
• PR should not have do-not-merge/ label attached.
• If Applicable, issue must have kind/ and severity/ labels attached at least.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This pull request adds a bounded sender queue that evicts oldest entries, expires stale metrics, and retries transient failures.

Changes:

• Introduces SenderQueue and PendingPayload.
• Integrates expiry, eviction telemetry, and retry handling.
• Adds unit tests and a performance benchmark.

File summaries

File	Review summary
tests/unit/dogstatsd/test_statsd.py	One nit regarding a timing-dependent queue test.
tests/performance/test_sender_queue_benchmark.py	Reviewed with no final comments.
datadog/dogstatsd/sender_queue.py	One critical producer-notification issue and one moderate shutdown requeue issue.
datadog/dogstatsd/base.py	One critical shutdown issue and three moderate expiry, batching, and retry issues.

Review details

Suppressed comments (2)

datadog/dogstatsd/base.py:1792

• Queue-mode retry only handles sent is None, but _xmit_packet_attempt() returns False for socket.timeout; _get_uds_socket() can raise that exception when a UDS connect attempt times out. A connection timeout therefore falls through to the writer-drop path and permanently loses the queued payload instead of letting the sender queue retry it. Distinguish connect timeouts from send timeouts or propagate connect timeouts as transient failures in queue mode.

        if sent is None and queue_mode:
            # Connection trouble, and the caller is the background sender
            # queue: let it requeue the payload and retry once reconnected,
            # instead of dropping it here.
            return None

datadog/dogstatsd/sender_queue.py:196

• Stop is allowed past the capacity limit in put(), but requeue_front() counts that sentinel in len(self._deque). During shutdown, if the sender has an in-flight payload and stop() appends Stop, a failed send sees the queue as full and drops the payload instead of requeueing it; this loses metrics that shutdown is supposed to drain. Coordinate sentinel insertion with in-flight work or make the requeue path ignore the shutdown sentinel when enforcing capacity.

            if self._maxsize > 0 and len(self._deque) >= self._maxsize:
                self._on_drop_queue_full(item)
                self._finish_task_locked()
                return

• Files reviewed: 4/4 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[StephenWakely]

This will be addressed in a followup PR that will place a timeout on these functions.