Skip to content

Add bounded queue support for enqueue option#1431

Open
GangEunzzang wants to merge 1 commit intoDelgan:masterfrom
GangEunzzang:feature/bounded-enqueue
Open

Add bounded queue support for enqueue option#1431
GangEunzzang wants to merge 1 commit intoDelgan:masterfrom
GangEunzzang:feature/bounded-enqueue

Conversation

@GangEunzzang
Copy link

PR Title

Add bounded queue support for enqueue option (#1419)

Summary

This PR adds support for bounded queues when using enqueue to prevent unbounded memory growth with slow sinks.

Fixes #1419

Changes

New enqueue options:

Usage Description
enqueue=True Unbounded queue (existing behavior)
enqueue=100 Bounded queue with max size 100, blocks when full
enqueue={"size": 100} Same as above
enqueue={"size": 100, "overflow": "block"} Blocks when full (default)
enqueue={"size": 100, "overflow": "drop"} Drops messages when full

Overflow policies:

  • "block" (default): Blocks logging call until queue has space (backpressure)
  • "drop": Silently drops the message when queue is full (non-blocking)

Example

from loguru import logger

# Bounded queue that blocks when full
logger.add("file.log", enqueue=100)

# Bounded queue that drops messages when full
logger.add("file.log", enqueue={"size": 100, "overflow": "drop"})

Test Plan

- Added 4 new test cases for bounded queue functionality
- All 19 existing enqueue tests pass
- Tested on macOS and Linux (Docker)

Add support for bounded queues to prevent unbounded memory growth
when using enqueue=True with slow sinks.

New enqueue options:
- enqueue=<int>: bounded queue with specified max size (blocking)
- enqueue={"size": N, "overflow": "block|drop"}: configurable overflow
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enqueue=True causes unbounded memory growth with slow sinks (missing backpressure)

1 participant