Fair Round Robin Implementation

[tejas7777]

Close: #119

Why we are doing this

Prevent noisy-neighbor delays by queueing work per branch and dispatching fairly across branches.

Design

1. dispatch_branch_task() receives webhook work and enqueues it into a Redis LIST for that branch.
2. Redis stores:
   • jobs:branch:{branch_id} (per-branch FIFO queue)
   • jobs:branches:active (set of branches with pending jobs)
   • jobs:dispatcher:cursor (last branch processed)
3. A dedicated dispatcher worker runs continuously:
   • reads active branches
   • rotates from the cursor (round-robin)
   • dispatches at most one job per branch per cycle
4. Backpressure check pauses dispatch when Celery broker queue depth is above threshold.
5. Dispatched jobs run task_send_webhooks, which performs the outbound HTTP calls and writes webhook logs.

Mermaid Diagram

flowchart TD
    A["TC2 webhook payload"] --> B["dispatch_branch_task"]
    B --> C["Redis list: jobs:branch:BRANCH_ID"]
    B --> D["Redis set: jobs:branches:active"]

    E["Dispatcher worker loop (job_dispatcher_task)"] --> D
    E --> F["Read cursor: jobs:dispatcher:cursor"]
    E --> G{"Broker queue below threshold?"}
    G -- "No" --> H["Sleep cycle_delay"]
    H --> E
    G -- "Yes" --> I["dispatch_cycle (round robin)"]

    I --> C
    I --> J["task.apply_async"]
    J --> K["Celery broker queue"]
    K --> L["task_send_webhooks"]
    L --> M["Client webhook endpoints"]
    L --> N["WebhookLog rows"]
    I --> O["Ack queue head"]
    O --> P["Update cursor"]
    P --> E

Loading

Expected Outcome

• Fairer cross-branch processing.
• Reduced impact from one branch generating heavy traffic.
• Controlled queue growth via dispatcher-side backpressure.

[HenryTraill]

Also can you improve the current workers queue too long error on chronos to include qlength

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.04878% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
chronos/worker.py	92.30%	2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[tejas7777]

Code coverage since the bot doesn't seem to be updating the message and the page gives 404:

[tejas7777]

We weren't setting task_soft_time_limit and task_time_limit earlier so by default they were None - they concern the normal celery workers and the dispatch worker itself as we run it with these flags --soft-time-limit=0 --time-limit=0

[HenryTraill]

Lets breakdown the events array and add each event to the new queues to make the round robin equal

[tejas7777]

Codex - xHigh reasoning review on backwards compatibility:

Backward Compatibility Sign-off (PR Summary)

Approved for rollout with backward compatibility, with one operational requirement: keep dispatcher running while round-robin queues are draining.

Why this is backward compatible

1. Task contract is unchanged: task_send_webhooks(payload, url_extension) (chronos/worker.py:195).
2. Old queued batched payloads are still supported because worker-side per-event split remains (chronos/worker.py:133).
3. New split payloads are the same serialized JSON payload shape consumed by the same worker task (chronos/worker.py:320).
4. Feature-flag fallback remains available: use_round_robin=False routes new ingress directly to Celery (chronos/views.py:82).

Mixed-version safety

1. Old jobs + new workers: compatible.
2. New jobs + old workers: compatible.
3. Mixed old/new jobs during rolling deploy: compatible.

Operational guardrails

1. If issues occur, set use_round_robin=False first (stop new per-branch queue writes).
2. Keep dispatcher up until jobs:branches:active is empty.
3. Rollback only after queue drain; otherwise queued per-branch jobs can be stranded.

Bottom line

No payload-format backward-compatibility break was identified for in-flight webhook tasks. Primary risk is operational sequencing (flag/dispatcher/rollback order), not task schema incompatibility.

[tejas7777]

Testing after changes:

• Creating new integration works

• Webhooks are sent

[tejas7777]

@HenryTraill We need to make sure on Heroku that the dispatcher process never auto scales past one single instance else if we have multiple dispatchers there is risk of duplicate webhooks.