Draft
Conversation
# Conflicts: # swatch-tally/ct/java/api/TallySwatchService.java
|
⛏️ Workflow Run 🧪 JUnit
Details
|
||||||||||||||||
- Add tallyOrgAsync() to TallySwatchService for async tally path (enqueues UPDATE_SNAPSHOTS to main tasks topic vs sync in-process) - Add nightlySnapshotTasksAreProducedToMainTasksTopic test to verify nightly tasks go to main tasks topic; use TASKS_TOPIC env for Clowder topic suffix - Add Given/When/Then structure and should* naming per component test standards Made-with: Cursor
Keep triggerHourlySnapshotsForAllOrgs and tallyOrgAsync (task queue isolation), add getInstancesByProduct Javadoc from incoming branch. Remove duplicates. Made-with: Cursor
- Add Task Queue Isolation section to TEST_PLAN.md (tally-task-queue-isolation-TC001, tally-task-queue-isolation-TC002) - Add task queue isolation to Scope - Annotate TallyTaskQueueIsolationComponentTest methods with @TestPlanName - Apply spotless formatting Made-with: Cursor
- CaptureSnapshotsTaskManagerTest: tallyOrgByHourlyWithThreadPoolExecutorUsesInMemoryQueue, nightlyAndHourlyUseDifferentTopics - TEST_PLAN: tally-task-queue-isolation-TC003 through TC006 (idempotency, no duplicates, no cross-topic leakage) - Apply spotless formatting Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira issue: SWATCH-4617
Description
Problem
When the nightly tally runs, it enqueues a large batch of tasks to the same Kafka topic that hourly tally uses. A single consumer pool processes that topic, so hourly tasks sit behind the nightly batch for 3–6 hours. That delays hourly usage metrics and triggers “remittance halted” alerts (e.g. no increase in
swatch_producer_metered_totalwithin the 3h window).Approach
Isolate hourly snapshot work from nightly by using a separate Kafka topic and dedicated consumer for hourly tally tasks. Nightly tasks stay on the existing tasks topic; hourly tasks go to a new topic and are consumed by a second listener. Same application and task logic, different topic/consumer so hourly is never blocked by the nightly backlog.
Changes (conceptual)
New topic
platform.rhsm-subscriptions.tally-hourly-tasks(same partition/replica layout as the main tasks topic). Declared in swatch-tally’s ClowdApp.Producer (swatch-tally)
CaptureSnapshotsTaskManagernow enqueuesUPDATE_HOURLY_SNAPSHOTSto the hourly topic (from new config) and keeps enqueueingUPDATE_SNAPSHOTS(nightly) to the existing tasks topic. One producer, topic chosen by task type.Consumer (swatch-tally)
A second Kafka listener (
tallyHourlyTaskProcessor) subscribes only to the hourly topic with its own consumer group (swatch-tally-hourly-processor). SameTallyTaskFactoryand task execution; no double-processing.Config
New
rhsm-subscriptions.tally-hourly-tasksblock (topic + consumer group). Optional env (e.g.TALLY_HOURLY_TASKS_TOPIC) for Clowder; defaults work for local/bonfire.Tests
Unit test updated so hourly descriptors use the hourly topic. New component test (
TallyTaskQueueIsolationComponentTest) verifies that triggering “hourly for all orgs” producesUPDATE_HOURLY_SNAPSHOTSon the hourly topic and that single-org nightly producesUPDATE_SNAPSHOTSon the main tasks topic.Result
Hourly and nightly tally run on separate streams. Hourly tasks are processed by the dedicated consumer and no longer wait behind the nightly batch, which should reduce delayed hourly metrics and related alerts. No changes to swatch-billable-usage or downstream; isolation is entirely within swatch-tally’s task routing and consumers.