feat: Implement provider attempt tracking to prevent immediate exclusion on assignment failure

[onahprosper]

Description

This PR implements provider attempt tracking with retry logic to prevent immediate exclusion of providers when they fail to process orders. Instead of immediately adding providers to the exclude list on assignment failure, the system now tracks how many times each provider has been attempted for an order and only excludes them after 3 failed attempts.

Background:
Previously, when a provider failed to accept or fulfill an order (indicated by order_request expiration), they were immediately added to the exclude list. This was too aggressive and prevented providers from retrying after temporary issues like network connectivity problems or temporary service downtime.

Changes:

• Added Redis-based attempt tracking using hash \order_provider_attempts_{orderID}\\ with provider ID as field and count as value
• Implemented \IncrementProviderAttemptCount\\ to track attempts when order_request expires
• Added \getProviderAttemptCount\\ helper to check current attempt count for a provider
• Moved attempt count check to after rate/balance validation but before actual assignment
• Only exclude providers if they have been attempted 3+ times
• Use refund timeout (OTC vs regular) as TTL for attempt tracking data
• Removed immediate exclusion on assignment failure in \assignOtcOrder\\ and \sendOrderRequest\\
• Added attempt count check before OTC order assignment and regular order assignment

Impact:

• Providers can retry orders up to 3 times before being excluded
• Temporary failures (network issues, temporary downtime) don't permanently exclude providers
• Only valid assignment attempts are counted (after all validations pass)
• Attempt tracking automatically expires based on refund timeout, ensuring cleanup

References

• Closes feat: Implement provider attempt tracking to prevent immediate exclusion on assignment failure #649

Testing

Manual Testing Steps:

1. Create an order and assign it to a provider
2. Wait for order_request to expire (or manually expire it)
3. Verify that the provider is still eligible for reassignment (not excluded)
4. Repeat steps 2-3 two more times
5. On the 4th reassignment attempt, verify that the provider is excluded

Test Scenarios:

• Provider fails 1-2 times: should still be eligible for assignment
• Provider fails 3+ times: should be excluded
• Rate/balance validation failures: should NOT count as attempts
• OTC orders: should use \OrderRefundTimeoutOtc\\ for TTL
• Regular orders: should use \OrderRefundTimeout\\ for TTL
• Attempt tracking data: should expire after refund timeout

Environment:

• Go 1.21+

• Redis (for attempt tracking)

• PostgreSQL (for order data)

• This change adds test coverage for new/changed/fixed functionality

Checklist

• I have added documentation and tests for new/changed functionality in this PR
• All active GitHub checks for tests, formatting, and security are passing
• The correct base branch is being used, if not \main\\

By submitting a PR, I agree to Paycrest's Contributor Code of Conduct and Contribution Guide.

Summary by CodeRabbit

• Improvements
  • Enhanced order assignment logic with automatic provider rotation tracking
  • Refined handling for different order types during provider assignment
  • Optimized provider request management with attempt-based exclusion

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Implements provider attempt tracking for order reassignment. The system tracks provider assignment attempts per order using Redis, incrementing counts when order requests expire. During reassignment, providers with 3+ attempts are excluded. TTLs differ for OTC versus regular orders.

Changes

Cohort / File(s)	Summary
Provider Attempt Tracking services/priority_queue.go	Added IncrementProviderAttemptCount method and private getProviderAttemptCount helper to track provider attempts per order using Redis hash order_provider_attempts_<orderID>. Implements pre-assignment checks in both OTC and regular matchRate paths—if a provider has 3+ attempts, it's added to order_exclude_list_<orderID> with appropriate TTL (OrderRefundTimeoutOtc or OrderRefundTimeout) and skipped. Includes providerId in Redis order request data for tracking. Sets provider in database after balance reservation in sendOrderRequest.
Attempt Counting on Reassignment tasks/tasks.go	Modified ReassignStaleOrderRequest to use shared PriorityQueueService instance and determine orderType ("regular" or "otc") from order metadata. Calls IncrementProviderAttemptCount when order request expires, logging warnings on failure but continuing. Replaced direct service instantiation with shared instance for consistent state management.

Sequence Diagram

sequenceDiagram
    participant OrderTask as Order Reassignment Task
    participant PQService as PriorityQueueService
    participant Redis as Redis Cache
    participant DB as Database

    OrderTask->>PQService: Initial AssignPaymentOrder(orderID, providerID)
    PQService->>Redis: Check order_exclude_list_<orderID>
    Redis-->>PQService: Not in exclude list
    PQService->>DB: Assign provider
    DB-->>PQService: Success
    
    Note over OrderTask: Order request expires<br/>(provider didn't accept)
    
    OrderTask->>PQService: ReassignStaleOrderRequest
    OrderTask->>PQService: IncrementProviderAttemptCount(orderID, providerID, orderType)
    PQService->>Redis: HINCRBY order_provider_attempts_<orderID> providerID 1
    Redis-->>PQService: Attempt count = 3
    PQService->>Redis: Set TTL on attempt tracking (based on orderType)
    Redis-->>PQService: TTL updated
    
    OrderTask->>PQService: matchRate (reassignment)
    PQService->>Redis: getProviderAttemptCount(orderID, providerID)
    Redis-->>PQService: Attempt count = 3
    PQService->>Redis: Add to order_exclude_list_<orderID>
    Redis-->>PQService: Excluded
    PQService-->>OrderTask: Retry with next provider

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 Three chances, no more, no less,
We track each provider's request,
When attempts reach three, they take a rest,
Fair retries for the order's quest! 🔄

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main feature: implementing provider attempt tracking to prevent immediate exclusion on assignment failure.
Description check	✅ Passed	The description covers purpose, background, technical changes, impact, manual testing steps, and checklist items; it follows the template structure with all major sections completed.
Linked Issues check	✅ Passed	All acceptance criteria from issue #649 are met: attempt tracking via Redis hash, 3-attempt threshold, validation failures not counted, TTL-based expiry, and removal of immediate exclusion logic.
Out of Scope Changes check	✅ Passed	All code changes in services/priority_queue.go and tasks/tasks.go are directly related to implementing the attempt tracking feature; no unrelated modifications detected.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tasks/tasks.go (1)

1294-1342: Potential bypass of attempt count check when ProviderID is set.

When ReassignStaleOrderRequest calls AssignPaymentOrder with ProviderID set (line 1328), the attempt count check in matchRate is bypassed. In AssignPaymentOrder (lines 437-500 in priority_queue.go), if ProviderID is set and not in the exclude list, the code directly calls assignOtcOrder or sendOrderRequest without checking the provider's attempt count.

This means a provider could be retried indefinitely as long as they're not added to the exclude list, defeating the purpose of the 3-attempt limit for orders with a preferred provider.

Consider either:

1. Adding attempt count check before lines 472-476 and 479-482 in priority_queue.go, or
2. Setting ProviderID to empty string after incrementing attempts, so the order goes through matchRate which has the attempt check

Option 2: Clear ProviderID after incrementing attempts

 		err = priorityQueueService.IncrementProviderAttemptCount(ctx, order.ID.String(), providerID, orderType)
 		if err != nil {
 			logger.WithFields(logger.Fields{
 				"Error":      fmt.Sprintf("%v", err),
 				"OrderID":    order.ID.String(),
 				"ProviderID": providerID,
 			}).Warnf("Failed to increment provider attempt count")
 			// Continue anyway - not critical
 		}
+		// Clear provider ID so order goes through matchRate which checks attempt counts
+		providerID = ""
 	}

🧹 Nitpick comments (1)

services/priority_queue.go (1)

758-781: Provider tracking fallback consideration.

The provider is set in the database for attempt tracking when the order request expires. If this fails (logged as warning), the providerId is still available in Redis (line 836), but ReassignStaleOrderRequest in tasks/tasks.go retrieves the provider from the database relation (order.Edges.Provider), not from the Redis hash.

Consider whether the Redis providerId field should be used as a fallback in ReassignStaleOrderRequest when order.Edges.Provider is nil, to ensure attempts are tracked even if the database update fails.

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b1b237c and 4299fe6.

📒 Files selected for processing (2)

• services/priority_queue.go
• tasks/tasks.go

🧰 Additional context used

🧬 Code graph analysis (2)

tasks/tasks.go (2)

services/priority_queue.go (1)

• NewPriorityQueueService (39-43)

ent/paymentorder/paymentorder.go (2)

• OrderType (313-313)
• OrderTypeOtc (320-320)

services/priority_queue.go (2)

storage/redis.go (1)

• RedisClient (13-13)

utils/logger/logger.go (4)

• WithFields (76-109)
• Fields (73-73)
• Infof (125-130)
• Warnf (133-138)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: build-and-test

🔇 Additional comments (7)

services/priority_queue.go (6)

7-7: LGTM!

The new imports for strconv (parsing attempt counts) and redis (checking redis.Nil error) are correctly added and necessary for the new attempt tracking functionality.

Also applies to: 25-25

---

674-700: LGTM!

The implementation correctly uses atomic HIncrBy for concurrent-safe increment, applies appropriate TTLs based on order type, and refreshes TTL on each increment as specified in the requirements. The logging provides good observability.

---

702-719: LGTM!

The method correctly handles the redis.Nil case by returning 0 (provider hasn't been attempted yet), which is the expected behavior for the attempt tracking logic.

---

595-598: LGTM!

Adding providerId to the Redis order request data enables tracking which provider was assigned when the order request expires, which is essential for the attempt tracking feature.

---

1062-1092: LGTM!

The attempt count check for OTC orders is correctly placed after rate validation but before assignment, as specified in the requirements. The fail-open approach (allowing assignment if count retrieval fails) is appropriate to avoid blocking orders due to Redis issues. The 3+ attempts threshold and exclusion logic are correctly implemented.

---

1108-1142: LGTM!

The attempt count check for regular orders is correctly placed after balance validation but before assignment. The implementation mirrors the OTC logic with the appropriate TTL (OrderRequestValidity*2 for regular orders). The existing recursive call to AssignPaymentOrder on failure allows the system to try another provider.

tasks/tasks.go (1)

1299-1303: LGTM!

The order type determination correctly maps the enum value to the string expected by IncrementProviderAttemptCount. Using the constant paymentorder.OrderTypeOtc ensures type safety.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}