Refactor replicate object with async await - #2780
Conversation
Hello sylvainsenechal,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
| /** | ||
| * Runs an async task for each item in a collection, up to `limit` at a time. | ||
| * On error, no new tasks are started, but already-running tasks are awaited | ||
| * before the function resolves. Returns [firstError, results] so that callers |
There was a problem hiding this comment.
Return the first error only : Keeping same behavior as before with mapLimitWaitPendingIfError.
Could've returned all errors as a list but not needed now and prefer to keep existing behavior.
Cannot aggregate errors neither, as we need to keep the original error to check its properties to determine if its retryable or not
There was a problem hiding this comment.
refactored with async await, in the file runTaskWithConcurrency
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
... and 3 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.5 #2780 +/- ##
===================================================
+ Coverage 75.43% 75.48% +0.05%
===================================================
Files 201 202 +1
Lines 13928 13949 +21
===================================================
+ Hits 10506 10529 +23
+ Misses 3412 3410 -2
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
406a3d4 to
7f5778f
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
7f5778f to
5680800
Compare
| } | ||
| } | ||
|
|
||
| _refreshSourceEntry(sourceEntry, log, cb) { |
There was a problem hiding this comment.
Partial hierarchy migration: ReplicateObject._refreshSourceEntry was migrated to async _refreshSourceEntry(sourceEntry, log) (no callback), but this override still uses the callback signature (sourceEntry, log, cb). Not broken today because MultipleBackendTask.processQueueEntry calls it with a callback, but if any code path ever calls await this._refreshSourceEntry(sourceEntry, log) on a MultipleBackendTask instance, cb will be undefined and the method will throw.
Consider migrating this override to async in the same PR to keep the hierarchy consistent, per the "migrate when you touch" rule for class methods.
this commit is mandatory as multipleBackend is a class that extends replicateObject and some functions are overloaded Issue: BB-803
5680800 to
64afb79
Compare
| if (err.$metadata?.httpStatusCode === 404) { | ||
| return doneOnce(err); | ||
| // eslint-disable-next-line no-param-reassign | ||
| err.origin = 'source'; |
There was a problem hiding this comment.
Behavioral change: the old stream error handler returned errors.ObjNotFound (no .origin property). In _handleReplicationOutcome (line 1034), the ObjNotFound check matched but err.origin === 'source' was false, so it entered _processQueueEntryRetryFull. With err.origin = 'source' now set, _handleReplicationOutcome instead skips the entry immediately via done().
Old path: source 404 during streaming → retry full → next attempt likely gets a send-level 404 (no ObjNotFound) → marked FAILED
New path: source 404 during streaming → skipped silently (no FAILED status published)
The new behavior is arguably more correct (the source object is gone, retrying is pointless), but it changes replication status reporting in a production path — entries that were previously marked FAILED now produce no status at all.
| }); | ||
| throw errors.BadRole; | ||
| } | ||
| } |
There was a problem hiding this comment.
Hierarchy contract mismatch: ReplicateObject._setupRolesOnce now returns [sourceRole, targetRole] (line 348), but this override returns nothing. _setupRoles resolves to undefined, which would throw TypeError: undefined is not iterable if destructured like ReplicateObject.processQueueEntry does (const [, targetRole] = await this._setupRoles(...)). Currently safe because MultipleBackendTask._setupClients ignores the result, but returning the roles would keep the hierarchy contract consistent.
There was a problem hiding this comment.
Migrations here aren't the main point of this pr, but were kinda forced as MultipleBackend extends ReplicateObject.
I did the minimum required migration on this file
| await this._deleteOrphans(destEntry, destLocations, log); | ||
| throw err; | ||
| } | ||
| return this._handleReplicationOutcome(null, sourceEntry, destEntry, kafkaEntry, log, done); |
There was a problem hiding this comment.
_handleReplicationOutcome(null, …, done) is called from inside the try: if it throws after calling done, the catch calls it again and done fires twice. We can compute the error in the try/catch and call the handler exactly once after it. (same at line 909 and in _processQueueEntryRetryFull)
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
The following reviewers are expecting changes from the author, or must review again: |
3 participants
Issue: BB-803
Review hints :
Imo this refactor is super overdue and not so complicated to do.
Refactor replicate object with async await