POST bulk and skip duplicates

[ffiorentinolabid]

Feature Description

Insert a batch of documents, skipping duplicates (on unique indexes, i.e. E11000 duplicate key error).

Desired solution

Expose ordered option on POST /bulk endpoints. Default value is true, and mongo stop operation on first invalid object in the queue, otherwise if specified as false the operation would continue and it insert all valid documents, returning an array writeErrors

Possible alternatives

It can be managed also with PATCH /bulk, with upsert: true, maybe the same requested on #11

[danibix95]

Hi! Thank you for the feedback.

Allowing unordered operation when performing bulk operations, such as insertMany, may enable skipping duplicate key errors. However, this would open some issues to be taken into account.
In particular, in case an insert operation fails for any reason other than duplicate key, such operation may be lost without allowing the caller to know of the failure (the REST API should return a HTTP 200 code because errors were skipped).

Moreover, it is not recommended to tie the REST interface to the underlying result of the MongoDB bulk operation, since

• the coupling between them would increase, potentially leading to more complex situation to be maintained in the future
• the errors details returned by MongoDB may change over time in an unforeseen manner

[ffiorentinolabid]

Hi! Thank you for the feedback.

Allowing unordered operation when performing bulk operations, such as insertMany, may enable skipping duplicate key errors. However, this would open some issues to be taken into account. In particular, in case an insert operation fails for any reason other than duplicate key, such operation may be lost without allowing the caller to know of the failure (the REST API should return a HTTP 200 code because errors were skipped).

Moreover, it is not recommended to tie the REST interface to the underlying result of the MongoDB bulk operation, since

• the coupling between them would increase, potentially leading to more complex situation to be maintained in the future
• the errors details returned by MongoDB may change over time in an unforeseen manner

I agree to not tie REST interface to underlying operation but I guess that in some cases the interface can (and already act like this for other things) just act as a "proxy" allowing to specify advanced options and returning direct messages from the database, without worrying about mapping or managing changes.

In any case, do you think that any other solution for skipping duplicates could be available?

[danibix95]

Hi! Could a bulk operation that performs a replace action over documents instead of an insert be a possible solution?

[ffiorentinolabid]

Do you mean an upsert bulk like suggested in the request that I've linked?

[danibix95]

I was thinking more to a bulk operation that performs only replace actions instead of insert ones.

[ffiorentinolabid]

If the replace does not have upsert option it means that it must be known in advance the duplicates, and that does not cover what I have described

[danibix95]

Probably I did not stated clearly, but I was referring to an endpoint like

PUT /<collection-name>/bulk

where documents provided in the body could either replace existing ones or being inserted as new (using the corresponding upsert flag set to true). Using the PUT request method would highlight the fact that existing documents would be replaced instead of throwing an error.

This should partially cover your use case

[ffiorentinolabid]

@danibix95 This could be fine, I agree

[ffiorentinolabid]

@danibix95 Any update on this topic? Any service enhancement done or ongoing that could help with the specified use case?

[albertotessarotto]

At the moment, there have been no improvements to the service useful for this described scenario, and no developments in this direction are currently in roadmap.
We are considering a refactor of the service in the future in order to make it more performant, and with this in mind in that time we will consider all issues related to new features proposals on crud and consider whether to include them in such developments.