Potential Daklapack Order Loss

[cassidysymons]

While placing the first round of kit orders for the Skin Biome Initiative, I experienced a timeout error. After reconciling with Daklapack, we discovered that they received (and processed) an order that we did not have any record of in our database. While this kit was caught before it was actually shipped, it appears there's a situation where an order can be transmitted but not stored in our database. I see a couple of different issues and potential courses of action:

1. Most importantly, we should evaluate how to ensure that our database has a record of every order at the time of transmission. Even if we experience a disconnect or other error during order transmission, having a record of the order would allow us to reconcile the status via their API later. The other potential scenario here is that we end up with record of an order that doesn't reach Daklapack at all, which is highly preferable to the opposite (where Daklapack receives an order that we have no record of).

2. Less importantly, it would be nice to avoid the potential for timeouts here altogether. This is not a high priority, as I don't anticipate having to place huge batches of kit orders on a frequent basis. But if we do identify a future need for frequent manual batch ordering, it would be appropriate to ingest all of the orders and allow Celery to meter out the actual orders, rather than the current model, which places all of the orders in serial while the HTTP request between Flight and Private API is still open.

[wasade]

That is quite unexpected. Good catch. Did the timeout occur in this block? https://github.com/biocore/microsetta-private-api/blob/master/microsetta_private_api/admin/admin_impl.py#L605-L621 If so, I wonder whether moving the commit above the post would suffice but I'm unsure about downstream assumptions.

For (2), we have to assume timeouts can occur even with metering. Should we use retries?