DatafileManager.importMapFiles question

[dlmarion]

The Tablet metadata is updated in DatafileManager.importMapFiles here outside of the tablet lock. Then, the datafileSizes map is modified inside of the Tablet lock here. Tablet.compareTabletInfo is called from a thread in the TabletServer periodically. Does updating the tablet metadata outside of the tablet lock in importMapFiles make it more likely that the compareTabletInfo check would see a file in the tablet metadata but not in datafileSizes? It seems to me that if compareTabletInfo ran between Tablet line 265 and line 269, then it would report missing files that may not be missing.

[ivakegg]

I would expect that this is certainly the case. What is more interesting is that if the lock fails to be acquired or some failure occurs after updating the metadata but before the datafileSizes gets updated. In that case the metadata will continually show to be different than the in-memory. Is there additional error recovery that can be done to handle that situation?

[dlmarion]

Any other thing that grabs the tablet lock, not just the call to compareTabletInfo, could hold up a bulk import and leave the tablet in a state where datafileSizes is out of sync with the tablet metadata. If the Thrift RPC that calls importMapFiles times out while waiting for the tablet lock, then this would be bad, right? I think we might want to update that metadata table inside the tablet lock.

[keith-turner]

Does updating the tablet metadata outside of the tablet lock in importMapFiles make it more likely that the compareTabletInfo check would see a file in the tablet metadata but not in datafileSizes?

compareTabletInfo relies not on the tablet lock, but on counters incremented here and incremented here.

compareTabletInfo does the following.

1. Get a copy of a tablets update counters
2. Read the tablets metadata
3. compare metadata read to what tablet has in memory
4. Get another copy of the tablets update counters
5. If the counters are exactly the same it knows its check did not overlap in time with an update. If it did not overlap in time with an update and things differ then it logs an error message.

The counter increments must cover the in memory update update and metadata update. There was a problem where the counter increments were not covering everything that was fixed in #3392.

[keith-turner]

If the Thrift RPC that calls importMapFiles times out while waiting for the tablet lock, then this would be bad, right? I think we might want to update that metadata table inside the tablet lock.

The intention is to avoid any I/O while holding the tablet lock and only do non blocking things like update in memory data structs. All scans get the tablet lock when they start to determine what files and in memory maps to use and also reserve those resources while holding the lock. Anything that did I/O while holding the tablet lock would negatively impact scans. While the intention is not do I/O while holding the tablet lock its hard to analyze the code and ensure this does not happen because code gets the lock and then calls functions. Over time one of these functions calls could have I/O added. A lot of code logs using log4j while holding the tablet lock, its possible log4j could block on I/O also.

[ivakegg]

Whatever the case is here, there will always be the possibility that the metadata may get updated but the in-memory data files will not. In the case that I have witnessed, the metadata tablet was extremely busy. The result was that the metadata got updated however the socket timed out on the client side and hence the client did not update the in-memory. I am wondering if there is a way in which the tserver can remedy this situation when it is detected. Alternatively, the metadata update is actually the trigger for the tserver to update its in-memory view. Then maybe we can avoid this half-brain scenario.

[dlmarion]

I believe the issue here is an exception happening between line 265 and line 274. I think the simplest way to handle an exception here is to unload the tablet.

https://github.com/apache/accumulo/blob/26a71d062702ad703f445ab144880c4c0e323c6e/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L248C10-L280