Batch-invalidate files to improve latency when many files have changed (such as after branch switches)

[stuhood]

Currently, the InvalidationWatcher handles one event at a time. When thousands of files have changed (such as after a branch switch), this can lag behind by seconds to minutes (as observed by watching .pants.d/pants.log: particularly in debug mode).

To handle multiple events at a time, the event handler thread could attempt to drain the queue of events (by looping on Receiver::try_receive) each time it receives one event. The handle_event method would then become handle_events, which would allow for a larger set of invalidations to hit the impl Invalidatable for Graph instance at a time.

[huonw]

For the large batch invalidations like a branch switch, do the events all appear "instantly"? I imagine they'll generally incrementally appear over a short period, rather than being pushed as a single large batch from the OS.

If this is the case, a try_recv call straight away might happen too quickly and miss the following events? Thus, brainstorming:

• we could set a deadline slightly in the future (e.g. first receive time + 10ms) and loop on the recv_deadline method instead?
• or, maybe this isn't a problem... if there's a large batch, we might only pick up one event to start, but while we're processing that invalidation, the other events will appear, and the next loop through the event handler will pick them all up (so we do a batch of size 1 and then a batch of size n - 1, which is probably good enough)?

[stuhood]

we could set a deadline slightly in the future (e.g. first receive time + 10ms) and loop on the recv_deadline method instead?

That's cheap enough to do, so yea: probably worth doing.

But as mentioned in the description, in debug mode, user-space can already lag behind the kernel by a lot, so I don't think that we need to do much/anything to make the batching worthwhile.