Ignore spurious PollerCompletionQueue errors in AsyncioExecutor

[maffoo]

When using grpc with asyncio from multiple threads, spurious PollerCompletionQueue errors are printed due to multiple event loops listening a socket to be notified of grpc events. More than one event loop may be woken when the completion queue writes a byte to the notification socket, but only one of the loops receives the data and the others raise a BlockingIOError. This doesn't actually cause a problem in the grpc still works with asyncio in multiple threads, but lots of spurious error messages are printed by the default exception handler. (See grpc/grpc#25364 for discussion of the issue.)

This configures the asyncio event loop used for grpc with an exception handler that ignores these BlockingIOErrors from the PollerCompletionQueue so that we don't spam the logs with scary-looking messages when using grpc with asyncio from multiple threads.

[verult]

Is this function called somewhere?

[maffoo]

Oops, forgot to actually set this as the exception handler. Fixed.

[maffoo]

Note: I filed grpc/grpc#36096 to fix these errors upstream. We'll see what comes of that, but in any case it will be a while before we can rely on that and so I think silencing these errors in our own event loops makes sense.

[dstrain115]

@maffoo Is this still something we want to do? Should we close this old PR or push it over the finish line?

[maffoo]

I still think this is nice to have since the error messages look quite scary when printed even though everything is working fine. The error messages only happen when using grpc from multiple threads, which is somewhat niche, but note that this can happen even for users who are not themselves using multiple threads, for example if they use something like a cloud library that itself makes grpc calls in a background thread.

If we decide to go ahead I will rebase this to get it up to date with main.

[pavoljuhas]

LGTM if still relevant. Thank you!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.65%. Comparing base (02941bf) to head (be0bc43).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6492      +/-   ##
==========================================
- Coverage   98.65%   98.65%   -0.01%     
==========================================
  Files        1106     1106              
  Lines       95869    95871       +2     
==========================================
+ Hits        94581    94582       +1     
- Misses       1288     1289       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.