add timeout when waiting for chiapos to return #19510
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is likely only a small benefit, but it will log an error sooner and cause a FarmingInfo message to be sent to the farmer when the timeout fires, which is possibly helpful in diagnostics. It also upgrades from a warning to an error in cases where it takes such a long time for the call to return.
I believe the python thread that is "occupied" doing the actual disk lookups via chiapos will continue to be occupied after this timeout, so I don't think this frees the thread to be used by another lookup. That is the thread is still blocked waiting for a chiapos response.
This just allows us to log the error sooner and return a message to the farmer rather than waiting for however long it takes the actual
blocking_lookup
call to return.My reading of the overall code is that once all the available threads (default 30) get blocked, your harvester won't be able to process any SPs and they will pile up here as they get scheduled by
run_in_executor
- possibly there should be some limit set as to how many SPs can get piled up here before we stop doing that. This would need to add in some tracking of the tasks outstanding and if too many, stop scheduling them.