Skip to content

rsky-relay: clear events before polling & rsky-relay: don't try to reconnect if failed to requestCrawl#106

Merged
rudyfraser merged 5 commits into
blacksky-algorithms:mainfrom
utkarshgupta137:rsky-relay/fix-epoll
May 15, 2025
Merged

rsky-relay: clear events before polling & rsky-relay: don't try to reconnect if failed to requestCrawl#106
rudyfraser merged 5 commits into
blacksky-algorithms:mainfrom
utkarshgupta137:rsky-relay/fix-epoll

Conversation

@utkarshgupta137

Copy link
Copy Markdown
Contributor

Summary

Unlike mio crate, the polling crate doesn't automatically clear the events vec before waiting. This means that we weren't using epoll properly and were always iterating all the connections every time. This also made back-filling unsymmetrical because we were always trying to read from the connections in the same order.
Also, stop trying to reconnect crawler if the error occurred during connection phase. This fixes loops for some hosts that seem to be offline (yet the bsky relay listHosts lists them as active). In the future, a more advanced retry logic could be added to handle these cases once again.
Also tweaked some trace levels where appropriate.

Changes

  • Feature implementation
  • Bug fix
  • Documentation update
  • Other (please specify):

Checklist

  • I have tested the changes (including writing unit tests).
  • I confirm that my implementation aligns with the canonical Typescript implementation and/or atproto spec
  • I have updated relevant documentation.
  • I have formatted my code correctly
  • I have provided examples for how this code works or will be used

@rudyfraser rudyfraser merged commit 2a76939 into blacksky-algorithms:main May 15, 2025
6 checks passed
@utkarshgupta137 utkarshgupta137 deleted the rsky-relay/fix-epoll branch May 16, 2025 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants