Skip to content

fix: replace panic with retry loop in yellowstone slot tracker#136

Open
civa wants to merge 1 commit intorpcpool:mainfrom
civa:main
Open

fix: replace panic with retry loop in yellowstone slot tracker#136
civa wants to merge 1 commit intorpcpool:mainfrom
civa:main

Conversation

@civa
Copy link
Copy Markdown

@civa civa commented Apr 6, 2026

Problem:
Current behavior is a panic on grpc error. anything can cause a grpc error and it shouldnt halt the code forever. its an issue we face in prod already

2026-04-06T02:22:08.289462Z  INFO yellowstone_jet_tpu_client::core: Spawning connection for remote peer: EvnRmnMrd69kFdbLMxWkTn1icZ7DCceRhvmb2SJXqDo4, attempt: 1, source: Prediction
2026-04-06T02:22:16.399815Z ERROR yellowstone_jet_tpu_client::yellowstone_grpc::slot_tracker: Yellowstone slot tracker stream error: Status { code: Cancelled, message: "stream terminated by user", source: None }
thread 'tokio-rt-worker' (657774) panicked at /Users/civa/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/yellowstone-jet-tpu-client-0.3.1/src/yellowstone_grpc/slot_tracker.rs:70:17:
Box<dyn Any>
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2026-04-06T02:22:16.400170Z ERROR yellowstone_jet_tpu_client::yellowstone_grpc::sender: Yellowstone TPU sender dependency task 'slot-tracker' has failed with Err(JoinError::Panic(Id(35), ...))
thread 'tokio-rt-worker' (657774) panicked at /Users/civa/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/yellowstone-jet-tpu-client-0.3.1/src/yellowstone_grpc/schedule.rs:40:45:
load: PoisonError(411318601)

With this PR - it will reconnect with a capped(15s) exponential backoff

2026-04-06T02:24:12.139024Z  INFO yellowstone_jet_tpu_client::core: Spawning connection for remote peer: 9rkJMARqK6VBkcxGfKBAwnA44gPAfGxPbPsfsggFNDSQ, attempt: 1, source: Prediction
2026-04-06T02:24:12.251716Z ERROR yellowstone_jet_tpu_client::yellowstone_grpc::slot_tracker: Yellowstone slot tracker stream error: Status { code: Cancelled, message: "stream terminated by user", source: None }
2026-04-06T02:24:12.251743Z  WARN yellowstone_jet_tpu_client::yellowstone_grpc::slot_tracker: Yellowstone slot tracker reconnecting in 1s...
2026-04-06T02:24:13.676870Z  INFO yellowstone_jet_tpu_client::yellowstone_grpc::slot_tracker: Yellowstone slot tracker reconnected
2026-04-06T02:24:13.740345Z  WARN yellowstone_jet_tpu_client::core: Connection attempt 1 to remote peer: [HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU] failed, retrying...
2026-04-06T02:24:13.740397Z  INFO yellowstone_jet_tpu_client::core: Spawning connection for remote peer: HEL1USMZKAL2odpNBj2oCjffnFGaYwmbGmyewGv1e2TU, attempt: 2, source: Reattempt
2026-04-06T02:24:14.539056Z  INFO yellowstone_jet_tpu_client::core: Spawning connection for remote peer: 7EzbSahSfSjeRexHcNDLDpzHBAGBLjLKtjbmuoQnEtjE, attempt: 1, source: Prediction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant