Problem decoding pools on stream start

[johnwhitton]

This replaces propeller-heads/tycho-indexer#699

Streaming of all states

Recently, when streaming, we have not been getting all states sent in the first block. We are using a wide TVL range to get all activity on the chain. Below are examples for base and unicahain (a smaller chain), which always stream all states.

• Unichain
  • Command cargo run -p solver_driver --bin arbitrager -- --chain unichain --block-count 20 --min-tvl 1 --max-tvl 1 --max-hops 3 --info
  • First block 21:49:33 INFO 📡 Received stream message: block=27422610, new_pairs=61, states=61
• Base streaming on the first block for all states for limited TVL - key change --max-tvl 15
  • Command cargo run -p solver_driver --bin arbitrager -- --chain base --block-count 100 --min-tvl 1 --max-tvl 15 --max-hops 4 --info
  • First Block 01:03:30 INFO 📡 Received stream message: block=35696628, new_pairs=803, states=803
• Base steaming no states - key change is --max-tvl 1
  • Command cargo run -p solver_driver --bin arbitrager -- --chain base --block-count 10 --min-tvl 1 --max-tvl 1 --max-hops 4 --info --route-eval --clear-db
  • First Block 01:08:11 INFO 📡 Received stream message: block=35696771, new_pairs=0, states=0

Note similar symptoms as base occurs on Ethereum (depending on the TVL values)

Below is the latest example
01:04:04 ERROR StateDecodingFailure pool="0x596192bb6e41802428ac943d2f1476c1af25cc0e000000000000000000000659" error=Unable to set up vm state on the engine: Fatal error: Simulation reverted for unknown reason: Failed to decode: 0xdca284ad
01:04:04 WARN ⚠️ State decode failure (recoverable): Stream decode error: Unable to set up vm state on the engine: Fatal error: Simulation reverted for unknown reason: Failed to decode: 0xdca284ad
01:04:04 ERROR 🚨 FIRST-BLOCK DATA LOSS: block=23438736 has zero new_pairs AND zero states - likely Tycho upstream filtering issue
01:04:04 WARN 📡 PARTIAL BLOCK DATA: block=23438736, new_pairs=0, states=0 (total_blocks=1, avg_pairs=0.0, avg_states=0.0)

Additional Information

Arbitrager CLI

     Running `target/debug/arbitrager --help`
Production streaming runner: tokens→pools→graph→routes→evaluation→execution

Usage: arbitrager [OPTIONS]

Options:
      --chain <CHAIN>
          Chain to analyze (base, ethereum, unichain) [default: base]
      --data-dir <DATA_DIR>
          Data directory root; DB path will be <data_dir>/<chain>
      --block-count <BLOCK_COUNT>
          Number of blocks to stream before stopping (0 = keep-alive) [default: 0]
      --min-tvl <MIN_TVL>
          TVL thresholds (ETH units) to bound pool selection [default: 0]
      --max-tvl <MAX_TVL>
          [default: 0]
      --max-hops <MAX_HOPS>
          Maximum path length (edges) for route generation [default: 3]
      --create-routes
          Create routes in batch mode and exit (no streaming)
      --route-eval
          Evaluate discovered routes
      --dry-run
          Dry-run execution (build calldata, no send). If false and route-eval is on, auto-exec
      --clear-db
          Clear DB at start (danger: removes datadir/chain RocksDB)
      --info
          Logging shortcuts
      --debug

      --trace

      --quiet

      --route-id <ROUTE_ID>
          Execute only a specific route ID (overrides profitability check)
      --test
          Use test data directory instead of production data directory
      --no-preflight
          Skip preflight checks and send transactions directly to chain
      --no-initial-route-eval
          Skip evaluating all preloaded DB routes on startup (avoid initial burst)
      --analyzer-queue-capacity <ANALYZER_QUEUE_CAPACITY>
          Route analyzer queue capacity override (default 2000)
  -h, --help
          Print help

Stream setup

  pub async fn build_stream_with_protocols(
        hostname: &str,
        chain: Chain,
        protocols: &[String],
        tvl_filter: ComponentFilter,
        api_key: &str,
        tokens: std::collections::HashMap<tycho_common::Bytes, tycho_common::models::token::Token>,
    ) -> Result<
        impl futures::Stream<
            Item = Result<tycho_simulation::protocol::models::Update, StreamDecodeError>,
        >,
    > {

[johnwhitton]

Another error this morning on base (run with TVL 1 and then 15) : this pool looks to have no liquidity
https://basescan.org/address/0x03b71d2c31a3b0dcd1d29ef1e67421aaac3c0ee3#readContract

09:05:32  INFO ✅ Protocol stream built successfully with 2 protocols
09:05:34 ERROR StateDecodingFailure pool="0x03b71d2c31a3b0dcd1d29ef1e67421aaac3c0ee3" error=Missing attributes tick_liquidities
09:05:34  WARN ⚠️  State decode failure (recoverable): Stream decode error: Missing attributes tick_liquidities
09:05:34 ERROR 🚨 FIRST-BLOCK DATA LOSS: block=36013492 has zero new_pairs AND zero states - likely Tycho upstream filtering issue
09:05:34  WARN 📡 PARTIAL BLOCK DATA: block=36013492, new_pairs=0, states=0 (total_blocks=1, avg_pairs=0.0, avg_states=0.0)

[johnwhitton]

For now I've just changed the error to a warning to allow processing of all valid states
decoder.rs

                  // Construct state from snapshot
                    if let Some(state_decode_f) = self.registry.get(protocol.as_str()) {
                        match state_decode_f(
                            snapshot,
                            header.clone(),
                            account_balances.clone(),
                            self.state.clone(),
                        )
                        .await
                        {
                            Ok(state) => {
                                new_components.insert(id.clone(), state);
                            }
                            Err(e) => {
                                // JW Test skipping these to get around error
                                // if self.skip_state_decode_failures {
                                    warn!(pool = id, error = %e, "StateDecodingFailure");
                                    continue 'outer;
                                // } else {
                                //     error!(pool = id, error = %e, "StateDecodingFailure");
                                //     return Err(StreamDecodeError::Fatal(format!("{e}")));
                                // }
                            }
                        }