Failed testnet restarts (2025-12-03 and 2025-12-06)

[willhickey]

Issue to collect information about the two failed testnet restarts. More details coming soon

[tigarcia]

On testnet restart attempt number 2, I was waiting for restart with the validator pubkey 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN . The plan was for my validator to be the last one to come online once the rest of the cluster saw over 72% of stake online. 141vSYKGRPNGieSrGJy8EeDVBcbjSr6aWkimNgrNZ6xN had 10.5% of stake, so taking my node online should have taken us well over the 80% mark. I started my validator up with the following args:

[2025-12-11T06:27:46.565716810Z INFO  agave_validator::commands::run::execute] Starting validator with: ArgsOs {
        inner: [
            "/home/sol/agave-v3/bin/agave-validator",
            "--rpc-port",
            "8899",
            "--identity",
            "/home/sol/identity.json",
            "--authorized-voter",
            "/home/sol/staked-identity.json",
            "--block-verification-method",
            "unified-scheduler",
            "--vote-account",
            "/home/sol/vote-account-keypair.json",
            "--log",
            "/mnt/data/logs/solana-validator.log",
            "--ledger",
            "/mnt/ledger",
            "--accounts",
            "/mnt/data/accounts",
            "--snapshots",
            "/mnt/data/snapshots",
            "--dynamic-port-range",
            "9000-12500",
            "--entrypoint",
            "entrypoint.testnet.solana.com:8001",
            "--entrypoint",
            "entrypoint2.testnet.solana.com:8001",
            "--entrypoint",
            "entrypoint3.testnet.solana.com:8001",
            "--expected-genesis-hash",
            "4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY",
            "--wal-recovery-mode",
            "skip_any_corrupted_record",
            "--limit-ledger-size",
            "--no-snapshot-fetch",
            "--vote-use-quic",
            "true",
            "--experimental-retransmit-xdp-cpu-cores",
            "1",
            "--experimental-poh-pinned-cpu-core",
            "10",
            "--experimental-retransmit-xdp-interface",
            "enp65s0f0np0",
            "--experimental-retransmit-xdp-zero-copy",
            "--bind-address",
            "147.28.165.17",
            "--wait-for-supermajority",
            "374301609",
            "--expected-shred-version",
            "6045",
            "--expected-bank-hash",
            "Bn8ULiSxvGdyedFFENjrcLU4PUUqX9NGASHHjuSMAKA1",
        ],
    }

The validator then crashed with the following error:

[2025-12-11T06:28:30.263917871Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at core/src/validator.rs:1591:22:
    failed to create xdp retransmitter: \"failed to raise CAP_NET_ADMIN capability: caps error: capset failure: Operation not permitted (os error 1)\"" location="core/src/validator.rs:1591:22" version="3.1.4 (src:6dcc39fc; feat:2086771155, client:Agave)"

The issue was that the binary on my machine did not have the CAP_NET_ADMIN privileges for XDP. I realized what the problem was and then restarted without the XDP flags. I also removed the --wait-for-supermajority flag. I restarted with the following args

[2025-12-11T06:31:47.319769966Z INFO  agave_validator::commands::run::execute] Starting validator with: ArgsOs {
        inner: [
            "/home/sol/agave-v3/bin/agave-validator",
            "--rpc-port",
            "8899",
            "--identity",
            "/home/sol/identity.json",
            "--authorized-voter",
            "/home/sol/staked-identity.json",
            "--block-verification-method",
            "unified-scheduler",
            "--vote-account",
            "/home/sol/vote-account-keypair.json",
            "--log",
            "/mnt/data/logs/solana-validator.log",
            "--ledger",
            "/mnt/ledger",
            "--accounts",
            "/mnt/data/accounts",
            "--snapshots",
            "/mnt/data/snapshots",
            "--dynamic-port-range",
            "9000-12500",
            "--entrypoint",
            "entrypoint.testnet.solana.com:8001",
            "--entrypoint",
            "entrypoint2.testnet.solana.com:8001",
            "--entrypoint",
            "entrypoint3.testnet.solana.com:8001",
            "--expected-genesis-hash",
            "4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY",
            "--wal-recovery-mode",
            "skip_any_corrupted_record",
            "--limit-ledger-size",
            "--no-snapshot-fetch",
            "--expected-shred-version",
            "6045",
            "--expected-bank-hash",
            "Bn8ULiSxvGdyedFFENjrcLU4PUUqX9NGASHHjuSMAKA1",
        ],
    }

Between the crash and my restart, I would have expect the cluster to proceed without my stake and continue voting. approximately 72% of stake should have been online.