preload memory recovery test (deterministic simulation) failed with Streaming vnode mapping not found for fragment ... #23393
Description
Buildkite URL
Description and Insights
2022-08-12T18:14:54.988749Z ERROR node{id=5 name="frontend-2"}:task{id=749335}:handle_query{mode="simple query" session_id=0 sql=INSERT INTO region (r_regionkey, r_name, r_comment) VALUES (0, 'AFRICA', 'lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '), (1, 'AMERICA', 'hs use ironic, even requests. s'), (2, 'ASIA', 'ges. thinly even pinto beans ca'), (3, 'EUROPE', 'ly final courts cajole furiously final excuse'), (4, 'MIDDLE EAST', 'uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl')}:distributed_execute{query_id="da5603f0-10be-4c3c-81f8-31b7a3827482" epoch=BatchQueryEpoch { epoch: Some(Committed(BatchQueryCommittedEpoch { epoch: 2824135965933568, hummock_version_id: 14060 })) }}:stage{otel.name="Stage da5603f0-10be-4c3c-81f8-31b7a3827482-1" query_id="da5603f0-10be-4c3c-81f8-31b7a3827482" stage_id=1}: risingwave_frontend::scheduler::distributed::stage: Failed to schedule tasks error=Streaming vnode mapping not found for fragment 1035 query_id=QueryId { id: "da5603f0-10be-4c3c-81f8-31b7a3827482" } stage_id=1
2022-08-12T18:14:54.988749Z ERROR node{id=5 name="frontend-2"}:task{id=749335}:handle_query{mode="simple query" session_id=0 sql=INSERT INTO region (r_regionkey, r_name, r_comment) VALUES (0, 'AFRICA', 'lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '), (1, 'AMERICA', 'hs use ironic, even requests. s'), (2, 'ASIA', 'ges. thinly even pinto beans ca'), (3, 'EUROPE', 'ly final courts cajole furiously final excuse'), (4, 'MIDDLE EAST', 'uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl')}:distributed_execute{query_id="da5603f0-10be-4c3c-81f8-31b7a3827482" epoch=BatchQueryEpoch { epoch: Some(Committed(BatchQueryCommittedEpoch { epoch: 2824135965933568, hummock_version_id: 14060 })) }}: risingwave_frontend::scheduler::distributed::query: query stage failed error=Streaming vnode mapping not found for fragment 1035 query_id=QueryId { id: "da5603f0-10be-4c3c-81f8-31b7a3827482" } stage_id=1
2022-08-12T18:14:54.988749Z ERROR node{id=5 name="frontend-2"}:task{id=749335}:handle_query{mode="simple query" session_id=0 sql=INSERT INTO region (r_regionkey, r_name, r_comment) VALUES (0, 'AFRICA', 'lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '), (1, 'AMERICA', 'hs use ironic, even requests. s'), (2, 'ASIA', 'ges. thinly even pinto beans ca'), (3, 'EUROPE', 'ly final courts cajole furiously final excuse'), (4, 'MIDDLE EAST', 'uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl')}: pgwire::pg_protocol: error when process message error=Failed to run the query: Scheduler error: Streaming vnode mapping not found for fragment 1035
2022-08-12T18:14:54.988749Z INFO node{id=5 name="frontend-2"}:task{id=749335}:handle_query{mode="simple query" session_id=0 sql=INSERT INTO region (r_regionkey, r_name, r_comment) VALUES (0, 'AFRICA', 'lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '), (1, 'AMERICA', 'hs use ironic, even requests. s'), (2, 'ASIA', 'ges. thinly even pinto beans ca'), (3, 'EUROPE', 'ly final courts cajole furiously final excuse'), (4, 'MIDDLE EAST', 'uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl')}: pgwire_query_log: status="err" time=0ms
2022-08-12T18:14:54.993171Z DEBUG node{id=11 name="client"}:task{id=718}: risingwave_simulation::slt::runner: Record Statement { loc: Location { file: "e2e_test/batch/../tpch/insert_region.slt.part", line: 1, upper: Some(Location { file: "e2e_test/batch/tpch.slt", line: 59, upper: None }) }, conditions: [], connection: Default, sql: "INSERT INTO region (r_regionkey, r_name, r_comment) VALUES\n(0,'AFRICA','lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '),\n(1,'AMERICA','hs use ironic, even requests. s'),\n(2,'ASIA','ges. thinly even pinto beans ca'),\n(3,'EUROPE','ly final courts cajole furiously final excuse'),\n(4,'MIDDLE EAST','uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl');", expected: Ok, retry: None } finished in 13.383939ms
thread '<unnamed>' panicked at /risingwave/src/tests/simulation/src/slt.rs:445:21:
statement failed: db error: ERROR: Failed to run the query
Caused by these errors (recent errors listed first):
1: Scheduler error
2: Streaming vnode mapping not found for fragment 1035
[SQL] INSERT INTO region (r_regionkey, r_name, r_comment) VALUES
(0,'AFRICA','lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to '),
(1,'AMERICA','hs use ironic, even requests. s'),
(2,'ASIA','ges. thinly even pinto beans ca'),
(3,'EUROPE','ly final courts cajole furiously final excuse'),
(4,'MIDDLE EAST','uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl');
at e2e_test/batch/../tpch/insert_region.slt.part:1
at e2e_test/batch/tpch.slt:59
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/std/src/panicking.rs:697:5
1: core::panicking::panic_fmt
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/core/src/panicking.rs:75:14
2: panic_display<sqllogictest::runner::TestError>
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/core/src/panicking.rs:268:5
3: panic_cold_display<sqllogictest::runner::TestError>
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/core/src/panic.rs:100:13
4: {async_fn#0}<risingwave_simulation::client::RisingWave, risingwave_simulation::slt::run_slt_task::{async_fn#0}::{closure_env#0}, sqllogictest::column_type::DefaultColumnType>
at ./src/tests/simulation/src/slt.rs:445:21
5: {async_fn#0}
at ./src/tests/simulation/src/slt.rs:545:67
6: {async_block#2}
at ./src/tests/simulation/src/main.rs:274:52
7: {async_block#0}<risingwave_simulation::main::{closure#0}::{async_block#0}::{async_block_env#2}>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/task/mod.rs:645:28
8: poll<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_simulation::main::{closure#0}::{async_block#0}::{async_block_env#2}>>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.4.0/src/runnable.rs:455:68
9: run<core::pin::Pin<alloc::boxed::Box<async_task::runnable::{impl#6}::spawn_local::Checked<madsim::sim::task::{impl#10}::spawn_inner::{closure#0}::{async_block_env#0}<risingwave_simulation::main::{closure#0}::{async_block#0}::{async_block_env#2}>>, alloc::alloc::Global>>, (), madsim::sim::task::{impl#10}::spawn_inner::{closure_env#1}<risingwave_simulation::main::{closure#0}::{async_block#0}::{async_block_env#2}>, alloc::sync::Weak<madsim::sim::task::TaskInfo, alloc::alloc::Global>>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-task-4.4.0/src/raw.rs:563:17
10: {closure#0}
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/task/mod.rs:287:50
11: do_call<madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}, ()>
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/std/src/panicking.rs:589:40
12: catch_unwind<(), madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}>
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/std/src/panicking.rs:552:19
13: catch_unwind<madsim::sim::task::{impl#3}::run_all_ready::{closure_env#0}, ()>
at /rustc/28f1c807911c63f08d98e7b468cfcf15a441e34b/library/std/src/panic.rs:359:14
14: run_all_ready
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/task/mod.rs:287:17
15: block_on<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/task/mod.rs:240:18
16: block_on<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/runtime/mod.rs:129:19
17: {closure#0}<risingwave_simulation::main::{closure#0}::{async_block_env#0}>
at ./.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/madsim-0.2.31/src/sim/runtime/builder.rs:141:38
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
context: node=11 "client", task=718 (spawned at /risingwave/src/tests/simulation/src/cluster.rs:657:21)
note: run with `MADSIM_TEST_SEED=3` environment variable to reproduce this error