How to resolve "node is not the leader"

[ran83816apple]

Hello everyone,

I deployed single mode pyroscope v2.2.0 with v2 architecture. After running about one night, the log indicates error "node is not the leader". On the other hand, I also deployed single mode pyroscope v2.2.0 using v1 architecture and it runs correctly for 22 hours. Should I change configuration for v2 architecture? Please help. Thanks. @kavu @vbehar @appleboy

pyroscope configuration:

target: all                        # 单二进制模式
architecture_storage: "v2"

server:
  http_listen_port: 4040
  grpc_listen_port: 9095

# V2 元数据存储（单节点 Raft）
metastore:
  raft:
    dir: /data/pyroscope/v2/metastore/raft   # 只保留 dir，模板里没有 bootstrap_expect_peers
  data_dir: /data/pyroscope/v2/metastore/data

# V2 段数据落的对象存储
storage:
  backend: filesystem
  filesystem:
    dir: /data/pyroscope/storage

# V2 写入路径，单节点 ring 用 inmemory
segment_writer:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

# 多租户关闭（单节点建议）
multitenancy_enabled: false

log shows error:

ts=2026-07-30T01:05:52.560353946Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.5604358Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=236.465µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.560750567Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
ts=2026-07-30T01:05:52.658607866Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.658752895Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=239.177µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.65914919Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
ts=2026-07-30T01:05:52.756024389Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.756124375Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=265.545µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.75663447Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
ts=2026-07-30T01:05:52.845029928Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.84515799Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=280.844µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.845677473Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
ts=2026-07-30T01:05:52.945982301Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.946112421Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=257.581µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:52.946648761Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
ts=2026-07-30T01:05:53.041634207Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:53.041781346Z caller=grpc_logging.go:76 level=warn method=/metastore.v1.CompactionService/PollCompactionJobs duration=394.17µs msg=gRPC err="rpc error: code = Unavailable desc = node is not the leader"
ts=2026-07-30T01:05:53.042503305Z caller=methods.go:50 component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=localhost server_address=localhost:9095 server_resolved_address=localhost:9095
2026-07-30T01:05:53.130Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-07-30T01:05:53.130Z [INFO]  raft: entering candidate state: node="Node at 127.0.0.1:9099 [Candidate]" term=30516
2026-07-30T01:05:53.130Z [DEBUG] raft: pre-voting for self: term=30516 id=localhost:9099
2026-07-30T01:05:53.130Z [DEBUG] raft: calculated votes needed: needed=1 term=30516
2026-07-30T01:05:53.130Z [DEBUG] raft: pre-vote received: from=localhost:9099 term=30516 tally=0
2026-07-30T01:05:53.130Z [DEBUG] raft: pre-vote granted: from=localhost:9099 term=30516 tally=1
2026-07-30T01:05:53.130Z [INFO]  raft: pre-vote successful, starting election: term=30516 tally=1 refused=0 votesNeeded=1
ts=2026-07-30T01:05:53.130929322Z caller=recovery.go:81 component=metastore msg="recovery already stopped"
ts=2026-07-30T01:05:53.130994458Z caller=cleaner.go:82 component=metastore msg="index cleaner already stopped"
2026-07-30T01:05:53.268Z [DEBUG] raft: voting for self: term=30516 id=localhost:9099
2026-07-30T01:05:53.391Z [DEBUG] raft: vote granted: from=localhost:9099 term=30516 tally=1
2026-07-30T01:05:53.391Z [INFO]  raft: election won: term=30516 tally=1
2026-07-30T01:05:53.391Z [INFO]  raft: entering leader state: leader="Node at 127.0.0.1:9099 [Leader]"
2026-07-30T01:05:53.391Z [ERROR] raft: failed to commit logs: error="unable to store logs within log store, err: \"non-monotonic log entries: tried to append index 11091 after 11091\""
2026-07-30T01:05:53.391Z [INFO]  raft: entering follower state: follower="Node at 127.0.0.1:9099 [Follower]" leader-address= leader-id=
ts=2026-07-30T01:05:53.391554144Z caller=recovery.go:71 component=metastore msg="recovery started"
ts=2026-07-30T01:05:53.39161065Z caller=compaction_service.go:96 level=error component=metastore msg="failed to prepare compaction plan" err="rpc error: code = Unavailable desc = node is not the leader"