[FEATURE] Add follower-read consistency modes for raftstore

[feichai0017]

🚀 Is your feature request related to a problem?

Distributed reads are leader-centric today, which can increase tail latency and leader pressure. There is no explicit follower-read consistency mode.

💡 Proposed Solution

Add follower-read modes with clear consistency levels.

Suggested modes:

• strong (leader/read-index),
• bounded-stale,
• follower-prefer (when safe).

🎨 Alternatives Considered

1. Keep all reads on leader.
2. Add ad-hoc follower reads without explicit consistency contract.

An explicit mode contract is safer and easier to reason about.

📝 Technical Details / Implementation Plan

• Define read consistency enum/options in RPC path.
• Implement follower-read admission checks (applied index/lease/read index constraints).
• Add integration tests covering stale-read bounds and failover behavior.

🧐 Additional Context

This would improve horizontal read scalability for multi-replica deployments.

[wchwawa]

Hi! I’d like to pick up 91.

This is a bit big, so I’ll split it into a few small PRs (3+):
1. Add a ReadConsistency option (API/RPC plumbing only) so default stays the same (leader/strong).
2. Add one extra mode first (likely bounded-stale) + basic integration tests (staleness + failover).
3. Add follower-prefer last, together with the admission checks (applied index / lease / read-index).

I'll start with PR1. Let me know if you prefer specific semantics/naming for the modes.

[feichai0017]

Thanks for proposing this feature. Here is a concrete implementation plan for #91.

1) API contract (clear and minimal for v1)

Use two dimensions (consistency vs routing preference), so semantics stay explicit:

• ReadConsistency
  • STRONG (default, linearizable)
  • BOUNDED_STALE (bounded by lag/freshness checks)
• ReadPreference
  • LEADER_ONLY (default)
  • FOLLOWER_PREFER

This avoids mixing “consistency level” and “routing preference” into one enum.

For bounded stale requests, add request-side budgets:

• max_stale_read_index
• max_stale_read_ms

Backward compatibility:

• old clients (no new fields) => STRONG + LEADER_ONLY (same behavior as today).

2) Server execution model

STRONG + LEADER_ONLY

Keep current behavior:

• leader check
• ReadIndex
• wait applied
• local read

STRONG + FOLLOWER_PREFER

Allow follower serving with ReadIndex path:

• follower issues ReadIndex (forwarded to leader by raft)
• follower waits applied index
• follower executes local read
  If follower path fails/timeout, client may retry/fallback to leader.

BOUNDED_STALE (+ preference)

No ReadIndex round-trip in fast path. Serve local read only if admission passes:

• role is follower (or leader if explicitly allowed)
• commit_index - applied_index <= max_stale_read_index
• now - last_leader_contact <= max_stale_read_ms
• not in unstable role transitions
  If admission fails, return a clear region-level rejection (stale_command for v1).

3) Architecture changes needed

Current validateCommand in store enforces leader for all commands, including reads.
We should split validation into:

• base validation: region/epoch/key range/peer existence
• write-leader validation: only for write paths
• read policy admission: based on consistency + preference

This is the key structural change to support follower reads cleanly.

4) Package-level change list

pb/

• pb/kvrpcpb.proto
  • add ReadConsistency, ReadPreference
  • extend Context with read fields + stale budgets
• pb/raftcmdpb.proto
  • extend CmdHeader to carry read fields internally
• regenerate *.pb.go

raftstore/kv/

• service.go
  • buildHeader must forward new read fields
  • keep default behavior unchanged when fields are absent
• service_test.go
  • add request-plumbing + default-compat tests

raftstore/store/

• command_service.go
  • refactor validation split (base vs write-leader vs read admission)
  • implement read-path policy dispatch
• add focused tests for policy admission and fallback semantics

raftstore/peer/

• peer.go
  • track lastLeaderContact (updated on leader append/heartbeat/snapshot messages)
  • expose helpers for stale admission checks
• peer_test.go
  • leader contact freshness and lag-based admission tests

raftstore/client/

• client.go
  • add read options (new methods), keep existing methods as wrappers
  • follower selection for FOLLOWER_PREFER
  • fallback logic for strong follower-read failures
• client_test.go
  • route/fallback/retry coverage

optional metrics/docs

• add counters for follower-read attempt/success/fallback/reject reason
• document exact semantics and defaults

5) PR split

1. PR1: proto + plumbing only (no behavior change)

   • enums/fields + header propagation + compatibility tests

2. PR2: store read-path refactor + STRONG follower ReadIndex support

   • split validation
   • enable strong follower read path
   • integration tests (partition/failover)

3. PR3: client follower-prefer routing + fallback

   • follower target selection
   • fallback to leader for strong mode

4. PR4: bounded-stale admission

   • lag/freshness checks
   • stale rejection semantics + tests

5. PR5: metrics + docs cleanup