Skip to content

[manager] bound P2P host cache queries - #284

Draft
charpty wants to merge 12 commits into
mainfrom
agent/pr277-review-fixes
Draft

[manager] bound P2P host cache queries#284
charpty wants to merge 12 commits into
mainfrom
agent/pr277-review-fixes

Conversation

@charpty

@charpty charpty commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Standalone follow-up to #277, rebased onto the latest codex/mu-main at ce7a6c1f.

This PR contains only the remaining P2P query scalability and correctness work. It does not add protobuf/API compatibility code and does not include the previously explored storage availability, unavailable-backend event filtering, or service-metrics fallback changes.

Why this change is needed

With p2p_host_count > 0, both ordinary and Mamba queries still materialized two request-wide key -> host -> set<spec> graphs. At one million keys this creates an object graph proportional to key * host * spec, dominating both RSS and latency. Caching around the old reducer would not remove that asymptotic cost, so the two P2P reducers are replaced rather than patched locally.

Implementation

  • Add MetaIndexer::PrefixVisitOrder::ORDERED: backend chunks are read in bounded parallel windows, while stateful visitors consume callbacks in absolute key order. Existing callers keep the concurrent default.
  • Ordinary P2P interns host names to request-local dense IDs, projects only one key at a time, and maintains peer-prefix intersections only for the dynamic final top-N hosts.
  • Mamba P2P uses up to three bounded ordered phases: local scoring, independent F* full-group peer planning plus L* state-coverage voting, and final validation/projection only when an actual peer plan is selected.
  • Empty spec groups remain vacuously satisfied but allocate no peer tracker or fetch range. When every F/L group is empty, Mamba returns after the single local scan.
  • Preserve each full group's independent selected fetch range even beyond the shortest combined match prefix, so p2p_1_fetch remains exact; repeated block-key values are de-duplicated.
  • Revalidate each selected plan against the final metadata view. Hard read errors are propagated and inconsistent peer plans fail closed instead of returning a partial result.
  • Backends without progressive local reads perform one compact batch read and replay that same batch for later Mamba phases, preserving one backend I/O rather than introducing repeated remote reads.
  • Aggregate backend wall time across all progressive Mamba phases instead of leaving the metric at the final scan only.

The production diff is concentrated in meta_searcher.cc (the two old reducers are replaced) and meta_indexer.{h,cc} (the ordered visitor). Most of the remaining added lines are the independent oracle, edge tests, and manual benchmark.

Preserved semantics and edge coverage

  • dynamic top-N selection and lexical tie-breaking
  • independent full-group peers and one state-coverage peer
  • F* / L* group classification from current mu-main, with no legacy-name fallback
  • Eagle pop boundaries, including selected fetches at the popped boundary
  • empty spec groups, multiword masks over 64 specs, medium/status/URI filtering
  • duplicate block-key fetched-count de-duplication
  • metadata misses, hard errors, and metadata changes between Mamba phases
  • non-progressive backend single-read behavior
  • no final pass when no peer is selected

A deterministic materialized map/set oracle compares every output field for ordinary and Mamba queries over 6000 keys, 7 hosts, three medium filters, Eagle on/off, and top-N 0/1/3/7.

Performance evidence

Strict same-machine A/B on mu-main@16809002, Release/O2, four query workers, pure-local metadata, one million keys, and a peer-covered local gap at key 1024:

Scenario Before This PR Change
ordinary P2P p50 632.07 ms 162.81 ms -74.2%
Mamba P2P p50 635.87 ms 370.01 ms -41.8%
peak RSS 1,340,012 KiB 586,120 KiB -56.3% (about -736 MiB)

After rebasing to ce7a6c1f, the final manual smoke benchmark passed and measured 176.30 ms ordinary / 407.05 ms Mamba on the shared development machine. These values are regression evidence, not an online SLA.

Validation

  • external full ASAN: 110 passed, 1 repository-configured skip, 0 failed
  • internal full ASAN with --config=vcns, including Vcns/TairMempool/Pace tests: 113 passed, 1 repository-configured skip, 0 failed
  • final Release million-key benchmark passed and validated local, p2p_1_fetch, and p2p_1_total_match
  • formatting, diff checks, secret-pattern scan, one-commit scope, and clean worktree verified

Deliberate trade-offs

  • Pure-local Mamba uses up to three bounded scans to remove the request-wide object graph. Later phases revisit only ranges needed by selected plans, so the local metadata LRU/revisit instrumentation can observe internal reads.
  • Non-progressive backends retain one compact request batch in memory. Their memory is therefore not fully streaming, but they avoid repeated remote I/O and no longer create the two additional per-key map/set graphs.
  • Exact distinct fetched-key reporting requires storage proportional to the number of fetched block keys; the large key * host * spec state is removed.

The branch contains one commit and remains Draft for review.

@charpty
charpty force-pushed the agent/pr277-review-fixes branch 2 times, most recently from fb02386 to dbe8064 Compare August 11, 2026 07:59
@charpty charpty changed the title [manager] address PR 277 compatibility and host query feedback [manager] bound P2P host cache queries and preserve APIs Aug 11, 2026
@charpty
charpty force-pushed the codex/mu-main branch 3 times, most recently from 46ce54f to 9083822 Compare August 11, 2026 08:08
@charpty
charpty force-pushed the agent/pr277-review-fixes branch from dbe8064 to b8aeeb2 Compare August 11, 2026 11:28
@charpty charpty changed the title [manager] bound P2P host cache queries and preserve APIs [manager] bound P2P host cache queries Aug 11, 2026
@charpty
charpty force-pushed the agent/pr277-review-fixes branch 4 times, most recently from f03c1d5 to 7ac5de5 Compare August 11, 2026 15:16
lucky-zzz and others added 10 commits August 12, 2026 13:52
…host-state queries

Consolidate ReportEvent snapshot semantics and fused local RMW with the GetHostCacheState compact streaming, cancellation, LRU batching, jemalloc startup preload, tests, benchmarks, and maintenance documentation.
Flatten ReportEvent aggregation, reduce URI/copy and local-lock overhead, and preserve validation, ordering, and capacity semantics.

Harden EventReportBackend shutdown synchronization and expand correctness, sanitizer, and performance coverage and documentation.
Optimize the pure-local ReportEvent pipeline end to end: parse large HTTP bodies in request-owned buffers, use SIMD ASCII and NUL scans, flatten and intern request mutations, canonicalize versioned URIs without repeated parsing, inline the dominant one-spec task shape, and remove redundant temporary allocations.

Fuse exact single-location local RMW while preserving capacity, key-count, lifecycle, partial-failure and non-local backend semantics. Reuse LRU handles and bounded scratch, align metadata and local-LRU shard hashing, retire immutable values outside locks, and retain strict validation for every generic caller.

Accelerate GetHostCacheState with a bounded 4096-key probe and 16384-key parallel suffix, monotonic cancellation, ordinary and Mamba bitsets, request-scoped visibility parsing, serving-only filtering and strict non-NOENT error propagation. Preserve full materialization for P2P semantics and fail closed for malformed recovered EventReport URIs.

Add parser, URI, backend, capacity, lifecycle, concurrency, randomized differential, cancellation and ordered-error coverage. Document performance results, design invariants and rollback boundaries. Preserve the latest mu-main ReportEvent metrics commit unchanged.

Release validation on the merged mu-main tree passes 106 executable Bazel tests with one GPU-only skip. Pure-local measurements include about 106.32ms p50 for a 1M-key all-hit query with 4 workers, 84.38ms with 8 workers, and about 0.46ms for first-window cancellation; shared-host results are regression data, not an online SLA.
Include the requested instance_id when a per-instance service metrics collector cannot be resolved. Keep the diagnostic concise and avoid logging additional request context.
Keep reporter-owned locations out of generic physical deletion without granting false predicted-key credit when ordinary and EventReport locations coexist.

Document that EventReport bytes remain in the group-wide capacity watermark while being excluded from per-storage-type watermarks. Add regression coverage ensuring reporter locations cannot satisfy migration cold-tier spec coverage.
@charpty
charpty force-pushed the agent/pr277-review-fixes branch from 7ac5de5 to ac22c67 Compare August 12, 2026 12:42
@charpty
charpty force-pushed the agent/pr277-review-fixes branch from ac22c67 to 5420d75 Compare August 12, 2026 13:57
Base automatically changed from codex/mu-main to main August 13, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants