[KV Offload] Add self-describing events for tier promotions

[Change72]

Purpose

Related to #38260 and follows up on #43468.

#43468 added opt-in self-describing KV events, but only populated event metadata from the GPU-to-CPU store path. A secondary-to-primary tier promotion bypasses that path, so its CPU BlockStored event can otherwise retain the legacy placeholder payload.

A placeholder carries only the chunk's opaque hash — no tokens, parent hash, or block size — so KV-event consumers generally cannot index the promoted copy, and the worker holding it stays invisible to routing/reuse decisions.

This PR keeps the fix in the connector layer:

• record request metadata for non-miss results in both lookup helpers;
• confirm metadata on a primary HIT, GPU-to-CPU store, or CPU BlockStored event;
• retain speculative metadata while any observing request remains active;
• clean unconfirmed metadata when its owners finish;
• let only CPU events confirm or remove CPU metadata;
• translate queued pre-reset events before clearing their metadata; and
• allow self_describing_kv_events with TieringOffloadingSpec.

The existing full-attention filter remains the only group filter. Sliding-window and SSM groups retain placeholder payloads. This does not change offloading managers, LookupResult, the lookup contract, or the KV-event wire format.

Reset-time flushing is enabled only for self-describing events. It preserves per-medium ordering; cross-medium ordering is not part of the event contract.

Aside from that reset-time flush, which applies to single-tier deployments as well, the default single-tier CPUOffloadingSpec behavior is unchanged; with self-describing disabled, event output is unchanged on all paths.

Known limitations

Locally requested CPU promotions are best-effort self-describing. Events may retain placeholders when every local speculator finishes before translation, during a pending-removal/re-promotion generation race, or when an externally initiated promotion has no locally recorded metadata. Consumers must ignore removals for unknown hashes.

Duplicate-work check

This is not a duplicate. I reviewed #38260 and searched open PRs by issue reference, self_describing_kv_events, OffloadingEventsTracker, and promotion BlockStored keywords. No open PR implements connector-side promotion metadata tracking.

#48281 is the only open PR referencing #38260. It adds optional FS/OBJ locality metadata and is orthogonal to CPU promotion tracking and tiered self-describing events.

Test Plan

HF_HUB_OFFLINE=1 PYTHONPATH=. .venv/bin/python -m pytest \
  tests/v1/kv_connector/unit/offloading_connector/test_events.py \
  tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py -q

.venv/bin/pre-commit run --files \
  docs/features/kv_offloading_usage.md \
  tests/v1/kv_connector/unit/offloading_connector/test_events.py \
  tests/v1/kv_connector/unit/offloading_connector/test_scheduler.py \
  vllm/distributed/kv_transfer/kv_connector/v1/offloading/events.py \
  vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py \
  vllm/v1/kv_offload/tiering/spec.py

Model evaluation is not applicable because this changes KV-event metadata and bookkeeping only; it does not affect model execution, outputs, or accuracy.

Test Result

• Workstation pytest: 113 passed, 17 warnings
• Relevant-file pre-commit, including ruff, markdownlint, and mypy 3.10: passed
• Workstation manual mypy 3.12 (--all-files --hook-stage manual): passed

AI Assistance

AI assistance was used for investigation, implementation, tests, and adversarial review. The human submitter reviewed the design and production code and will complete the full line-by-line review before marking this draft ready.

[mergify]

Documentation preview: https://vllm--48679.org.readthedocs.build/en/48679/