Merge pull request #9 from aerospike-community/lyndon/phase1-closure

Phase 1 closure: ecosystem benchmarks, layerwise test, docs

Author: lyndonbauto

AGENTS.md

@@ -2,91 +2,61 @@
 
 Aerospike remote storage backend for [LMCache](https://github.com/LMCache/LMCache). LMCache caches LLM attention KV tensors; this package implements the durable shared tier via LMCache's `RemoteConnector` plugin contract.
 
-**Status:** Pre-implementation (design + executable plan only; no `src/` yet). Do not invent APIs or skip the plan's verification gates.
+**Status:** Phase 1 (`ConnectorAdapter` + `AerospikeRemoteConnector`) is implemented on `main`. Phase 2/3 remain design-only in `DESIGN.md`.
 
 ## Read order
 
 1. **This file** — workflow, pitfalls, and verification expectations.
 2. `[DESIGN.md](DESIGN.md)` — authoritative contract: data model, config, error handling, phases.
-3. `[IMPLEMENTATION_PLAN.md](IMPLEMENTATION_PLAN.md)` — step-by-step build guide (S0–S16). **When implementing, the plan wins over stale design text** until S16 reconciles `DESIGN.md`.
+3. `[IMPLEMENTATION_PLAN.md](IMPLEMENTATION_PLAN.md)` — step-by-step build guide (S0–S16).
 
-For Aerospike client/modeling rules, use the vetted skills in [aerospike/agent-skills](https://github.com/aerospike/agent-skills) (especially `[skills/aerospike-development/](https://github.com/aerospike/agent-skills/tree/main/skills/aerospike-development)`). Do not guess Aerospike APIs or namespace defaults.
+For Aerospike client/modeling rules, use [aerospike/agent-skills](https://github.com/aerospike/agent-skills) (especially `skills/aerospike-development/`). Do not guess Aerospike APIs or namespace defaults.
 
 ## Phases (scope discipline)
 
-
-| Phase | Surface                                         | In this repo today                |
-| ----- | ----------------------------------------------- | --------------------------------- |
-| 1     | `ConnectorAdapter` + `RemoteConnector` (Python) | **Only phase to implement now**   |
-| 2     | `StoragePluginInterface`, `L2AdapterInterface`  | Architectural in `DESIGN.md` only |
-| 3     | C++ `ConnectorBase` / `libaerospike`            | Architectural in `DESIGN.md` only |
-
+| Phase | Surface | In this repo today |
+| ----- | ------- | ------------------ |
+| 1 | `ConnectorAdapter` + `RemoteConnector` (Python) | **Implemented** |
+| 2 | `StoragePluginInterface`, `L2AdapterInterface` | Architectural in `DESIGN.md` only |
+| 3 | C++ `ConnectorBase` / `libaerospike` | Architectural in `DESIGN.md` only |
 
 Stay inside Phase 1 unless the user explicitly expands scope.
 
-## Implementation workflow
-
-1. Start at **S0** in `IMPLEMENTATION_PLAN.md` and proceed **in order** (S0 → S16).
-2. **Stop if a step's verify gate fails** — fix before continuing; do not batch steps.
-3. Mark **⚠ DESIGN-CORRECTION** items as non-negotiable (they were verified against upstream LMCache `dev` and the real Aerospike Python client).
-4. After S16, reconcile `DESIGN.md` with the implementation (grep for `exists_many`, `post_init`, `shape0`, stale caps).
-
-### Critical upstream facts (do not regress)
-
-- `**post_init()` is never called** on remote connectors. Server record-size discovery runs in the connector constructor (`_ensure_limits`), not in `post_init`.
-- **Adapter has a no-arg `__init__`**. LMCache instantiates `AerospikeConnectorAdapter()` then calls `create_connector(context)`.
-- **Config/metadata** come from `context.local_cpu_backend` (`.config`, `.metadata`), with fallback to `context.config` / `context.metadata`.
-- **Connector is serde-agnostic.** `naive` / `cachegen` / `kivi` and MLA/layerwise key rewriting happen in `RemoteBackend` above the connector.
-- `**save_chunk_meta`:** when true, store one `md` bin (`RemoteMetadata.serialize()`); when false, use `self.meta_`* + `reshape_partial_chunk` on read (mirror `FSConnector`).
-- **Batch API:** `batch_read(keys, bins=...)` and `batch_write(BatchRecords([Write|Read|Remove, ...]))`. Do **not** use removed `exists_many` / `get_many` / `select_many`.
-- **TTL:** pin `aerospike>=14,<19` and centralize TTL in one helper (`meta={"ttl": N}`); namespace must have `nsup-period > 0` when using positive TTL.
-- **Default `target_segment_bytes` = 4 MiB** (LMCache byte-throughput sweet spot); clamp to server-discovered cap, not Aerospike's 1–10 KiB ops sweet spot.
-
-### Package layout (target)
+## Package layout
 
 ```text
-src/lmcache_aerospike/
-  adapter.py connector.py client.py config.py keys.py
-  sharding.py limits.py serde.py policies.py errors.py metrics.py
-tests/unit/ tests/integration/ tests/bench/
-docker/docker-compose.yml docker/aerospike.conf
+src/lmcache_aerospike/     # published on PyPI
+tests/unit/ tests/integration/
+benchmarks/                # NOT published — ai-ecosystem-benchmark + micro harness
+docker/ scripts/
 ```
 
-## Verification (once code exists)
-
+## Verification
 
-| Scope          | Command                                         | Notes                                                        |
-| -------------- | ----------------------------------------------- | ------------------------------------------------------------ |
-| Preflight (S0) | `python scripts/preflight.py`                   | Confirms LMCache + Aerospike client symbols                  |
-| Unit           | `pytest tests/unit -q`                          | No network; mock `aerospike.Client`                          |
-| Integration    | `RUN_INTEGRATION=1 pytest tests/integration -q` | Requires `docker compose -f docker/docker-compose.yml up -d` |
-| Bench / vLLM   | per S15                                         | Optional; gated by env vars                                  |
+| Scope | Command | Notes |
+| ----- | ------- | ----- |
+| Preflight (S0) | `python scripts/preflight.py` | LMCache + Aerospike client symbols |
+| Unit | `pytest tests/unit -q` | No network |
+| Integration | `./scripts/start_aerospike_ce.sh` then `pytest tests/integration -q` | Live CE |
+| Ecosystem bench | `pip install -r benchmarks/requirements.txt` then `python benchmarks/run.py --profile smoke` | Not in CI by default |
+| Micro bench | `RUN_BENCH=1 pytest benchmarks/micro --benchmark-only` | FakeClient only |
 
+Pinned versions: `IMPLEMENTATION_PLAN.md` §0.2.
 
-Pinned versions are in `IMPLEMENTATION_PLAN.md` §0.2 — do not change without re-running S0.
+## Critical upstream facts (do not regress)
 
-## Aerospike skills (external)
-
-Before changing client usage, policies, TTL, batching, or record sizing, read the relevant reference from [agent-skills](https://github.com/aerospike/agent-skills/tree/main/skills/aerospike-development/references/). `DESIGN.md` §8.3 lists the ones this design depends on.
-
-For local Docker CE setup in integration tests, see [aerospike-getting-started/SKILL.md](https://github.com/aerospike/agent-skills/blob/main/skills/aerospike-getting-started/SKILL.md).
-
-## LMCache upstream (re-verify when unsure)
-
-- `[RemoteConnector](https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/connector/base_connector.py)`
-- `[ConnectorAdapter` / `ConnectorContext](https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/connector/__init__.py)`
-- `[RemoteBackend](https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/remote_backend.py)` — serde, `init_connection`, no `post_init`
-- `[FSConnector](https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/connector/fs_connector.py)` — `save_chunk_meta` pattern
-- `[redis_connector.py](https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/connector/redis_connector.py)` — `batched_contains` consecutive-prefix semantics
+- **`post_init()` is never called** on remote connectors. Discovery runs in `_ensure_limits()` during construction.
+- **Adapter has a no-arg `__init__`**. Config/metadata from `context.local_cpu_backend` (fallback `context.config` / `context.metadata`).
+- **Connector is serde-agnostic.** MLA/layerwise key rewriting happens in `RemoteBackend` above the connector.
+- **`save_chunk_meta`:** one `md` bin when true; `meta_*` + `reshape_partial_chunk` when false (mirror `FSConnector`).
+- **Batch API:** `batch_read` / `batch_write(BatchRecords(...))` — not `exists_many` / `get_many`.
+- **TTL:** `aerospike>=14,<19`, single `_put_meta` helper; `nsup-period > 0` for positive TTL.
+- **Default `target_segment_bytes` = 4 MiB** (LMCache byte-throughput); clamp to server cap.
 
 ## What not to do
 
-- Do not implement Phase 2/3 features (pin/unpin L2, native C++, controller metadata, EE-only APIs) in Phase 1 code paths.
-- Do not create a new `aerospike.Client` per request; use `AerospikeClientHolder` ref-counting.
-- Do not use CDTs or secondary indexes for chunk payload storage.
-- Do not hardcode 8 MiB record caps; discover `max-record-size` / `write-block-size` at construction.
+- Do not implement Phase 2/3 features in Phase 1 paths.
+- Do not add benchmark-only deps to `pyproject.toml` `[project]` dependencies (use `benchmarks/requirements.txt`).
+- Do not create a new `aerospike.Client` per request; use `AerospikeClientHolder`.
+- Do not hardcode record-size caps; discover at construction.
 - Do not commit unless the user asks.
-
-## Commits and PRs
-
-Follow repository commit style from `git log`. Summarize *why* in commit messages. For PRs, include unit-test results and note whether integration tests were run.

IMPLEMENTATION_PLAN.md

@@ -1288,24 +1288,23 @@ each is updated).
 
 ## Final acceptance checklist (Phase 1 "done")
 
-Mirrors DESIGN §1.5, with corrections. All must be true:
+Mirrors DESIGN §1.5, with corrections.
 
-- [ ] `pip install -e ".[dev]"` works on Python 3.10–3.13; `import lmcache_aerospike` clean.
-- [ ] `pytest tests/unit -q` fully green (no network).
-- [ ] Discovery runs at construction; bad namespace/`nsup-period 0`/out-of-range
+- [x] `pip install -e ".[dev]"` works on Python 3.10–3.13; `import lmcache_aerospike` clean.
+- [x] `pytest tests/unit -q` fully green (no network).
+- [x] Discovery runs at construction; bad namespace/`nsup-period 0`/out-of-range
       cap fail fast with the typed errors (proven by integration tests).
-- [ ] Round-trip byte-exact for `256B,64KiB,1MiB,4MiB,16MiB,64MiB` with correct
-      `nseg` for the discovered cap.
-- [ ] `batched_contains` returns the consecutive-prefix count
-      (`[T,T,F,T] -> 2`), matching the Redis connector's semantics.
-- [ ] TTL expiry and pinned (`-1`) behavior verified on a real CE node.
-- [ ] No use of `exists_many`/`get_many`/`select_many`; batch ops use
+- [x] Round-trip byte-exact for CI sizes (512B, 64KiB); larger sizes via
+      `RUN_LARGE_INTEGRATION=1` locally (`1MiB`–`64MiB`).
+- [x] `batched_contains` consecutive-prefix semantics (unit + integration coverage).
+- [x] TTL expiry and pinned (`-1`) behavior verified on a real CE node.
+- [x] No use of `exists_many`/`get_many`/`select_many`; batch ops use
       `batch_read`/`batch_write` + `BatchRecords`.
-- [ ] TTL set in exactly one helper; client version pinned.
-- [ ] `save_chunk_meta` honored (layerwise/MLA round-trips).
-- [ ] vLLM smoke shows cache reuse across a worker restart (if run).
-- [ ] Bench harness reports p50/p95/p99 + bytes/s and the segment-size decision.
-- [ ] `DESIGN.md` reconciled with this plan.
+- [x] TTL set in exactly one helper; client version pinned.
+- [x] `save_chunk_meta` honored; layerwise `LayerCacheEngineKey` integration smoke.
+- [ ] vLLM smoke shows cache reuse across a worker restart (optional; `RUN_VLLM=1`).
+- [x] Bench harness: `benchmarks/run.py` (ecosystem) + `benchmarks/micro/` (pytest-benchmark).
+- [x] `DESIGN.md` reconciled with this plan (v0.2 status on `main`).
 
 ## Appendix A — Upstream facts this plan relies on (so you can re-verify)
 

README.md

@@ -85,11 +85,17 @@ Large payloads (16–64 MiB): `RUN_LARGE_INTEGRATION=1`.
 
 ## Benchmarks
 
+Benchmark code lives under **`benchmarks/`** (not included in the PyPI wheel).
+
 ```bash
-RUN_BENCH=1 pytest tests/bench -v --benchmark-only
+pip install -r benchmarks/requirements.txt
+./scripts/start_aerospike_ce.sh && source .aerospike-ci.env
+python benchmarks/run.py --profile smoke
 ```
 
-See [`tests/bench/README.md`](tests/bench/README.md).
+Micro (FakeClient, no server): `RUN_BENCH=1 pytest benchmarks/micro --benchmark-only`
+
+See [`benchmarks/README.md`](benchmarks/README.md).
 
 ## License
 

benchmarks/README.md

@@ -0,0 +1,66 @@
+# Benchmarks
+
+Harnesses for **lmcache-aerospike** live under `benchmarks/` and are **not** shipped on PyPI
+(only `src/lmcache_aerospike/` is packaged). Deps with VCS URLs live in
+`benchmarks/requirements.txt` for the same reason as [adk-aerospike](https://github.com/aerospike-community/adk-aerospike).
+
+| Harness | Runner | Use when |
+|---|---|---|
+| **Ecosystem** (`run.py`) | [ai-ecosystem-benchmark](https://github.com/aerospike-community/ai-ecosystem-benchmark) | Coordinated-omission-safe QPS/latency on a live Aerospike CE node |
+| **Micro** (`micro/`) | pytest-benchmark + FakeClient | Fast, no-server connector overhead / segment-size sweep |
+
+## Ecosystem harness (recommended)
+
+### Install
+
+```bash
+pip install -e .
+pip install -r benchmarks/requirements.txt
+./scripts/start_aerospike_ce.sh
+set -a && source .aerospike-ci.env && set +a
+```
+
+### Run
+
+```bash
+python benchmarks/run.py --list-profiles
+python benchmarks/run.py --list-workloads
+python benchmarks/run.py --profile smoke
+python benchmarks/run.py --profile kv_chunk_smoke --results-dir benchmarks/results
+```
+
+Connection URI format (override with `--uri`):
+
+```text
+aerospike://127.0.0.1:3000/lmcache?set=bench_eco_kv&num_tokens=128&target_segment_bytes=4194304
+```
+
+### Workloads
+
+| Workload | Tests | Intent |
+|---|---|---|
+| `kv_hotpath` | `aerospike_kv_put`, `get_hit`, `get_miss`, `exists` | Steady KV remote-cache loop |
+| `kv_chunk` | `aerospike_kv_put_large`, `get_large` | ~4 MiB payload (multi-segment) |
+
+### Profiles
+
+| Profile | Workload | Notes |
+|---|---|---|
+| `smoke` | `kv_hotpath` | Laptop sanity, ~5 s per test |
+| `kv_chunk_smoke` | `kv_chunk` | 4 MiB target segment band |
+
+## Micro harness (no Aerospike)
+
+Uses `FakeClient` from unit tests — not the ecosystem runner.
+
+```bash
+pip install -r benchmarks/requirements.txt
+RUN_BENCH=1 pytest benchmarks/micro -v --benchmark-only
+```
+
+See [micro/README.md](micro/README.md).
+
+## Data isolation
+
+Workloads default to `bench_eco_*` sets under the `lmcache` namespace. They do not use
+`it_chunks` integration-test data. Tear down by deleting the bench sets or the namespace prefix.

benchmarks/micro/README.md

@@ -0,0 +1,14 @@
+# Micro benchmarks (FakeClient)
+
+In-process pytest-benchmark tests — no Aerospike server.
+
+```bash
+pip install -r benchmarks/requirements.txt
+RUN_BENCH=1 pytest benchmarks/micro -v --benchmark-only
+```
+
+Optional live spot-check (Aerospike CE up):
+
+```bash
+RUN_BENCH=1 RUN_BENCH_LIVE=1 pytest benchmarks/micro -v -k live --benchmark-only
+```

tests/bench/conftest.py benchmarks/micro/conftest.pytests/bench/conftest.py renamed to benchmarks/micro/conftest.py

@@ -1,4 +1,4 @@
-"""Fixtures for in-process connector benchmarks (FakeClient, no Aerospike)."""
+"""Fixtures for in-process connector micro-benchmarks (FakeClient)."""
 
 from __future__ import annotations
 
@@ -45,7 +45,6 @@ def _resolved(target_segment_bytes: int) -> ResolvedLimits:
 
 @pytest.fixture
 def bench_connector(fake_client: FakeClient, request):
-    """Connector with configurable target_segment_bytes via indirect param."""
     target = getattr(request, "param", 4 * 1024 * 1024)
     resolved = _resolved(target)
     backend = FakeLocalCPUBackend(alloc_size=8 * 1024 * 1024)

tests/bench/test_connector_bench.py benchmarks/micro/test_connector_micro.pytests/bench/test_connector_bench.py renamed to benchmarks/micro/test_connector_micro.py

@@ -1,13 +1,4 @@
-"""pytest-benchmark harness for connector put/get (S15).
-
-Uses FakeClient (no network). Run:
-
-  RUN_BENCH=1 pytest tests/bench -v --benchmark-only
-
-Optional live Aerospike:
-
-  RUN_BENCH=1 RUN_BENCH_LIVE=1 pytest tests/bench -v -k live
-"""
+"""pytest-benchmark micro harness (no Aerospike). See benchmarks/micro/README.md."""
 
 from __future__ import annotations
 
@@ -18,7 +9,7 @@
 
 pytestmark = pytest.mark.skipif(
     os.environ.get("RUN_BENCH") != "1",
-    reason="Set RUN_BENCH=1 to run performance benchmarks",
+    reason="Set RUN_BENCH=1 to run micro benchmarks",
 )
 
 LIVE = os.environ.get("RUN_BENCH_LIVE") == "1"
@@ -60,7 +51,6 @@ def run():
 
 
 def test_segment_size_sweep(benchmark, segment_sweep_connector):
-    """Sensitivity check across target_segment_bytes (1–8 MiB)."""
     conn, key, mo, target, payload_size, _loop = segment_sweep_connector
     conn._put_sync_impl(key, mo)
 
@@ -78,7 +68,8 @@ def run():
 
 @pytest.mark.skipif(not LIVE, reason="Set RUN_BENCH_LIVE=1 with Aerospike CE up")
 def test_live_put_get(benchmark):
-    """Optional bench against scripts/start_aerospike_ce.sh (not run in CI)."""
+    import asyncio
+
     from tests.integration.helpers import (
         build_connector,
         chunk_byte_size,
@@ -88,8 +79,6 @@ def test_live_put_get(benchmark):
         payload_pattern,
     )
 
-    import asyncio
-
     conn, backend, _, _ = build_connector(num_tokens=128)
     try:
         chunk_len = chunk_byte_size(backend.metadata)

benchmarks/profiles/kv_chunk_smoke.json

@@ -0,0 +1,12 @@
+{
+  "description": "4 MiB target payload put/get (multi-segment on most CE nodes).",
+  "workload": "kv_chunk",
+  "connection_string": "aerospike://127.0.0.1:3000/lmcache?set=bench_eco_kv&target_segment_bytes=4194304",
+  "queries_per_second": 8,
+  "scheduler_thread_count": 1,
+  "worker_thread_count": 2,
+  "runtime_per_function": 5,
+  "workload_params": {
+    "target_payload_bytes": 4194304
+  }
+}

benchmarks/profiles/smoke.json

@@ -0,0 +1,12 @@
+{
+  "description": "Quick KV hot-path sanity (~30s). Requires Aerospike CE (start_aerospike_ce.sh).",
+  "workload": "kv_hotpath",
+  "connection_string": "aerospike://127.0.0.1:3000/lmcache?set=bench_eco_kv&num_tokens=128",
+  "queries_per_second": 20,
+  "scheduler_thread_count": 1,
+  "worker_thread_count": 4,
+  "runtime_per_function": 5,
+  "workload_params": {
+    "key_count": 16
+  }
+}

benchmarks/requirements.txt

@@ -0,0 +1,5 @@
+# Ecosystem harness (benchmarks/run.py). Not part of the PyPI wheel.
+# Install package first: pip install -e .
+git+https://github.com/aerospike-community/ai-ecosystem-benchmark.git
+pytest>=7
+pytest-benchmark>=4