Merge pull request #16 from aerospike-community/lyndon/bench-l2-setup

Add lmcache bench l2 harness for AerospikeL2Plugin

Author: lyndonbauto

.gitignore

@@ -24,3 +24,4 @@ htmlcov/
 .DS_Store
 .claude/
 benchmarks/results/
+benchmarks/l2/.env.local

AGENTS.md

@@ -39,6 +39,7 @@ docker/ scripts/
 | Unit | `pytest tests/unit -q` | No network |
 | Integration | `./scripts/start_aerospike_ce.sh` then `pytest tests/integration -q` | Live CE |
 | Ecosystem bench | `pip install -r benchmarks/requirements.txt` then `python benchmarks/run.py --profile smoke` | Not in CI by default |
+| L2 bench | `./scripts/setup_l2_bench.sh` then `./benchmarks/l2/run.sh` (LMCache `dev` + live CE) | Not in CI by default |
 | Micro bench | `RUN_BENCH=1 pytest benchmarks/micro --benchmark-only` | FakeClient only |
 
 Pinned versions: `IMPLEMENTATION_PLAN.md` §0.2.

README.md

@@ -123,6 +123,13 @@ pip install -r benchmarks/requirements.txt
 python benchmarks/run.py --profile smoke
 ```
 
+**L2 adapter** ([`lmcache bench l2`](https://docs.lmcache.ai/cli/bench_l2.html), requires LMCache `dev`):
+
+```bash
+./scripts/setup_l2_bench.sh
+# ./benchmarks/l2/run.sh   # after Aerospike is up
+```
+
 Micro (FakeClient, no server): `RUN_BENCH=1 pytest benchmarks/micro --benchmark-only`
 
 See [`benchmarks/README.md`](benchmarks/README.md).

benchmarks/README.md

@@ -7,6 +7,7 @@ Harnesses for **lmcache-aerospike** live under `benchmarks/` and are **not** shi
 | Harness | Runner | Use when |
 |---|---|---|
 | **Ecosystem** (`run.py`) | [ai-ecosystem-benchmark](https://github.com/aerospike-community/ai-ecosystem-benchmark) | Coordinated-omission-safe QPS/latency on a live Aerospike CE node |
+| **L2** (`l2/`) | [`lmcache bench l2`](https://docs.lmcache.ai/cli/bench_l2.html) + `AerospikeL2Plugin` | Store / lookup / load through the MP L2 adapter API (needs LMCache `dev`) |
 | **Micro** (`micro/`) | pytest-benchmark + FakeClient | Fast, no-server connector overhead / segment-size sweep |
 
 ## Ecosystem harness (recommended)
@@ -49,6 +50,19 @@ aerospike://127.0.0.1:3000/lmcache?set=bench_eco_kv&num_tokens=128&target_segmen
 | `smoke` | `kv_hotpath` | Laptop sanity, ~5 s per test |
 | `kv_chunk_smoke` | `kv_chunk` | 4 MiB target segment band |
 
+## L2 harness (`lmcache bench l2`)
+
+Benchmarks `AerospikeL2Plugin` via LMCache’s official L2 bench CLI (not PyPI 0.4.x).
+
+```bash
+./scripts/setup_l2_bench.sh          # LMCache dev + bench deps (once)
+./scripts/start_aerospike_ce.sh
+set -a && source .aerospike-ci.env && set +a
+./benchmarks/l2/run.sh               # when ready
+```
+
+See [`l2/README.md`](l2/README.md).
+
 ## Micro harness (no Aerospike)
 
 Uses `FakeClient` from unit tests — not the ecosystem runner.

benchmarks/l2/README.md

@@ -0,0 +1,94 @@
+# L2 adapter benchmark (`lmcache bench l2`)
+
+End-to-end store / lookup / load benchmarks for [`AerospikeL2Plugin`](../../src/lmcache_aerospike/l2_plugin.py) using the same [`parse_args_to_l2_adapters_config`](https://docs.lmcache.ai/mp/configuration.html) + `create_l2_adapter` path as production MP mode.
+
+Upstream docs: [lmcache bench l2](https://docs.lmcache.ai/cli/bench_l2.html).
+
+## Requirements
+
+| Piece | Why |
+| ----- | --- |
+| **LMCache `dev`** | PyPI `lmcache` 0.4.x does not ship `lmcache bench l2` or `L2StoreResult`. |
+| **This package** (`pip install -e .`) | Provides `type: plugin` → `AerospikeL2Plugin`. |
+| **torch + openai** | Bench CLI and `TensorMemoryObj` payloads (see `benchmarks/l2/requirements.txt`). |
+| **Aerospike CE** (for real runs) | `./scripts/start_aerospike_ce.sh` → `.aerospike-ci.env`. |
+
+`native_storage_ops` is optional for bench: `benchmarks/l2/bootstrap.py` installs the same Python fallback used in integration tests when the extension module is missing.
+
+## One-time setup
+
+```bash
+# LMCache dev clone (default: ../LMCache next to this repo)
+git clone https://github.com/LMCache/LMCache.git ../LMCache
+cd ../LMCache && git checkout dev
+
+cd /path/to/lmcache-aerospike
+./scripts/setup_l2_bench.sh
+```
+
+Override the LMCache path:
+
+```bash
+LMCACHE_SRC=/path/to/LMCache ./scripts/setup_l2_bench.sh
+```
+
+Preflight only:
+
+```bash
+python3 scripts/preflight_l2_bench.py
+```
+
+## Running (when you are ready)
+
+Start Aerospike and load connection env (dynamic port is written to `.aerospike-ci.env`):
+
+```bash
+./scripts/start_aerospike_ce.sh
+set -a && source .aerospike-ci.env && set +a
+```
+
+Smoke (store → lookup → load, defaults from [bench_l2 docs](https://docs.lmcache.ai/cli/bench_l2.html)):
+
+```bash
+./benchmarks/l2/run.sh
+```
+
+Stress profile and single operation:
+
+```bash
+./benchmarks/l2/run.sh --profile stress
+./benchmarks/l2/run.sh --only store
+./benchmarks/l2/run.sh -- --no-skip-verify   # round-trip byte check on last round
+```
+
+Direct CLI (equivalent; `run.sh` sets `L2_ADAPTER_JSON` from `adapters/` + `.aerospike-ci.env`):
+
+```bash
+python3 benchmarks/l2/bootstrap.py
+export L2_ADAPTER_JSON="$(cat benchmarks/l2/adapters/aerospike_smoke.json)"
+lmcache bench l2 --l2-adapter "$L2_ADAPTER_JSON" --num-keys 32 --in-flight 1
+```
+
+## Layout
+
+| Path | Purpose |
+| ---- | ------- |
+| `adapters/*.json` | Plugin specs (`type: plugin`, `AerospikeL2Plugin`). Hosts/port overridden by `run.sh`. |
+| `profiles/*.env` | Default CLI flag bundles (`BENCH_L2_EXTRA_ARGS`). |
+| `bootstrap.py` | `native_storage_ops` fallback + `init_remote_metadata_info(1)`. |
+| `run.sh` | Sources Aerospike env, renders adapter JSON, runs `lmcache bench l2`. |
+| `env.example` | Optional `benchmarks/l2/.env.local` knobs. |
+
+Bench records use set **`kv_chunks_bench_l2`** by default (not `it_chunks` or `kv_chunks_l2_it`).
+
+## Cold-cache / isolated operations
+
+For adapters backed by the OS page cache, run operations separately and drop caches between runs (see upstream note in [bench_l2](https://docs.lmcache.ai/cli/bench_l2.html)):
+
+```bash
+./benchmarks/l2/run.sh --only store
+sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
+./benchmarks/l2/run.sh --only lookup
+```
+
+Aerospike has no local page cache; the default combined `run.sh` pass is usually representative.

benchmarks/l2/adapters/aerospike_smoke.json

@@ -0,0 +1,10 @@
+{
+  "type": "plugin",
+  "module_path": "lmcache_aerospike.l2_plugin",
+  "class_name": "AerospikeL2Plugin",
+  "adapter_params": {
+    "hosts": "127.0.0.1:3000",
+    "namespace": "lmcache",
+    "set": "kv_chunks_bench_l2"
+  }
+}

benchmarks/l2/adapters/aerospike_stress.json

@@ -0,0 +1,11 @@
+{
+  "type": "plugin",
+  "module_path": "lmcache_aerospike.l2_plugin",
+  "class_name": "AerospikeL2Plugin",
+  "adapter_params": {
+    "hosts": "127.0.0.1:3000",
+    "namespace": "lmcache",
+    "set": "kv_chunks_bench_l2",
+    "default_ttl_seconds": 86400
+  }
+}

benchmarks/l2/bootstrap.py

@@ -0,0 +1,63 @@
+"""Runtime bootstrap for ``lmcache bench l2`` against AerospikeL2Plugin."""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+import sys
+from pathlib import Path
+
+
+def _repo_root() -> Path:
+    return Path(__file__).resolve().parents[2]
+
+
+def _lmcache_src() -> Path:
+    env = os.environ.get("LMCACHE_SRC")
+    if env:
+        return Path(env).expanduser().resolve()
+    return _repo_root().parent / "LMCache"
+
+
+def install_native_storage_ops_fallback() -> None:
+    """Use LMCache test fallback when ``lmcache.native_storage_ops`` is incomplete."""
+    try:
+        native = __import__("lmcache.native_storage_ops", fromlist=["Bitmap"])
+        if hasattr(native, "Bitmap") and hasattr(native, "TTLLock"):
+            return
+    except Exception:
+        pass
+
+    utils = _lmcache_src() / "tests/v1/storage_backend/raw_block_test_utils.py"
+    if not utils.is_file():
+        raise RuntimeError(
+            "lmcache.native_storage_ops is unavailable and LMCache test utils were not found.\n"
+            f"  Expected: {utils}\n"
+            "  Set LMCACHE_SRC to your LMCache dev clone, or build LMCache with native extensions."
+        )
+    spec = importlib.util.spec_from_file_location("lmcache_raw_block_test_utils", utils)
+    if spec is None or spec.loader is None:
+        raise RuntimeError(f"Could not load LMCache test utils from {utils}")
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    mod.install_native_storage_ops_fallback()
+
+
+def bootstrap() -> None:
+    install_native_storage_ops_fallback()
+    from lmcache.v1.protocol import init_remote_metadata_info
+
+    init_remote_metadata_info(1)
+
+
+def main() -> None:
+    bootstrap()
+    print("OK: L2 bench bootstrap")
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except Exception as exc:
+        print(exc, file=sys.stderr)
+        sys.exit(1)

benchmarks/l2/env.example

@@ -0,0 +1,16 @@
+# Copy to benchmarks/l2/.env.local (optional) or export in your shell.
+# After ./scripts/start_aerospike_ce.sh, prefer sourcing the repo root file:
+#   set -a && source .aerospike-ci.env && set +a
+
+# Aerospike (overrides defaults in adapters/*.json when using benchmarks/l2/run.sh)
+# AEROSPIKE_TEST_HOST=127.0.0.1
+# AEROSPIKE_TEST_PORT=3000
+# AEROSPIKE_TEST_NAMESPACE=lmcache
+# Dedicated set so bench data does not collide with integration tests (it_chunks, kv_chunks_l2_it).
+AEROSPIKE_BENCH_L2_SET=kv_chunks_bench_l2
+
+# Optional: pass adapter JSON directly instead of --adapter aerospike_smoke.json
+# L2_ADAPTER_JSON='{"type":"plugin","module_path":"lmcache_aerospike.l2_plugin",...}'
+
+# LMCache source for scripts/setup_l2_bench.sh (default: sibling ../LMCache)
+# LMCACHE_SRC=/path/to/LMCache

benchmarks/l2/profiles/smoke.env

@@ -0,0 +1,2 @@
+# Default CLI flags for a quick store -> lookup -> load pass (see docs.lmcache.ai/cli/bench_l2.html).
+BENCH_L2_EXTRA_ARGS="--num-keys 32 --in-flight 1 --data-size-kb 256 --rounds 1 --warmup-rounds 1"