docs: enhance AGENTS with project context and performance guardrails (#277)

## Summary

Follow-up to #276 — adds Intel-specific technical detail and performance
guardrails based on code review.

## Changes

**Root AGENTS.md**
- Add project description, tech stack (C++20, MKL, OpenMP, pybind11)
- Add core principle: _performance over simplicity_ in hot paths
- Add quick start commands (build/test/Python)
- Add directory map for navigation

**`.github/copilot-instructions.md`**
- Add concrete authoring rules (e.g. don't refactor full files for
2-line changes)
- Add contribution expectations: backward compat, pre-commit, regression
tests
- Expand source-of-truth: explicitly list `mkl.cmake`,
`multi-arch.cmake`, `numa.cmake`, `pyproject.toml`
- Add negative constraint: no `std::iostream` in perf-critical headers

**`include/svs/AGENTS.md`**
- Add perf constraints: memory alignment, SIMD dispatch rules, hot path
rules
- Add failure modes: template instantiation impact, ABI compat

**`bindings/python/AGENTS.md`**
- Add pybind11 rules: GIL release, dtype/shape validation, memory
ownership
- Add common failure modes: wrapper sync on C++ signature changes, edge
cases, lifetime management

**`cmake/AGENTS.md`**
- Document Intel-specific modules: `mkl.cmake` (threading),
`multi-arch.cmake` (ISA dispatch), `numa.cmake`, `openmp.cmake`
- Add guardrail: no removal of optimization flags without benchmark
validation

**`benchmark/AGENTS.md`**
- Add constraints: warmup/iteration stability, dataset provenance,
baseline comparisons
- Add reporting requirements: median/min/max/stddev

**`include/svs/quantization/AGENTS.md`** (new)
- Quantization guardrails: type safety, precision bounds, SIMD
alignment, codec compat
- Common failure modes: float arithmetic in hot paths, misaligned
access, codebook sync

## Notes
- No mutable details (versions, flags) hardcoded in instruction files
- All changes are additive — no policy removed from original PR

---------

Co-authored-by: Mihai Capotă <mihai@mihaic.ro>

Author: napetrov

.github/copilot-instructions.md

@@ -3,21 +3,29 @@
 This file is the canonical instruction set for Copilot behavior in this repository
 (required flow, precedence, and fallback rules).
 
-`AGENTS.md` is reserved for AGENTS-aware tooling context and should not duplicate
-Copilot policy text.
+`AGENTS.md` provides project context and tech stack overview.
 
 ## Mandatory
-1. Read root `AGENTS.md` first.
+1. Read root `AGENTS.md` for project context.
 2. Read the nearest directory `AGENTS.md` for edited files.
 3. If multiple apply, use the most specific.
 
 ## Authoring rules
-- Keep suggestions minimal and scoped.
+- Keep suggestions minimal and scoped (do not refactor entire files for 2-line changes).
 - Use source-of-truth files for mutable details.
 - Do not invent or hardcode versions/flags/matrices.
+- Avoid `std::iostream` in performance-critical headers.
+
+## Contribution expectations
+- Preserve backward compatibility for public API (`include/svs/`)
+- Prefer additive API evolution (avoid duplication of existing functionality)
+- Pair public API changes with tests and documentation
+- For bug fixes: add regression tests
+- Run `pre-commit run --all-files` before proposing changes
 
 ## Source-of-truth files
-- `CMakeLists.txt`, `cmake/*.cmake`
-- `.github/workflows/`
-- `.clang-format`, `.pre-commit-config.yaml`
-- `include/svs/`, `bindings/python/`
+- Build: `CMakeLists.txt`, `cmake/*.cmake` (incl. `cmake/mkl.cmake`, `cmake/multi-arch.cmake`, `cmake/numa.cmake`)
+- Dependencies: `bindings/python/pyproject.toml`, `bindings/python/setup.py`
+- CI: `.github/workflows/`
+- Style: `.clang-format`, `.pre-commit-config.yaml`
+- API: `include/svs/`, `bindings/python/`

AGENTS.md

@@ -1,12 +1,28 @@
-# AGENTS.md
+# AGENTS.md — ScalableVectorSearch
 
-This file is for AGENTS-aware tools.
+## What this project is
+High-performance C++ library for vector similarity search at billion scale. Uses Intel MKL, AVX-512/multi-arch SIMD dispatch, quantization, NUMA-aware memory, OpenMP threading. Python bindings via pybind11. Archetype: **C++** (with Python bindings).
 
-Scope:
-- Tool-specific guidance for AGENTS-aware agents.
-- Repository context that is not Copilot-specific.
+Tech stack: C++20, Intel MKL, OpenMP, pybind11, CMake.
+Core principle: **Performance over simplicity** in hot paths.
 
-For Copilot policy (required flow, source-of-truth, fallback rules), see:
-- `.github/copilot-instructions.md` (canonical for Copilot behavior)
+## How to work
+- Backward compatibility is default for public API (`include/svs/`)
+- Performance-critical: avoid allocations in hot loops, respect memory alignment
+- For SIMD dispatch changes: consult `cmake/multi-arch.cmake` and `include/svs/multi-arch/`
+- Python bindings: update `bindings/python/` and ensure GIL release for blocking MKL calls
 
-Do not duplicate Copilot policy here.
+## Quick start
+Build: `cmake -B build && cmake --build build`
+Test: `ctest --test-dir build`
+Python: `pip install -e bindings/python/`
+
+## Directory AGENTS files
+- `.github/AGENTS.md` — CI/CD
+- `benchmark/AGENTS.md` — performance benchmarks
+- `bindings/AGENTS.md` — C++/Python bindings (router)
+- `cmake/AGENTS.md` — build system
+- `include/svs/AGENTS.md` — public C++ API
+- `tests/AGENTS.md` — test suite
+
+For Copilot/agent behavior policy: `.github/copilot-instructions.md`

benchmark/AGENTS.md

@@ -5,3 +5,9 @@ Benchmark harness and performance-eval code.
 - Keep runs reproducible (inputs/config in repo).
 - Separate harness changes from algorithm changes when possible.
 - If shared with tests, validate impacted test targets.
+
+## Performance benchmarking constraints
+- **Warmup and iterations:** Do not change warmup rounds or iteration counts without justification. These values are tuned for CI stability.
+- **Input datasets:** Use fixed seeds and document dataset provenance. Do not commit large binary datasets without approval.
+- **Baseline comparisons:** When adding new benchmarks, provide baseline comparisons against existing algorithms.
+- **Reporting:** Report median, min, max, stddev — not just mean. Flag outliers and variance sources (e.g., NUMA effects, Turbo Boost).

bindings/python/AGENTS.md

@@ -2,7 +2,17 @@
 
 Python package, extension bindings, and Python-facing behavior.
 
-- Source of truth: files under `bindings/python/` (+ cross-check with `CMakeLists.txt`).
+- Source of truth: `bindings/python/pyproject.toml`, `setup.py`, `CMakeLists.txt`.
 - Keep API compatibility by default.
 - Preserve clear dtype/shape validation and error messages.
 - Add tests for every user-visible behavior change.
+
+## pybind11 integration
+- **GIL release:** For blocking MKL calls or long-running C++ operations, release GIL with `py::call_guard<py::gil_scoped_release>()`.
+- **Type validation:** Explicitly validate NumPy dtypes (float32, float64, uint8 for quantized) and shapes before passing to C++.
+- **Memory ownership:** Document lifetime expectations for array views vs copies in docstrings.
+
+## Common failure modes
+- Forgetting to update pybind11 wrappers when C++ signatures change in `include/svs/`.
+- Not testing edge cases: empty arrays, mismatched dtypes, out-of-bounds indices.
+- Returning raw pointers without proper lifetime management — use `py::return_value_policy`.

cmake/AGENTS.md

@@ -5,4 +5,14 @@ Build modules, dependency wiring, and feature toggles.
 - `CMakeLists.txt` + `cmake/*.cmake` are authoritative.
 - Keep option names/defaults stable unless task requires change.
 - Prefer additive options over rewrites.
-- Validate option/target changes against CI workflows.
+- Validate option/target changes against CI workflows (`.github/workflows/`).
+
+## Intel-specific modules
+- **`cmake/mkl.cmake`:** MKL linkage (static vs dynamic threading). Do not hardcode MKL versions. When changing linkage mode, validate threading behavior in tests.
+- **`cmake/multi-arch.cmake`:** AVX-512 / SIMD ISA dispatch. Do not hardcode `-march` or ISA flags outside this file. Changes must align with `include/svs/multi-arch/` runtime dispatch code.
+- **`cmake/numa.cmake`:** NUMA-aware memory allocation. Respect NUMA topology assumptions in performance-critical code.
+- **`cmake/openmp.cmake`:** Threading model. Do not assume specific OpenMP version or runtime without checking source-of-truth.
+
+## Guardrails
+- Do not remove optimization flags without justification and benchmark validation.
+- Keep CMake minimum version conservative unless a new feature is required across all CI targets.

include/svs/AGENTS.md

@@ -3,6 +3,16 @@
 Public C++ headers and API contracts.
 
 - Backward compatibility is default.
-- Prefer additive API evolution.
+- Prefer additive API evolution (avoid duplicating existing functionality).
 - Keep headers self-contained and aligned with CMake C++ standard.
 - Pair public API changes with tests and docs.
+
+## Performance-critical constraints
+- **Memory alignment:** Respect alignment requirements for SIMD operations (use SVS-specific allocators).
+- **SIMD dispatch:** Changes to `include/svs/multi-arch/` must align with `cmake/multi-arch.cmake` ISA dispatch logic.
+- **Hot paths:** Avoid heap allocations, virtual calls, and `std::iostream` in performance-critical code.
+- **MKL integration:** Do not hardcode MKL versions or assume threading model; check `cmake/mkl.cmake`.
+
+## Failure modes to avoid
+- Adding template instantiations without checking compile-time impact.
+- Changing header-only code without validating ABI compatibility.

include/svs/quantization/AGENTS.md

@@ -0,0 +1,13 @@
+# AGENTS.md — include/svs/quantization/
+
+Quantization algorithms and data types (scalar quantization).
+
+- **Type safety:** Explicitly validate quantized dtype (uint8, int8, uint16) before operations.
+- **Precision:** Changes to quantization schemes must preserve reconstruction error bounds. Document expected accuracy impact.
+- **SIMD alignment:** Quantized data structures must respect SIMD lane widths (16/32/64 bytes for AVX-512).
+- **Codec compatibility:** Do not break serialization format without versioning and migration path.
+
+## Common failure modes
+- Using `float` arithmetic in quantization hot paths (use integer SIMD intrinsics).
+- Misaligned memory access in quantized vector loads/stores.
+- Forgetting to update codebook when changing quantization levels.