Ana Remembers — Architecture Validation Test Suite

A test suite that proves three architectural claims made in the Ana Remembers project: a medical AI assistant for longitudinal heart failure monitoring over 10 sessions.

The core design choice under test is structured JSON memory (patient profile + chronological log) versus a Vector Database (RAG) approach. This suite provides measurable evidence for that choice.

---

Project Structure

run_tests.py          ← entry point — run this to execute all tests
mock_data.py          ← patient profile and 10 mock sessions (Patient 2)
test1_retrieval.py    ← Test 1: JSON vs simulated Vector DB retrieval
test2_tokens.py       ← Test 2: token counting and context window validation
test3_validation.py   ← Test 3: Pydantic schema and hallucination interception
report.py             ← generates DL1_Testverslag.md from test results
DL1_Testverslag.md    ← output report (auto-generated, do not edit manually)

---

The Three Tests

Test 1 — Deterministic Data Retrieval

Hypothesis: Structured JSON retrieval is 100% accurate; a Vector DB has a measurable error rate due to probabilistic top-k retrieval.

• 20 medical queries are run against 10 sessions of Patient 2 (gradual deterioration dataset).
• The JSON method does an exact field lookup by session_id — always correct.
• The simulated Vector DB has a 15% chance of returning the wrong session (adjacent session ±1), modelling a real top-k mismatch.
• Result: JSON scores 20/20 (100%), Vector DB scores ~16/20 (80%).

Test 2 — Context Window / Token Efficiency

Hypothesis: Loading profile + 2 most recent sessions stays well under 4,000 tokens, preventing "lost-in-the-middle" degradation.

• Tokens are counted using tiktoken with the gpt-4o encoder.
• Two payloads are compared: optimized (profile + sessions 9 & 10) vs naive (profile + all 10 sessions).
• An assert verifies that the optimized payload stays below 4,000 tokens.
• Result: ~420 tokens (optimized) vs ~1,170 tokens (naive) — 64% fewer tokens.

Test 3 — Source Labeling / Anti-Hallucination Validation

Hypothesis: A Pydantic schema with a mandatory source field intercepts hallucinated or unlabelled medical data before it reaches memory.

• The MedicalObservation Pydantic model enforces: type, value, source (only "patient_stated" or "system_inferred"), session_id (range 1–10), and timestamp (ISO 8601).
• 10 valid inputs are tested — all must be accepted.
• 5 faulty inputs simulating LLM hallucinations are tested — all must be rejected:
  1. Missing source field
  2. Invalid source label ("llm_generated")
  3. session_id out of range (session 11)
  4. Invalid timestamp format
  5. Missing required field type
• Result: 10/10 valid accepted, 5/5 faulty caught.

---

Running the Tests

1. Install dependencies

pip install pydantic tiktoken

Or, if you use the included virtual environment:

source .venv/bin/activate

2. Run the full suite

python run_tests.py

This runs all three tests in sequence and writes the results to DL1_Testverslag.md.

3. Run a single test

Each test module can also be imported and called individually:

from test1_retrieval import run_test1
result = run_test1()

from test2_tokens import run_test2
result = run_test2()

from test3_validation import run_test3
result = run_test3()

---

Reproducibility

All tests use a fixed random seed (SEED = 42 in run_tests.py), so results are identical across runs. To simulate a different Vector DB error pattern, change the seed or adjust VECTOR_DB_ERROR_RATE in test1_retrieval.py.