Skip to content

construction.md

Latest commit

 

History

History
94 lines (62 loc) · 4.76 KB

construction.md

File metadata and controls

94 lines (62 loc) · 4.76 KB
Raw

StockBench memory benchmark — construction guide

This directory (bench/) implements an offline StockBench harness whose primary purpose is memory behavior validation, not trading performance measurement.

Purpose

We verify whether TradingAgents accumulates, persists, recalls, and uses memory across repeated trading-day propagations when backed by the same on-disk memory roots.

Although StockBench can support numeric evaluation (e.g., simulated fills against OHLCV), TradingAgents itself produces narrative outputs plus a coarse rating-style label rather than explicit trade sizing or execution prices. The TradingAgents paper does not define numeric trade execution targets.

Therefore this benchmark does not claim to evaluate realized trading performance, alpha, or directional accuracy. Outcome-style metrics (returns, accuracy bands, portfolio KPIs) are not part of the bench contract. The canonical human-readable output is bench/results.md, generated by bench/metrics.py from existing run artifacts.

Layout

Path Role
bench/harness.py CLI entry: prepares or executes StockBench sessions under bench/stockbench/runs/.
bench/metrics.py Writes bench/results.md — memory coverage tables and qualitative case studies only.
bench/config/memory-on.json, memory-off.json Condition-specific overrides (e.g., LLM config).
bench/stockbench/tasks.json Symbols, calendar window, sessions, simulator rules.
bench/stockbench/runs/memory-on/ Per-session outputs plus shared-memory/ for the persisted agent/user stores when memory is shared.
bench/stockbench/runs/memory-off/session-*/memory/isolated/ One isolated memory root per decision (date × symbol) when memory is off.
bench/stockbench_adapter.py Offline data routing and StockBench bar loading (no live APIs).
bench/stockbench_portfolio.py Multi-day propagate loop; feeds decisions into memory and optional reflection flush.
bench/.cache/ Local cache dir referenced by graph config.

Prerequisites

  • Repository root on PYTHONPATH (the harness inserts TradingAgents/ automatically).
  • Environment variables for your LLM backend as used elsewhere in this repo (often via .env).
  • StockBench offline OHLCV available where StockBenchOfflineAdapter expects it (run StockBench pre-cache / data layout per your StockBench install).

Running

Dry-run (no LLM calls; validates tasks and writes lightweight metadata under a temporary staging dir):

cd TradingAgents
python bench/harness.py --showcase stockbench --condition all

Execute full sessions (writes daily_decisions.json, memory_snapshot.json, tradingagents-results/, and memory trees):

python bench/harness.py --showcase stockbench --condition all --execute

Force overwrite prior outputs:

python bench/harness.py --showcase stockbench --condition memory-on --execute --force

Clear shared or isolated memory before an execute run:

python bench/harness.py --showcase stockbench --condition memory-on --execute --force --reset-memory

Regenerate the English memory report:

python bench/metrics.py

Output: bench/results.md (never under stockbench/results/).

Memory-on vs memory-off protocol

Memory-on

  • Persistent stores (ticker_knowledge, recommendations, reflections, analyst_cache, macro_notes, …) live under a single shared directory, by default bench/stockbench/runs/memory-on/shared-memory/.
  • All symbols and dates read/write the same knowledge cards and reflection corpora unless you override shared_memory_root in config.

Memory-off

  • Each trading-day decision uses a fresh isolated subtree: session-*/memory/isolated/<date>-<SYMBOL>/.
  • Cross-day or cross-symbol recall is intentionally suppressed so differences vs memory-on are attributable to memory isolation, not different prompts.

Artifacts kept for analysis

After an execute run, retain at least:

  • daily_decisions.json — coarse ratings and previews per day/symbol.
  • memory_snapshot.json — summarizer-friendly stats (kc_in_prompt, reflection counts, recent decisions slice).
  • session_state.json — dry-run metadata or minimal execute metadata (portfolio outcome metrics are not persisted by the harness).
  • tradingagents-results/<SYMBOL>/TradingAgentsStrategy_logs/full_states_log_*.json — qualitative reasoning traces.
  • session-*/preflight.json — offline data validation metadata.

Do not expect portfolio_nav.json, trades.json, or portfolio_metrics.json from the harness; those were removed from the bench contract.

CLI compatibility

--showcase all is equivalent to stockbench (the only showcase). Older showcase names are removed.