Skip to content

Latest commit

 

History

History
812 lines (776 loc) · 94.5 KB

WORKLOG.md

File metadata and controls

812 lines (776 loc) · 94.5 KB

Worklog - TPOT Analyzer

Phase 4.0: Tweet Classification + Content-Aware Clustering

  • [2026-02-25 16:06 UTC] Second pass: eliminate remaining React hook dependency warnings (Codex GPT-5)

    • Assumptions
      • The remaining lint warnings are stale-dependency declarations (not behavioral bugs), so explicit dependency alignment should be behavior-preserving.
    • Predicted outcome
      • npm run lint becomes fully warning-free; ClusterCanvas/ClusterView tests remain green.
    • Confidence
      • 0.80
    • Fallback plan
      • If dependency expansion caused regressions, revert to ref-backed reads for volatile values and isolate render-effect dependencies by extracting stable selectors.
    • Changes (files + why)
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:253: include minZoomProp/maxZoomProp in focus-camera effect dependencies.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:1373: include full render-effect inputs (canExpandNode, centeredNodeId, highlightedMemberAccountId, palette fields, zoomMode) to remove stale closure risk.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:1633: include onSelectionChange and selectedSet in hover/selection callback dependencies.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:2067: include logHybridZoom, zoom bounds, and zoomMode in wheel-handler effect dependencies.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:71: derive stable expandedList, collapsedList, expandedCount, and focusLeafValue for fetch effect.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:168 and :266: replace direct expanded.size reads with expandedCount to match declared dependencies.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:300: align fetch-effect dependency list with the derived values used to build the request payload.
    • Verification
      • cd tpot-analyzer/graph-explorer && npm run lint → clean (0 warnings, 0 errors).
      • cd tpot-analyzer/graph-explorer && npx vitest run --silent=true src/ClusterCanvas.test.jsx src/ClusterView.integration.test.jsx src/ClusterView.test.jsx → 53 passed.

  • [2026-02-25 15:54 UTC] Frontend lint-gate stabilization: Vitest globals + targeted dead-code cleanup (Codex GPT-5)

    • Hypothesis
      • Current npm run lint failures are dominated by config drift (test globals missing) and stale locals from refactors; a narrow cleanup should restore lint as a reliable quality gate without changing runtime behavior.
    • Changes (files + why)
      • tpot-analyzer/graph-explorer/eslint.config.js:7: ignore generated artifacts (coverage, playwright-report, test-results) and add test-file globals (browser + node + vitest) so test suites lint under the right environment contract.
      • tpot-analyzer/graph-explorer/src/clusterCanvasConfig.js:1 (NEW): extract ZOOM_CONFIG from component file to satisfy react-refresh/only-export-components.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:3,133: import shared ZOOM_CONFIG and use optional catch binding to remove unused error symbol.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.test.jsx:262,458: drop unused rerender binding and import zoom defaults from clusterCanvasConfig.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.memoryleak.test.jsx:42: remove unused getInternalSizes helper.
      • tpot-analyzer/graph-explorer/src/ClusterView.integration.test.jsx:32,404: remove unused mock param and unused post-call count variable.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:800: remove unused collapsingCluster assignment in semantic-collapse path.
      • tpot-analyzer/graph-explorer/src/ClusterView.test.jsx:1: remove unused imports (afterEach, act).
      • tpot-analyzer/graph-explorer/src/Discovery.jsx:6,74,110: remove unused normalizeHandle/queryState and wire setSelectedAutocompleteIndex from seed-input hook.
      • tpot-analyzer/graph-explorer/src/hooks/useSeedInput.js:118: export setSelectedAutocompleteIndex so Discovery can control hover index.
      • tpot-analyzer/graph-explorer/src/GraphExplorer.jsx:77: remove unused graphSettings destructure.
      • tpot-analyzer/graph-explorer/src/accountsApi.test.js:1: remove unused afterEach import.
      • tpot-analyzer/graph-explorer/src/data.js.test.js:382: remove unused cache-key temp vars.
      • tpot-analyzer/graph-explorer/src/hooks/useModelSettings.test.js:2: remove unused waitFor import.
      • tpot-analyzer/graph-explorer/src/hooks/useRecommendations.js:31,113: remove dead query-advance branch + unused cached ego state setters; keep behavior on subgraph mode unchanged.
      • tpot-analyzer/graph-explorer/src/hooks/useRecommendations.test.js:147,300,348,760: align destructuring with actual assertions (fix undefined/unused result variables).
      • tpot-analyzer/graph-explorer/src/hooks/useAccountManager.test.js:829 and tpot-analyzer/graph-explorer/src/hooks/useSeedInput.test.js:635: replace global with globalThis for lint-safe timer spies.
    • Verification
      • cd tpot-analyzer/graph-explorer && npm run lint → passes with 0 errors (5 existing react-hooks/exhaustive-deps warnings remain in ClusterCanvas.jsx and ClusterView.jsx).
      • cd tpot-analyzer/graph-explorer && npx vitest run src/ClusterCanvas.memoryleak.test.jsx src/ClusterCanvas.test.jsx src/ClusterView.integration.test.jsx src/ClusterView.test.jsx src/accountsApi.test.js src/data.js.test.js src/hooks/useModelSettings.test.js src/hooks/useRecommendations.test.js src/hooks/useAccountManager.test.js src/hooks/useSeedInput.test.js → 10 files, 358 tests passed.

  • [2026-02-25 13:19 UTC] Post-review hardening: interpret endpoint guardrails + labeling UI contract fixes (Codex GPT-5)

    • Hypothesis
      • Four regressions identified in review can be fixed with narrow changes: undeclared YAML dependency, open-ended interpret endpoint access/model override, metrics field mismatch in labeling UI, and slider normalization drift causing intermittent 400s on label submit.
    • Changes (files + why)
      • tpot-analyzer/requirements.txt:2: add PyYAML==6.0.2 so src/api/routes/golden.py imports are reproducible in fresh environments.
      • tpot-analyzer/src/api/routes/golden.py:22: add interpret access constants/env contracts (GOLDEN_INTERPRET_ALLOWED_MODELS, GOLDEN_INTERPRET_ALLOW_REMOTE).
      • tpot-analyzer/src/api/routes/golden.py:63: add allowlist + loopback checks (_allowed_interpret_models, _is_loopback_request, _enforce_interpret_access) to prevent arbitrary remote model spend by default.
      • tpot-analyzer/src/api/routes/golden.py:355: validate threadContext type and return explicit 400 when malformed.
      • tpot-analyzer/src/api/routes/golden.py:410: map access denials to HTTP 403 with descriptive error.
      • tpot-analyzer/graph-explorer/src/Labeling.jsx:17: replace per-field float rounding with integer-thousandths normalization, guaranteeing distributions sum to 1.0 in backend-compatible precision.
      • tpot-analyzer/graph-explorer/src/Labeling.jsx:260: fix metrics binding to labeledCount and show labeled/total progress.
      • tpot-analyzer/graph-explorer/src/labelingApi.js:32: remove unused lucidity argument from submitLabel API helper.
      • tpot-analyzer/tests/test_golden_routes.py:207: add /api/golden/interpret tests for remote deny-by-default, model allowlist rejection, and successful loopback flow with mocked OpenRouter response.
      • tpot-analyzer/docs/index.md:47: include ADR 010/011 in latest ADR index list to avoid documentation drift.
    • Verification
      • cd tpot-analyzer && .venv/bin/pytest -q tests/test_golden_routes.py tests/test_api_autocomplete.py → 18 passed.
      • cd tpot-analyzer/graph-explorer && npm run build → successful production build.
      • cd tpot-analyzer/graph-explorer && npx eslint src/Labeling.jsx src/labelingApi.js → passes for touched frontend files.

  • [2026-02-25 14:10 UTC] MVP A backend curation loop scaffold (Codex GPT-5)

    • Hypothesis
      • We need a normalized, persistent tweet-label/prediction schema and API layer before dashboard work; this should unlock fixed splits, queue ranking, and Brier evaluation with deterministic behavior.
    • Changes (files + why)
      • tpot-analyzer/src/data/golden/constants.py:1 (NEW): define simulacrum axis constants, split/status enums, and queue weighting contract.
      • tpot-analyzer/src/data/golden/schema.py:1 (NEW): add Option-B normalized schema + helper math/validation utilities.
      • tpot-analyzer/src/data/golden/base.py:1 (NEW): add split bootstrap, candidate retrieval with context-cache hydration, and single-reviewer label upsert history.
      • tpot-analyzer/src/data/golden/predictions.py:1 (NEW): add prediction ingest, disagreement/entropy queue scoring, and queue retrieval.
      • tpot-analyzer/src/data/golden/evals.py:1 (NEW): add Brier evaluation run storage and metrics summary assembly.
      • tpot-analyzer/src/data/golden/store.py:1 (NEW): compose focused mixins into GoldenStore.
      • tpot-analyzer/src/data/golden_store.py:1: compatibility re-export shim for existing import ergonomics.
      • tpot-analyzer/src/api/routes/golden.py:1 (NEW): add /api/golden/candidates|labels|queue|predictions/run|eval/run|metrics endpoints.
      • tpot-analyzer/src/api/server.py:24 and tpot-analyzer/src/api/server.py:116: register golden_bp in app factory startup.
      • tpot-analyzer/tests/test_golden_routes.py:1 (NEW): route-level regression tests for split bootstrap, label state changes, queue disagreement, and eval pass criteria.
      • tpot-analyzer/scripts/verify_mvp_a.py:1 (NEW): human-friendly ✓/✗ verification script for end-to-end MVP A backend flow.
      • tpot-analyzer/docs/adr/009-golden-curation-schema-and-active-learning-loop.md:1 (NEW): capture schema/contract decision and constraints.
      • tpot-analyzer/docs/index.md:1: update docs index review date + add ADR 009/008 entries.
      • tpot-analyzer/docs/ROADMAP.md:32: mark MVP A backend loop item completed under Phase 4 LLM Classification.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m py_compile src/data/golden/constants.py src/data/golden/schema.py src/data/golden/base.py src/data/golden/predictions.py src/data/golden/evals.py src/data/golden/store.py src/data/golden_store.py src/api/routes/golden.py scripts/verify_mvp_a.py tests/test_golden_routes.py → passed.
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_golden_routes.py -q → 3 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_mvp_a --help → CLI usage renders.

  • [2026-02-25] Architecture pivot: tweet-level LLM classification as account fingerprinting (Claude Sonnet 4.6)

    • Context
      • Identified that graph-structure-only clustering cannot capture TPOT's vibe/aesthetic-based community boundaries. TPOT membership is defined by epistemic style (l1/l2/l3 simulacrum axis) and social function, not follow patterns alone. PageRank as a discovery signal is anti-correlated with TPOT membership (TPOT valorizes obscurity).
    • Decisions (see ADR 008)
      • Adopt two-layer architecture: (1) content-aware embedding via tweet classification (universal, runs once), (2) per-user semantic labeling via exemplar annotation (configurable, per-user).
      • Use LLM few-shot classification (OpenRouter, frontier model) with a human-curated golden dataset (taxonomy.yaml) to classify each tweet on three orthogonal axes: epistemic/simulacrum (l1/l2/l3), functional/social (aggression, dialectics, etc.), topic (meditation, alignment, etc.).
      • Treat liked tweets as a separate passive-engagement signal, distinct from posted tweets.
      • Active learning loop: golden dataset grows via human arbitration of high-entropy "scissor tweets"; Brier score tracks calibration per axis per model.
      • 334 anchor accounts serve as embedding scaffold for the broader follow/following graph.
    • Data pipeline built
      • src/archive/fetcher.py: streams community archive JSON per account from Supabase blob storage; atomic temp→rename cache; exponential backoff retry (4 attempts, 2/4/8/16s).
      • src/archive/store.py: parses tweets + likes + note-tweets into data/archive_tweets.db; skips retweets; INSERT OR IGNORE for safe re-runs.
      • scripts/fetch_archive_data.py: parallel download (N workers) + serial DB write; resume-safe (skips ok/not_found in fetch_log, retries error).
      • scripts/verify_archive_vs_cache.py: data quality comparison between archive and scraped cache.db.
    • Status: archive fetch in progress (316 accounts, ~halfway). Retry pass queued after first run completes (uses fixed streaming fetcher).
    • Next
      • Run verify_archive_vs_cache.py once fetch completes
      • Build golden dataset (collaborative: human labels tweets, we craft examples per category)
      • Build LLM eval harness + Brier score script
      • Build classification pipeline with budget controls
      • Recompute account fingerprints + clustering

Phase 1.0: Setup & Infrastructure

  • [2025-10-14] Initial setup, codebase_investigator analysis.

  • [2025-10-14] Established AGENTS.md and docs/ROADMAP.md.

  • [2025-12-08] Architectural Refactoring (Gemini 3 Pro)

    • Hierarchy Decomposition: Split monolithic src/graph/hierarchy.py (701 LOC) into a modular package:
      • models.py: Dataclasses for clusters/edges.
      • traversal.py: Tree navigation logic.
      • layout.py: PCA and edge connectivity math.
      • builder.py: Main orchestration logic.
    • API Refactoring: Refactored server.py (God Object, 1119 LOC) into:
      • src/api/services/: Dependency-injected AnalysisManager and CacheManager to replace global state.
      • src/api/routes/: Functional slices (Blueprints) for core, graph, analysis, discovery, accounts.
      • src/api/server.py: A lightweight Application Factory pattern (~100 LOC).
    • Verification: verify_setup.py passed.

  • [2025-12-12] Phase 1.1 (WIP): Account search → teleport → tag single accounts

    • Backend (teleport plumbing)
      • tpot-analyzer/src/api/cluster_routes.py:296 Parse focus_leaf query param and include it in the cache key (_make_cache_key) so /api/clusters and /members stay consistent.
      • tpot-analyzer/src/graph/hierarchy/builder.py:44 Add focus_leaf_id parameter to build_hierarchical_view and apply it before greedy expansions.
      • tpot-analyzer/src/graph/hierarchy/focus.py:1 Add deterministic “reveal leaf by splitting along the path” helper (reveal_leaf_in_visible_set) to guarantee a target leaf becomes visible when budget allows.
    • Backend (search + tagging)
      • tpot-analyzer/src/api/routes/accounts.py:69 Add GET /api/accounts/search?q= using snapshot metadata for handle/name prefix search (autocomplete semantics).
      • tpot-analyzer/src/api/routes/accounts.py:103 Add tag CRUD endpoints scoped by ego:
        • GET /api/accounts/<id>/tags
        • POST /api/accounts/<id>/tags with {tag, polarity: in|not_in}
        • DELETE /api/accounts/<id>/tags/<tag>
        • GET /api/tags for autocomplete.
      • tpot-analyzer/src/data/account_tags.py:1 Introduce SQLite-backed AccountTagStore persisted at tpot-analyzer/data/account_tags.db.
      • tpot-analyzer/src/api/routes/accounts.py:200 Add GET /api/accounts/<id>/teleport_plan which selects a base cut n that guarantees revealing the target leaf within budget (returns focus_leaf to use in /api/clusters).
    • Frontend (wiring)
      • tpot-analyzer/graph-explorer/src/data.js:703 Add focus_leaf param propagation to fetchClusterView and fetchClusterMembers; fix timing propagation so Stage logs can show per-attempt timings.
      • tpot-analyzer/graph-explorer/src/AccountSearch.jsx:1 Add search dropdown UI that calls /api/accounts/search and triggers teleport.
      • tpot-analyzer/graph-explorer/src/AccountTagPanel.jsx:1 Add “IN / NOT IN” single-account tagging UI backed by the new account tag endpoints.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:220 Add teleport flow: clear state → apply teleport_plan (visibleTarget + focusLeaf) → auto-explode leaf → center camera on the member node; integrate AccountTagPanel in the side panel (ClusterView.jsx:1115).
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:98 Add member-node hit testing + highlight/centering hooks so teleport can visually focus the selected account.
    • Repo hygiene
      • .gitignore Add tpot-analyzer/data/account_tags.db* to keep local tag DB/WAL out of git.
    • Verification
      • tpot-analyzer/scripts/verify_search_teleport_tagging.py:1 Add a human-friendly verification script that exercises /api/clusters, /api/accounts/search, teleport_plan, and tag CRUD with ✓/✗ output.

  • [2025-12-13] TDD backfill: regression tests for teleport + tagging

    • Backend unit/integration tests
      • tpot-analyzer/tests/test_hierarchy_focus_leaf.py:1 Covers deterministic leaf reveal (reveal_leaf_in_visible_set) including budget exhaustion behavior.
      • tpot-analyzer/tests/test_account_tags_store.py:1 Covers AccountTagStore CRUD and tag normalization (case-insensitive keys).
      • tpot-analyzer/tests/test_accounts_search_teleport_tags.py:1 Covers /api/accounts/search, /api/accounts/<id>/teleport_plan, and tag endpoints using Flask’s test client (no network).
    • Frontend component tests (Vitest)
      • tpot-analyzer/graph-explorer/src/AccountSearch.test.jsx:1 Covers debounce + selection → onPick.
      • tpot-analyzer/graph-explorer/src/AccountTagPanel.test.jsx:1 Covers tag load + add/remove flows with mocked API calls.

  • [2025-12-13] Phase 2.0 (WIP): On-demand cluster tag summary + heuristic suggested label

    • Goal / UX
      • Compute tag summary only when a cluster is selected (keeps /api/clusters payload small).
      • Suggested label heuristic uses score = inCount - notInCount and picks the top tag with score > 0.
    • Backend
      • tpot-analyzer/src/data/account_tags.py:110 Add AccountTagStore.list_tags_for_accounts(...) with chunking to avoid SQLite variable limits.
      • tpot-analyzer/src/api/cluster_routes.py:582 Add GET /api/clusters/<cluster_id>/tag_summary returning tagCounts + suggestedLabel (uses cached view when available).
    • Frontend
      • tpot-analyzer/graph-explorer/src/data.js:876 Add fetchClusterTagSummary(...).
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:525 Fetch tag summary on cluster select + after tag edits; render Tag summary panel and “Apply suggested label”.
    • Tests + Verification
      • tpot-analyzer/tests/test_cluster_tag_summary.py:1 Covers tag summary counts + ego requirement.
      • tpot-analyzer/scripts/verify_search_teleport_tagging.py:317 Extend verification to hit /tag_summary and confirm member-tag appears at the cluster level.

  • [2025-12-16] Stabilization: make test suite green + tighten contracts

    • Autocomplete/search semantics
      • tpot-analyzer/src/api/routes/accounts.py:1 Keep /api/accounts/search prefix-only (autocomplete semantics); remove dead _rank_search_hit.
      • tpot-analyzer/tests/test_api_autocomplete.py:1 Align fixtures with documented expectations (5 eigen* accounts).
    • Enricher testability + observability
      • tpot-analyzer/src/shadow/enricher.py:1 Fail fast if policy isn’t an EnrichmentPolicy; add warning logs when scrape-history lookup fails but proceed “fail-open”.
      • tpot-analyzer/tests/conftest.py:1 Return a real EnrichmentPolicy fixture (no Mock() truthiness surprises).
      • tpot-analyzer/tests/test_account_status_tracking.py:1 Use real EnrichmentPolicy in HybridShadowEnricher tests.
    • Result
      • cd tpot-analyzer && .venv/bin/python -m pytest -q → 307 passed, 7 skipped (green).
    • Dev tooling
      • tpot-analyzer/graph-explorer/package.json:1 Add test:e2e:mock (auto-starts Vite) and test:e2e:mock:no-server for reusing an already-running dev server.
      • tpot-analyzer/scripts/run_e2e.sh:1 Add a repo-level Playwright runner for mock/full/ui/debug modes.
      • tpot-analyzer/scripts/verify_browser_binaries.py:1 Add a browser-binary/cache verification script; docs in tpot-analyzer/docs/diagnostics/BROWSER_BINARIES.md.

  • [2025-12-16] E2E: lock in Phase 1 loop (mocked)

    • UI deep-link correctness
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:213 Gate URL-sync effect on urlParsed so StrictMode mount doesn’t overwrite initial URL params before parsing (fixes flaky deep-link behavior in tests and manual reloads).
    • Mocked E2E coverage (Playwright)
      • tpot-analyzer/graph-explorer/e2e/teleport_tagging_mock.spec.ts:1 Add “search → teleport → tag → tag summary → apply suggested label” regression test with fully mocked backend.
      • tpot-analyzer/graph-explorer/e2e/cluster_mock.spec.ts:1 Stabilize cluster-view E2E by clicking nodes deterministically via canvas test helpers; add real API-failure assertion (HTTP 500).
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:225 Expose window.__CLUSTER_CANVAS_TEST__ helpers in dev mode only (Playwright uses these for stable node selection).
    • Runner updates
      • tpot-analyzer/graph-explorer/package.json:1 Update mock runner scripts to execute all e2e/*mock*.spec.ts tests.
      • tpot-analyzer/scripts/run_e2e.sh:1 Update mock runner to execute all e2e/*mock*.spec.ts tests.
    • Verification
      • cd tpot-analyzer/graph-explorer && npm run test:e2e:mock (9 passed)
      • cd tpot-analyzer/graph-explorer && npx vitest run (7 passed)

  • [2025-12-16] UI polish: force-morph expand/collapse transitions

    • Goal: remove “teleporting” feel when the cluster layout changes by using a short-lived D3 force simulation to morph nodes toward their new positions.
    • Implementation
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:384 Replace linear “move-to-target” tweening with a force-directed morph (forceX/forceY to target + forceCollide + forceManyBody) and snap to exact targets at the end for deterministic focus/teleport behavior.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:832 Keep exploded member dots visually attached to their parent cluster during morphs by translating them by the parent’s display delta.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:185 E2E helper window.__CLUSTER_CANVAS_TEST__ returns displayed positions (settled/morphed), keeping Playwright clicks stable while nodes move.
    • Verification
      • cd tpot-analyzer/graph-explorer && npm run test:e2e:mock (9 passed)
      • cd tpot-analyzer/graph-explorer && npx vitest run (7 passed)
      • cd tpot-analyzer && .venv/bin/python -m pytest -q (307 passed, 7 skipped)

  • [2025-12-17] Observability: request-id correlation + disk-first logs

    • Goal: eliminate “paste logs into chat” by making request timings + correlation easy to inspect via logs/api.log and logs/frontend.log.
    • Backend
      • tpot-analyzer/src/api/request_context.py:1 Add ContextVar-backed request id (req_id) + RequestIdFilter for log record injection.
      • tpot-analyzer/src/api/server.py:57 Add before_request/after_request hooks:
        • Generate/accept X-Request-ID (or reqId query) and echo it in the response header.
        • Emit access logs with dur_ms per request (skip /api/log to DEBUG to avoid spam).
      • tpot-analyzer/src/api/server.py:136 Make api.log location stable via TPOT_LOG_DIR (defaults to tpot-analyzer/logs regardless of CWD); keep file handler at DEBUG and include req=<id> in the formatter for grepability.
      • tpot-analyzer/src/api/log_routes.py:1 Write frontend log events to ${TPOT_LOG_DIR}/frontend.log and include req_id in each JSONL entry.
    • Dev scripts
      • tpot-analyzer/scripts/start_dev.sh:7 Default API_LOG_LEVEL=DEBUG, CLUSTER_LOG_LEVEL=DEBUG, TPOT_LOG_DIR=$PROJECT_ROOT/logs; route Vite stdout to logs/vite.log so logs/frontend.log is reserved for POST /api/log.
      • tpot-analyzer/StartGraphExplorer.command:17 Ensure TPOT_LOG_DIR is set for the backend launch path.
    • Verification & tooling
      • tpot-analyzer/scripts/verify_api_observability.py:1 Add a ✓/✗ script that checks /api/health + X-Request-ID + api.log correlation + /api/log → frontend.log.
      • tpot-analyzer/scripts/tail_cluster_logs.py:1 Add a tail/filter helper (--clusters, --req <id>) for api.log and frontend.log.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest -q (309 passed, 9 skipped)
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_api_observability (requires backend running)

  • [2025-12-17] Repo hygiene: canonical docs + E2E naming

    • Docs canonicalization
      • Repo root: replace docs/ with a symlink to tpot-analyzer/docs/ (keep a single canonical doc tree).
      • Repo root: remove legacy graph-explorer/ folder (keep only tpot-analyzer/graph-explorer/).
    • E2E consistency (Playwright)
      • tpot-analyzer/graph-explorer/e2e/cluster_real.spec.ts:1 Rename from cluster.spec.ts; update backend startup instructions to python -m scripts.start_api_server.
      • tpot-analyzer/graph-explorer/playwright.config.ts:1 Update backend-start comment to match scripts.start_api_server.
      • tpot-analyzer/scripts/run_e2e.sh:1 Update full-backend runner to execute cluster_real.spec.ts.
      • tpot-analyzer/graph-explorer/package.json:1 Add test:e2e:real convenience script.
      • tpot-analyzer/docs/TEST_AUDIT.md:1 Update E2E table to reference cluster_real.spec.ts.
      • tpot-analyzer/CODEX_TASK_E2E_TESTS.md:1 Update “Current State” to reference cluster_real.spec.ts.
    • Test suite consolidation
      • tpot-analyzer/tests/test_seed_profile_counts.py:1 Move root test into analyzer suite; remove sys.path hacks.
      • tpot-analyzer/tests/test_shadow_store_migration.py:1 Move root test into analyzer suite; add TPOT_LEGACY_DB_PATH override + @pytest.mark.integration.
    • Browser-binary docs
      • tpot-analyzer/docs/diagnostics/BROWSER_BINARIES.md:1 Document Playwright system browser usage + PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 for restricted-network installs.
      • tpot-analyzer/docs/FEATURES_INTENT.md:1 Align “Disk-friendly E2E” note with system-browser approach.
    • Verification
      • cd tpot-analyzer && python3 -m pytest -q tests/test_seed_profile_counts.py tests/test_shadow_store_migration.py → 3 passed, 1 xfailed

  • [2025-12-17] Bugfix: /api/seeds 500 + GraphExplorer crash

    • Root cause
      • tpot-analyzer/src/api/routes/accounts.py:329 was returning a Python set (load_seed_candidates() result) through jsonify, causing TypeError: Object of type set is not JSON serializable and repeated GET /api/seeds 500 in the UI.
      • tpot-analyzer/graph-explorer/src/GraphExplorer.jsx:1216 rendered graphStats.totalNodes, which could become an array/object when /api/graph-data returns directed_nodes as a node list → React “Objects are not valid as a React child”.
    • Fix
      • tpot-analyzer/src/api/routes/accounts.py:329 now returns the canonical seed/settings state (get_seed_state()), and POST /api/seeds supports both settings updates and seed list save/activate.
      • tpot-analyzer/graph-explorer/src/GraphExplorer.jsx:415 normalizes data.graph.nodes arrays into {[id]: node} maps and coerces directed_nodes/directed_edges/undirected_edges into counts for safe rendering.
      • tpot-analyzer/graph-explorer/src/data.js:313 logs numeric counts instead of dumping arrays.
    • Tests & verification
      • tpot-analyzer/tests/test_api_seeds_endpoint.py:1 adds unit tests for /api/seeds GET/POST.
      • tpot-analyzer/graph-explorer/src/GraphExplorer.test.jsx:1 adds a regression test ensuring GraphExplorer renders summary counts when backend returns directed_nodes arrays.
      • tpot-analyzer/scripts/verify_graph_explorer_boot.py:1 adds a ✓/✗ boot check script for /api/seeds and /api/graph-data.
    • Note
      • tpot-analyzer/graph-explorer/src/GraphExplorer.jsx is >300 LOC (monolith); keep future changes minimal and plan a thin-slice decomposition.

  • [2025-12-22] Hybrid zoom diagnostics: scroll-to-expand instrumentation

    • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx:106,1417,1431,1484,1532 Add opt-in HybridZoom debug logging (?hz_log=1 or localStorage.hybridZoomLog=1) capturing wheel inputs, modifier-zoom path, zoom-mode transitions, and centered-node screen diagnostics for hypothesis falsification.
    • tpot-analyzer/scripts/verify_hybrid_zoom_logging.py:1 Add a ✓/✗ verification script that scans logs/frontend.log for HybridZoom diagnostics and summarizes last payloads.
    • tpot-analyzer/docs/ROADMAP.md:48 Track follow-up to decompose ClusterCanvas.jsx into smaller components.

  • [2025-12-26] Self-evaluating expansion with strategy scoring and caching

    • Goal / Design
      • Replace deterministic expansion heuristics with a "self-evaluating" system that tries all applicable strategies and scores their outputs
      • Weights are on evaluation signals (size entropy, collapse ratio, fragmentation, edge separation, tag coherence), NOT on strategy selection
      • Enables user-tunable weights to define what "good structure" means
      • Precomputation + caching delivers instant expansion clicks
    • Backend (expansion scoring)
      • tpot-analyzer/src/graph/hierarchy/expansion_scoring.py:1 (NEW, 465 LOC) Structure-aware scoring:
        • compute_structure_score() - evaluates expansion outputs on 5 weighted signals
        • StructureScoreWeights - user-tunable weights (default equal)
        • StructureScoreBreakdown - detailed component scores + human-readable reason
        • compute_size_entropy(), compute_collapse_ratio(), compute_fragmentation_ratio(), compute_edge_separation_fast(), compute_tag_coherence() - individual signal computations
        • ScoredStrategy and rank_strategies() for ranking strategies by score
      • tpot-analyzer/src/graph/hierarchy/expansion_strategy.py:600 Added:
        • execute_louvain_local() - local Louvain community detection on induced subgraph
        • evaluate_all_strategies() - runs all applicable strategies and scores results
        • get_best_expansion() - convenience wrapper returning top-ranked strategy
    • Backend (expansion caching)
      • tpot-analyzer/src/graph/hierarchy/expansion_cache.py:1 (NEW, 277 LOC) Caching infrastructure:
        • ExpansionCache - LRU cache with configurable TTL-based expiry
        • CachedExpansion - stores ranked strategies with metadata (compute time, member count)
        • ExpansionPrecomputer - background thread for ahead-of-time computation
        • compute_and_cache_expansion() - on-demand compute + cache
        • trigger_precompute_for_visible_clusters() - queue visible clusters for background precompute
      • tpot-analyzer/src/graph/hierarchy/__init__.py - exports all new symbols
    • Tests
      • tpot-analyzer/tests/test_expansion_scoring.py:1 (NEW) 23 tests for scoring components
      • tpot-analyzer/tests/test_expansion_strategy.py:407 Added 10 tests for evaluate_all_strategies()
      • tpot-analyzer/tests/test_expansion_cache.py:1 (NEW) 21 tests for cache infrastructure
    • Dependencies
      • Added python-louvain package for local community detection
    • Builder integration
    • tpot-analyzer/src/graph/hierarchy/builder.py:197 Replaced should_use_local_expansion() with compute_and_cache_expansion() for clusters >= 10 members
    • Added local_expansion_strategies dict to track which strategy was used for each expansion
    • Virtual clusters now include expansion_strategy field for UI display (e.g., "louvain", "core_periphery", "tag_split")
    • Falls back to dendrogram expansion if scored expansion produces single cluster

  • [2025-12-30] Test coverage hardening (goodhart audit + steelman plan) — WIP

    • Pre-flight review
      • Read: tpot-analyzer/docs/TEST_AUDIT.md, tpot-analyzer/docs/tasks/fix-goodharted-tests.md, tpot-analyzer/docs/TESTING_METHODOLOGY.md, tpot-analyzer/docs/ROADMAP.md, tpot-analyzer/docs/index.md.
      • Hypotheses:
        • H1: Logic reimplementation in tests (e.g., ClusterView utilities, skip-coverage logic) allows behavior regressions to pass.
        • H2: Mock-call assertions in enricher orchestration tests hide missing side effects (upserts/metrics).
        • H3: Cache-internal assertions in cluster route tests bypass response contracts and miss payload regressions.
        • H4: Production-data dependencies and skip markers reduce determinism and inflate green counts.
        • H5: Oversized UI modules suppress granular tests; modularity will improve coverage focus.
    • Completed changes (line numbers)
      • tpot-analyzer/scripts/verify_test_inventory.py:1: add goodhart inventory script with ✓/✗ output + samples.
      • tpot-analyzer/scripts/verify_backend_intent.py:1: add backend verification script for fixtures + helper checks.
      • tpot-analyzer/tests/fixtures/create_test_cache_db.py:1: deterministic cache.db builder for API tests.
      • tpot-analyzer/tests/fixtures/__init__.py:1: mark fixtures as a package for imports.
      • tpot-analyzer/tests/conftest.py:243: add temp_snapshot_dir fixture wiring SNAPSHOT_DIR/CACHE_DB_PATH.
      • tpot-analyzer/tests/test_api.py:11: use deterministic cache fixture + stronger payload assertions.
      • tpot-analyzer/src/shadow/enricher.py:1168 and tpot-analyzer/src/shadow/enricher.py:2371: use + expose _compute_skip_coverage_percent.
      • tpot-analyzer/tests/test_shadow_enricher_utils.py:743: assert skip coverage via helper (no reimplementation).
      • tpot-analyzer/tests/test_cluster_routes.py:362: assert cache hit returns identical payload; label tests use real store.
      • tpot-analyzer/docs/ROADMAP.md:14: capture pending test-hardening gaps (shadow enricher + fixture data).
      • tpot-analyzer/tests/helpers/recording_shadow_store.py:1: add recording store to assert enrichment side effects without mock-call checks.
      • tpot-analyzer/tests/helpers/__init__.py:1: add helper package for test utilities.
      • tpot-analyzer/tests/test_shadow_enricher_orchestration.py:22: switch orchestration tests to recording store + persisted outcome assertions.
      • tpot-analyzer/tests/test_shadow_enricher_orchestration.py:443: align delta refresh test with list-specific policy (only following refreshes).
      • tpot-analyzer/src/graph/builder.py:76: default archive node provenance for graph payloads.
    • Remaining planned changes
      • tpot-analyzer/graph-explorer/src/ClusterView.test.jsx: remove reimplementation tests, replace with user-flow assertions.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.memoryleak.test.jsx: make leak checks deterministic via test hooks.
      • tpot-analyzer/graph-explorer/src/ClusterCanvas.jsx, tpot-analyzer/graph-explorer/src/ClusterView.jsx, tpot-analyzer/graph-explorer/src/GraphExplorer.jsx: decompose into <300 LOC modules.
      • tpot-analyzer/docs/index.md: update doc index for new ADR/scripts once created.
      • tpot-analyzer/docs/adr/006-testability-refactor.md (NEW): record decisions on testability and modularization.
    • Verification
      • cd tpot-analyzer && python3 -m pytest tests/test_shadow_enricher_orchestration.py -q → ERROR ModuleNotFoundError: No module named 'sqlalchemy'.
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_shadow_enricher_orchestration.py tests/test_shadow_enricher_utils.py::TestZeroCoverageEdgeCase tests/test_cluster_routes.py::TestClusterLabelEndpoints tests/test_api.py -q → 28 passed, 1 warning (LibreSSL warning).

  • [2026-02-09 11:26 UTC] Bugfix: Discovery crash from hook initialization order (Codex GPT-5)

    • Hypothesis
      • Discovery crashes because useAccountManager references fireChange in a dependency array before fireChange is initialized, triggering a Temporal Dead Zone ReferenceError during render.
    • Changes (line numbers + why)
      • tpot-analyzer/graph-explorer/src/hooks/useAccountManager.js:26: move fireChange declaration before first use so render-time dependency evaluation cannot read an uninitialized const.
      • tpot-analyzer/graph-explorer/src/hooks/useAccountManager.js:27: update effect dependency to [onAccountChange] so callback-ref sync tracks the real input prop, not the internal notifier.
      • tpot-analyzer/graph-explorer/src/hooks/useAccountManager.js:66: change markAccountPending dependencies from onAccountChange to fireChange to match the closure and avoid stale/misleading deps.
    • Verification
      • cd tpot-analyzer/graph-explorer && npm run build → success (Vite production build completed; existing non-blocking chunk-size/import warnings remain).

  • [2026-02-09 11:54 UTC] Bugfix: /api/subgraph/discover 500 from route/signature drift (Codex GPT-5)

    • Hypothesis
      • Discovery route refactor drifted from src.api.discovery contracts: validate_request tuple output was not unpacked and discover_subgraph was invoked without required graph/pagerank_scores, producing deterministic HTTP 500.
    • Changes (line numbers + why)
      • tpot-analyzer/src/api/routes/discovery.py:22: add _load_graph_result() to restore route-side graph loading (in-memory snapshot -> snapshot loader -> cache.db fallback) so discovery has a directed graph input.
      • tpot-analyzer/src/api/routes/discovery.py:45: add _resolve_seed_handles() to resolve frontend username seeds to graph node ids before discovery scoring.
      • tpot-analyzer/src/api/routes/discovery.py:71: fix request validation flow by unpacking (parsed_request, errors) and returning structured 400 validation responses.
      • tpot-analyzer/src/api/routes/discovery.py:108: compute PageRank and call discover_subgraph(directed_graph, parsed_request, pagerank_scores) with the required signature to remove the route-level TypeError.
      • tpot-analyzer/tests/test_api.py:199: add regression test covering username-seed discovery success path (POST /api/subgraph/discover returns ranked payload).
      • tpot-analyzer/tests/test_api.py:235: add regression test covering invalid discovery payload path (400 with validation details instead of 500).
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_api.py -q → 9 passed.

  • [2026-02-09 17:24 UTC] Docs hygiene cleanup pass (Codex GPT-5)

    • Hypothesis
      • Active docs still had stale references to retired scripts/paths and a missing canonical runbook, causing onboarding drift and broken local commands.
    • Changes (line numbers + why)
      • tpot-analyzer/docs/guides/TEST_MODE.md:5: remove literal legacy script path from active docs while preserving migration guidance.
      • tpot-analyzer/docs/WORKLOG.md:261: update stale future-task path from docs/BACKEND_IMPLEMENTATION.md to docs/reference/BACKEND_IMPLEMENTATION.md.
      • tpot-analyzer/docs/WORKLOG.md:249: add this timestamped cleanup entry to preserve doc-hygiene rationale and verification trace.
    • Verification
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 6/6 checks passing.

  • [2026-02-09 17:45 UTC] Docs hygiene cleanup pass #2: historical docs modernization (Codex GPT-5)

    • Hypothesis
      • Historical docs still contained runnable stale commands/scripts (create_test_fixtures.py, start_test_backend.sh, run_all_tests.sh, scripts/api_server.py) that could mislead operators despite canonical docs being fixed.
    • Changes (line numbers + why)
      • tpot-analyzer/docs/tasks/E2E_TESTS.md:36: add modernization note with current runnable fixture/bootstrap/backend commands.
      • tpot-analyzer/docs/tasks/E2E_TESTS.md:76: mark legacy task-script filenames as historical/superseded and map to current implementations.
      • tpot-analyzer/docs/tasks/E2E_TESTS.md:339: replace stale fixture/backend verification commands with create_test_cache_db bootstrap + scripts.start_api_server workflow.
      • tpot-analyzer/docs/archive/BUGFIXES.md:5: add historical context banner clarifying current backend entrypoint.
      • tpot-analyzer/docs/archive/BUGFIXES.md:119: update backend checklist command to .venv/bin/python -m scripts.start_api_server.
      • tpot-analyzer/docs/index.md:58: add explicit historical-note bullets for docs/tasks/E2E_TESTS.md and docs/archive/BUGFIXES.md.
    • Verification
      • cd tpot-analyzer && rg -n "python scripts/create_test_fixtures\\.py|\\.venv/bin/python3 scripts/api_server\\.py" docs/tasks/E2E_TESTS.md docs/archive/BUGFIXES.md → no direct stale runnable commands remain.
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 6/6 checks passing.

  • [2026-02-09 17:55 UTC] Docs hygiene cleanup pass #3: CI gate + broad historical sweep (Codex GPT-5)

    • Hypothesis
      • Docs hygiene checks should run in CI, and historical-doc drift should be enforced across all docs/tasks/*.md and docs/archive/*.md, not only two specific files.
    • Changes (line numbers + why)
      • tpot-analyzer/.github/workflows/test.yml:19: add Verify docs hygiene step so PRs fail when docs drift regressions are introduced.
      • tpot-analyzer/scripts/verify_docs_hygiene.py:43: expand historical sweep to all markdown files under docs/tasks/ and docs/archive/.
      • tpot-analyzer/scripts/verify_docs_hygiene.py:154: add contextualization check so legacy script references are required to be marked as historical/superseded.
      • tpot-analyzer/docs/PLAYBOOK.md:122: add Docs Release Checklist with required verification and doc-index/worklog hygiene steps.
    • Verification
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passing.
      • cd tpot-analyzer && rg -n "api_server\\.py|create_test_fixtures\\.py|start_test_backend\\.sh|run_all_tests\\.sh" docs/tasks docs/archive → only contextualized historical/superseded references remain.

  • [2026-02-09 18:12 UTC] Phase A-D sprint: discovery reliability + API contracts + service tests (Codex GPT-5)

    • Hypothesis
      • Discovery had edge-case contract drift (non-object JSON shape handling, unknown seed reporting, debug cache leakage), and frontend contract paths required explicit backend parity and repeatable verification tooling.
    • Changes (line numbers + why)
      • tpot-analyzer/src/api/routes/discovery.py:45: extend seed resolver to return unresolved inputs so unknown handles can be surfaced deterministically.
      • tpot-analyzer/src/api/routes/discovery.py:90: fix request-shape validation to reject non-object JSON bodies instead of coercing to {}.
      • tpot-analyzer/src/api/routes/discovery.py:128: return NO_VALID_SEEDS with concrete unknown_handles when no valid seeds remain after resolution.
      • tpot-analyzer/src/api/discovery.py:75: add debug into discovery cache key to prevent debug payload leakage across requests.
      • tpot-analyzer/tests/test_discovery_endpoint_matrix.py:42: add discovery regression matrix for seed normalization, unknown-seed behavior, request-shape validation, caching, and pagination.
      • tpot-analyzer/scripts/verify_discovery_endpoint.py:70: add human-friendly discovery verifier (✓/✗, metrics, next steps).
      • tpot-analyzer/scripts/restart_and_smoke_backend.sh:1: add backend restart + discovery smoke helper for local operator workflows.
      • tpot-analyzer/src/api/services/signal_feedback_store.py:21: add in-memory feedback service for discovery signal feedback and quality aggregation.
      • tpot-analyzer/src/api/server.py:106: inject SIGNAL_FEEDBACK_STORE into app config for route access.
      • tpot-analyzer/src/api/routes/analysis.py:123: add /api/metrics/performance contract endpoint used by frontend API wrappers.
      • tpot-analyzer/src/api/routes/analysis.py:171: add /api/signals/feedback and /api/signals/quality endpoints to match Discovery UI calls.
      • tpot-analyzer/src/api/services/cache_manager.py:48: expose cache-size helpers for performance diagnostics endpoint.
      • tpot-analyzer/tests/test_api_contract_routes.py:18: add contract tests for new performance/signals endpoints.
      • tpot-analyzer/tests/test_signal_feedback_store.py:7: add service behavior tests for feedback aggregation math.
      • tpot-analyzer/tests/test_cache_manager.py:7: add service behavior tests for graph/discovery cache storage + clear semantics.
      • tpot-analyzer/scripts/verify_api_contracts.py:75: add frontend/backend API route parity verifier.
      • tpot-analyzer/scripts/verify_api_services_tests.py:29: add route/service regression verification bundle script.
      • tpot-analyzer/.github/workflows/test.yml:22: add API contract verification step in CI.
      • tpot-analyzer/docs/PLAYBOOK.md:72: document new discovery/API verification commands.
      • tpot-analyzer/docs/ROADMAP.md:18: mark discovery regression verifier and API contract verifier work as implemented.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_api_contract_routes.py tests/test_discovery_endpoint_matrix.py tests/test_api.py::test_subgraph_discover_endpoint_resolves_username_seed tests/test_api.py::test_subgraph_discover_endpoint_rejects_invalid_payload -q → 14 passed.
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_cache_manager.py tests/test_signal_feedback_store.py tests/test_api_contract_routes.py tests/test_discovery_endpoint_matrix.py -q → 18 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_api_services_tests → regression bundle passed (18 tests).
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_api_contracts → Contract gaps: 0.
    • Constraints
      • Local sandbox denied port binding for live backend start (Operation not permitted) while validating scripts/restart_and_smoke_backend.sh; static/syntax checks and test-client verification were used instead.

  • [2026-02-09 23:17 UTC] Recent-activity + twitterapi.io subset verifier (Codex GPT-5)

    • Hypothesis
      • Given current X app gating on follows endpoints, post recency/cadence is still high-signal and available.
      • We need an external relationship-audit comparator to quantify whether local shadow_edge is complete, incorrect, or a strict subset for sampled accounts.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/fetch_recent_activity.py:1 (NEW): adds batched search/recent harvesting for selected accounts; computes last_tweet_at, 7d/30d volume, median/mean post gap, engagement means, and logs request/rate-limit + estimated Post:Read spend.
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:1 (NEW): adds subset audit against twitterapi.io followers/followings endpoints with explicit overlap/coverage/precision metrics and missing/extra samples per account.
      • tpot-analyzer/docs/ROADMAP.md:39 records follow-up for standardizing third-party relationship-audit key wiring and adapter behavior.
    • Verification
      • cd tpot-analyzer && .venv/bin/python scripts/fetch_recent_activity.py --usernames adityaarpitha eigenrobot --dry-run → query batching preview succeeded.
      • cd tpot-analyzer && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --sample-size 1 → expected key-missing diagnostic with concrete env var names.
      • cd tpot-analyzer && .venv/bin/python - <<'PY' ... ast.parse(...) ... PY → syntax parse passed for both new scripts.
    • Discovered constraints
      • tpot-analyzer/.env currently includes X_BEARER_TOKEN but no twitterapi.io key variable; verifier now fails loudly with remediation steps.
      • Live X tests continue to show follows endpoint enrollment gating for the current app/token.

  • [2026-02-10 02:20 UTC] twitterapi.io verifier: per-account checkpoint writes (Codex GPT-5)

    • Hypothesis
      • Full seed-list audits are long-running; writing output only at the end risks losing progress visibility and partial results if interrupted.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:360: add _mean_coverage(...) helper to compute running/final average coverage from current results safely.
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:377: add write_report(...) helper to persist a full JSON report payload with explicit progress (completed_accounts, total_accounts, is_complete).
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:535: write checkpoint report after each account and print checkpoint written (n/total) so humans can monitor in real time.
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:547: reuse write_report(...) for final report write to keep checkpoint/final schema identical.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -u scripts/verify_shadow_subset_against_twitterapiio.py --api-key "$API_KEY" --usernames adityaarpitha --max-pages 1 --page-size 200 --wait-on-rate-limit --output data/outputs/twitterapiio_shadow_audit/checkpoint_smoke.json → checkpoint message emitted and run succeeded.
      • cd tpot-analyzer && jq '{progress, targets, total_remote_requests, average_coverage_pct}' data/outputs/twitterapiio_shadow_audit/checkpoint_smoke.json → shows progress.completed_accounts=1, is_complete=true.

  • [2026-02-10 02:24 UTC] Decomposition: split twitterapi.io verifier into modular helpers (Codex GPT-5)

    • Hypothesis
      • scripts/verify_shadow_subset_against_twitterapiio.py grew past maintainable size (~575 LOC), increasing review/debug overhead and violating file-size decomposition guidance.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:1: refactor to thin orchestrator (CLI flow + human-readable progress output) while keeping flags/output behavior stable.
      • tpot-analyzer/scripts/shadow_subset_audit/cli.py:1: extract argument parsing + API key resolution to isolate entrypoint config concerns.
      • tpot-analyzer/scripts/shadow_subset_audit/remote.py:1: extract remote endpoint schema parsing, pagination, and rate-limit handling logic.
      • tpot-analyzer/scripts/shadow_subset_audit/local_db.py:1: extract sqlite target selection + local follower/following resolution queries.
      • tpot-analyzer/scripts/shadow_subset_audit/reporting.py:1: extract overlap math and JSON report/checkpoint writer.
      • tpot-analyzer/scripts/shadow_subset_audit/constants.py:1: centralize default paths/key env candidates/shared symbols.
      • tpot-analyzer/scripts/shadow_subset_audit/models.py:1: extract RemoteResult dataclass for typed handoff between helpers.
      • tpot-analyzer/scripts/shadow_subset_audit/normalize.py:1: extract username normalization helper for consistent local/remote parsing.
    • Verification
      • cd tpot-analyzer && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --help → usage output renders successfully with expected options.
      • cd tpot-analyzer && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --sample-size 1 → expected missing-key diagnostic preserved (Checked env vars: ...).
      • cd tpot-analyzer && .venv/bin/python scripts/fetch_recent_activity.py --dry-run --usernames user_a → dry-run query preview works after script staging.
      • cd tpot-analyzer && .venv/bin/python scripts/verify_docs_hygiene.py → docs hygiene verifier still passes.

  • [2026-02-10 02:57 UTC] Observability hardening: frequent checkpoints + runtime diagnostics for twitterapi.io audit (Codex GPT-5)

    • Hypothesis
      • Per-account-only checkpoints hide progress/failure cause during long runs; we need request-level observability and persisted runtime state so interruptions/failures are diagnosable.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:29: keep entrypoint thin and add robust failure/interruption checkpoint persistence in main() while delegating audit loop logic.
      • tpot-analyzer/scripts/shadow_subset_audit/runner.py:19: extract main account/relation loop and trigger periodic + boundary checkpoints during in-flight remote fetches.
      • tpot-analyzer/scripts/shadow_subset_audit/console.py:61: centralize checkpoint writes so each write includes progress + runtime metadata and emits explicit checkpoint reason.
      • tpot-analyzer/scripts/shadow_subset_audit/observability.py:42: add AuditRuntime state model (status, current account/relation, remote request counters, failure reason, recent event history) plus logger wiring.
      • tpot-analyzer/scripts/shadow_subset_audit/remote.py:103: add structured remote events (request_start, response, request_exception, rate_limit_wait, page_complete, relation_*) via callback for live checkpoint triggers and diagnostics.
      • tpot-analyzer/scripts/shadow_subset_audit/reporting.py:47: add optional runtime payload in report JSON so checkpoint files preserve run-state observability.
      • tpot-analyzer/scripts/shadow_subset_audit/cli.py:52: add runtime observability flags (--checkpoint-every-requests, --checkpoint-min-seconds, --max-event-history, --log-level, --log-file).
      • tpot-analyzer/scripts/shadow_subset_audit/constants.py:14: include API_KEY in env fallback candidates to match current local .env key naming.
    • Verification
      • cd tpot-analyzer && .venv/bin/python - <<'PY' ... ast.parse(...) ... PY → syntax parse passed for updated/new audit modules.
      • cd tpot-analyzer && set -a && source .env >/dev/null 2>&1 && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --usernames adityaarpitha --max-pages 1 --page-size 50 --sample-output-count 3 --wait-on-rate-limit --checkpoint-every-requests 1 --checkpoint-min-seconds 1 --max-event-history 20 --output data/outputs/twitterapiio_shadow_audit/checkpoint_observability_smoke.json → emitted multiple in-flight checkpoints and wrote runtime metadata + .log file.
      • cd tpot-analyzer && jq '.runtime | keys' data/outputs/twitterapiio_shadow_audit/checkpoint_observability_smoke.json → runtime observability fields present (status, request counters, checkpoint counters, recent events, failure/termination fields).
    • Constraints
      • Sandbox DNS cannot resolve api.twitterapi.io in this session, so smoke verification confirmed observability behavior on request exceptions rather than successful remote payload fetches.

  • [2026-02-10 03:39 UTC] Resume support: avoid redundant twitterapi.io calls after interruption (Codex GPT-5)

    • Hypothesis
      • Interrupted audits currently restart from account 1 and re-fetch already completed accounts; loading prior output and skipping completed usernames will reduce wasted API spend.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/shadow_subset_audit/cli.py:82: add --resume-from-output flag to opt into resumable execution.
      • tpot-analyzer/scripts/shadow_subset_audit/resume.py:50 (NEW): add robust resume-state loader that parses prior output JSON, filters rows to current target set, de-duplicates usernames, and derives carried-forward request totals from saved per-account remote metadata.
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:72: load resume state when requested, pre-seed results + total_remote_requests, and skip completed usernames before remote calls.
      • tpot-analyzer/scripts/verify_shadow_subset_against_twitterapiio.py:98: add resume_noop_complete path when all targets are already complete, writing a final checkpoint without issuing new API requests.
      • tpot-analyzer/scripts/shadow_subset_audit/runner.py:19: accept total_targets, completed_offset, and starting_total_remote_requests so progress/index/checkpoint math remains correct across resumed runs.
    • Verification
      • cd tpot-analyzer && .venv/bin/python - <<'PY' ... ast.parse(...) ... PY → syntax parse passed for updated files (verify_shadow_subset_against_twitterapiio.py, runner.py, resume.py, cli.py).
      • cd tpot-analyzer && set -a && source .env >/dev/null 2>&1 && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --usernames adityaarpitha --resume-from-output --output data/outputs/twitterapiio_shadow_audit/checkpoint_smoke.json --max-pages 1 --page-size 20 --timeout-seconds 10 → resume no-op path triggered (remaining=0, checkpoint reason resume_noop_complete).
      • cd tpot-analyzer && set -a && source .env >/dev/null 2>&1 && .venv/bin/python scripts/verify_shadow_subset_against_twitterapiio.py --usernames adityaarpitha eigenrobot --resume-from-output --output data/outputs/twitterapiio_shadow_audit/checkpoint_smoke.json --max-pages 1 --page-size 20 --timeout-seconds 10 → skipped pre-completed adityaarpitha; resumed at [2/2] @eigenrobot and preserved carried-forward request count.

  • [2026-02-10 05:15 UTC] ADR draft: shared tagging + anchor-conditioned TPOT membership (Codex GPT-5)

    • Hypothesis
      • Current local-only tagging is sufficient for single-user exploration but cannot support collaborative active learning or Chrome-extension labeling workflows.
      • A shared backend and explicit membership formulation are required before implementation can proceed safely.
    • Changes (line numbers + why)
      • tpot-analyzer/docs/adr/006-shared-tagging-and-tpot-membership.md:1: add a proposed ADR defining the two-surface architecture (graph navigation + feed labeling), shared tag storage decision, anchor-conditioned TPOT probability model, migration outline, and blocking questions.
      • tpot-analyzer/docs/index.md:6: update last-reviewed date to keep doc-index freshness explicit.
      • tpot-analyzer/docs/index.md:45: add direct link to ADR 006 under Architecture and Specs for discoverability.
      • tpot-analyzer/docs/ROADMAP.md:33: add feature backlog items for anchor-conditioned TPOT scoring and uncertainty-driven active learning.
      • tpot-analyzer/docs/ROADMAP.md:57: add infrastructure backlog items for shared tag storage migration and Chrome-extension integration.
      • tpot-analyzer/docs/WORKLOG.md:396: add this timestamped entry for ADR intent and traceability.
    • Verification
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passing.

  • [2026-02-10 05:30 UTC] Phase 1 implementation: extension feed ingestion + account exposure summaries (Codex GPT-5)

    • Hypothesis
      • Capturing in-feed tweet impressions gives stronger semantic signal than follow edges alone for "who is this account?" modeling.
      • A thin ingestion + summary layer can ship safely now and feed later TPOT membership scoring iterations.
    • Changes (line numbers + why)
      • tpot-analyzer/src/data/feed_signals.py:1 (NEW): add SQLite store for extension event ingest with idempotent event keys, validation, raw payload retention, and tweet rollup upserts.
      • tpot-analyzer/src/data/feed_signals_queries.py:1 (NEW): extract summary/top-exposure query + keyword extraction helpers to keep feed_signals.py under 300 LOC and preserve modularity.
      • tpot-analyzer/src/api/routes/extension.py:1 (NEW): add /api/extension/feed_events, /api/extension/accounts/<id>/summary, and /api/extension/exposure/top endpoints with scope validation (ego, workspace_id) and explicit error handling.
      • tpot-analyzer/src/api/server.py:26: register extension_bp so extension APIs are available in app factory startup paths.
      • tpot-analyzer/tests/test_feed_signals_store.py:1 (NEW): add store-level tests for dedupe, rollup counts, keyword extraction, and invalid day windows.
      • tpot-analyzer/tests/test_extension_routes.py:1 (NEW): add route-level tests for validation, ingest roundtrip, account summary, and top exposure listing.
      • tpot-analyzer/scripts/verify_extension_feed_ingest.py:1 (NEW): add human-friendly ✓/✗ verifier for extension ingestion + summary endpoints.
      • tpot-analyzer/docs/reference/DATABASE_SCHEMA.md:363: document feed_events and feed_tweet_rollup schemas and intent.
      • tpot-analyzer/docs/ROADMAP.md:37: capture follow-up for embedding + recency/ranking-bias normalization on extension-captured content.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_feed_signals_store.py tests/test_extension_routes.py tests/test_api_contract_routes.py -q → 7 passed.
    • cd tpot-analyzer && .venv/bin/python -m scripts.verify_extension_feed_ingest --help → CLI usage renders.
    • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passing.

  • [2026-02-10 06:17 UTC] Phase 1.1 implementation: open-mode policy controls + continuous firehose + tag-scope purge (Codex GPT-5)

    • Hypothesis
      • We can safely keep localhost ingestion in open mode while still making privacy boundaries explicit via allowlist toggles and tag-scoped purge.
      • Continuous append-only firehose mirroring gives Indra's Net-compatible raw telemetry without blocking existing summary APIs.
    • Changes (line numbers + why)
      • tpot-analyzer/src/data/feed_scope_policy.py:101 (NEW): add scoped policy table/store with defaults (open, infinite, continuous) plus allowlist/firehose toggles.
      • tpot-analyzer/src/data/feed_firehose.py:14 (NEW): add append-only NDJSON firehose writer for continuous raw event mirroring.
      • tpot-analyzer/src/data/feed_signals_admin.py:49 (NEW): add admin queries for raw event paging, inserted-key fetch, and scoped deletion.
      • tpot-analyzer/src/data/feed_signals.py:148: extend ingest to optionally return inserted event keys for downstream firehose selection.
      • tpot-analyzer/src/data/account_tags.py:110: add positive-tag account-id lookup helpers for tag-based allowlist and purge.
      • tpot-analyzer/src/api/routes/extension.py:35: add /api/extension/settings, /feed_events/raw, /feed_events/purge_by_tag, and ingest policy/firehose orchestration.
      • tpot-analyzer/src/api/routes/extension_runtime.py:26 (NEW): extract extension route singleton dependency wiring to keep route module <300 LOC.
      • tpot-analyzer/src/api/routes/extension_utils.py:13 (NEW): extract shared request validation + ingest auth helpers.
      • tpot-analyzer/src/api/routes/extension_read_routes.py:14 (NEW): extract summary/top read routes to preserve modularity.
      • tpot-analyzer/tests/test_extension_routes.py:40: add route coverage for settings updates, firehose filtering, raw-event reads, tag-scope purge, and guarded mode auth.
      • tpot-analyzer/tests/test_feed_scope_policy_store.py:9 (NEW): add policy-store behavior/validation tests.
      • tpot-analyzer/tests/test_feed_signals_admin_store.py:10 (NEW): add admin-store raw/purge tests.
      • tpot-analyzer/tests/test_feed_signals_store.py:9: assert inserted-event-key collection during ingest.
      • tpot-analyzer/tests/test_account_tags_store.py:9: assert new tag-to-account lookup helpers.
      • tpot-analyzer/scripts/verify_extension_feed_ingest.py:95: extend verifier to check settings API, raw-event API, firehose output, and tag-scope purge.
      • tpot-analyzer/docs/reference/DATABASE_SCHEMA.md:414: document feed_scope_policy table and firehose stream path semantics.
      • tpot-analyzer/docs/ROADMAP.md:65: add follow-up items for firehose relay worker and storage/privacy-boundary verification.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_extension_routes.py tests/test_feed_signals_store.py tests/test_feed_signals_admin_store.py tests/test_feed_scope_policy_store.py tests/test_account_tags_store.py tests/test_api_contract_routes.py -q → 17 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_extension_feed_ingest --help → CLI usage renders.
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passing.

  • [2026-02-10 07:05 UTC] Phase 1.2 implementation: spectator firehose relay worker (Codex GPT-5)

    • Hypothesis
      • Spectator-mode raw streams (X, YouTube shorts, Instagram, etc.) should remain outside main prayer-detection DB and be relayed via a checkpointed firehose worker.
      • A byte-offset checkpoint + retry/backoff transport is sufficient to keep relay durable under local restarts and endpoint hiccups.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/relay_firehose_to_indra.py:1 (NEW): add thin CLI for continuous/once relay execution with batching, retry, and participant filtering controls.
      • tpot-analyzer/scripts/firehose_relay/models.py:1 (NEW): add typed models for records/read results/send results/checkpoint state.
      • tpot-analyzer/scripts/firehose_relay/state.py:1 (NEW): add atomic checkpoint load/save helpers for relay_checkpoint.json.
      • tpot-analyzer/scripts/firehose_relay/reader.py:1 (NEW): add incremental NDJSON reader with truncation/rotation handling + parse-error accounting.
      • tpot-analyzer/scripts/firehose_relay/transport.py:1 (NEW): add HTTP POST transport with explicit retry/backoff behavior.
      • tpot-analyzer/scripts/firehose_relay/worker.py:20 (NEW): add relay loop that skips participant events by default, forwards spectator batches, tracks lag, and persists checkpoint metrics.
      • tpot-analyzer/scripts/verify_firehose_relay.py:1 (NEW): add human-friendly ✓/✗ verifier with local mock endpoint and concrete forwarding metrics.
      • tpot-analyzer/tests/test_firehose_relay_worker.py:8 (NEW): add relay tests for spectator-only forwarding and checkpoint resume semantics.
      • tpot-analyzer/docs/ROADMAP.md:65: mark firehose relay worker item complete and record implementation artifacts.
      • tpot-analyzer/docs/reference/DATABASE_SCHEMA.md:450: document relay checkpoint file contract and metrics fields.
      • tpot-analyzer/docs/PLAYBOOK.md:84: document relay verification/run commands for operations.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_firehose_relay_worker.py tests/test_extension_routes.py tests/test_feed_scope_policy_store.py tests/test_feed_signals_admin_store.py tests/test_feed_signals_store.py tests/test_account_tags_store.py tests/test_api_contract_routes.py -q → 19 passed.
      • cd tpot-analyzer && .venv/bin/python scripts/verify_firehose_relay.py → all checks passed (events_forwarded_total=2, events_skipped_participant_total=1).
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passing.

Upcoming Tasks

  1. Unit Test Backfill: The refactor moved code, but existing tests in test_api.py are integration tests dependent on a live DB. We need unit tests for the new services/ and routes/ that mock the managers.
  2. Documentation Update: docs/reference/BACKEND_IMPLEMENTATION.md needs to be updated to reflect the new modular architecture.
  3. Frontend Alignment: Ensure graph-explorer API calls match the new route structure (URLs remained mostly the same, but need verification).

  • [2026-02-10 15:15 UTC] Relay default endpoint alignment: localhost firehose ingest (Codex GPT-5)

    • Hypothesis
      • Relay ergonomics improve if the CLI defaults to the now-live Indra endpoint (http://localhost:7777/api/firehose/ingest) instead of requiring a mandatory flag/env setup each run.
    • Changes (line numbers + why)
      • tpot-analyzer/scripts/relay_firehose_to_indra.py:21: add DEFAULT_INDRA_FIREHOSE_ENDPOINT constant and _default_endpoint_url() helper.
      • tpot-analyzer/scripts/relay_firehose_to_indra.py:43: switch --endpoint-url default to _default_endpoint_url() so unset/blank env falls back to localhost ingest path.
      • tpot-analyzer/tests/test_relay_firehose_cli.py:1 (NEW): add unit tests for default endpoint, env override, and blank-env fallback behavior.
      • tpot-analyzer/docs/PLAYBOOK.md:96: update operator command to use no-flag default relay run; keep explicit override example for non-default endpoints.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_relay_firehose_cli.py tests/test_firehose_relay_worker.py -q → 5 passed (plus existing LibreSSL urllib3 warning).

  • [2026-02-17 19:00 UTC] Phase 0/1 foundation: observation-aware adjacency controls + diagnostics (Codex GPT-5)

    • Hypothesis
      • The current clustering path treats unobserved edges as absent edges; introducing an optional observation-aware IPW adjacency path (behind settings flags) should improve math fidelity under incomplete graph coverage while preserving safe default behavior.
    • Changes (files + why)
      • tpot-analyzer/src/graph/observation_model.py:1 (NEW): add observation completeness + inverse-probability weighting helpers and stats summarization.
      • tpot-analyzer/src/api/cluster_routes.py:1: wire optional adjacency build path (obs_weighting=off|ipw), maintain legacy cache compatibility, and include observation metadata in cluster responses.
      • tpot-analyzer/src/graph/seeds.py:1: add graph settings schema flags (hierarchy_engine, membership_engine, obs_weighting, obs_p_min, obs_completeness_floor) with value clamping/validation.
      • tpot-analyzer/config/graph_settings.json:1: add matching default settings entries so behavior is explicit and operator-tunable.
      • tpot-analyzer/src/graph/__init__.py:1: export observation helpers for shared graph-module usage.
      • tpot-analyzer/tests/test_observation_model.py:1 (NEW): test completeness math, IPW weighting behavior, clipping, and summary output.
      • tpot-analyzer/tests/test_graph_settings_flags.py:1 (NEW): test settings defaults/sanitization for new graph math flags.
      • tpot-analyzer/scripts/verify_phase0_baseline.py:1 (NEW): add baseline verifier with explicit ✓/✗ checks and next-step guidance.
      • tpot-analyzer/scripts/verify_observation_weighting.py:1 (NEW): add observation/IPW verifier with metrics reporting.
      • tpot-analyzer/docs/adr/007-observation-aware-clustering-membership.md:1 (NEW): record decision and rollout plan for observation-aware math and membership evolution.
      • tpot-analyzer/docs/index.md:1: link ADR 007 and refresh docs index metadata.
      • tpot-analyzer/docs/ROADMAP.md:1: track partial-observability benchmarking and uncertainty follow-ups.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_observation_model.py tests/test_graph_settings_flags.py tests/test_api_seeds_endpoint.py tests/test_cluster_routes.py -q → 41 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_phase0_baseline → all checks passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_observation_weighting --mode off → passed.
    • cd tpot-analyzer && .venv/bin/python -m scripts.verify_observation_weighting --mode ipw --p-min 0.01 --completeness-floor 0.01 → passed (nodes=95303, edges=322621, mean_completeness=0.6907, clipped_pairs=63268).

  • [2026-02-17 19:07 UTC] Phase 1.1 implementation: GRF membership endpoint + anchor aggregation (Codex GPT-5)

    • Hypothesis
      • We can ship a principled TPOT membership path now by solving a GRF/harmonic labeling system from ego-scoped account-tag anchors, while keeping existing hierarchy rendering unchanged.
    • Changes (files + why)
      • tpot-analyzer/src/graph/membership_grf.py:1 (NEW): add sparse GRF solver (compute_grf_membership) with regularized Laplacian solve, uncertainty outputs, and convergence metadata.
      • tpot-analyzer/src/data/account_tags.py:171: add list_anchor_polarities(ego=...) to aggregate per-account anchor polarity by sign of net tags.
      • tpot-analyzer/src/api/cluster_routes.py:149: add membership helpers (engine gate, anchor resolution, anchor digest, coverage estimate).
      • tpot-analyzer/src/api/cluster_routes.py:846: add GET /api/clusters/accounts/<account_id>/membership endpoint returning probability, CI, uncertainty/evidence, anchor counts, and solver metadata (cache-backed).
      • tpot-analyzer/src/graph/__init__.py:31: export GRF module symbols.
      • tpot-analyzer/tests/test_membership_grf.py:1 (NEW): solver behavior tests (balanced chain, connectivity bias, missing-anchor validation).
      • tpot-analyzer/tests/test_cluster_membership_endpoint.py:1 (NEW): route contract tests for disabled engine, missing anchors, and cache-hit behavior.
      • tpot-analyzer/tests/test_account_tags_store.py:44: extend store tests for anchor-polarity aggregation.
      • tpot-analyzer/scripts/verify_membership_grf.py:1 (NEW): add human-friendly phase verifier with ✓/✗ checks + metrics + next steps.
      • tpot-analyzer/docs/ROADMAP.md:30: record shipped GRF endpoint and add calibration/UI integration follow-ups.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_membership_grf.py tests/test_cluster_membership_endpoint.py tests/test_account_tags_store.py -q → 8 passed.
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_cluster_routes.py tests/test_api_seeds_endpoint.py tests/test_observation_model.py tests/test_graph_settings_flags.py tests/test_membership_grf.py tests/test_cluster_membership_endpoint.py -q → 47 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_membership_grf → all checks passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_phase0_baseline && .venv/bin/python -m scripts.verify_observation_weighting --mode off → all checks passed.

  • [2026-02-18 07:20 UTC] Phase 1.2 implementation: ClusterView membership panel wiring (Codex GPT-5)

    • Hypothesis
      • Showing account-level GRF membership in the existing right sidebar (selected account section) will add actionable TPOT context without changing cluster navigation behavior.
    • Changes (files + why)
      • tpot-analyzer/graph-explorer/src/data.js:620: add fetchAccountMembership(...) API helper for GET /api/clusters/accounts/<id>/membership?ego=... with explicit error propagation.
      • tpot-analyzer/graph-explorer/src/AccountMembershipPanel.jsx:1 (NEW): add focused presentational panel for probability, CI, uncertainty, coverage, anchor counts, and high-uncertainty warning.
      • tpot-analyzer/graph-explorer/src/ClusterDetailsSidebar.jsx:9: render membership panel in selected-account block and thread loading/error/membership props.
      • tpot-analyzer/graph-explorer/src/ClusterView.jsx:450: add loadMembership flow, abort handling, selected-account membership effect, and refresh on tag changes.
      • tpot-analyzer/graph-explorer/src/ClusterView.test.jsx:161: update ./data mock to include fetchAccountMembership.
      • tpot-analyzer/graph-explorer/src/ClusterView.integration.test.jsx:651: add end-to-end UI test for member-click → membership fetch → panel render.
      • tpot-analyzer/graph-explorer/src/AccountMembershipPanel.test.jsx:1 (NEW): add component tests for guidance/loading/data render states.
      • tpot-analyzer/scripts/verify_membership_ui.py:1 (NEW): add human-friendly verification script for frontend membership wiring and artifact presence.
      • tpot-analyzer/docs/ROADMAP.md:44: mark membership-panel integration complete and add decomposition follow-up for ClusterView.jsx/data.js.
    • Verification
      • cd tpot-analyzer/graph-explorer && npx vitest run src/AccountMembershipPanel.test.jsx src/ClusterView.test.jsx src/ClusterView.integration.test.jsx → 41 passed.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_membership_ui → 12/12 checks passed.

  • [2026-02-18 09:30 UTC] Phase 1 remediation kickoff: fetcher resource cleanup + real-DB test isolation (Codex GPT-5)

    • Hypotheses
      • CachedDataFetcher.close() leaks SQLite handles because SQLAlchemy engine pools are never disposed.
      • A subset of tests reads shared data/cache.db by default, introducing environment-coupled flakiness (disk I/O error) during full-suite execution.
    • Changes (line numbers + why)
      • tpot-analyzer/src/data/fetcher.py:64: update close() to always dispose the SQLAlchemy engine/pool and clear owned HTTP client references; this releases SQLite file descriptors deterministically.
      • tpot-analyzer/tests/test_shadow_coverage.py:13: add _require_real_cache_db() opt-in gate (TPOT_RUN_REAL_DB_TESTS) for shared-db coverage tests.
      • tpot-analyzer/tests/test_shadow_coverage.py:108: gate test_low_coverage_detection() on explicit shared-db opt-in.
      • tpot-analyzer/tests/test_shadow_coverage.py:146: gate test_archive_vs_shadow_coverage() on explicit shared-db opt-in.
      • tpot-analyzer/tests/test_shadow_coverage.py:203: gate test_coverage_script_runs() on explicit shared-db opt-in.
      • tpot-analyzer/tests/test_shadow_enricher_utils.py:26: add REAL_DB_REQUIRED skip marker for integration classes that depend on shared data/cache.db.
      • tpot-analyzer/tests/test_shadow_enricher_utils.py:666: apply real-db opt-in marker to account-ID migration integration class.
      • tpot-analyzer/tests/test_shadow_enricher_utils.py:782: apply real-db opt-in marker to multi-run freshness integration class.
      • tpot-analyzer/scripts/verify_test_isolation.py:1 (NEW): add human-friendly verifier with ✓/✗ checks for fetcher-handle release, default skip behavior, optional real-db smoke test, and explicit next-step guidance.
      • tpot-analyzer/docs/ROADMAP.md:18: add follow-up for a dedicated opt-in shared-db regression lane to keep TPOT_RUN_REAL_DB_TESTS coverage visible outside default suites.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_shadow_coverage.py tests/test_shadow_enricher_utils.py -q → 33 passed, 6 skipped.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_test_isolation → all checks passed (max_handles=0, expected skip count matched, opt-in smoke test passed).
      • cd tpot-analyzer && .venv/bin/python -m pytest -q --maxfail=20 → 543 passed, 12 skipped, 3 xfailed.

  • [2026-02-18 09:45 UTC] Phase 2 remediation: discovery BFS depth progression fix (Codex GPT-5)

    • Hypothesis
      • Discovery depth expansion is capped at one hop because BFS frontier state is computed after mutating the visited-node set, causing current_layer to become empty before hop 2.
    • Changes (line numbers + why)
      • tpot-analyzer/src/api/discovery.py:245: compute next_frontier before updating subgraph_nodes, break early on empty frontier, and carry frontier forward correctly so depth>1 traverses as intended.
      • tpot-analyzer/tests/test_discovery_logic.py:1 (NEW): add unit regressions that verify two-hop inclusion and strict depth boundary behavior on a deterministic in-memory graph.
      • tpot-analyzer/scripts/verify_discovery_depth.py:1 (NEW): add phase verification script with explicit ✓/✗ checks, metrics, and next-step guidance for discovery traversal.
      • tpot-analyzer/docs/ROADMAP.md:27: add follow-up item to harden expansion-test dependency handling for missing python-louvain (community) in local/CI environments.
    • Verification
      • cd tpot-analyzer && python3 -m pytest tests/test_discovery_logic.py tests/test_discovery_endpoint_matrix.py tests/test_api.py -q → 20 passed.
      • cd tpot-analyzer && python3 -m scripts.verify_discovery_depth → all checks passed (checks_passed=3/3).
      • cd tpot-analyzer && python3 -m pytest -q --maxfail=20 → discovery tests passed; full run reported unrelated existing failures from missing optional dependency (ModuleNotFoundError: community) in expansion modules (tests/test_expansion_cache.py, tests/test_expansion_strategy.py).

  • [2026-02-18 10:42 UTC] Engineering guardrails doc: empirical bug patterns → enforceable invariants (Codex GPT-5)

    • Hypothesis
      • Capturing recent failures as architecture guardrails (symptom, generator, invariant, guardrail, migration policy) will reduce repeat regressions and make future agent work more consistent under high-velocity iteration.
    • Changes (line numbers + why)
      • tpot-analyzer/docs/reference/ENGINEERING_GUARDRAILS.md:1 (NEW): add canonical guardrails document with traced entries for discovery BFS frontier collapse, fetcher resource lifecycle leak, real-DB test coupling, and optional dependency drift (community/python-louvain).
      • tpot-analyzer/docs/index.md:6: update doc index review date.
      • tpot-analyzer/docs/index.md:25: add Engineering Guardrails to canonical operational docs for discoverability.
    • Verification
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passed.

  • [2026-02-21 09:45 UTC] Phase 2 remediation follow-up: strict Louvain dependency contract (Codex GPT-5)

    • Hypothesis
      • Expansion strategy failures (ModuleNotFoundError: community) are caused by undeclared dependency contract drift; pinning python-louvain in canonical requirements and adding an explicit verifier will make environments reproducible.
    • Changes (line numbers + why)
      • tpot-analyzer/requirements.txt:8: add python-louvain==0.16 to make the Louvain backend an explicit required dependency for environments created from project requirements.
      • tpot-analyzer/scripts/verify_louvain_dependency_contract.py:1 (NEW): add human-friendly verifier with ✓/✗ checks for requirements pin presence and import/execute behavior of community_louvain.best_partition.
      • tpot-analyzer/docs/ROADMAP.md:26: mark Louvain dependency-contract hardening item complete with implementation reference.
      • tpot-analyzer/docs/reference/ENGINEERING_GUARDRAILS.md:142: update optional dependency drift guardrail to reference concrete pin + verifier artifacts.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_louvain_dependency_contract → all checks passed (checks_passed=2/2).
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_expansion_strategy.py::TestExecuteLouvainLocal::test_finds_communities -q → 1 passed.
      • cd tpot-analyzer && .venv/bin/python -m pytest tests/test_expansion_strategy.py tests/test_expansion_cache.py -q → 45 passed.
      • cd tpot-analyzer && .venv/bin/python -m pytest -q --maxfail=20 → 545 passed, 12 skipped, 3 xfailed.
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passed.

  • [2026-02-22 04:28 UTC] Developer-experience hardening: venv-enforced local test entrypoints + CI contract gate (Codex GPT-5)

    • Hypothesis
      • Preexisting interpreter drift (python3 vs .venv/bin/python) and implicit CI path assumptions make dependency failures hard to diagnose; codifying local and CI execution contracts will keep test signals reliable.
    • Changes (line numbers + why)
      • tpot-analyzer/Makefile:1 (NEW): add canonical local test/verification targets with .venv/bin/python default (verify-louvain-contract, test-smoke, test) and explicit missing-venv diagnostics.
      • tpot-analyzer/.github/workflows/test.yml:15: add project-directory resolver (. vs tpot-analyzer) to make workflow commands path-stable across checkout layouts.
      • tpot-analyzer/.github/workflows/test.yml:29: add Verify Louvain dependency contract step before test execution.
      • tpot-analyzer/.github/workflows/test.yml:48: run pytest via python -m pytest to enforce interpreter consistency in CI.
      • tpot-analyzer/scripts/verify_test_runner_contract.py:1 (NEW): add human-friendly ✓/✗ contract verifier for Makefile venv defaults and CI Louvain/pre-pytest guard wiring.
      • tpot-analyzer/docs/PLAYBOOK.md:66: document new local contract/entrypoint commands (make verify-louvain-contract, make test-smoke, verify_test_runner_contract).
      • tpot-analyzer/docs/guides/QUICKSTART.md:177: update quick command reference to make test.
      • tpot-analyzer/README.md:267: add canonical make-based test entrypoints to Testing section.
      • tpot-analyzer/docs/ROADMAP.md:100: mark make target standardization item complete.
    • Verification
      • cd tpot-analyzer && python3 -m scripts.verify_test_runner_contract → all checks passed (checks_passed=8/8).
      • cd tpot-analyzer && make verify-louvain-contract → all checks passed (checks_passed=2/2).
      • cd tpot-analyzer && make test-smoke → 7 passed.
      • cd tpot-analyzer && make test → 545 passed, 12 skipped, 3 xfailed.
      • cd tpot-analyzer && python3 -m scripts.verify_docs_hygiene → 9/9 checks passed.

  • [2026-02-25 22:28 IST] Merge-in: test-coverage-hardening branch integration (Codex GPT-5)

    • Context
      • Preserved and integrated work from test-coverage-hardening (WIP checkpoint bbfcae8) into main, then resolved merge conflicts while preserving newer mainline ClusterView architecture.
    • Imported test-hardening payload
      • Frontend tests hardened to avoid call-count coupling:
        • tpot-analyzer/graph-explorer/src/AccountSearch.test.jsx
        • tpot-analyzer/graph-explorer/src/AccountTagPanel.test.jsx
        • tpot-analyzer/graph-explorer/src/ClusterCanvas.memoryleak.test.jsx
        • tpot-analyzer/graph-explorer/src/ClusterCanvas.test.jsx
        • tpot-analyzer/graph-explorer/src/ClusterView.integration.test.jsx
      • Backend test hardening:
        • tpot-analyzer/tests/test_api.py
        • tpot-analyzer/tests/test_hierarchy_builder.py
        • tpot-analyzer/tests/test_list_scraping.py
        • tpot-analyzer/tests/test_shadow_archive_consistency.py
        • tpot-analyzer/tests/test_shadow_coverage.py
        • tpot-analyzer/tests/test_shadow_enricher_orchestration.py
        • tpot-analyzer/tests/test_shadow_enricher_utils.py
        • tpot-analyzer/tests/test_x_api_client.py
    • Regression repairs applied during integration
      • tpot-analyzer/tests/test_shadow_coverage.py:246:
        • switched subprocess invocation to explicit script path + project-root cwd for deterministic execution.
      • tpot-analyzer/tests/test_shadow_enricher_utils.py:572 and tpot-analyzer/tests/test_shadow_enricher_utils.py:636:
        • added no-op set_pause_callback/set_shutdown_callback to fake Selenium workers to satisfy HybridShadowEnricher constructor contract.
    • Verification
      • cd /Users/aditya/Documents/Ongoing Local/Project 2 - Map TPOT-wt-test-coverage && /Users/aditya/Documents/Ongoing Local/Project 2 - Map TPOT/tpot-analyzer/.venv/bin/pytest -q tpot-analyzer/tests/test_api.py tpot-analyzer/tests/test_hierarchy_builder.py tpot-analyzer/tests/test_list_scraping.py tpot-analyzer/tests/test_shadow_archive_consistency.py tpot-analyzer/tests/test_shadow_coverage.py tpot-analyzer/tests/test_shadow_enricher_orchestration.py tpot-analyzer/tests/test_shadow_enricher_utils.py tpot-analyzer/tests/test_x_api_client.py → 108 passed, 1 warning.
      • cd /Users/aditya/Documents/Ongoing Local/Project 2 - Map TPOT-wt-test-coverage && /Users/aditya/Documents/Ongoing Local/Project 2 - Map TPOT/tpot-analyzer/.venv/bin/python tpot-analyzer/scripts/verify_test_inventory.py → skip markers/call-count/reimplementation markers resolved; internal-state assertions and >300 LOC debt remain (tracked in ROADMAP).

  • [2026-03-05 11:13 IST] Onboarding docs dry-run remediation: broken links + runnable commands (Codex GPT-5)

    • Hypothesis
      • New contributors are blocked by dead README links and a few stale quickstart commands; updating docs to only existing paths and validated CLI invocations should make first-run onboarding reproducible.
    • Changes (line numbers + why)
      • ../README.md:5: remove missing grok-probe project link so root docs no longer point to a non-existent directory.
      • ../README.md:12: clarify that this checkout currently includes tpot-analyzer as the active project entrypoint.
      • tpot-analyzer/README.md:285: point coverage baseline reference to existing historical file (docs/archive/test-coverage-baseline.md).
      • tpot-analyzer/README.md:294: point testing-principles reference to canonical guide (docs/TESTING_METHODOLOGY.md).
      • tpot-analyzer/README.md:382: update center-user fix link to docs/archive/CENTER_USER_FIX.md (live path).
      • tpot-analyzer/README.md:383: update bugfix log link to docs/archive/BUGFIXES.md (live path).
      • tpot-analyzer/README.md:384: update test-mode link to docs/guides/TEST_MODE.md (live path).
      • tpot-analyzer/docs/guides/QUICKSTART.md:27: replace placeholder clone URL with neutral <your-repo-url> template.
      • tpot-analyzer/docs/guides/QUICKSTART.md:91: switch cookie setup command to module form (python -m scripts.setup_cookies).
      • tpot-analyzer/docs/guides/QUICKSTART.md:94: replace invalid --max-seeds flag with valid --max-scrolls.
      • tpot-analyzer/docs/guides/QUICKSTART.md:105: switch spectral build command to module form (python -m scripts.build_spectral) to avoid ModuleNotFoundError: src.
      • tpot-analyzer/docs/guides/QUICKSTART.md:178: switch rebuild command to module form (python -m scripts.refresh_graph_snapshot) for consistency with package entrypoints.
      • tpot-analyzer/docs/guides/QUICKSTART.md:206: switch troubleshooting spectral command to module form (python -m scripts.build_spectral).
      • tpot-analyzer/graph-explorer/README.md:8: align Python prerequisite with project docs (Python 3.9+) to remove version mismatch.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_docs_hygiene → 9/9 checks passing.
      • cd tpot-analyzer && .venv/bin/python -m scripts.build_spectral --help → CLI usage renders (module entrypoint valid).
      • cd tpot-analyzer && .venv/bin/python -m scripts.enrich_shadow_graph --help | rg "max-scrolls|--center" → expected flags present.
      • cd /Users/aditya/Documents/Ongoing Local/Project 2 - Map TPOT && python3 <link-check snippet> over onboarding docs (README.md, tpot-analyzer/README.md, docs/index.md, docs/guides/QUICKSTART.md, docs/PLAYBOOK.md, graph-explorer/README.md) → NO_BROKEN_LINKS.

  • [2026-03-05 14:40 IST] Dev onboarding stabilization: optional NetworKit + baseline snapshot path (Codex GPT-5)

    • Hypothesis
      • First-run onboarding failures are caused by (a) mandatory NetworKit build in base dependencies and (b) quickstart using spectral build as baseline despite missing snapshot preconditions.
      • Moving NetworKit to an explicit optional dependency file and using refresh_graph_snapshot as the default quickstart path should make baseline setup reproducible on clean machines.
    • Changes (line numbers + why)
      • tpot-analyzer/requirements.txt:1: remove networkit==11.0 from base install contract so pip install -r requirements.txt no longer requires native NetworKit build toolchain.
      • tpot-analyzer/requirements-performance.txt:1 (NEW): add optional performance dependency manifest with networkit==11.0 and explicit install intent for post-onboarding acceleration.
      • tpot-analyzer/docs/guides/QUICKSTART.md:38: add optional performance-install note (requirements-performance.txt) after baseline dependency install.
      • tpot-analyzer/docs/guides/QUICKSTART.md:102: switch Step 4 baseline from python -m scripts.build_spectral to python -m scripts.refresh_graph_snapshot.
      • tpot-analyzer/docs/guides/QUICKSTART.md:120: mark spectral generation as "Advanced (optional)" instead of baseline onboarding requirement.
      • tpot-analyzer/docs/guides/QUICKSTART.md:216: update "No graph data" troubleshooting command to python -m scripts.refresh_graph_snapshot.
      • tpot-analyzer/scripts/verify_dev_onboarding.py:1 (NEW): add phase verifier with explicit ✓/✗ checks for dependency contract and quickstart onboarding command invariants.
      • tpot-analyzer/docs/PLAYBOOK.md:71: add python3 -m scripts.verify_dev_onboarding to canonical verification command set.
      • tpot-analyzer/docs/ROADMAP.md:261: add follow-up item for explicit offline/local-only snapshot mode to prevent unexpected Supabase refresh during onboarding.
    • Verification
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_dev_onboarding → 6/6 checks passing.
      • cd tpot-analyzer && .venv/bin/python -m scripts.verify_docs_hygiene → 9/9 checks passing.
      • tmpdir=$(mktemp -d /tmp/tpot-req-smoke.XXXXXX) && cd "$tmpdir" && python3 -m venv .venv && .venv/bin/pip install -r /Users/aditya/Documents/Ongoing\ Local/Project\ 2\ -\ Map\ TPOT/tpot-analyzer/requirements.txt → base requirements install completed successfully (no NetworKit compile failure).
      • cd tpot-analyzer && .venv/bin/python -m scripts.refresh_graph_snapshot --output-dir /tmp/tpot-onboard-check-data --frontend-output /tmp/tpot-onboard-check-analysis.json → snapshot refresh completed and emitted expected artifacts.