Skip to content

Commit d0684be

Browse files
sdspiegclaude
andcommitted
Move analytics export to local desktop, add freshness banner
The VPS pipeline lacks statsmodels/scipy and was silently failing on the analytics export step. Rather than install heavy scientific Python on the VPS, scripts/export_analytics_data.py now runs locally on the WSL desktop where statsmodels + RTX 4090 already live; the resulting analytics_*.json files plus a public/data/last_refreshed_analytics.json stamp are committed to this repo and picked up by the next VPS pipeline run. The Causal Analytics tab now reads the stamp on mount and shows "last refreshed Thu, 28 May 2026 21:55 UTC (0.0 days ago)" — turns yellow with a "stale, please re-run" warning when >7 days old. Wrapper: ~/.local/bin/update_redlines_analytics.sh on the desktop. See README.md "Local-compute analytics" section for the full rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8334623 commit d0684be

3 files changed

Lines changed: 79 additions & 2 deletions

File tree

README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,32 @@ VPS (daily 03:00 UTC) GitHub Actions GitHub Pa
6767
| `rls_second_pass.py` | same directory | 2nd pass: RRLS taxonomy annotation |
6868
| `nts_second_pass.py` | same directory | 2nd pass: NTS taxonomy annotation |
6969

70+
## Local-compute analytics (statsmodels / scipy)
71+
72+
The heavy `scripts/export_analytics_data.py` step (VAR, Granger, IRF, LP, event-study via `statsmodels` + `scipy`) does **not** run on the VPS — the VPS pipeline at `/stratbase/apps/webapps/redlines-dashboard-pipeline/run_pipeline.sh` intentionally skips it. Instead it runs on the WSL desktop where the dependencies + the RTX 4090 already live, the result JSONs (`analytics_*.json`) are committed to this repo, and the next VPS pipeline run picks them up via `git pull` and bundles them into the vite build that ships to gh-pages. Same pattern as the other static JSONs (Oryx, UkrDailyUpdate, KIU, Kaggle missiles).
73+
74+
Refresh whenever the redlines DB has new annotations you want reflected on the Causal Analytics tab:
75+
76+
```bash
77+
~/.local/bin/update_redlines_analytics.sh # full: export + commit + push
78+
~/.local/bin/update_redlines_analytics.sh --dry-run # export only
79+
~/.local/bin/update_redlines_analytics.sh --no-push # commit but don't push
80+
```
81+
82+
The wrapper:
83+
84+
1. `git pull` the local clone at `/home/stephan/src/redlines-dashboard`
85+
2. Loads DB creds from `/mnt/g/My Drive/SYSTEM_CREDENTIALS.env` (the `PG_WARDATASETS_*` block — same host, port 5432; the export script then targets both the `redlines` and `war_datasets` DBs)
86+
3. Runs `scripts/export_analytics_data.py` (typically 3-6 min)
87+
4. Writes `public/data/last_refreshed_analytics.json` with the UTC timestamp + host + elapsed seconds
88+
5. Commits the changed `analytics_*.json` files + the stamp, pushes to both `origin` (hcss-utils) and `upstream` (sdspieg)
89+
6. Next VPS pipeline run (cron `0 4 * * *` UTC) picks them up and deploys — or trigger immediately:
90+
`ssh root@138.201.62.161 'bash /stratbase/apps/webapps/redlines-dashboard-pipeline/run_pipeline.sh'`
91+
92+
A small banner on the Causal Analytics tab reads `last_refreshed_analytics.json` and shows the freshness ("Causal Analytics JSONs last refreshed Thu, 28 May 2026 21:55 UTC (0.0 days ago)"); after 7 days the banner turns yellow with a "stale, please re-run" hint.
93+
94+
Tradeoff: the VPS pipeline is no longer fully self-contained — the analytics JSONs only refresh when the desktop runs the wrapper. For the current cadence (annotation passes are weekly at best) this is fine.
95+
7096
## Dashboard Tabs
7197

7298
1. **Overview** - Stat cards, funnel charts, slope chart (RRLS vs NTS rank comparison), by-source and by-database breakdowns

scripts/export_analytics_data.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
#!/usr/bin/env python3
2-
"""Export causal analytics data (Granger, VAR/IRF, LP, event study) to JSON."""
2+
"""Export causal analytics data (Granger, VAR/IRF, LP, event study) to JSON.
3+
4+
Runs LOCALLY on the WSL desktop, NOT in the VPS daily pipeline. The VPS does
5+
not have statsmodels + scipy installed; this script's outputs arrive in the
6+
repo via `git pull` from a desktop-side push. See README.md §"Local-compute
7+
analytics" and HANDOFF.md (2026-05-28 entry) for the architectural rationale.
8+
9+
Wrapper: ~/.local/bin/update_redlines_analytics.sh
10+
"""
311

412
import json
513
import os

src/components/Analytics.tsx

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,49 @@
1+
import { useEffect, useState } from 'react';
2+
3+
type Stamp = { refreshed_at_utc: string; source_host?: string; export_elapsed_s?: number };
4+
5+
function daysAgo(iso: string): number {
6+
return (Date.now() - new Date(iso).getTime()) / 86_400_000;
7+
}
8+
9+
function StalenessBanner({ stamp }: { stamp: Stamp | null }) {
10+
if (!stamp) return null;
11+
const days = daysAgo(stamp.refreshed_at_utc);
12+
const stale = days > 7;
13+
return (
14+
<div style={{
15+
maxWidth: '800px',
16+
margin: '0 auto 1rem',
17+
padding: '0.75rem 1rem',
18+
fontSize: '0.85rem',
19+
color: stale ? '#ffd166' : '#9ec5fe',
20+
background: stale ? 'rgba(255, 209, 102, 0.08)' : 'rgba(158, 197, 254, 0.05)',
21+
border: `1px solid ${stale ? 'rgba(255, 209, 102, 0.3)' : 'rgba(158, 197, 254, 0.2)'}`,
22+
borderRadius: '6px',
23+
textAlign: 'center',
24+
}}>
25+
{stale && '⚠ '}
26+
Causal Analytics JSONs last refreshed{' '}
27+
<strong>{new Date(stamp.refreshed_at_utc).toUTCString().replace(/:\d{2} GMT$/, ' UTC')}</strong>
28+
{' '}({days.toFixed(1)} days ago{stale ? ' — stale, please re-run' : ''})
29+
</div>
30+
);
31+
}
32+
133
export default function Analytics() {
34+
const [stamp, setStamp] = useState<Stamp | null>(null);
35+
36+
useEffect(() => {
37+
const base = import.meta.env.BASE_URL ?? '/';
38+
fetch(`${base}data/last_refreshed_analytics.json`)
39+
.then(r => (r.ok ? r.json() : null))
40+
.then(setStamp)
41+
.catch(() => setStamp(null));
42+
}, []);
43+
244
return (
345
<div className="tab-content">
46+
<StalenessBanner stamp={stamp} />
447
<div style={{
548
maxWidth: '800px',
649
margin: '4rem auto',
@@ -82,4 +125,4 @@ export default function Analytics() {
82125
</div>
83126
</div>
84127
);
85-
}
128+
}

0 commit comments

Comments
 (0)