fix(tsdb): stop the /api/tsdb/stats flash storm; soak evidence for execute_from_psram (#42)

* fix(server): stop /api/tsdb/stats generating a flash storm

Building that response is the only flash access anywhere on the HTTP
path, and on the S3 flash access is what crashes us: every esp_flash op
runs spi_flash_disable_interrupts_caches_and_other_cpu, which stalls the
other core. FlashLock stops httpd and the history writer being inside
flash simultaneously, but nothing stopped them ALTERNATING, and a burst
of cache-disables from both cores faults the writer inside IDF's own
cross-core stall coordination.

Reproduced on the rig, same build, 4 concurrent loops each:
  4x /api/tsdb/stats -> crash in ~11s (writer faults in lfs_file_flush)
  4x /api/status     -> survives, at a HIGHER request rate
/api/status touches no flash. That is the entire difference, which is
why more locking would not have helped.

So stop generating the traffic: build the body at most once per minute,
serve it from PSRAM in between, and have the Diagnostics view fetch it
on open instead of on every 10s tick. Everything in the response only
moves when the writer commits.

Uses psram_string for the cache, not PSRAMString — the latter owns a raw
pointer with no copy constructor, so caching one and copying it out
would double-free.

Does NOT fix the underlying fault, which still reboots the device at
roughly its documented rate. It removes the way the UI provoked it.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* fix(server): serve /api/tsdb/stats from RAM, never from flash

The previous commit's 60s response cache was not enough. It survived the
kill-load on an idle device for 4 min, then died at 13:29:34 — the exact
instant of a writer commit — when a cache rebuild landed on top of it.
Caching made the collision rarer, not impossible.

Both sides take FlashLock and they still collide; why, is not yet
understood. So stop relying on the lock: remove one of the two parties.
TigoHistory now keeps a StatsSnapshot in RAM, refreshed by the writer
task from inside its own flash batch (after commit_journal_, on slot
assignment, and once at init). The HTTP handler copies that under a
plain std::mutex and formats it. /api/tsdb/stats now touches no flash at
any request rate, which makes it exactly as safe as /api/status — the
endpoint we proved survives unlimited hammering.

Trade: the figures are as fresh as the last commit rather than live. The
response carries snapshot_age_ms and the page renders 'sampled Nm ago'
instead of implying they're current.

Fixtures updated for the new field per CLAUDE.md; screenshots regenerated.

Still does NOT fix the underlying flash-vs-cache fault, which reboots the
device roughly daily on its own.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* perf(history): halve flash exposure — snapshot every 60 min, not 30

Every snapshot writes 4 TSDB databases (8 fsync'd writes: a data block
and a header rewrite each) plus a journal-commit marker file. Each of
those forces spi_flash_disable_interrupts_caches_and_other_cpu, which
cuts the I-cache on both cores and stalls the other one — the window the
'Fault - Unknown' crash happens in. Exposure scales with how often that
fires, so halving the cadence roughly halves the windows.

Cost: hourly rather than half-hourly panel resolution. Retention goes UP
(~225 days per panel, from ~112). period_e_kwh is energy-since-last-
snapshot so lifetime totals are unaffected by the change.

Cadence now lives in one constant, tigo_history.h:kSnapshotIntervalMin,
with the reasoning attached and the log line derived from it — the last
change to this number left five comments restating the old one, one of
which still said '5-min' after the move to 30. Those are now gone.

This is a probability reduction, not a fix, and unlike the last two
changes it cannot be verified with the 11s reproducer — the background
crash needs a multi-day soak to show any rate change, against samples
spanning 13.5-42h. Do not claim an improvement from one clean run.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* chore(history): log commit duration at INFO — it IS the crash exposure

Every mid-file write makes LittleFS copy from the write offset to EOF a
byte at a time (lfs_file_flush, lfs.c:3376), and esp_tsdb rewrites the
header at offset 0 on every record — so each commit drags most of each
file through that loop. On this rig that is ~850 KB per commit (system
262 KB + 3x198 KB panels). Both decoded crash PCs (lfs.c:3380 and
lfs.c:3598) are inside that loop, which is simply where the writer
spends nearly all its time.

The writer already measured this, at DEBUG, while the deploy config runs
at INFO — so the single most diagnostic number the component produces
was invisible. One observed commit ran 81 s (11:17:28 tick, dead at
11:18:49) while we assumed commits were milliseconds.

No behaviour change; makes the next commit measurable.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* docs(issue): retract execute_from_psram from the ruled-out list

It was excluded on the grounds that the crash 'recurs without it' — which
only tests it as a CAUSE. Nobody ever enabled it and measured it as a
CURE, and it is the only candidate found that removes the mechanism
instead of shrinking the window.

IDF 5.5.5 compiles cache_disable out entirely when
CONFIG_SPIRAM_FETCH_INSTRUCTIONS && CONFIG_SPIRAM_RODATA are set, which
is exactly what this flag sets. Verified in the built sdkconfig.h, with
CONFIG_SPI_FLASH_SHARE_SPI1_BUS undefined so no earlier branch wins;
spi1_start then takes a plain mutex. That deletes three frames from the
14:25 backtrace by construction.

'Harmful on S3' was in the notes with nothing behind it. ESPHome's own
S3 test config sets this flag.

Enabled on the rig's deploy config (not in-repo). Soak result pending —
this records the reasoning, not a result.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* fix(ui): derive the chart resolution label instead of hardcoding it

The power chart said "Power · 5-minute resolution" while the firmware was
sampling hourly — 12x wrong. The cadence has moved twice (5 -> 30 min on
2026-07-23, 30 -> 60 min on 2026-07-28) and the label was never part of
either change, because nothing connected it to kSnapshotIntervalMin.

Both history endpoints now emit interval_min, and app.html renders the
label from it. Hardcoding "60-minute" would have gone stale again the
first time we walk the dial back after the flash-crash soak.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

* fix(esp32): ship execute_from_psram — the actual flash-crash fix

The soak build has run 142 h against prior MTTFs of 13.5/42.0/14.0 h, but
the flag doing the work lived only in the deploy YAML. Merging without
this would have shipped the mitigations and left the cure behind.

Three layers, deliberately redundant:

- codegen (tigo_monitor/__init__.py) sets CONFIG_SPIRAM_FETCH_INSTRUCTIONS
  and CONFIG_SPIRAM_RODATA directly, so existing users get the fix without
  editing anything. Gated on S3 + PSRAM, matching how ESPHome gates its own
  execute_from_psram option (esp32/__init__.py:544). Verified against
  test-local-tigomonitor.yaml, which sets no such flag: both symbols land in
  the built sdkconfig.h, and neither SPI_FLASH_AUTO_SUSPEND nor
  SPI_FLASH_SHARE_SPI1_BUS is defined, so the SPIRAM pair alone satisfies
  SPI_FLASH_CACHE_NO_DISABLE.
- boards/esp32s3-atoms3r.yaml sets the flag explicitly, so the reference
  config states the requirement rather than inheriting it silently.
- site/boards.js does the same for generated configs (drift test pairs the
  two, so they cannot diverge).

Costs ~1.7 MiB of PSRAM, out of the heap, holding relocated instructions
and rodata.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

Author: RAR

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Docs screenshots generate themselves.** The guides now show the real device UI — dashboard, history, topology, node table, diagnostics — rendered during the docs build by driving the actual `app.html` against synthetic API responses. No firmware, no hardware, and nothing committed: the images are rebuilt from the current UI every time, so they cannot drift from it. All data is invented, with IPs and MACs from the IETF documentation ranges and a reserved fake-serial prefix, enforced by a test so a real serial, SSID or address can't reach a published page.
 
 ### Changed
+- **History snapshots every 60 minutes instead of 30.** Each snapshot writes four databases and forces a LittleFS journal commit, and every one of those flash writes opens a window where the ESP32-S3 disables its instruction cache and stalls the other core — the window in which this device's long-standing `Fault - Unknown` crash happens. Halving the cadence roughly halves the number of windows. Per-panel history now spans ~225 days rather than ~112, at hourly resolution rather than half-hourly. Lifetime energy totals are unaffected: each row stores energy since the previous snapshot, not a running total. This is a probability reduction, not a fix — the underlying fault is still there.
 - **Docs rewritten for first-time installers.** A new **Start Here** guide covers what to buy, what the jargon means, and the five steps from parts on the desk to a live dashboard — the "Get Started" button used to drop you into a high-voltage warning and a terminal-block diagram. The sidebar is now ordered by when you need something rather than alphabetically, the Config Builder explains what each field is for, and the reference-heavy pages say up front whether you need to read them. No behaviour changed.
 - **Docs site recoloured to the brand mark.** Link and accent colours moved from green to the mark's silicon indigo and busbar silver; amber keeps its one meaning — this part is live. The blueprint grid behind the headline is now a PV array receding toward the horizon — cells, busbars and module frames, drawn in CSS gradients so it stays sharp at any size and re-tints with the theme.
 - **New brand mark.** The green-to-blue tile is replaced by an optimizer emitting a telemetry frame, in silicon indigo and amber. It's drawn at three sizes rather than scaled — the device favicon, the device UI sidebar, and the docs — so it stays legible in a browser tab. The docs site had no logo or favicon configured before and now carries the same mark the device serves.
@@ -19,6 +20,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Boards without PSRAM are no longer offered.** The Config Builder listed the M5Stack AtomS3/AtomS3 Lite and a generic ESP32 DevKit, both of which lack PSRAM — so it could hand you a ready-to-flash YAML for a board the project doesn't support. Both are gone from the builder, along with their `boards/*.yaml` reference configs, and a test now blocks a PSRAM-less board from being added back.
 
 ### Fixed
+- **The flash-write crash appears to be fixed, not just made rarer.** The device now executes from PSRAM (`execute_from_psram`), and firmware configures this itself — you don't have to know it exists. Every history commit takes an ESP-IDF lock that disables the instruction cache across both cores; with code running from flash, that stall raced the WiFi/BLE radio ISRs and faulted. ESP-IDF skips the cache-disable entirely when instructions and read-only data live in PSRAM instead, which removes the race rather than shrinking its window. Previous builds survived 13.5, 42.0 and 14.0 hours before faulting; the current one has run **142 hours and counting** with no movement in any memory watermark, and with the per-panel databases already at full size — so it has been running at the worst-case flash cost the whole time, not easing into it. Costs about 1.7 MB of the 8 MB PSRAM, which moves out of the heap to hold the relocated code. Applies to ESP32-S3 boards with PSRAM configured; the reference config and the Config Builder set the flag explicitly as well.
+
+- **Opening the Diagnostics page could reboot the device.** The page asked for `/api/tsdb/stats` every 10 seconds, and building that response was the only thing in the whole web UI that made the device read flash. On the ESP32-S3 every flash access stalls the other core, and doing it from the web server while the history writer is mid-write faults the writer. Reproduced on demand: four concurrent callers killed the device in about eleven seconds, while four concurrent callers of `/api/status` — same load, no flash — ran indefinitely. The figures now come from a snapshot the history writer takes during its own commit, so serving them touches no flash at any request rate. They can be up to one snapshot interval old, and the page now says how old ("sampled 4m ago") rather than implying they're live. This does not fix the underlying flash-versus-cache fault, which can still reboot the device roughly once a day; it removes the way the UI was provoking it.
 - **Opening the web UI could reboot the device.** History pages read the TSDB from the web server's task while the 30-minute snapshot writes it from another — and on the ESP32-S3 every flash access, read included, cuts the instruction cache on both cores, so the two collide and fault (`Fault - Unknown`). All LittleFS access now goes through one lock. History requests can queue briefly (a month-range query holds the flash ~2.3 s) and return an error rather than hang if they wait more than 15 s.
 
 ## [2.0.0-beta.5] - 2026-07-25

boards/esp32s3-atoms3r.yaml

@@ -17,6 +17,19 @@ esp32:
   framework:
     type: esp-idf
     version: recommended
+    advanced:
+      # XIP-from-PSRAM — the fix for the tsdb flash-write crash
+      # (docs/tsdb-flash-crash-issue.md). Every history commit takes ESP-IDF's
+      # cross-core cache-disable; with code running from flash that stall raced
+      # WiFi/BLE-coex ISRs and faulted, MTTF 13.5-42 h. This flag sets
+      # SPIRAM_FETCH_INSTRUCTIONS + SPIRAM_RODATA, which compiles the
+      # cache-disable out entirely (IDF 5.5.5 spi_flash_os_func_app.c:29).
+      #
+      # tigo_monitor's codegen sets the same two sdkconfig symbols, so this line
+      # is belt-and-braces — it keeps the reference config honest about what the
+      # firmware actually requires. Costs ~1.7 MiB of PSRAM, taken out of the
+      # heap to hold the relocated instructions and rodata.
+      execute_from_psram: true
     components:
       # Time-series database for persistent history
       # (see https://rar.github.io/esphome-tigomonitor/guides/tsdb-integration/).

components/tigo_monitor/__init__.py

@@ -2,8 +2,10 @@
 import esphome.codegen as cg
 import esphome.config_validation as cv
 from esphome.components import uart, time as time_, esp32
+from esphome.components.esp32.const import VARIANT_ESP32S3
+from esphome.components.psram import DOMAIN as PSRAM_DOMAIN
 from esphome.const import CONF_ID, CONF_UART_ID, CONF_TIME_ID, CONF_NAME
-from esphome.core import coroutine
+from esphome.core import coroutine, CORE
 
 DEPENDENCIES = ['uart']
 
@@ -92,3 +94,26 @@ def to_code(config):
     if esp32.idf_version() >= cv.Version(6, 0, 0):
         esp32.add_idf_component(name="espressif/cjson", ref="^1.7.19")
 
+    # XIP-from-PSRAM. This is not a performance tuning knob — it is what keeps
+    # the history writer from bricking the device.
+    #
+    # Every tsdb commit takes ESP-IDF's cross-core cache-disable
+    # (spi_flash_disable_interrupts_caches_and_other_cpu). With code executing
+    # from flash, that stall raced WiFi/BLE-coex radio ISRs and produced the
+    # "Fault - Unknown" crashes in docs/tsdb-flash-crash-issue.md — mean
+    # time-to-failure measured at 13.5-42 h on an AtomS3R.
+    #
+    # IDF 5.5.5 spi_flash_os_func_app.c:29 compiles cache_disable out entirely
+    # when SPIRAM_FETCH_INSTRUCTIONS && SPIRAM_RODATA are both set, which is
+    # exactly the pair ESPHome's `execute_from_psram: true` sets. Setting them
+    # here means users get the fix without having to know it exists; a config
+    # that already sets the flag lands on the same two values, so there is no
+    # conflict.
+    #
+    # Gated the same way ESPHome gates its own option (esp32/__init__.py:544):
+    # S3-only, and PSRAM must be configured. It costs ~1.7 MiB of PSRAM, which
+    # moves out of the heap to hold relocated instructions and rodata.
+    if esp32.get_esp32_variant() == VARIANT_ESP32S3 and PSRAM_DOMAIN in CORE.config:
+        esp32.add_idf_sdkconfig_option("CONFIG_SPIRAM_FETCH_INSTRUCTIONS", True)
+        esp32.add_idf_sdkconfig_option("CONFIG_SPIRAM_RODATA", True)
+

components/tigo_monitor/tigo_history.cpp

@@ -59,7 +59,7 @@ static int16_t enc_kwh_(float kwh) { return enc_clamp_(kwh * 100.0f); }
 static int16_t enc_temp_(float c) { return enc_clamp_(c); }
 static int16_t enc_dhz_(float hz) { return enc_clamp_(hz * 10.0f); }
 
-// Schema for system.tsdb — system + per-inverter rollups, 5-min cadence.
+// Schema for system.tsdb — system + per-inverter rollups, one row per snapshot.
 // Order is fixed once data has been written; appending requires migration.
 // See https://rar.github.io/esphome-tigomonitor/guides/tsdb-integration/ for unit/scaling reference.
 static const char *kSystemParamNames[] = {
@@ -94,7 +94,7 @@ static const char *kPanelParamNames[kPanelsPerDb] = {
 static constexpr size_t kSystemFileBytes = 1 * 1024 * 1024;
 
 // 192 KB per panel DB × 3 = 576 KB. At 36 B/record (16×2-byte params + 4-byte
-// ts) that's ~5400 records per DB ≈ 18 days of 5-min data per panel.
+// ts) that's ~5400 records per DB — ~225 days at the current kSnapshotIntervalMin.
 static constexpr size_t kPanelFileBytes = 192 * 1024;
 
 static constexpr const char *kPanelMapPath = "/tsdb/panel_map.json";
@@ -131,6 +131,11 @@ bool TigoHistory::init() {
     next_free_slot_ = 0;
   }
   initialized_ = true;
+
+  // Populate the snapshot before anything can serve it, so /api/tsdb/stats
+  // has real numbers from boot rather than an empty table until the first
+  // scheduled commit. Still inside this function's FlashLock.
+  refresh_stats_snapshot_();
   return true;
 }
 
@@ -171,7 +176,7 @@ void TigoHistory::commit_journal_() {
   //
   // Rewriting a tiny marker file is a dir-entry metadata update, which forces that
   // commit — the same side effect save_slot_map_ relies on for panel_map.json.
-  // One small file per 5-min snapshot; LittleFS wear-levels the block.
+  // One small file per snapshot; LittleFS wear-levels the block.
   static uint32_t commit_counter = 0;
   FILE *f = fopen(kJournalMarkerPath, "wb");
   if (f == nullptr) {
@@ -195,7 +200,7 @@ bool TigoHistory::init_system_db_() {
   // PSRAM-backed buffer pool. The S3-PICO-1 has 8 MB octal PSRAM — internal
   // RAM is the constraint (we land at ~110 KB free at low water with the
   // SPA + tsdb stack). Buffer access is fast enough on octal PSRAM that
-  // tsdb_write latency stays well under the 5-min snapshot budget.
+  // tsdb_write latency stays well under the snapshot budget.
   cfg.alloc_strategy = TSDB_ALLOC_PSRAM;
   cfg.use_paged_allocation = false;
   cfg.page_size = 0;
@@ -381,6 +386,11 @@ uint8_t TigoHistory::get_or_assign_slot(const std::string &barcode_last6) {
   // of a barcode) and the cost of losing one is "panel rejoins as a new
   // slot, history splits in two", which we want to avoid.
   save_slot_map_();
+
+  // Slot count and a newly-opened panel DB both show up in /api/tsdb/stats.
+  // Refresh here (still under FlashLock, flash already hot) so the Diagnostics
+  // table tracks discovery instead of lagging until the next scheduled commit.
+  refresh_stats_snapshot_();
   return slot;
 }
 
@@ -402,7 +412,7 @@ bool TigoHistory::start_writer_task() {
   if (task_ != nullptr) {
     return true;  // already running
   }
-  // Queue depth 4 — at 5-min cadence we should never be more than 1 deep.
+  // Queue depth 4 — at the snapshot cadence we should never be more than 1 deep.
   // Extra headroom absorbs transient flash slowdowns without dropping samples.
   queue_ = xQueueCreate(4, sizeof(EncodedRow));
   if (queue_ == nullptr) {
@@ -567,6 +577,40 @@ int TigoHistory::iterate_panel(uint8_t slot, uint32_t start_ts, uint32_t end_ts,
   return count;
 }
 
+void TigoHistory::refresh_stats_snapshot_() {
+  // Build into a local first: tsdb_get_stats_h can block on the per-DB mutex
+  // for up to 5 s, and holding stats_mutex_ across that would make a reader
+  // wait on the writer for no reason. The published copy swaps in at the end.
+  StatsSnapshot snap;
+  snap.valid = true;
+  snap.updated_ms = (uint32_t) (esp_timer_get_time() / 1000);
+
+  size_t total = 0, used = 0;
+  if (esp_littlefs_info("tsdb", &total, &used) == ESP_OK) {
+    snap.fs_total = total;
+    snap.fs_used = used;
+  }
+  snap.slot_count = slot_map_.size();
+  snap.next_free_slot = next_free_slot_;
+
+  auto grab = [](tsdb_t *db, StatsSnapshot::Db &out) {
+    if (db == nullptr) return;  // lazily-unopened panel DB: no row at all
+    out.present = true;
+    out.error = tsdb_get_stats_h(db, &out.stats);
+    out.available = (out.error == ESP_OK);
+  };
+  grab(system_db_, snap.system);
+  for (size_t i = 0; i < kNumPanelDbs; ++i) grab(panel_db_[i], snap.panels[i]);
+
+  std::lock_guard<std::mutex> guard(stats_mutex_);
+  stats_snapshot_ = snap;
+}
+
+void TigoHistory::copy_stats_snapshot(StatsSnapshot &out) {
+  std::lock_guard<std::mutex> guard(stats_mutex_);
+  out = stats_snapshot_;
+}
+
 void TigoHistory::writer_task_entry_(void *arg) {
   static_cast<TigoHistory *>(arg)->writer_task_loop_();
 }
@@ -604,8 +648,8 @@ void TigoHistory::writer_task_loop_() {
     // between two writes is just as fatal as one landing mid-write.
     //
     // Deliberately scoped BELOW xQueueReceive: holding this across the
-    // portMAX_DELAY receive would block every reader for the full 30-minute
-    // gap between snapshots. It releases when the iteration ends.
+    // portMAX_DELAY receive would block every reader for the whole
+    // inter-snapshot gap. It releases when the iteration ends.
     //
     // portMAX_DELAY, not a timeout: dropping a snapshot loses data, and the
     // only holders are bounded (a query, a slot save).
@@ -638,13 +682,26 @@ void TigoHistory::writer_task_loop_() {
     // unmount (which on this rig isn't sticking).
     this->commit_journal_();
 
+    // Refresh the diagnostics snapshot here, while we are still holding
+    // FlashLock and the flash is already hot. This is the ONLY place the
+    // numbers behind /api/tsdb/stats are read from flash — doing it from the
+    // HTTP task is what crashed the device (see StatsSnapshot in the header).
+    this->refresh_stats_snapshot_();
+
     UBaseType_t hwm = uxTaskGetStackHighWaterMark(nullptr);
     if (err != ESP_OK) {
       ESP_LOGW(TAG, "tsdb_write @ %lu failed after %u ms: %s (stack hwm %u B)",
                (unsigned long) row.timestamp, (unsigned) t_sys,
                esp_err_to_name(err), (unsigned) (hwm * sizeof(StackType_t)));
     } else {
-      ESP_LOGD(TAG,
+      // INFO, not DEBUG: this duration is the single most useful number this
+      // component produces. Every mid-file write makes LittleFS copy from the
+      // write offset to EOF, a byte at a time (lfs.c:3376 in lfs_file_flush),
+      // and esp_tsdb rewrites the header at offset 0 on EVERY record — so a
+      // commit drags most of each file through that loop. Both decoded crash
+      // PCs sit inside it. How long this takes IS the crash exposure, and at
+      // the default INFO log level it was invisible.
+      ESP_LOGI(TAG,
                "tsdb_write @ %lu ok (sys %u ms, panels %u ms, stack hwm %u B)",
                (unsigned long) row.timestamp, (unsigned) t_sys,
                (unsigned) panel_total_ms,