revert(history): drop the esp_tsdb fork pin — it changed nothing

Measured on the rig, before and after the single-fsync patch:

  tsdb_write ok (sys 5135 ms, panels 15854 ms)   # 2.3.0,   8 fsyncs
  tsdb_write ok (sys 5349 ms, panels 17269 ms)   # patched, 4 fsyncs

No improvement. The reasoning was wrong: removing the first fsync doesn't
remove flash work, it defers it to the trailing lfs_file_sync, which then
flushes the block and the header together. Same erases, same COW churn.
fsync cost tracks outstanding dirty work, not call count — the LD_PRELOAD
counter measured the wrong thing.

So the pin bought nothing and cost fork maintenance. Back to the registry
in the board YAML, the Config Builder and the deploy config. Upstream PR
retracted.

Also strips every claim built on that model: the "twice as fast" CHANGELOG
entry (deleted), "~10.5 s" in the header comment, "~10 s to commit" in the
build warning, and the docs' commit-duration figures — all back to the
measured ~21 s.

The 21 s is now explicitly documented as NOT understood, with both dead
theories named so the next person doesn't re-derive them: it is not
proportional to file size (1 MB and 192 KB DBs cost the same), and it is
not fsync count (halving it changed nothing). ~5.3 s per database, per
commit, cause unknown. Docs carry a caution box saying so.

Claude-Session: https://claude.ai/code/session_01RgSnMCa3JQigphGnPbazdw

Author: RAR

CHANGELOG.md

@@ -22,8 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 - **Updating the firmware could crash the device.** The history writer is meant to stand down while an update is being written, since both are writing the same flash chip — but it only checked for an update at the moment it started a snapshot, and a snapshot takes ten to twenty seconds to write. An update beginning during one was ignored until it was too late, and the two collided. Starting an update now waits for any snapshot in progress to finish before transferring, so the collision can't happen. If you update at the wrong moment you'll see a short pause and a "waited N ms" message before the transfer begins; that's the wait working. An update is refused outright if a snapshot somehow doesn't finish within 45 seconds, rather than proceeding into a known crash.
 
-- **Saving history is twice as fast.** Each snapshot writes four databases, and every write was syncing to flash twice — once for the data, once for the record header — when the second sync already covered both. On LittleFS a sync is a full journal commit costing seconds, and that cost turned out to be per sync rather than per byte: a 1 MB database and a 192 KB one took the same time, whether appending or overwriting. Dropping the redundant sync takes a snapshot from about 21 seconds to about 10. That matters beyond the writing itself, because the device holds its history lock for the whole commit — so it's also how long a chart request could be stuck waiting. Sent upstream as [zakery292/esp_tsdb#5](https://github.com/zakery292/esp_tsdb/pull/5); until it lands in a release the firmware builds against a pinned fork.
-- **History pages no longer fail when a snapshot is being written.** Requests gave up after 15 seconds, which was less than a snapshot took, so any chart loaded at the wrong moment returned an error. They now wait up to 30 seconds and get their data.
+- **History pages no longer fail when a snapshot is being written.** Requests gave up after 15 seconds, which is less than a snapshot takes, so any chart loaded at the wrong moment returned an error. They now wait up to 30 seconds and get their data.
 
 - **The flash-write crash appears to be fixed, not just made rarer.** The device now executes from PSRAM (`execute_from_psram`), and firmware configures this itself — you don't have to know it exists. Every history commit takes an ESP-IDF lock that disables the instruction cache across both cores; with code running from flash, that stall raced the WiFi/BLE radio ISRs and faulted. ESP-IDF skips the cache-disable entirely when instructions and read-only data live in PSRAM instead, which removes the race rather than shrinking its window. Previous builds survived 13.5, 42.0 and 14.0 hours before faulting; the current one has run **142 hours and counting** with no movement in any memory watermark, and with the per-panel databases already at full size — so it has been running at the worst-case flash cost the whole time, not easing into it. Costs about 1.7 MB of the 8 MB PSRAM, which moves out of the heap to hold the relocated code. Applies to ESP32-S3 boards with PSRAM configured; the reference config and the Config Builder set the flag explicitly as well. Note that the 142-hour run was at the old hourly cadence, and the new default writes twice as often — so the fix now carries more load than the run that demonstrated it.
 

boards/esp32s3-atoms3r.yaml

@@ -33,20 +33,10 @@ esp32:
     components:
       # Time-series database for persistent history
       # (see https://rar.github.io/esphome-tigomonitor/guides/tsdb-integration/).
-      # 2.1.0 was the first upstream release with the handle-based multi-instance
+      # 2.1.0 is the first upstream release with the handle-based multi-instance
       # API and the wrapped-ring query fix this project depends on; the caret
-      # resolved to 2.3.0.
-      #
-      # TEMPORARILY PINNED TO A FORK. v2.3.0 fsyncs twice per record write — once
-      # for the data block, once for the header — and on LittleFS each fsync is a
-      # full journal commit costing ~2.6 s regardless of file size. Four databases
-      # per snapshot made that a ~21 s stall, long enough to block every history
-      # query behind it. The branch drops the redundant first fsync (8 fsyncs per
-      # snapshot -> 4). Upstream PR: zakery292/esp_tsdb#5 — revert this to
-      # `zakery292/esp_tsdb^2.4.0` once it lands in a release.
-      - name: zakery292/esp_tsdb
-        source: https://github.com/RAR/esp_tsdb.git
-        ref: perf/single-fsync-per-write
+      # resolves to 2.3.0.
+      - zakery292/esp_tsdb^2.1.0
       # LittleFS — backing filesystem for the tsdb partition
       - joltwallet/littlefs^1.16
     sdkconfig_options:

components/tigo_monitor/__init__.py

@@ -38,8 +38,7 @@ def _warn_history_wear(config):
     """Flag intervals that buy resolution with flash life.
 
     Each snapshot commits four databases, and every commit stalls the writer for
-    seconds while holding the filesystem lock — measured at ~21 s on upstream
-    esp_tsdb 2.3.0, ~10.5 s with the redundant fsync removed. That cost is per
+    seconds while holding the filesystem lock — measured at ~21 s. That cost is per
     commit, so a shorter interval spends proportionally more of the device's
     time in it, and flash wear rises the same way. Runs as a validator rather
     than in to_code so `esphome config` surfaces it too.
@@ -48,7 +47,7 @@ def _warn_history_wear(config):
     if minutes < 15:
         _LOGGER.warning(
             "history_interval is %d min, %.1fx more often than the 30 min "
-            "default. Each snapshot takes ~10 s to commit and holds the "
+            "default. Each snapshot takes ~21 s to commit and holds the "
             "filesystem for all of it, so history pages queue behind it that "
             "much more often, and flash wear rises by the same factor. "
             "Per-panel history also shrinks to ~%d days.",

components/tigo_monitor/tigo_history.h

@@ -46,17 +46,20 @@ namespace tigo_monitor {
 //
 // Two costs still scale with it, and neither was fixed by that flag:
 //
-//   * A commit stalls the writer for seconds and holds the flash lock the whole
-//     time, so history queries queue behind it. Measured at ~21 s (sys 5135 ms
-//     + panels 15854 ms) against upstream esp_tsdb 2.3.0; ~10.5 s once the
-//     redundant fsync is gone (zakery292/esp_tsdb#5, currently pinned as a
-//     fork). The cost is per fsync and per database — NOT per byte: a 1 MB DB
-//     and a 192 KB DB measured the same, appending or evicting.
+//   * A commit stalls the writer ~21 s and holds the flash lock the whole time,
+//     so history queries queue behind it (sys 5349 ms + panels 17269 ms,
+//     esp_tsdb 2.3.0, panel rings full).
 //   * Flash wear scales with 1/interval, since the write volume per commit is
 //     roughly fixed. The absolute figure has not been measured; don't quote one.
 //
-// The floor is 4 fsyncs per snapshot (one per open database), so going below
-// ~10 s needs either fewer databases or syncing every Nth snapshot.
+// The 21 s is NOT understood, and two plausible explanations have been measured
+// and killed. It is not proportional to bytes: a 1 MB DB and a 192 KB DB cost
+// the same, appending or evicting. It is not the fsync count either: halving
+// fsyncs per write (8 -> 4 per snapshot) changed nothing, because the removed
+// sync's work simply moved to the remaining one. Whatever dominates is per
+// commit and per database, ~5.3 s each. Anyone shortening the interval should
+// know this cost is currently a fixed, unexplained tax — measure before
+// theorising, and don't trust a third guess without numbers.
 //
 // Retention scales with the interval too, since each DB holds a fixed record
 // count: the 5404-record panel rings span ~112 days at 30 min, ~37.5 at 10;

site/boards.js

@@ -10,13 +10,8 @@ export const BOARDS = [
     // out ESP-IDF's cross-core cache-disable, which was racing WiFi/BLE-coex
     // ISRs on every history commit (MTTF 13.5-42 h). Costs ~1.7 MiB of PSRAM.
     frameworkAdvanced: { enable_idf_experimental_features: false, execute_from_psram: true },
-    frameworkComponents: ['joltwallet/littlefs^1.16'],
-    // Temporarily a fork pin: upstream v2.3.0 fsyncs twice per record write, and
-    // on LittleFS each fsync is a ~2.6 s journal commit — four DBs per snapshot
-    // made a ~21 s stall that blocked every history query behind it. Branch drops
-    // the redundant one. Revert to 'zakery292/esp_tsdb^2.4.0' in
-    // frameworkComponents once zakery292/esp_tsdb#5 ships.
-    hostedComponent: { source: 'https://github.com/RAR/esp_tsdb.git', ref: 'perf/single-fsync-per-write' },
+    frameworkComponents: ['zakery292/esp_tsdb^2.1.0', 'joltwallet/littlefs^1.16'],
+    hostedComponent: null,
     sdkconfig: {
       CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240: 'y',
       CONFIG_UART_ISR_IN_IRAM: 'y',

site/src/content/docs/guides/tsdb-integration.md

@@ -80,23 +80,29 @@ esp32:
 `history_interval` accepts 5 to 1440 minutes and defaults to 30. Everything
 scales linearly with it, in both directions:
 
-| Interval | Per-panel history | System history | Device time spent committing | Chart resolution |
-|----------|-------------------|----------------|------------------------------|------------------|
-| 10 min | ~5 weeks | ~7 months | ~1.8% | finest |
-| **30 min (default)** | **~4 months** | **~2 years** | **~0.6%** | balanced |
-| 60 min | ~7.5 months | ~3.7 years | ~0.3% | coarse |
+| Interval | Per-panel history | System history | Device time spent writing | Chart resolution |
+|----------|-------------------|----------------|---------------------------|------------------|
+| 10 min | ~5 weeks | ~7 months | ~3.5% | finest |
+| **30 min (default)** | **~4 months** | **~2 years** | **~1.2%** | balanced |
+| 60 min | ~7.5 months | ~3.7 years | ~0.6% | coarse |
 
 The low end costs more than the resolution alone suggests. Writing a snapshot
-takes about 10 seconds — four databases, each needing a filesystem sync that
-LittleFS implements as a full journal commit — and the device holds the history
-lock for all of it, so chart requests arriving mid-commit have to wait it out
-(they allow 30 s before giving up). Flash wear rises by the same factor. Values
-under 15 minutes log a build-time warning.
-
-That cost is per commit rather than per byte: a 1 MB database and a 192 KB one
-were measured taking the same time, whether appending or overwriting. So it
-scales with how *often* you write, not how much you store — which is exactly why
-the interval is the lever that matters.
+takes about **21 seconds**, and the device holds its history lock for all of it,
+so chart requests arriving mid-write have to wait it out (they allow 30 s before
+giving up). Flash wear rises by the same factor. Values under 15 minutes log a
+build-time warning.
+
+That 21 seconds is a fixed cost per snapshot, not proportional to how much data
+you store — a 1 MB database and a 192 KB one measure the same. So the interval
+controls how much of the device's time goes into writing history, and it's the
+only lever that does.
+
+:::caution[Known limitation]
+21 seconds is slower than it should be and the cause isn't yet understood. Two
+likely explanations were tested and ruled out: it doesn't scale with file size,
+and it isn't the number of filesystem syncs. Until that's solved, treat short
+intervals as genuinely expensive rather than merely finer.
+:::
 
 Changing the interval later is safe: `period_e_*` stores energy since the
 previous snapshot rather than a running total, so lifetime figures stay correct