perf(history): pin the sidecar-header esp_tsdb — 21.4 s commits to 0.63 s

Upstream esp_tsdb rewrites the database header in place at offset 0 on
every commit. LittleFS stores a file as a CTZ skip-list of block
*addresses*, so modifying byte N forces every block after N to be
rewritten — overwrite cost is linear at ~20.4 ms/KB (measured across a
190x size range; appends stay flat at ~170 ms whatever the size).
Rewriting offset 0 is therefore the most expensive write the filesystem
offers, and it grew as the databases filled.

That accounted for 15.2 s of a 21.4 s commit — 4 databases x ~3.8 s —
with the remaining ~6.2 s in wrapped-ring block overwrites.

The fork writes the header to an alternating sidecar (<db>.h0/.h1) so the
hot path appends instead of rewriting. Rig, 132 commits: median 633 ms,
max 1,019 ms, no upward drift as the databases fill. It also adds
tsdb_peek_span, which the rolling-files work would need.

Pinned by SHA, not branch: a branch can move under a build and this is
the config people copy. The Config Builder mirrors it, so generated
configs match the reference (same shape the P4 board already uses).

Knock-on corrections, all previously stating 21 s as a fact of life:
- the guide's cost table (~1.2% of device time at the default -> ~0.04%)
  and its "cause isn't yet understood" caution, which is now answered
- the build-time warning under 15 min: the reason to think twice is now
  retention, not a multi-second stall
- the 5-minute floor's rationale — a duty-cycle guard at 21 s, a
  retention guard at 0.65 s (~19 days of panel history)

Verified: reference config compiles with the pin, and the fetched
dependency is confirmed to be the fork (sidecar present in tsdb_core.c,
tsdb_peek_span exported) rather than a silent registry fallback. Full
esp_tsdb host suite passes on the fork — sidecar, multi, resize, schema,
freecap, wrap — so it does not regress existing behaviour. 46/46 site
tests; docs build clean.

Not upstreamed yet; that stays a separate decision.

Author: RAR

CHANGELOG.md

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Docs screenshots generate themselves.** The guides now show the real device UI — dashboard, history, topology, node table, diagnostics — rendered during the docs build by driving the actual `app.html` against synthetic API responses. No firmware, no hardware, and nothing committed: the images are rebuilt from the current UI every time, so they cannot drift from it. All data is invented, with IPs and MACs from the IETF documentation ranges and a reserved fake-serial prefix, enforced by a test so a real serial, SSID or address can't reach a published page.
 
 ### Changed
+- **Writing history got about 30× faster — 21 seconds down to under one.** Each snapshot used to lock the filesystem for roughly 21 seconds, which is why chart requests could arrive mid-write and wait, and why the snapshot interval kept being coarsened. The cause turned out to be the storage library rewriting its database header at the very start of the file on every commit: LittleFS stores a file as a chain of block *addresses*, so changing the first bytes forces every later block to be rewritten too — the most expensive write the filesystem offers, and it got worse as the databases filled. The header now goes to a small companion file that is replaced whole instead, so the main file is only ever appended to. Measured on the reference rig across 132 snapshots: median 0.63 s, worst 1.02 s, and flat as the databases fill rather than creeping up. This needs a forked build of `esp_tsdb`, which the reference config and the Config Builder now pin by exact commit; the change has been offered to nobody yet, so it is a fork rather than a version bump. Short intervals are still a trade-off, but the reason is retention now, not write cost.
 - **History resolution is now yours to set, and twice as fine by default.** The snapshot interval had been coarsened twice — 5 to 30 minutes, then 30 to 60 — not because anyone wanted less detail, but because each snapshot's flash writes were what triggered the crash below, so the only lever available was doing it less often. With the underlying fault addressed it becomes a normal trade-off, so it's now a setting: `history_interval`, 5 to 1440 minutes, defaulting to 30. Lower is not free — each database holds a fixed number of readings however often you fill it, so twice the detail is half the span, and each snapshot costs the same few seconds to write regardless of how little changed. At the 30-minute default you get about 4 months of per-panel history and 2 years system-wide; at 10 minutes, about 5 weeks and 7 months, with three times the write load. Anything under 15 minutes says so at build time. Retuning later is safe — each row stores energy since the previous snapshot rather than a running total, so lifetime figures stay correct and existing history isn't invalidated.
 - **Docs rewritten for first-time installers.** A new **Start Here** guide covers what to buy, what the jargon means, and the five steps from parts on the desk to a live dashboard — the "Get Started" button used to drop you into a high-voltage warning and a terminal-block diagram. The sidebar is now ordered by when you need something rather than alphabetically, the Config Builder explains what each field is for, and the reference-heavy pages say up front whether you need to read them. No behaviour changed.
 - **Docs site recoloured to the brand mark.** Link and accent colours moved from green to the mark's silicon indigo and busbar silver; amber keeps its one meaning — this part is live. The blueprint grid behind the headline is now a PV array receding toward the horizon — cells, busbars and module frames, drawn in CSS gradients so it stays sharp at any size and re-tints with the theme.

boards/esp32s3-atoms3r.yaml

@@ -33,10 +33,24 @@ esp32:
     components:
       # Time-series database for persistent history
       # (see https://rar.github.io/esphome-tigomonitor/guides/tsdb-integration/).
-      # 2.1.0 is the first upstream release with the handle-based multi-instance
-      # API and the wrapped-ring query fix this project depends on; the caret
-      # resolves to 2.3.0.
-      - zakery292/esp_tsdb^2.1.0
+      #
+      # Pinned to a fork, not the registry's 2.3.0, for the sidecar header.
+      # Upstream writes the database header in place at offset 0 on every
+      # commit. LittleFS stores a file as a skip-list of block addresses, so
+      # modifying byte N rewrites every block after N — overwrite cost is linear
+      # at ~20.4 ms/KB, measured across a 190x size range. That made a commit
+      # 21.4 s (15.2 s of it header rewrites) and growing as the databases fill.
+      #
+      # The fork writes the header to an alternating sidecar (<db>.h0/.h1), so
+      # the hot path appends instead of rewriting: 21.4 s -> 633 ms median,
+      # 1,019 ms max over 132 commits on the rig, with no drift as they fill.
+      # It also adds tsdb_peek_span. Neither is upstream yet.
+      #
+      # An immutable SHA rather than a branch: a branch can move under a build,
+      # and this is the reference config people copy.
+      - name: zakery292/esp_tsdb
+        source: https://github.com/RAR/esp_tsdb.git
+        ref: 3fb785ffe0e280e7645a598f1cf82f6babe72143
       # LittleFS — backing filesystem for the tsdb partition
       - joltwallet/littlefs^1.16
     sdkconfig_options:

components/tigo_monitor/__init__.py

@@ -49,20 +49,20 @@
 def _warn_history_wear(config):
     """Flag intervals that buy resolution with flash life.
 
-    Each snapshot commits four databases, and every commit stalls the writer for
-    seconds while holding the filesystem lock — measured at ~21 s. That cost is per
-    commit, so a shorter interval spends proportionally more of the device's
-    time in it, and flash wear rises the same way. Runs as a validator rather
-    than in to_code so `esphome config` surfaces it too.
+    Each snapshot commits four databases while holding the filesystem lock —
+    ~0.65 s measured with the sidecar-header esp_tsdb the reference config pins
+    (it was ~21 s before that). The dominant cost of a short interval is now
+    retention rather than write time: each database holds a fixed row count, so
+    twice the detail is half the span. Flash wear still scales with frequency.
+    Runs as a validator rather than in to_code so `esphome config` surfaces it.
     """
     minutes = config[CONF_HISTORY_INTERVAL]
     if minutes < 15:
         _LOGGER.warning(
             "history_interval is %d min, %.1fx more often than the 30 min "
-            "default. Each snapshot takes ~21 s to commit and holds the "
-            "filesystem for all of it, so history pages queue behind it that "
-            "much more often, and flash wear rises by the same factor. "
-            "Per-panel history also shrinks to ~%d days.",
+            "default. Per-panel history shrinks to ~%d days, and flash wear "
+            "rises by the same factor. (Each snapshot itself is cheap — ~0.65 s "
+            "— so the cost here is span and wear, not stalling.)",
             minutes,
             30 / minutes,
             round(5404 * minutes / 1440),
@@ -83,8 +83,9 @@ def _warn_history_wear(config):
     cv.Optional(CONF_NIGHT_MODE_TIMEOUT, default=60): cv.int_range(min=1, max=1440),  # 1 minute to 24 hours
     cv.Optional(CONF_STALE_TIMEOUT, default=10): cv.int_range(min=0, max=1440),  # minutes; 0 disables staleness zeroing
     # How often on-flash history is written. Bounds mirror kMin/kMaxSnapshotIntervalMin
-    # in tigo_history.h. The floor is not arbitrary: a commit holds the flash lock
-    # ~21 s with the panel rings full, so 5 min is already ~7% duty cycle.
+    # in tigo_history.h. The 5-minute floor predates the sidecar header (a commit
+    # then held the flash lock ~21 s, making 5 min a ~7% duty cycle); at ~0.65 s
+    # it is now a retention guard — 5 min leaves only ~19 days of panel history.
     cv.Optional(CONF_HISTORY_INTERVAL, default=30): cv.int_range(min=5, max=1440),
 }).extend(cv.polling_component_schema('30s')).extend(uart.UART_DEVICE_SCHEMA), _warn_history_wear)
 

site/boards.js

@@ -10,8 +10,16 @@ export const BOARDS = [
     // out ESP-IDF's cross-core cache-disable, which was racing WiFi/BLE-coex
     // ISRs on every history commit (MTTF 13.5-42 h). Costs ~1.7 MiB of PSRAM.
     frameworkAdvanced: { enable_idf_experimental_features: false, execute_from_psram: true },
-    frameworkComponents: ['zakery292/esp_tsdb^2.1.0', 'joltwallet/littlefs^1.16'],
-    hostedComponent: null,
+    // Fork-pinned for the sidecar header: upstream rewrites the db header in
+    // place at offset 0 every commit, which LittleFS turns into a rewrite of
+    // every block to EOF (~20.4 ms/KB). 21.4 s -> 633 ms median on the rig.
+    // Mirrors boards/esp32s3-atoms3r.yaml; SHA, not a branch, so it cannot move
+    // under a build.
+    frameworkComponents: ['joltwallet/littlefs^1.16'],
+    hostedComponent: {
+      source: 'https://github.com/RAR/esp_tsdb.git',
+      ref: '3fb785ffe0e280e7645a598f1cf82f6babe72143',
+    },
     sdkconfig: {
       CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240: 'y',
       CONFIG_UART_ISR_IN_IRAM: 'y',

site/src/content/docs/guides/tsdb-integration.md

@@ -82,32 +82,37 @@ scales linearly with it, in both directions:
 
 | Interval | Per-panel history | System history | Device time spent writing | Chart resolution |
 |----------|-------------------|----------------|---------------------------|------------------|
-| 10 min | ~5 weeks | ~7 months | ~3.5% | finest |
-| **30 min (default)** | **~4 months** | **~2 years** | **~1.2%** | balanced |
-| 60 min | ~7.5 months | ~3.7 years | ~0.6% | coarse |
-
-The low end costs more than the resolution alone suggests. Writing a snapshot
-takes about **21 seconds**, and the device holds its history lock for all of it,
-so chart requests arriving mid-write have to wait it out (they allow 30 s before
-giving up). Flash wear rises by the same factor. Values under 15 minutes log a
-build-time warning.
-
-That 21 seconds is a fixed cost per snapshot, not proportional to how much data
-you store — a 1 MB database and a 192 KB one measure the same. So the interval
-controls how much of the device's time goes into writing history, and it's the
-only lever that does.
-
-:::caution[Known limitation]
-21 seconds is slower than it should be and the cause isn't yet understood. Two
-likely explanations were tested and ruled out: it doesn't scale with file size,
-and it isn't the number of filesystem syncs. Until that's solved, treat short
-intervals as genuinely expensive rather than merely finer.
-:::
+| 10 min | ~5 weeks | ~7 months | ~0.1% | finest |
+| **30 min (default)** | **~4 months** | **~2 years** | **~0.04%** | balanced |
+| 60 min | ~7.5 months | ~3.7 years | ~0.02% | coarse |
+
+Writing a snapshot takes about **0.65 seconds**, and the device holds its history
+lock for all of it, so a chart request arriving mid-write waits that long. Flash
+wear still rises as the interval shortens. Values under 15 minutes log a
+build-time warning — retention, not write cost, is now the reason to think twice.
+
+The main cost of a short interval is span: each database holds a fixed number of
+rows however often you fill it, so twice the detail is half the history.
 
 Changing the interval later is safe: `period_e_*` stores energy since the
 previous snapshot rather than a running total, so lifetime figures stay correct
 across a change and existing history is not invalidated.
 
+:::note[Why this needs a forked esp_tsdb]
+A snapshot used to take **21 seconds**, and the reference config pins a fork of
+`esp_tsdb` to avoid it. Upstream writes the database header in place at offset 0
+on every commit. LittleFS stores a file as a skip-list of block *addresses*, so
+changing byte N forces every block after N to be rewritten — overwrite cost is
+linear at about 20.4 ms/KB, while appends stay flat at ~170 ms whatever the size.
+Rewriting the header at offset 0 is therefore the most expensive write the
+filesystem offers, and it accounted for 15.2 s of the 21 s.
+
+The fork writes the header to an alternating sidecar file (`<db>.h0` / `<db>.h1`)
+so the hot path appends instead. Measured on the reference rig over 132 commits:
+median 633 ms, maximum 1,019 ms, with no upward drift as the databases fill. The
+change is not upstream yet, so the board config pins the fork by commit SHA.
+:::
+
 > **Board note:** the `board:` value above (`m5stack-atoms3`) is an example. The reference rig for this project is the **AtomS3R** — set `board:` to whatever board you actually run so you don't flash the wrong target.
 
 The TSDB code is conditionally compiled — without those two dependencies on the include path, `tigo_history.h` short-circuits and the History / TSDB-stats endpoints don't exist. You can run the rest of the component without TSDB; you just lose persistent history.