Skip to content

Export (and later import) TSDB history — a repartition silently destroys it #47

Description

@RAR

The gap

There is no way to get history out of the device. /api/history/power and
/api/history/panel serve chart data, and the SPA renders it — but nothing
packages it for download, and nothing can put it back.

That matters because any partition-layout change destroys it. The tsdb
partition moves, LittleFS reformats, and the series is gone. This is not a rare
event:

  • upgrading from 1.x (stock layout → tigo-8mb.csv)
  • turning on cca_source: ble on an 8 MB board (→ tigo-8mb-ble.csv)
  • any future app-slot resize

The node table already has Export/Import (Nodes view → /api/nodes/import), and
it works well. History has no equivalent, so the same upgrade that preserves your
panel names silently discards months of production data.

Hit this on the reference rig on 2026-08-13: a 2.0 → 2.25 MB app-slot change
meant losing 8,493 system records (~47 days) and 16,520 panel records across 40
slots. They were recoverable only by scraping the four range= endpoints by
hand into JSON, which is not something a user would think to do — or know they
needed to, before it was too late.

Proposed

Export — a button in Diagnostics (next to the TSDB stats table) that
downloads the whole series as one JSON file: system, per-inverter, and every
occupied panel slot, with the slot→barcode map so it survives re-slotting.
Mirror the node-table export UX exactly; users already know that pattern.

Import — restore into an empty or partial database. Harder, and worth
splitting into its own issue if it complicates this one:

  • Records are a fixed-capacity ring; an import larger than max_records has to
    be rejected or downsampled, not silently truncated at an arbitrary end.
  • Timestamps must merge rather than append, or a restore after some running
    time interleaves badly.
  • Writes go through the history writer's queue and lock, so a bulk import
    can't bypass the OTA quiesce or hold the flash long enough to trip the
    cache-disable fault (see docs/tsdb-flash-crash-issue.md).

Export alone is worth shipping first — it turns an irreversible loss into a
recoverable one, even if the recovery is initially "you still have the data".

Notes

  • tsdb_peek_span (in our esp_tsdb fork) already reads a database's time span
    cheaply, which an export UI could use to show what it is about to pull.
  • Streaming matters: range=year on the system DB is 287 KB of JSON. Building
    that in PSRAM in one string is probably fine, but it is the largest response
    the server would ever produce.
  • Whatever ships, the upgrade docs should point at it — getting-started.mdx
    currently tells users to export the node table and accept that history is
    lost.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions