Quarterly schedule-update tick can be silently skipped when hardware retries run long, permanently dropping a period's actuals

[johanzander]

Summary

The quarterly schedule-update job (app.py update_schedule_quarterly, CronTrigger(minute="0,15,30,45")) can silently miss its scheduled run — and permanently drop a whole 15-minute period's actuals — when the previous run is still busy retrying a slow inverter hardware write when the next tick fires.

Evidence (production, 2026-07-27, beta b25)

1. Add-on auto-updated b24→b25 at 00:26:58, restarting the container mid-period-1 (00:15-00:30).
2. On restart (00:27:06), a health-check safety net (battery_system_manager.py:2946, "No schedule exists yet... retrying the initial schedule build") fired an early schedule build for period 1, immediately followed by start()'s own normal call to the same period. Each hit the pre-existing number/set_value Supervisor timeout issue and retried with backoff.
3. Between the two overlapping calls, the "apply period 1 to hardware" step stayed busy from 00:27:06 to 00:31:56 (log: Applied period settings for period 1 (00:15) at both 00:29:33 and 00:31:56).
4. The job is registered with misfire_grace_time=30 (app.py:337-341) and default max_instances=1. The 00:30:00 tick (which would run as "period 2" and record period 1 as its just-completed previous period) landed inside that busy window, missed its 30s grace, and was coalesced away by APScheduler. There is no Updating battery schedule for period 2 log line anywhere in that day's log — direct proof it never ran.
5. The next tick to run was 00:45:00, as "period 3" — which only records period 2 as previous. Period 1 (00:15-00:30) was never anyone's "previous period" and was never recorded.
6. Confirmed live via GET /api/export-debug-data?compact=false: historical_periods has exactly one null entry for the day — period 1 — with all others (including period 0, correctly InfluxDB-backfilled at startup) present.
7. User-visible effect: the "Incomplete Historical Data" dashboard banner, persisting for the rest of the day since nothing ever retries a mid-day skipped tick.

Root cause

• misfire_grace_time=30 is too tight for a job whose critical section can legitimately take minutes (hardware-write retries with exponential backoff).
• APScheduler's default coalesce=True silently drops a missed run rather than surfacing it — no warning/error is logged for this specific job (unlike apply_discharge_inhibit, which does log ... skipped: maximum number of running instances reached because it's a minute='*' job that collides constantly).
• Separately, the health-check "retrying the initial schedule build" path (battery_system_manager.py:2942-2948) can race with start()'s own scheduled call for the same period right after a restart, doubling the busy window.

Suggested directions (not decided)

• Increase misfire_grace_time for the quarterly job to comfortably exceed worst-case hardware-retry duration, and/or decouple the synchronous "apply to hardware" step from the job's critical section so a slow retry can't block the next tick from recording data.
• Make missed quarterly ticks visible (log a WARNING, or track via runtime_failure_tracker) instead of failing silently.
• Consider a deliberate backfill/retry path for a period that's found missing when the next tick runs (today only period N-1 relative to "current" is ever collected — there's no sweep for older still-missing periods).

Related

Not the same bug as the "moved not fixed" 23:55→00:00 historical_store.clear() timing fix (#396) — that was about premature clearing before midnight. This is a mid-day skipped-tick gap unrelated to the clear timing. Also related to the broader HistoricalDataStore persistence gap (in-memory only, no disk backing) tracked separately.