backlog: queue diagnostics and byte-limit safety net#24881
Open
joluxer wants to merge 6 commits into
Open
Conversation
BacklogLoop previously left backlog_timer at the value written by CommandHandler at the start of ExecuteCommand -- millis() at command start time plus P_BACKLOG_DELAY. After a long execution (rule-trigger chain) that point was already past: TimeReached() returned true immediately on the next iteration, draining the queue without pause regardless of SetOption34. A nested CmndBacklog reset the timer to millis() during execution, compounding the problem: the queue drained as fast as commands were enqueued. BacklogLoop now unconditionally sets backlog_timer after each timed drain step. The inter-command interval becomes deterministic: exactly P_BACKLOG_DELAY between drain steps, regardless of what CommandHandler or nested Backlog commands wrote during ExecuteCommand. Exception: CmndDelay called from within a drain sets backlog_delay_guard (detected via backlog_mutex=true). BacklogLoop preserves the timer for that one step and clears the guard. External commands can still shorten an active delay -- that matches original Tasmota behaviour.
Introduce SettingsParam(index) as a named function returning a uint8_t reference to Settings->param[index]. Adds bounds protection: an index >= PARAM8_SIZE returns a reference to a stable zero byte instead of an out-of-bounds access. Replace three direct Settings->param[] accesses in the SetOption handler (CmndSetoption, GetOption, CmndSetoptionBase) to use the new accessor.
Move the backlog queue, timer, and control flags out of TasmotaGlobal_t and the global .ino scope into a dedicated translation unit (support_backlog.cpp). The state was accessible from any .ino file; the new module limits mutation to explicit paths through the Backlog:: API. All internal state is held in an anonymous namespace. The public interface is declared in support_backlog.h. A one-line support_backlog.ino adapter keeps BacklogLoop() visible to the unity build. Removes from TasmotaGlobal_t: backlog_timer, backlog_nodelay, backlog_mutex, backlog_delay_guard, backlog_no_mqtt_response. Removes global LList<char*> backlog. Adds SuppressMqttResponse() to support.ino as the designated mutation path for TasmotaGlobal.no_mqtt_response from backlog context. No intended behavior change.
SetOption166 (default 0) controls whether external commands (MQTT, Serial, Button, ...) extend the Backlog drain window after execution. SO166=0: CommandHandler sets the drain timer to millis() + BACKLOG_EXT_DELAY (100 ms, separate from SO34, overridable at build time via user_config_override.h). SO166=1: BacklogLoop is the sole timer owner; external commands do not stall the drain. The inter-command delay within a sequence (SO34) is unchanged. Each queue entry now carries a flavor byte (bit0=nodelay, bit1=no_mqtt_resp) baked in from staged flags at enqueue time. D_CMND_NODELAY is resolved in CmndBacklog() and ExecuteCommandBlock() at enqueue rather than stored as a queue token. Loop() reads the flavor byte per drain step; _nodelay and _no_mqtt_resp each split into _staged (set by callers) and _current (set from the flavor byte, read by IsNodelay()). ExecuteCommandBlock() resets staged flags to plain-Backlog defaults before its tokenisation loop: IF blocks no longer inherit ambient state from a preceding CmndBacklog() call. ScheduleDelay() is a no-op when _nodelay_current is set. Default-configuration behavior: the inter-command delay (SO34) is unchanged. The post-external-command drain window changes from SO34 to BACKLOG_EXT_DELAY (100 ms vs. 200 ms default) -- this is intentional and separately configurable.
Backlog16 N: set runtime chunk size for Backlog21-29 paged content dump (default 20; non-persistent). Backlog17 0/1: enable per-drain-step trace log at INFO level (BLG: tag). Logs queue depth before drain, flavor byte, and command string per entry. Build flag BACKLOG_TRACE_SOURCE extends the trace with the command source. Backlog20: JSON statistics snapshot covering depth, timer state, staged and current flags, per-operation counters, high-watermark values, and runtime config (SO34/SO166, chunk size). Bytes/Discarded/MaxBytes report 0 in this commit; the follow-up byte-limit commit fills them with real values. Backlog21-29: paged queue content as JSON. Page = index - 21, chunk size from Backlog16. Entries appear on individual log lines and in an assembled MQTT/Web-Console response. The entry header is generalized to accommodate the optional BACKLOG_TRACE_SOURCE source byte. A compile-time offset replaces the previous hardcoded value, keeping the conditional out of every access site. The source parameter on EnqueueCmd/InsertCmd is unconditional with a "not annotated" default. A running depth counter alongside the linked-list length serves as a cross-check; underflow is logged at error level. Per-operation counters for drain, enqueue, insert, and mutex-skip feed the Backlog20 snapshot.
Without a size bound the queue can grow until a malloc fails mid-sequence, leaving partial command sets in flight and the node unreachable. The byte limit trades completeness for availability: a sequence that would exceed the limit is rejected whole instead of partially enqueued. The node stays reachable and reports the rejection in the log. Empirically confirmed on ESP8266: 30+ min under sustained MQTT load without crash; limit adjustment under load drains the queue cleanly. Backlog18 N: set the byte limit at runtime (0 resets to the compile-time default). Clamped to a minimum confirmed stable on ESP8266 under load. Platform-dependent compile-time defaults are in tasmota.h; override via user_config_override.h. Accounting covers the command string, the linked-list node, and heap-allocator bookkeeping per allocation - the overhead a string-length check alone would miss. The formula is centralized so that a change to the overhead model touches one place. LList gains a public node-size constant so the overhead calculation stays inside the queue module rather than depending on LList internals. Per-command fallback checks guard the rules IF/ENDIF insertion path and any direct caller that bypasses the sequence-level gate. Backlog20 (added in the preceding diagnostics commit) now reports real byte counters and the active limit instead of placeholder zeros.
Owner
|
I strongly feel this feature needs to be enabled using a comple time The code size impact is way to large compared to it's functionality and use-case. Most of the time it won't be used considering the fact Tasmota is used some ten years now and there was never felt a use to monitor/trace backlog use simply because normally only small backlogs are being used. |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of the backlog improvement series documented in discussion #24776.
Implementation choices are open to discussion. If you'd prefer a different approach, please comment -- I'll revise.
Problem
The Backlog queue had no visibility into its own state. Depth, drain
counters, timer status, staged flags, and configuration values were
inaccessible at runtime, making it difficult to diagnose timing issues,
verify that sequences executed as intended, or confirm that behavioral
fixes from earlier PRs had the expected effect.
The queue also had no upper bound on heap consumption. Under sustained
MQTT load the queue can grow until a malloc failure leaves a partial
command sequence in flight, or until the node resets. Both outcomes can
be considered worse than refusing the sequence before it enters the queue.
Changes
Two commits, each independently buildable.
Commit 1: queue diagnostics
Backlog16 N- set the chunk size for paged queue content output(Backlog21-29, default 20 entries per page; runtime, non-persistent).
Backlog17 0/1- enable a per-drain-step trace log at INFO level.Each drained command is logged with the queue depth at that moment
and the entry's flavor byte. Build flag
BACKLOG_TRACE_SOURCEextendsthe trace with the originating command source per entry.
Backlog20- JSON statistics snapshot: current and counter-verifieddepth, byte usage, timer state, mutex state, staged and current
behavioral flags, per-operation counters (drained / enqueued /
inserted / discarded / mutex-skipped), high-watermark values, and
runtime config (SO34, SO166, chunk size). Bytes/Discarded/MaxBytes/
BytesLimit report 0 in this commit; the byte-limit commit fills them.
Backlog21-29- queue content as JSON, paged by chunk size. Each entryappears on its own log line and in the assembled MQTT/Web-Console
response.
The entry header is generalized to support the optional
BACKLOG_TRACE_SOURCEsource byte without scattering conditionalsacross access sites.
Commit 2: queue byte limit
Backlog18 N- set the byte limit at runtime (0 resets to thecompile-time default; clamped to a minimum confirmed stable on ESP8266
under sustained load). Platform-dependent compile-time defaults in
tasmota.h; override viauser_config_override.h.Capacity at default settings: ESP8266 4096 B ~= 97 short commands;
ESP32 32768 B ~= 780.
A whole-sequence-or-nothing gate in
CmndBacklogpre-checks the fullestimated cost before tokenising the string. A sequence that would
exceed the limit is rejected intact; the node stays reachable and logs
the rejection. Per-command fallback checks guard the IF/ENDIF insertion
path and any caller that bypasses the sequence gate.
Accounting covers command strings, linked-list node overhead, and
heap-allocator bookkeeping per allocation - the cost a string-length
check alone would miss.
Backlog20now reports real byte counters and the active limit.Memory impact (measured, per commit)
Ref: pr/backlog-behavior tip (85ba5553)
Requires pr/backlog-behavior: per-entry flavor byte and queue-access API from that PR are the structural prerequisite for the diagnostics and byte-limit accounting here.
Tested
Build-tested on ESP8266 (tasmota-4M) and ESP32 (tasmota32).
Target test on ESP8266 (ESP-12F): Backlog20 snapshot verified; Backlog17
drain trace verified; byte-limit rejection under sustained MQTT load
confirmed (30+ min without crash; limit adjustment under load drains the
queue cleanly without side effects).
Checklist