Reset the trace queue indexes when the queue shrinks

[AbhinavMir]

Problem

Log_pretrace() reallocates the trace queue when an application changes trace_settings.max_trace_entries. The queue holds max_trace_entries entries, so the valid indexes are 0 to max_trace_entries - 1.

The bounds check compares the stored ring indexes with max_trace_entries + 1. It therefore accepts max_trace_entries and max_trace_entries + 1. Both are past the end of the new queue.

A shrink to the current write index leaves next_index out of range. Log_trace() and Log_stackTrace() then write a whole traceEntry past the end of the allocation. The index also never wraps again, because the wrap test uses ==. Every later entry goes further past the end.

Change

Compare the indexes with max_trace_entries.

Test

test/test_unit_coverage.c gets a new case. It uses 5 slots of a 10 entry queue, shrinks the queue to 5 entries, and keeps logging. A trace callback counts the entries.

Without the fix:

• AddressSanitizer reports a heap buffer overflow in Log_trace(), 0 bytes to the right of the 1560 byte queue.
• A plain build aborts with double free or corruption (!prev).

With the fix the test passes in both builds.

Build

Built with gcc 12 on Debian 12. Static and shared, with OpenSSL, plus an AddressSanitizer build. No new compiler warnings.

test1 cases 1 to 5 and test4 cases 1 to 3 pass against a local broker.

The commit is signed off.