Optimize filtering

[aregtech]

Description:

With the current filter implementation, if the log viewer has large numbers of logs (for example, 100K entries), each filter change freezes the window for some tens of seconds (on app. 150:K entries, a simple filtering by log priority was freezing UI for approximately 40 secs.). This performance must be improved by following way:

1. Keep current model structure:

• LoggingModelBase — source, holding all logs.
• LogViewerFilter — proxy on top of it.

2. Make LogViewerFilter lightweight:

• Override filterAcceptRow() to only check against a precomputed bitset or list indicating which rows are accepted, not doing filtering work inline.
• This bitset/list is computed in the background thread asynchronously. Example of bit-sets QArray<bool> bitsets; -- this keeps only boolean values to indicate whether row is accepted or not when call filterAcceptRow().

3. Background filter worker:

• When filter criteria change:
• Post the filter criteria + current snapshot of logs to the background thread.
• Background thread filters logs chunk-by-chunk, builds a bitset or vector of accepted row indices.
• Support cancellation if new filter criteria arrive before finishing.

4. Update proxy model incrementally:

• When the worker thread completes filtering (or partial chunk), it emits a signal with updated filtered row info.
• The proxy model updates the internal acceptance bitset and calls beginResetModel() / endResetModel() or uses more fine-grained row insertions/removals if possible.
• This allows Qt view to repaint with new filtered data without blocking UI.

5. Optimize data() calls:

• Cache or precompute string representations where possible, e.g., keep thread IDs or priorities as integers and only convert to string when displaying (not in filtering).
• For filtering on numeric IDs, compare IDs directly, not strings.
• Implement efficient lazy string conversion, possibly store QString for common fields on log insert.

6. Handle live data during filtering:

• The background filter thread processes a snapshot of logs at the filter start.
• New logs arriving are appended to LiveLogsModel and stored, but filtering is only applied to them after the current filtering pass completes.
• You may queue new filtering jobs for newly arrived logs or batch updates.

[git-since2005]

I am finding more challenging contributions.
You have to decrease the content with method if you assign it to me I can do this for you.

[aregtech]

@git-since2005 , thanks for your interest in helping with this issue.

Could you clarify what you mean by:

You have to decrease the content with method

Are you referring to the log messages, the log data structure, or something else? I don’t think we should reduce the data itself — the key problems are:

1. String comparisons (e.g., log priorities) should use integers instead.
2. Filtering needs to run in a background thread to avoid UI freezing.
3. Filtering should be interruptible if the user changes the filter.

To reproduce and analyze:

• Clone and compile areg-sdk
• Run logcollector and mcrouter (for IPC examples)
• Start example 16_pubfsm from the products directory (runs until quit is typed).
• Optionally run multiple 16_pubclient instances to generate more logs.
• Open lusan to view/filter logs.

Please try to analyze the problem in this setup first. If you think you can address it, I’ll gladly assign it to you.

[git-since2005]

By the word decrease, i meant we can abstract the information through status codes and with various flags to know the reason of a particular log record

[aregtech]

Got it.
Yes, right now the filtering logic still works on text (I needed a fast solution). The log format already stores many fields as integers:

• Log priorities
• Cookie IDs (instead of process names)
• Thread IDs (instead of thread names)
• Scope IDs (instead of scope names)
• Timestamps and execution durations

My plan is to switch to integer comparisons for these. Filtering by scopes and time-frame doesn’t exist yet, but it’s coming. I agree this will allow much faster filtering without touching the text part.

What I don’t have are categories like “Hardware” or “Network,” because in my experience the simpler the API is, the more developers use it. I might add such categories later if there are enough requests for the feature.

The most challenging part remains filtering by log message text — this is the main bottleneck, and at the moment I don’t have a solution for it yet.

BTW, did you succeed in using the live log viewer?
You need to connect to logcollector to receive logs. See the screenshot from Live Logs navigation window. Sorry, I don't have documentation yet.

[aregtech]

@git-since2005 , I'm going to work on this issue. Do you want to join for pair programming? :)

[git-since2005]

Yes, I would love to work on this issue.