Optimize file indexing

[RuiMarioCosta]

dtl-viewer is slow when loading DLT files taking some time to load a file with just 500mb and becomes extremely slow files in the gigabytes. There are several parts that contribute to this but the main method is QDltFile::getMsg which is called from DltFileIndexer::indexFilter.

My plan is to first stablish a baseline to understand if the performance is actually improving or not and to do that I was thinking of using google's benchmark library which is fairly easy to use and well documented. The idea in this step is to create bechmarks for the biggest hotspots of the code. After that the optimization can start.

I already have some small benchmarks but nothing special but before putting more effort would this be a good idea?

[alexmucde]

@RuiMarioCosta Good idea to find the bottlenecks and possibilities for optimization

[RuiMarioCosta]

Hi @alexmucde I have a few questions, I'm still trying to understand the code so some questions might not make sense.

I have already profiled the code and see that for big files most of the time is spent on DltFileInderxer::indexFilter, especially on dltFile->getMsg. From what I see this function creates and saves the the vector with the filtered indexes to a .dix file.

1. I noticed that the QSltFile::getMsg(int index) is parsing the .dlt but I also saw that dlt_common.h exist which looks like a copy from the same files in dlt-daemon. Wouldn´t it be better to use the functions already implemented in dlt_common, like the dlt_file_init, dlt_file_open, dlt_file_read, dlt_file_message, etc to parse the file?

2. DltFileIndexer::index(int num) is also parsing the .dlt file but only searching for the current_message_pos to add to indexAllList. Wouldn´t it be better to parse the entire file once and then do the filtering and saving the filtered index after?

3. The loop in DltFileIndexer::indexFilter is constantly allocating a QSharedPointer<QSltMsg> but isn´t it better to allocate a large chunk of memory from the start? I still don´t understand every part of the code but for it seems that there are a lot of allocations and copies of data from one buffer to another.

4. Are the messages that appear in the GUI of the application the ones in DltFileIndexer -> QDltFile -> QCache<int, QDltMsg>?

Having looked at the code I have a suggestion which is to make a lib for the (de)serialization of the DLT object I think it would make the code much clearer and could be shared between dlt-viewer, dlt-daemon, etc. The same would apply to the mini (de)serialization of the Index file but this one doesn´t need to be an external lib since it's only used in this project.

[RuiMarioCosta]

@alexmucde, the best way that I see to improve the performance is using memory mapped files and use pointers to the memory region to avoid memory allocations and copies. Do you think this have any issue elsewhere in the code?

[alexmucde]

@RuiMarioCosta QCache is already used to cache the messages in memory. The Benefit is, that the messages are already parsed. The size of the cache can be set here:

But this cache does not really help when having big files, only the crolling is getting smoother.

Another improvement could be when the Filter index runs, already showing the results to the user, when the indexer is still running, so that the user has not to wait for the end. Thats the same like the Search function is already doing.

[sudomgamal]

I also see multiple operations done inside the GUI event loop (in the GUI related slots). This can impact the application responsiveness since the operations can take a lot of time specially in the cases mentioned by @RuiMarioCosta

Besides the performance improvements, we can use a worker queue to handle these operations outside the event loop. But maybe this can be tracked on another ticket.

[RuiMarioCosta]

@sudomgamal, maybe that can also be optimized, but from the profiling results, the program spends most of the time in the getMsg and setMsg. I'm writing a small POC to see if the ideas that I have can improve my concern is how to later incorporate them here.

I would really like this to be improved because just the other day, I had to open a ~3Gb file, and I think it took almost 10 minutes.