Performance Issue with over 800k registers, suggesting using sqlite to store and dynamically query the RDL database.

[isuckatdrifting]

Hi @amykyta3,

I am using the PeakRDL-html 2.10.1 and find performance issue on a website for around 800k registers (and more fields obviously) similar to #34.

There are two major issues I would like to propose:

1. webpage performance -- As shown below in a history build of the website, the synchronous data.js is occupying much of the time of the page loading, and significantly draws back the overall performance.

I found this issue on older releases of this tool (maybe two years ago), but I am not sure if it is resolved in the latest build due to the issue 2 below.

2. build performance -- our 800k register RDL build takes around 30 minutes on a decent server CPU (latest xeon-platinum), and it is an acceptable period of time actually, for manual build or CI. However, the webpage exporter takes even longer (roughly 4-5 hours). Currently the process is gonna run for hours and generates the same number of html files under the content folder (which is also a burden for the hard-drive actually).

In our local frontend environment, we see sqlite significantly improves the overall query performance of such large scale of register information. So I would like to suggest using sqlite to store the generated RDL database, and dynamically query and generate the detail pages and hierarchy trees, instead of statically generate them in the content folder.

I wonder if it is planned in the future release of this tool. Thank you.

[isuckatdrifting]

A quick update here -- seems issue 1 webpage browsing performance is pretty good after the webpage based on PeakRDL-html 2.10.1 is built.

Add another 2 minor issues here (though I managed to workaround it), in case anyone needs:

1. Seems the MathJax.js (pretty small) is retrieved online. I wonder if it can be placed with other js so that in environments without internet connection, the website can be generated out-of-the-box.
2. Seems the latest 2.10.1 build uses private class member js feature, which is only available in newer chrome/firefox 90+. I wonder some extra backward compatibility can be added (like firefox 78 on older linux distros such as centos).

[amykyta3]

Thanks for the feedback!
I agree that export speed and disk space is one area that I'd like to improve. One thought I had was condensing the content as you had suggested, and rendering it client-side. That should reduce the time spent exporting it, as well as reduce disk space consumed.

In your design of 800k registers, are many of your registers instances of the same type definition? Or are they all unique?
Currently the generator will dump an HTML content file for every register, regardless of whether it was identical to an existing one. It is not uncommon to have repeating structures in large designs, so I am looking at ways to remove the amount of redundant information written.

When you mention using sqlite, can you elaborate what you mean by that? Would this be client-side processing of sqlite? Or a server-side database?

[isuckatdrifting]

Hi Alex,

I would say these registers are different from each other in the scope of fields and register at block level, and are duplicated to some extent at SoC level. So I wonder how it would compress based on the literal meanings at block level, but I am sure that some for loops of duplication can be optimized anyway.

The reason I am referring to sqlite is that we actually use it in a lot of production flows as a means for compression and randomized access. I would like to take some indutrial flows as an example. ARM uses protobufs (a google format) to store the tarmac outputs and some RISC-V community uses sqlite-like DBs to similarily store the critical debugging information as an intermediate means between opening a large waveform and viewing some raw logs.

I am not an expert of sqlite in the website 'frontend' (though I think it is not a big deal to open tens or hundreds of megabytes of database in modern browsers and server machines), so I am not very sure whether to make it a server-side db or client-side one, but I am sure the RDL generation and rendering flow resembles a normal sqlite access -- store a large amount of random data (literally random but somehow organized), and render or iterate over the db afterwards.

[tsyw]

We found two issues

1. the side_bar expand/collapse will consume many time on create/remove nodes if the nodes number is too large.
2. The one addressmap have a huge number of sub addrmap, the sidebar unroll and the page loading consumes too much time.