performance scaling issues with 1M+ registers

[binarybob14]

I had referenced issue #16. So Im on the other end of this :). I am testing this with a system of 1Million+ (more like 3M) registers and its almost unusable in its current form and consumes 8GB or so of memory since it seems to be loading all the registers into memory for search. I see in #16 that you have a new version that adds full description search. I am not sure if we have that version yet (I dont think ours is doing description search). Would this new model help our scaling issues?

I have tried to use tricks that I think speed things up like using arrays of components rather than individually instantiate where I can, but our address map doesn't make that possible in many cases.

Do you have any ideas or plans to make this faster? Could we provide a switch that removes search maybe at least in the interim so that I can at least load and manually click through the tool with 1M+ registers? Should I file a different github issue for my scaling problem?

[amykyta3]

Holy smokes. 3 million registers! That's way more than I would ever have anticipated!

The new description search shouldn't be the issue here. Description search uses a pretty well optimized hash index that should scale pretty well. The one edge case with description search would be if all your registers have mostly the same description text in them - as that would probably bloat some parts of the index.

The older "path search" is definitely not very optimized since it performs a pretty naive linear search through the register model's tree. That could probably be improved if I indexed it.

Some follow up questions

• Is the initial page load slow? Or just the search operation?
• In the generated output directory...
  • How big is the js/data.js file?
  • How big is search/bkt_index.js?
  • In the search/ folder, how big is the largest bkt-*.json file?
  • I assume the content/ folder is what consumes the most amount of the ~8GB of space?
• Are many of your registers instances of the same type definition? Or are they all unique?

Regarding your design input, I would definitely recommend using arrays wherever possible. Not only will they make the model smaller, using arrays will make it far easier to manipulate in other scenarios. Millions of registers seems pretty excessive to me - it makes me wonder if maybe there is a better way to represent your address space or whether you are hitting some limitation that can be solved in a different way. I'm guessing these registers are auto-generated in some way, and not hand-coded?

[binarybob14]

The initial page takes a very long time to load. After that the search is slow but usable. In my browser it always crashes on load. For some people it finishes. We are on an older version that doesnt have search/

latest snapshot:
reg_html> du -sh *
12G content
112K css
24K favicon.png
8.6M fonts
24K index.html
31M js

30M js/data.js

We have a very large project ;) I don't have much flexibility in the address map as we have to work around IP limitations and we are pushing it beyond what it was designed for. There are some cases where I could use an array, but the numbering of the elements isn't always mapA_0, mapA_1, mapA_2, etc. its sometimes like map0, mapA_4, mapA_1, mapA_5, etc. so I dont know how to combine them. I would like to change the naming but the design always wins over address map niceness :). This is due to physical hierarchy.

regarding auto-generated. some of its external IP through IPxact (sometimes auto generated), some of its hand coded and some of its hand-coded + automation. Its a larger team.

[amykyta3]

Thanks for the info.
Since you don't have the search/ folder, it looks like you are using version 2.2.0 or older which was before content search was implemented.
I'd recommend upgrading to v2.5.0 or newer, since I made some improvements to page load time. Older versions would stall the page contents until all of the sidebar elements were loaded. With large designs like yours, that would definitely freeze the browser. In v2.5.0, I changed it to load that content in the background.

Here is the changelog if you need more information: https://github.com/SystemRDL/PeakRDL-html/releases

I'll look into improving the hier path search search since that will still be slow for large projects.

[amykyta3]

I ran some tests with a design with 1M registers and made some additional improvements that should significantly help on your end.
Try the latest v2.8.0 release. (Reminder: if you're upgrading from an older version, you'll want to change how you import peakrdl-html. See details in #31)

The changes should further improve the initial page load, as well as significantly reduce the amount of RAM used once the content finishes loading. I realized I was loading an absurd amount of hidden DOM elements which is probably why your browser was crashing.

No changes were made to how search works, so that will still be slow. I need to plan out a proper indexing scheme for that in order to speed it up.

[binarybob14]

Thanks for looking at this! We are going to run that version. Will let you know.