A local issue for tracking development issues in the main queue that are crucial for the stability of HDV

[landreev]

I'm going to maintain a (check)list of the development efforts that are deemed super important for speeding things up/improving overall stability of our prod. instance.

• Speeding up exports of unexportable monster datasets: Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality dataverse#11799 [in progress; PR in draft]
• Jim's improvements addressing unacceptable performance in reindexing after making changes to role assignments in large collections (an exotic/edge case; but the performance degradation is really bad): QDR: Multiple transactions for permission reindexing dataverse#11822 [PR approved; ready for QA; I am testing it separately on the perf. system]
• A performance issue with re-harvesting of harvested datasets; is a major blocker in refreshing some large harvested collections that are now stale: Unacceptable performance deleting datasets and/or datasetversions in the prod. database - needs to be improved.  dataverse#11828
• Add a dedicated File page rate-limiting command: A dedicated rate-limiting command for the File page dataverse#11366 [a trivial PR I'm already including in my prod. deployment patch; but needs to be merged into the main source tree at some point]
• Expanding Steven's internal rate limiting subsystem to make it more fine-tunable [discussing this w/ Steven; haven't opened an issue yet]
• Adding Payara-level caching for rendered pages [discussing w/ Steven; no issue yet]

... work in progress - there is definitely more!