Reimplement scaling of query size #13
Description
Some repos with large records cause occasional issues with OpenSearch queries. This first surfaced as an issue with the responses containing incomplete JSON, and was approached with "auto-scaling" mechanism to reduce the number of hits returned from each query.
However, the initial implementation of this did not scale well to full production runs and so was abandoned in favour of just limiting to 1000 results per query. Bringing back larger queries with better scaling would dramatically increase the speed of generation and decrease load on OpenSearch.