Skip to content

Index update, Phase 2: Use new large_user_postcode index#572

Merged
KludgeKML merged 3 commits into
mainfrom
add-postcode-side-index
Apr 28, 2025
Merged

Index update, Phase 2: Use new large_user_postcode index#572
KludgeKML merged 3 commits into
mainfrom
add-postcode-side-index

Conversation

@KludgeKML
Copy link
Copy Markdown
Contributor

@KludgeKML KludgeKML commented Feb 11, 2025

The updates added in PR #556 included a naive method of identifying relevant ONSPD records to update (fetching a list which had to be filtered by a select statement), resulting in a notable bump in CPU usage by the worker threads. This PR moves the "size" element from the free-form JSON results field into a specific field which can be indexed and searched in a single search.

Speed of the query code as reported by benchmark:

Before:
0.473827 s / call

After
0.000685 s / call

(Speed improvement of 690x on this code, which is called once a second)

Obviously there's a cost in writing, because the index has to be updated, so it's not plain sailing, but the benefits should outweigh the cost, we can monitor that on CPU usage.

Integration worker CPU and memory load over the last two weeks - low before PR 556, high after it, then low again after this PR is temporarily deployed - CPU and memory load seem to be marginally better even than before PR 556

image

Actual index and migration task added in #610

https://trello.com/c/Pf89f3wr/350-improve-worker-cpu-usage-on-locations-api, Jira issue PNP-6398

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

⚠️ Coverage note: test suite is set to fail if coverage drops below 100%. If you need to merge in an emergency, you will have to temporarily change branch protection rules. ⚠️

Follow these steps if you are doing a Rails upgrade.

@KludgeKML KludgeKML changed the title Add postcode side index Add postcode large_user_postcode index Feb 11, 2025
@KludgeKML KludgeKML force-pushed the add-postcode-side-index branch from 974b49a to 3c31f6c Compare February 12, 2025 10:31
@KludgeKML KludgeKML marked this pull request as ready for review April 23, 2025 08:55
Copy link
Copy Markdown
Contributor

@sihugh sihugh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a nice change. I had a couple of questions:

  • is the migration quick enough to make it through a standard deploy? (We've had some recent ones where it's hit a timeout and had to be decoupled into schema updates and rake tasks, or some such arrangement)
  • can we drop any of the other indices after this one has gone in if we're worried about write performance?

@KludgeKML
Copy link
Copy Markdown
Contributor Author

Looks like a nice change. I had a couple of questions:

  • is the migration quick enough to make it through a standard deploy? (We've had some recent ones where it's hit a timeout and had to be decoupled into schema updates and rake tasks, or some such arrangement)

Good question - I think so, because I have deployed this to integration. Let me do that again, see how long the migration takes.

  • can we drop any of the other indices after this one has gone in if we're worried about write performance?

I think we're not worried per se about write performance, I just mention it for completeness. Writing is relatively stable (we have a fixed rate of 3 writes per second, except during ONSPD imports), and it's all done in workers so doesn't affect the end user. But we can certainly audit the indices somehow to confirm whether there are any vestigial ones.

@KludgeKML KludgeKML force-pushed the add-postcode-side-index branch from 3c31f6c to 71058b0 Compare April 23, 2025 12:03
@KludgeKML KludgeKML changed the title Add postcode large_user_postcode index Phase 2: Use new large_user_postcode index Apr 24, 2025
@KludgeKML KludgeKML force-pushed the add-postcode-side-index branch from 71058b0 to 8b61fa4 Compare April 24, 2025 15:06
@KludgeKML KludgeKML changed the title Phase 2: Use new large_user_postcode index Index update, Phase 2: Use new large_user_postcode index Apr 24, 2025
@KludgeKML KludgeKML force-pushed the add-postcode-side-index branch from 8b61fa4 to b482bf8 Compare April 28, 2025 08:46
@KludgeKML KludgeKML force-pushed the add-postcode-side-index branch from b482bf8 to c78722e Compare April 28, 2025 09:12
@KludgeKML KludgeKML merged commit bdbf26d into main Apr 28, 2025
9 checks passed
@KludgeKML KludgeKML deleted the add-postcode-side-index branch April 28, 2025 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants