Add quota limit for auto evaluations

[davidgisbey]

Description

If we run our auto-evaluation on every answer that's generated we could end up spending a significant amount of money as many metrics have multiple LLM calls. Each of which is run 3 times.

This adds a quota for evaluations per hour which we've set to 300. Each job that runs an evaluation:

1. check for the presence of the auto_evaluations_count key in the cache
2. if not found. It adds the key with a value of 1 and set's the cache to expire at the end of the hour
3. If it is found then it checks to see if the value >= to the threshold of 300

• If it is then it logs that the the quota is at the threshold and exits the jobs
• if it isn't then it increments the counter and runs the evaluation

I've added the functionality to the base worker so it can be reused across the workers and added a shared example to test it.

I've updated the AnswerTopicsJob to inherit from the base worker and updated it to adhere from the quota.

I've also added another shared example for retry logic to remove some duplication

Trello card

https://trello.com/c/lfwcnNaz/3045-add-quota-for-auto-evaluation-llm-requests

[kevindew]

Great job, just a few small changes

[davidgisbey]

Thanks for the review @kevindew i've made those changes with the exception of one comment i left.

[davidgisbey]

I ran into an interesting issue when testing on Integration where it wasn't working as implemented. I pushed up a new commit which outlines the issue and gives a few suggestions.

Any feedback the approach we should take would be great.

[kevindew]

Ooh tricky problem - thanks for outlining options. I think I like:

expires_in = (Time.current.end_of_minute - Time.current).to_i
count = Rails.cache.increment("auto_evaluations_count", expires_in:)

if count >= max_evaluations
  do the various logging things
  return false
end

Though I would suggest doing: if count > max_evaluations as we've incremented it before.

Did you have problems with expires_at? I notice that's changed from expires_in

[davidgisbey]

I did. it didn't seem to work so i had a look at the docs and increment expects expires_in

[davidgisbey]

this is required for the system specs that run auto evaluation because NullStore returns nil when you call increment which causes this line to blow up.

[kevindew]

I think rather than doing this - because it's pretty heavy, it'd be better to change the usage code to be:

# fallback to 1 in scenarios where we have a null cache (dev/test) and this returns nil
count = Rails.cache.increment("auto_evaluations_count", 1, expires_in:) || 1

[davidgisbey]

Thanks for the feedback @kevindew i've made those changes.

[kevindew]

I did. it didn't seem to work so i had a look at the docs and increment expects expires_in

So it should be fine because the the merged_options method is supposed to convert it to an expires_in

[kevindew]

I did. it didn't seem to work so i had a look at the docs and increment expects expires_in

So it should be fine because the the merged_options method is supposed to convert it to an expires_in

Hmm I've had a bit of a play with it and while it does work, there is an edge case you can hit where if you were very close to the end of an hour you'd get a negative number and an exception:

govuk-chat(prod)> Rails.cache.increment("test", expires_at: expiry)
(govuk-chat):37:in '<main>': Cache expiration time is invalid, cannot be negative: -16.07605283 (ArgumentError)

            raise error

which would be annoying to hit.

I then considered that it might be better to do expires_in but I realised that if we hit the same sort of edge case on that we'd end up with the cache potentially spanning more than one hour.

---

So, I've tried to think of a way to do it without these edge cases that might be a little more robust.

Rather than us relying on expiry I think we should set the key based on the time and then just allow it to expire after an hour. I think that's actually what Rack::Attack does if I remember right.

So something like:

key = "auto_evaluations_count_#{Time.beginning_of_hour.to_i}"
count = Rails.cache.increment("key", 1, expires_in: 1.hour)

I think this would also have an advantage that for every hour we get a fresh cache key and that is something explicit we can debug on if we hit problems, whereas relying on the TTL for MemCache is very cryptic because you can't look it up. I learnt this in my little terminal session when I accidentally ran fetch on a key once and then it seemed to never expire:

govuk-chat(prod)> Rails.cache.increment("test", expires_in: 10)
=> 13
govuk-chat(prod)> Rails.cache.increment("test", expires_in: 10)
=> 14
govuk-chat(prod)> Rails.cache.increment("test", expires_in: 10)
=> 15
govuk-chat(prod)> Time.now
=> 2026-01-08 10:41:27.049026017 +0000
govuk-chat(prod)> Time.now
=> 2026-01-08 10:41:43.991510293 +0000
govuk-chat(prod)> Rails.cache.increment("test", expires_in: 10)
=> 16

[davidgisbey]

@kevindewgood shout on that. I'll remember that approach going forward 🎉 I've made the requested change.

[kevindew]

Great stuff