[hma] Queue-based hashing & matching

[ThisIsMissEm]

Whilst HTTP Requests for doing hashing does work, it may be better to use a durable queue for requests, such that failures can be more easily retried. One such option would be using RabbitMQ, though plenty of other queue based messaging systems exist (kafka, AWS SQS, Google PubSub, etc).

This would be an entirely new way to interface with HMA, so wouldn't be a small amount of work, but could provide some reliability guarantees for production workloads.

Ideally there'd be one queue for incoming requests for hashing, an internal retry-queue (in the case that an error was encountered and the hashing can be retried), as well as an error output queue and a results output queue (these could be combined into a single queue). It is important that if the error is retryable automatically, that HMA does do that, instead of deferring the requeuing to the integrator).

Perhaps this could be built in a generic way as to allow various queue drivers under the hood. It would also require HMA to have more configuration for this type of integration.

(this is a ~low-medium priority for me personally, as in my integrations I'll probably need some queuing / worker processes anyway at the moment).

[Dcallies]

Fun fact, this was the core implementation of HMA1.0, which was needed to hit our target scale! We used async lambda backed by SQS.

@dougneal and I talked about this when redesigning HMA, and the position we came to is that we want the API to slot easily into a queue/async based solution, but that we would not put that detail into the base image.

Due to the additional complexity of handling a queue on the instance itself, and queue management ultimately needing to fallback to the queue service in case of server failure or shutdown, my ask is we not build queue handling into the core HMA.

For example, in HMA1.0, we tested two versions of queued hasing+matching using lambda:

1. The executor code would make a load balanced network call to an index instance, retrying locally
2. The executor instance loaded a copy of the index before accepting requests, then would match locally and make db calls to load the results of matching

Both had their tradeoffs.

I think this might make a good reference repo with an alternative deployment (@juanmrad did one or two of these).

In HMA1.0 we used terraform to set up SQS and Lambda, but it's a bit of a pain to learn.