use task groups for concurrent queue consumers

[zzstoatzz]

related to #16299

• use task groups to consume items off the in-memory queue concurrently (2 concurrent consumers appears to roughly double the rate of clearing the queue)
• add debug queue depth metrics

the queue depth metrics were very useful while debugging and I could imagine they might be useful to server operators in the future, which I why I opted to keep it in - happy to re-evaluate this choice. doing so made me think that we may want to add more server metrics in the future

[codspeed-hq]

CodSpeed Performance Report

Merging #16850 will not alter performance

_{Comparing statement-timeout (719f7ec) with main (180185e)}

Summary

✅ 2 untouched benchmarks

[chrisguidry]

This all makes sense to me! The tests look great

[daniel-prefect]

Docs changes look good

[zzstoatzz]

Docs changes look good

thanks! though this reminds me that I should have actually removed that new setting, this is removed now so there should be no doc-related changes here

[desertaxle]

I left a couple nits, but this looks like an awesome enhancement!