Noticed while doing #23018 that server/v2 doesn't server a swagger UI.
Is that something server/v2 should support out of the box?

The server in v1 is fairly simple: https://github.com/cosmos/cosmos-sdk/blob/d297747/server/swagger.go, hence super easily to add by chain themselves with the UI they prefer.