Epic: Integrate public API's into supervisor

[akosyakov]

Summary

Migrate supervisor from Server to Public API.

Context

Why we chose to migrate Supervisor to the new Public API? In order to validate Public API we started with #9757, but VS Code Desktop depends on SH installations which block migrations till Public API is reliable. VS Code as well does not have interesting use cases to validate usability and reliabiltiy of the API, i.e. pooling vs streaming, and does not generate enough traffic. We have also had incidents with supervisor working against Server API. We could not react promptly because we don't have any client observability [1], additionally jsonrpc library which we use on go client side has very verbose logging [2]. Because of all these reasons it looks sensible to migrate supervisor first:

• it has a tricky use cases, like listening to workspace status updates to promptly update UI state
• Used in each workspace and will generate enough traffic.
• Easy to migrate since it is self contained in each installation.
• Improves reliability by using client side grpc observability.
• Improves security by giving us control of logging.

Value

• Validate usability and reliability of Public API.
• Improve reliability and security of connection between supervisor and Public APi.

Acceptance Criteria

• Supervisor don't use Server API anymore, but completely relies on Public API.

Measurement

• 99% of requests from supervisor to Public API should be successful. SLO

Remaining Work

• Enable PublicAPI in supervisor experimentally, to dog food and check also stability of public API notion
  • Production configcat
• Use workspaceStatus stream rpc in supervisor #15403
  • Split PR into two: public-api and supervisor
  • Complete public-api-server instance update data transform
  • Use instance update in supervisor
• don't log request headers via jsonrpc2 golang library #15039 (closed when epic is delivered)
• server connection observability from supervisor #15041 (closed when epic is delivered)
• SLO for supervisor public-api (requests duration)

Descoped / Nice to have

• Use common-go grpc in supervisor PublicAPI
• Pass gitpod host as configcat contracture argument (don't know if it's worth, since we merged these two constructive method) Fix supervisor panic when tkns and publicapi Service is nil #15274 (comment)
• Add request state logging as a middleware of common-go grpc Use public api in supervisor #15112 (comment)

nit

• make public-api host inside serverapi/Service config Use public api in supervisor #15112 (comment)