Telemetry and Application Monitoring ADR

[coilysiren]

Ticket

Resolves #715

[lorenyu]

i think the ADR is conflating two things:

1. whether or not to use the OpenTelemetry protocol vs use vendor-specific protocols
2. which implementation of the OpenTelemetry protocol to use

i think those two things should be more clearly separated. for "whether to use OTLP", i think it's clear we should use OpenTelemetry protocol because:

• it abstracts away the monitoring service which decouples the monitoring service from the application, reducing lock-in
• most monitoring services accept it and some require it
  the downside is that (a) there's an extra layer of indirection which adds some complexity to the architecture and (b) in many cases requires a separate container rather than direct integration with the monitoring service through an SDK. (i'm not sure about (b) you may want to verify my reasoning).

for "which implementation of OTLP", fluentbit says they support the opentelemetry protocol so i think it's a choice between OpenTelemetry Collector and Fluent Bit (not sure if there are others). i'd like to see some pros/cons of the implementations, and happy to go with whatever you choose. can also ask others at Nava to see what they think.