[Kibana Debugging] Improve docs and make it easier to configure OTel + RUM (elastic#270588)

## Summary

Since we moved to OTel Traces, we should update our existing docs
(https://www.elastic.co/docs/extend/kibana/kibana-debugging#_instrumenting_with_elastic_apm)
with how to set up OTel in Kibana.

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [x] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

---------

Co-authored-by: Gerard Soldevila <gerard.soldevila@elastic.co>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 3 people

docs/extend/kibana-debugging.md

@@ -7,19 +7,98 @@ mapped_pages:
 
 For information about how to debug unit tests, refer to [Debugging Unit Tests](/extend/development-tests.md#debugging-unit-tests).
 
-
 ## Server Code [_server_code]
 
 `yarn debug` will start the server with Node’s inspect flag. {{kib}}'s development mode will start three processes on ports `9229`, `9230`, and `9231`. Chrome’s developer tools need to be configured to connect to all three connections. Add `localhost:<port>` for each {{kib}} process in Chrome’s developer tools connection tab.
 
+## Instrumenting with OTel Traces [_instrumenting_with_otel_traces]
+
+{{kib}} has built-in OpenTelemetry instrumentation for debugging and observability. The following sections cover tracing configuration.
+
+To enable OTel traces, apply the following configuration:
+
+```yaml
+telemetry.tracing:
+  enabled: true
+  sample_rate: 1 # 1 by default
+  exporters:
+    - proto:
+        url: <URL_TO_THE_OTLP_ENDPOINT>
+        headers:
+          authorization: 'ApiKey [REDACTED]'
+```
+
+The OTLP endpoint can be any OTel/EDOT collector. It is recommended to use the mOTLP (Managed OTLP) endpoint that is provided by Elastic Cloud. The instructions to retrieve the OTLP endpoint and the API Key are available in [Get started with traces and APM](https://www.elastic.co/docs/solutions/observability/apm/get-started).
+
+### Supported exporter protocols
+
+{{kib}} supports gRPC, protobuf and plain HTTP protocols to export the OTel Traces. The protocol to use is specified in the config via top property name in the exporter's declaration. The config syntax is the same for all.
+
+```yaml
+telemetry.tracing:
+  enabled: true
+  sample_rate: 1 # 1 by default
+  exporters:
+    - grpc:
+        url: <URL_TO_THE_GRPC_OTLP_ENDPOINT>
+        headers:
+          authorization: 'ApiKey [REDACTED]'
+    - proto:
+        url: <URL_TO_THE_PROTO_OTLP_ENDPOINT>
+        headers:
+          authorization: 'ApiKey [REDACTED]'
+    - http:
+        url: <URL_TO_THE_HTTP_OTLP_ENDPOINT>
+        headers:
+          authorization: 'ApiKey [REDACTED]'
+```
+
+:::::{tip}
+gRPC typically operates on a different port to the protobuf and http protocols.
+
+When using the mOTLP endpoint, the port is the same, but the endpoint changes:
+
+- gRPC uses the root path (https://my-ech-deployment.ingest.europe-west1.gcp.elastic-cloud.com:443)
+- Protobuf and HTTP use the path `/v1/traces` (https://my-ech-deployment.ingest.europe-west1.gcp.elastic-cloud.com:443/v1/traces)
+:::::
+
+### Enable OTel Traces on the server + Elastic RUM on the browser in {{kib}}
+
+:::::{important}
+OTel instrumentation is only available on the server side. For RUM observability, {{kib}} uses [Elastic RUM](https://www.elastic.co/docs/solutions/observability/apm/apm-agents/real-user-monitoring-rum).
+
+OTel instrumentation in the browser will be available in the future, once the [OTel for RUM](https://www.elastic.co/docs/solutions/observability/applications/otel-rum) is ready for production.
+:::::
+
+When `telemetry.tracing.enabled` is `true`, server-side Elastic APM is disabled by default, but the `kibana-frontend` (RUM) service remains active. Elastic APM and OpenTelemetry tracing cannot be enabled simultaneously; setting `elastic.apm.active: true` while OTel tracing is enabled causes Kibana to fail on startup. You can still collect browser traces with Elastic RUM.
+
+To enable Elastic RUM alongside OTel tracing, define the APM Server's URL in the Kibana config. [Any settings accepted by the agent](https://www.elastic.co/docs/reference/apm/agents/rum-js/configuration) are also accepted:
+
+```yaml
+elastic:
+  apm:
+    serverUrl: https://my-ech-deployment.apm.europe-west1.gcp.cloud.es.io:443
+    # RUM is enabled by default when telemetry.tracing.enabled is true; serverUrl is required to send data to your deployment.
+    # Below are optional
+    environment: localhost
+    transactionSampleRate: 1
+```
+
+:::::{tip}
+RUM sends traces from the browser without embedded credentials. Prefer an Elastic Cloud Hosted (ECH) APM endpoint configured for RUM intake. Serverless APM endpoints often require authenticated intake and are usually not suitable for RUM unless your deployment explicitly supports unauthenticated browser intake.
+:::::
 
 ## Instrumenting with Elastic APM [_instrumenting_with_elastic_apm]
 
+:::::{warning}
+This approach is deprecated. Use [OTel Traces](#_instrumenting_with_otel_traces) instead.
+:::::
+
 {{kib}} ships with the [Elastic APM Node.js Agent](https://github.com/elastic/apm-agent-nodejs) built-in for debugging purposes.
 
-With an application as varied and complex as Kibana has become, it’s not practical or scalable to craft all possible performance measurements by hand ahead of time. As such, we need to rely on tooling to help us catch things we may otherwise have missed.
+With an application as varied and complex as Kibana has become, it’s not practical or scalable to craft all possible performance measurements by hand ahead of time. As such, we need to rely on tooling to help us catch things we might otherwise have missed.
 
-For example, say you implement a brand new feature, plugin or service but don’t quite know how it will impact Kibana’s performance as a whole. APM allows us to not only spot that something is slow, but also hints at why it might be performing slowly. For example, if a function is slow on specific types of inputs, we can see where the time is spent by viewing the trace for that function call in the APM UI.
+For example, say you implement a brand new feature, plugin or service but do not know how it will impact Kibana’s performance as a whole. APM allows us to not only spot that something is slow, but also hints at why it might be performing slowly. For example, if a function is slow on specific types of inputs, we can see where the time is spent by viewing the trace for that function call in the APM UI.
 
 ![apm example trace](images/apm_example_trace.png)
 
@@ -47,57 +126,56 @@ APM [Real User Monitoring agent](apm-agent-rum-js://reference/index.md) is not a
 ELASTIC_APM_ACTIVE=true yarn start
 // activates both Node.js and RUM agent
 ```
-Once the agent is active, it will trace all incoming HTTP requests to {{kib}}, monitor for errors, and collect process-level metrics. The collected data will be sent to the APM Server and is viewable in the APM UI in {{kib}}.
 
+Once the agent is active, it will trace all incoming HTTP requests to {{kib}}, monitor for errors, and collect process-level metrics. The collected data will be sent to the APM Server and is viewable in the APM UI in {{kib}}.
 
 ## Running Kibana with the APM Agent Locally [_running_kibana_with_the_apm_agent_locally]
 
 The easiest and recommended way of running Kibana with the APM agent locally is to use the solution provided by the [apm-integration-testing](https://github.com/elastic/apm-integration-testing) repo. You’ll need [Docker](https://www.docker.com/community-edition), [Docker Compose](https://docs.docker.com/compose/install/) and [Python (version 3 preferred)](https://www.python.org/downloads) to use the tool.
 
-
 ### Quick start guide [_quick_start_guide]
 
 1. Clone the [elastic/apm-integration-testing](https://github.com/elastic/apm-integration-testing) repo.
 2. Change into the apm-integration-testing repo:
 
-    ```bash
-    cd apm-integration-testing
-    ```
+   ```bash
+   cd apm-integration-testing
+   ```
 
 3. Run {{es}} and the APM servers without running Kibana:
 
-    ```bash
-    ./scripts/compose.py start master --no-kibana
-    ```
+   ```bash
+   ./scripts/compose.py start master --no-kibana
+   ```
 
 4. Clone the [elastic/kibana](https://github.com/elastic/kibana) repo.
 5. Change into the {{kib}} repo:
 
-    ```bash
-    cd ../kibana
-    ```
+   ```bash
+   cd ../kibana
+   ```
 
 6. Change the elasticsearch credentials in your `kibana.yml` configuration file to match those needed by elasticsearch and the APM server (see the apm-integration-testing repo’s [README](https://github.com/elastic/apm-integration-testing#logging-in) for users provided to test different scenarios).
 7. Make sure that the APM agent is active and points to the local APM server by adding the following configuration settings to a config file under `config/kibana.dev.yml`:
 
-    Example `config/kibana.dev.yml` file:
+   Example `config/kibana.dev.yml` file:
 
-    ```yaml
-    elastic:
-      apm:
-        active: true
-        serverUrl: http://localhost:8200
-        secretToken: very_secret
-        centralConfig: true
-        breakdownMetrics: true
-        transactionSampleRate: 0.1
-    ```
+   ```yaml
+   elastic:
+     apm:
+       active: true
+       serverUrl: http://localhost:8200
+       secretToken: very_secret
+       centralConfig: true
+       breakdownMetrics: true
+       transactionSampleRate: 0.1
+   ```
 
 8. Start Kibana with APM active using:
 
-    ```bash
-    yarn start
-    ```
+   ```bash
+   yarn start
+   ```
 
 9. After Kibana starts up, navigate to the APM app, where you should see some transactions.
 
@@ -108,4 +186,3 @@ You can now continue doing what you want to in Kibana (e.g. install sample data
 ```bash
 ./scripts/compose.py stop
 ```
-

src/platform/packages/private/kbn-apm-config-loader/src/config.test.ts

@@ -17,6 +17,7 @@ import {
 } from './config.test.mocks';
 
 import { ApmConfiguration, CENTRALIZED_SERVICE_BASE_CONFIG } from './config';
+import { shouldInstrumentClient } from './rum_agent_configuration';
 
 describe('ApmConfiguration', () => {
   beforeEach(() => {
@@ -222,13 +223,10 @@ describe('ApmConfiguration', () => {
     it('ELASTIC_APM_KIBANA_FRONTEND_ACTIVE', () => {
       process.env.ELASTIC_APM_KIBANA_FRONTEND_ACTIVE = 'false';
       const config = new ApmConfiguration(mockedRootDir, {}, false);
-      const serverConfig = config.getConfig('servicesOverrides');
-      // @ts-ignore
-      expect(serverConfig.servicesOverrides).toEqual({
-        'kibana-frontend': {
-          active: false,
-        },
-      });
+      const frontendConfig = config.getConfig('kibana-frontend');
+      expect(frontendConfig.active).toBe(false);
+      expect(frontendConfig).not.toHaveProperty('servicesOverrides');
+      expect(shouldInstrumentClient(frontendConfig)).toBe(false);
     });
 
     it('does not override the environment from NODE_ENV if already set in the config file', () => {
@@ -476,4 +474,67 @@ describe('ApmConfiguration', () => {
       expect(config.isUsersRedactionEnabled()).toEqual(false);
     });
   });
+
+  describe('OTel tracing affects the default config', () => {
+    let config: ApmConfiguration;
+    const kibanaConfig = {
+      telemetry: {
+        enabled: true,
+        tracing: {
+          enabled: true,
+          sample_rate: 1,
+          exporters: [],
+        },
+      },
+    };
+
+    beforeAll(() => {
+      config = new ApmConfiguration(mockedRootDir, kibanaConfig, false);
+    });
+
+    it('sets "active: false" for the server service', () => {
+      const serverConfig = config.getConfig('serviceName');
+      expect(serverConfig.active).toBe(false);
+      expect(serverConfig).not.toHaveProperty('servicesOverrides');
+    });
+
+    it('sets "active: true" for the "kibana-frontend" service', () => {
+      const frontendConfig = config.getConfig('kibana-frontend');
+      expect(frontendConfig.active).toBe(true);
+      expect(frontendConfig).not.toHaveProperty('servicesOverrides');
+      expect(shouldInstrumentClient(frontendConfig)).toBe(true);
+    });
+
+    it('disables kibana-frontend when ELASTIC_APM_KIBANA_FRONTEND_ACTIVE=false', () => {
+      process.env.ELASTIC_APM_KIBANA_FRONTEND_ACTIVE = 'false';
+      const configWithEnv = new ApmConfiguration(mockedRootDir, kibanaConfig, false);
+      const frontendConfig = configWithEnv.getConfig('kibana-frontend');
+      expect(frontendConfig.active).toBe(false);
+      expect(frontendConfig).not.toHaveProperty('servicesOverrides');
+      expect(shouldInstrumentClient(frontendConfig)).toBe(false);
+    });
+
+    it('disables kibana-frontend when overridden in elastic.apm.servicesOverrides', () => {
+      const configWithYamlOverride = new ApmConfiguration(
+        mockedRootDir,
+        {
+          ...kibanaConfig,
+          elastic: {
+            apm: {
+              servicesOverrides: {
+                'kibana-frontend': {
+                  active: false,
+                },
+              },
+            },
+          },
+        },
+        false
+      );
+      const frontendConfig = configWithYamlOverride.getConfig('kibana-frontend');
+      expect(frontendConfig.active).toBe(false);
+      expect(frontendConfig).not.toHaveProperty('servicesOverrides');
+      expect(shouldInstrumentClient(frontendConfig)).toBe(false);
+    });
+  });
 });

src/platform/packages/private/kbn-apm-config-loader/src/config.ts

@@ -67,8 +67,12 @@ interface KibanaRawConfig {
   };
 }
 
+type ApmBaseConfiguration = AgentConfigOptions & {
+  servicesOverrides?: Record<string, AgentConfigOptions>;
+};
+
 export class ApmConfiguration {
-  private baseConfig?: AgentConfigOptions;
+  private baseConfig?: ApmBaseConfiguration;
   private kibanaVersion: string;
   private pkgBuild: Record<string, any>;
 
@@ -92,12 +96,17 @@ export class ApmConfiguration {
       serviceName,
     };
 
-    const serviceOverride = servicesOverrides[serviceName];
+    const serviceOverride = merge(
+      {},
+      baseConfig.servicesOverrides?.[serviceName],
+      servicesOverrides[serviceName]
+    );
     if (serviceOverride) {
       baseConfig = merge({}, baseConfig, serviceOverride);
     }
 
-    return baseConfig;
+    const { servicesOverrides: _servicesOverrides, ...resolvedConfig } = baseConfig;
+    return resolvedConfig;
   }
 
   public getTelemetryConfig(): TelemetryConfig {
@@ -123,6 +132,7 @@ export class ApmConfiguration {
           serviceVersion: this.kibanaVersion,
         },
         DEFAULT_CONFIG,
+        this.getElasticAPMConfigDerivedFromOTelConfig(),
         this.getUuidConfig(),
         this.getGitConfig(),
         this.getCiConfig(),
@@ -158,6 +168,32 @@ export class ApmConfiguration {
     return this.baseConfig;
   }
 
+  /**
+   * Retrieve the default Elastic APM config derived from the OTel config.
+   * Only applies when `telemetry.tracing.enabled` is true; otherwise returns an empty object.
+   *
+   * @returns The default Elastic APM config derived from the OTel config.
+   */
+  private getElasticAPMConfigDerivedFromOTelConfig(): ApmBaseConfiguration {
+    const telemetryConfig = this.getTelemetryConfig();
+    if (telemetryConfig.tracing.enabled) {
+      // We want to disable Elastic APM if OTel tracing is enabled.
+      // Note that this is only used for calculating our default config. The user can still override these.
+      return {
+        active: false,
+        contextPropagationOnly: false,
+        servicesOverrides: {
+          // Keep RUM active by default
+          'kibana-frontend': {
+            active: true,
+          },
+        },
+      };
+    }
+
+    return {};
+  }
+
   /**
    * Override some config values when specific environment variables are used
    */