[Discover][Traces] Add APM operation IDs to traces log fetching#268452
Merged
iblancof merged 14 commits intoMay 12, 2026
Merged
[Discover][Traces] Add APM operation IDs to traces log fetching#268452iblancof merged 14 commits into
iblancof merged 14 commits into
Conversation
iblancof
added
release_note:skip
|
Pinging @elastic/obs-exploration-team (Team:obs-exploration) |
…//github.com/iblancof/kibana into 5348-traces-in-discover-monitor-transactions
iblancof
changed the title
lukasolson
approved these changes
May 11, 2026
Contributor
lukasolson
left a comment
lukasolson
left a comment
There was a problem hiding this comment.
Minor notes below, LGTM!
…//github.com/iblancof/kibana into 5348-traces-in-discover-monitor-transactions
Contributor
💛 Build succeeded, but was flaky
Failed CI Steps
Test Failures
Metrics [docs]Module Count
Async chunks
Page load bundle
History
cc @iblancof |
Contributor
|
Starting backport for target branches: 9.4 |
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
May 12, 2026
…#268452) (#268809) # Backport This will backport the following commits from `main` to `9.4`: - [[Discover][Traces] Add APM operation IDs to traces log fetching (#268452)](#268452) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Irene Blanco","email":"irene.blanco@elastic.co"},"sourceCommit":{"committedDate":"2026-05-12T07:09:53Z","message":"[Discover][Traces] Add APM operation IDs to traces log fetching (#268452)\n\n## Summary\n\nRelates to https://github.com/elastic/observability-dev/issues/5348\n\nThis PR instruments two log-fetching code paths in the trace document\nviewer with APM execution context and explicit error capture, enabling\nRED metrics (rate, errors, duration) visibility in APM for those\noperations. Without operation IDs, all Kibana search requests appear\nidentically in APM's search/ese spans with no way to attribute latency\nor throughput to a specific feature.\n\n### fetchLogDocumentById\n\n`fetchLogDocumentById` is called when a user opens a log document from\nthe expanded waterfall flyout. Because this function owns the\n`data.search.search` call directly, we can instrument it fully:\n\n- An `executionContext` with `operation_id: fetch-log-by-id` is passed\nto the search call. The Kibana HTTP layer forwards this via the\n`x-kbn-context` header, and the server-side APM agent attaches it as a\n`kibana_meta_operation_id` label on the resulting span.\n- The search call is wrapped in a try-catch that calls\n`apm.captureError` with the same label, so failures appear in APM with\nthe correct operation context.\n- `toasts.addDanger` is replaced with `toasts.add({ color: 'danger', ...\n})` to avoid double error capture: `addDanger` internally calls\n`apm.captureError`, which would produce a duplicate error event\nalongside ours.\n\n### Logs embeddable (TraceContextLogEvents)\n\nThe correlated logs section in the trace document overview renders via\n`LazySavedSearchComponent`, which delegates all fetching to\n`SearchEmbeddable` internally. Because we do not own the fetch call,\nexplicit error capture is not feasible: `SearchEmbeddable` handles\nerrors internally and surfaces them as inline UI state, so failures\nnever propagate as rejections we can catch or as failed HTTP spans in\nAPM.\n\nFor latency and throughput, we thread `executionContext` with\n`operation_id: fetch-trace-context-logs` from `TraceContextLogEvents`\ndown through `LogEventsComponent` and `LazySavedSearchComponent` to\n`SearchEmbeddable`'s `parentApi`. The embeddable reads execution context\nfrom `parentApi` and includes it in every search request, tagging those\nspans with `kibana_meta_operation_id: fetch-trace-context-logs` on the\nserver.\n\nThis required adding `executionContext` as a typed prop to\n`LogEventsProps` and `SavedSearchComponentProps`, and declaring\n`@kbn/core-execution-context-common` in `kbn_references` for both\naffected packages.\n\n### How to test\n\nVerify the logs embeddable span label:\n\n- Start Kibana with a synthtrace scenario that includes log documents\ncorrelated to a trace (e.g. `node scripts/synthtrace\nlogs_traces_hosts`).\n- Open a trace document in the doc viewer and expand the Logs section.\n- In APM, filter search/ese spans by `labels.kibana_meta_operation_id:\nfetch-trace-context-logs` and confirm requests from the logs section\nappear with that label.\n\nVerify the fetch log span label and error capture:\n\n- From the expanded waterfall, click a log item to open the log document\nflyout.\n- In APM, filter by `labels.kibana_meta_operation_id: fetch-log-by-id`\nand confirm the fetch span appears.\n- To test error capture: temporarily break the fetch (e.g. pass an\ninvalid index) and confirm a single APM error appears with\n`labels.kibana_meta_operation_id: fetch-log-by-id`, not two.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"9fca0a10f22c136932d2a8ef8a6c82738897c1ba","branchLabelMapping":{"^v9.5.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","Team:obs-exploration","v9.4.0","v9.5.0"],"title":"[Discover][Traces] Add APM operation IDs to traces log fetching","number":268452,"url":"https://github.com/elastic/kibana/pull/268452","mergeCommit":{"message":"[Discover][Traces] Add APM operation IDs to traces log fetching (#268452)\n\n## Summary\n\nRelates to https://github.com/elastic/observability-dev/issues/5348\n\nThis PR instruments two log-fetching code paths in the trace document\nviewer with APM execution context and explicit error capture, enabling\nRED metrics (rate, errors, duration) visibility in APM for those\noperations. Without operation IDs, all Kibana search requests appear\nidentically in APM's search/ese spans with no way to attribute latency\nor throughput to a specific feature.\n\n### fetchLogDocumentById\n\n`fetchLogDocumentById` is called when a user opens a log document from\nthe expanded waterfall flyout. Because this function owns the\n`data.search.search` call directly, we can instrument it fully:\n\n- An `executionContext` with `operation_id: fetch-log-by-id` is passed\nto the search call. The Kibana HTTP layer forwards this via the\n`x-kbn-context` header, and the server-side APM agent attaches it as a\n`kibana_meta_operation_id` label on the resulting span.\n- The search call is wrapped in a try-catch that calls\n`apm.captureError` with the same label, so failures appear in APM with\nthe correct operation context.\n- `toasts.addDanger` is replaced with `toasts.add({ color: 'danger', ...\n})` to avoid double error capture: `addDanger` internally calls\n`apm.captureError`, which would produce a duplicate error event\nalongside ours.\n\n### Logs embeddable (TraceContextLogEvents)\n\nThe correlated logs section in the trace document overview renders via\n`LazySavedSearchComponent`, which delegates all fetching to\n`SearchEmbeddable` internally. Because we do not own the fetch call,\nexplicit error capture is not feasible: `SearchEmbeddable` handles\nerrors internally and surfaces them as inline UI state, so failures\nnever propagate as rejections we can catch or as failed HTTP spans in\nAPM.\n\nFor latency and throughput, we thread `executionContext` with\n`operation_id: fetch-trace-context-logs` from `TraceContextLogEvents`\ndown through `LogEventsComponent` and `LazySavedSearchComponent` to\n`SearchEmbeddable`'s `parentApi`. The embeddable reads execution context\nfrom `parentApi` and includes it in every search request, tagging those\nspans with `kibana_meta_operation_id: fetch-trace-context-logs` on the\nserver.\n\nThis required adding `executionContext` as a typed prop to\n`LogEventsProps` and `SavedSearchComponentProps`, and declaring\n`@kbn/core-execution-context-common` in `kbn_references` for both\naffected packages.\n\n### How to test\n\nVerify the logs embeddable span label:\n\n- Start Kibana with a synthtrace scenario that includes log documents\ncorrelated to a trace (e.g. `node scripts/synthtrace\nlogs_traces_hosts`).\n- Open a trace document in the doc viewer and expand the Logs section.\n- In APM, filter search/ese spans by `labels.kibana_meta_operation_id:\nfetch-trace-context-logs` and confirm requests from the logs section\nappear with that label.\n\nVerify the fetch log span label and error capture:\n\n- From the expanded waterfall, click a log item to open the log document\nflyout.\n- In APM, filter by `labels.kibana_meta_operation_id: fetch-log-by-id`\nand confirm the fetch span appears.\n- To test error capture: temporarily break the fetch (e.g. pass an\ninvalid index) and confirm a single APM error appears with\n`labels.kibana_meta_operation_id: fetch-log-by-id`, not two.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"9fca0a10f22c136932d2a8ef8a6c82738897c1ba"}},"sourceBranch":"main","suggestedTargetBranches":["9.4"],"targetPullRequestStates":[{"branch":"9.4","label":"v9.4.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.5.0","branchLabelMappingKey":"^v9.5.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/268452","number":268452,"mergeCommit":{"message":"[Discover][Traces] Add APM operation IDs to traces log fetching (#268452)\n\n## Summary\n\nRelates to https://github.com/elastic/observability-dev/issues/5348\n\nThis PR instruments two log-fetching code paths in the trace document\nviewer with APM execution context and explicit error capture, enabling\nRED metrics (rate, errors, duration) visibility in APM for those\noperations. Without operation IDs, all Kibana search requests appear\nidentically in APM's search/ese spans with no way to attribute latency\nor throughput to a specific feature.\n\n### fetchLogDocumentById\n\n`fetchLogDocumentById` is called when a user opens a log document from\nthe expanded waterfall flyout. Because this function owns the\n`data.search.search` call directly, we can instrument it fully:\n\n- An `executionContext` with `operation_id: fetch-log-by-id` is passed\nto the search call. The Kibana HTTP layer forwards this via the\n`x-kbn-context` header, and the server-side APM agent attaches it as a\n`kibana_meta_operation_id` label on the resulting span.\n- The search call is wrapped in a try-catch that calls\n`apm.captureError` with the same label, so failures appear in APM with\nthe correct operation context.\n- `toasts.addDanger` is replaced with `toasts.add({ color: 'danger', ...\n})` to avoid double error capture: `addDanger` internally calls\n`apm.captureError`, which would produce a duplicate error event\nalongside ours.\n\n### Logs embeddable (TraceContextLogEvents)\n\nThe correlated logs section in the trace document overview renders via\n`LazySavedSearchComponent`, which delegates all fetching to\n`SearchEmbeddable` internally. Because we do not own the fetch call,\nexplicit error capture is not feasible: `SearchEmbeddable` handles\nerrors internally and surfaces them as inline UI state, so failures\nnever propagate as rejections we can catch or as failed HTTP spans in\nAPM.\n\nFor latency and throughput, we thread `executionContext` with\n`operation_id: fetch-trace-context-logs` from `TraceContextLogEvents`\ndown through `LogEventsComponent` and `LazySavedSearchComponent` to\n`SearchEmbeddable`'s `parentApi`. The embeddable reads execution context\nfrom `parentApi` and includes it in every search request, tagging those\nspans with `kibana_meta_operation_id: fetch-trace-context-logs` on the\nserver.\n\nThis required adding `executionContext` as a typed prop to\n`LogEventsProps` and `SavedSearchComponentProps`, and declaring\n`@kbn/core-execution-context-common` in `kbn_references` for both\naffected packages.\n\n### How to test\n\nVerify the logs embeddable span label:\n\n- Start Kibana with a synthtrace scenario that includes log documents\ncorrelated to a trace (e.g. `node scripts/synthtrace\nlogs_traces_hosts`).\n- Open a trace document in the doc viewer and expand the Logs section.\n- In APM, filter search/ese spans by `labels.kibana_meta_operation_id:\nfetch-trace-context-logs` and confirm requests from the logs section\nappear with that label.\n\nVerify the fetch log span label and error capture:\n\n- From the expanded waterfall, click a log item to open the log document\nflyout.\n- In APM, filter by `labels.kibana_meta_operation_id: fetch-log-by-id`\nand confirm the fetch span appears.\n- To test error capture: temporarily break the fetch (e.g. pass an\ninvalid index) and confirm a single APM error appears with\n`labels.kibana_meta_operation_id: fetch-log-by-id`, not two.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"9fca0a10f22c136932d2a8ef8a6c82738897c1ba"}}]}] BACKPORT--> Co-authored-by: Irene Blanco <irene.blanco@elastic.co>
patrykkopycinski
pushed a commit
to patrykkopycinski/kibana
that referenced
this pull request
May 13, 2026
…tic#268452) ## Summary Relates to elastic/observability-dev#5348 This PR instruments two log-fetching code paths in the trace document viewer with APM execution context and explicit error capture, enabling RED metrics (rate, errors, duration) visibility in APM for those operations. Without operation IDs, all Kibana search requests appear identically in APM's search/ese spans with no way to attribute latency or throughput to a specific feature. ### fetchLogDocumentById `fetchLogDocumentById` is called when a user opens a log document from the expanded waterfall flyout. Because this function owns the `data.search.search` call directly, we can instrument it fully: - An `executionContext` with `operation_id: fetch-log-by-id` is passed to the search call. The Kibana HTTP layer forwards this via the `x-kbn-context` header, and the server-side APM agent attaches it as a `kibana_meta_operation_id` label on the resulting span. - The search call is wrapped in a try-catch that calls `apm.captureError` with the same label, so failures appear in APM with the correct operation context. - `toasts.addDanger` is replaced with `toasts.add({ color: 'danger', ... })` to avoid double error capture: `addDanger` internally calls `apm.captureError`, which would produce a duplicate error event alongside ours. ### Logs embeddable (TraceContextLogEvents) The correlated logs section in the trace document overview renders via `LazySavedSearchComponent`, which delegates all fetching to `SearchEmbeddable` internally. Because we do not own the fetch call, explicit error capture is not feasible: `SearchEmbeddable` handles errors internally and surfaces them as inline UI state, so failures never propagate as rejections we can catch or as failed HTTP spans in APM. For latency and throughput, we thread `executionContext` with `operation_id: fetch-trace-context-logs` from `TraceContextLogEvents` down through `LogEventsComponent` and `LazySavedSearchComponent` to `SearchEmbeddable`'s `parentApi`. The embeddable reads execution context from `parentApi` and includes it in every search request, tagging those spans with `kibana_meta_operation_id: fetch-trace-context-logs` on the server. This required adding `executionContext` as a typed prop to `LogEventsProps` and `SavedSearchComponentProps`, and declaring `@kbn/core-execution-context-common` in `kbn_references` for both affected packages. ### How to test Verify the logs embeddable span label: - Start Kibana with a synthtrace scenario that includes log documents correlated to a trace (e.g. `node scripts/synthtrace logs_traces_hosts`). - Open a trace document in the doc viewer and expand the Logs section. - In APM, filter search/ese spans by `labels.kibana_meta_operation_id: fetch-trace-context-logs` and confirm requests from the logs section appear with that label. Verify the fetch log span label and error capture: - From the expanded waterfall, click a log item to open the log document flyout. - In APM, filter by `labels.kibana_meta_operation_id: fetch-log-by-id` and confirm the fetch span appears. - To test error capture: temporarily break the fetch (e.g. pass an invalid index) and confirm a single APM error appears with `labels.kibana_meta_operation_id: fetch-log-by-id`, not two. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jcger
pushed a commit
that referenced
this pull request
May 26, 2026
) ## Summary Relates to elastic/observability-dev#5348 This PR instruments two log-fetching code paths in the trace document viewer with APM execution context and explicit error capture, enabling RED metrics (rate, errors, duration) visibility in APM for those operations. Without operation IDs, all Kibana search requests appear identically in APM's search/ese spans with no way to attribute latency or throughput to a specific feature. ### fetchLogDocumentById `fetchLogDocumentById` is called when a user opens a log document from the expanded waterfall flyout. Because this function owns the `data.search.search` call directly, we can instrument it fully: - An `executionContext` with `operation_id: fetch-log-by-id` is passed to the search call. The Kibana HTTP layer forwards this via the `x-kbn-context` header, and the server-side APM agent attaches it as a `kibana_meta_operation_id` label on the resulting span. - The search call is wrapped in a try-catch that calls `apm.captureError` with the same label, so failures appear in APM with the correct operation context. - `toasts.addDanger` is replaced with `toasts.add({ color: 'danger', ... })` to avoid double error capture: `addDanger` internally calls `apm.captureError`, which would produce a duplicate error event alongside ours. ### Logs embeddable (TraceContextLogEvents) The correlated logs section in the trace document overview renders via `LazySavedSearchComponent`, which delegates all fetching to `SearchEmbeddable` internally. Because we do not own the fetch call, explicit error capture is not feasible: `SearchEmbeddable` handles errors internally and surfaces them as inline UI state, so failures never propagate as rejections we can catch or as failed HTTP spans in APM. For latency and throughput, we thread `executionContext` with `operation_id: fetch-trace-context-logs` from `TraceContextLogEvents` down through `LogEventsComponent` and `LazySavedSearchComponent` to `SearchEmbeddable`'s `parentApi`. The embeddable reads execution context from `parentApi` and includes it in every search request, tagging those spans with `kibana_meta_operation_id: fetch-trace-context-logs` on the server. This required adding `executionContext` as a typed prop to `LogEventsProps` and `SavedSearchComponentProps`, and declaring `@kbn/core-execution-context-common` in `kbn_references` for both affected packages. ### How to test Verify the logs embeddable span label: - Start Kibana with a synthtrace scenario that includes log documents correlated to a trace (e.g. `node scripts/synthtrace logs_traces_hosts`). - Open a trace document in the doc viewer and expand the Logs section. - In APM, filter search/ese spans by `labels.kibana_meta_operation_id: fetch-trace-context-logs` and confirm requests from the logs section appear with that label. Verify the fetch log span label and error capture: - From the expanded waterfall, click a log item to open the log document flyout. - In APM, filter by `labels.kibana_meta_operation_id: fetch-log-by-id` and confirm the fetch span appears. - To test error capture: temporarily break the fetch (e.g. pass an invalid index) and confirm a single APM error appears with `labels.kibana_meta_operation_id: fetch-log-by-id`, not two. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relates to https://github.com/elastic/observability-dev/issues/5348
This PR instruments two log-fetching code paths in the trace document viewer with APM execution context and explicit error capture, enabling RED metrics (rate, errors, duration) visibility in APM for those operations. Without operation IDs, all Kibana search requests appear identically in APM's search/ese spans with no way to attribute latency or throughput to a specific feature.
fetchLogDocumentById
fetchLogDocumentByIdis called when a user opens a log document from the expanded waterfall flyout. Because this function owns thedata.search.searchcall directly, we can instrument it fully:executionContextwithoperation_id: fetch-log-by-idis passed to the search call. The Kibana HTTP layer forwards this via thex-kbn-contextheader, and the server-side APM agent attaches it as akibana_meta_operation_idlabel on the resulting span.apm.captureErrorwith the same label, so failures appear in APM with the correct operation context.toasts.addDangeris replaced withtoasts.add({ color: 'danger', ... })to avoid double error capture:addDangerinternally callsapm.captureError, which would produce a duplicate error event alongside ours.Logs embeddable (TraceContextLogEvents)
The correlated logs section in the trace document overview renders via
LazySavedSearchComponent, which delegates all fetching toSearchEmbeddableinternally. Because we do not own the fetch call, explicit error capture is not feasible:SearchEmbeddablehandles errors internally and surfaces them as inline UI state, so failures never propagate as rejections we can catch or as failed HTTP spans in APM.For latency and throughput, we thread
executionContextwithoperation_id: fetch-trace-context-logsfromTraceContextLogEventsdown throughLogEventsComponentandLazySavedSearchComponenttoSearchEmbeddable'sparentApi. The embeddable reads execution context fromparentApiand includes it in every search request, tagging those spans withkibana_meta_operation_id: fetch-trace-context-logson the server.This required adding
executionContextas a typed prop toLogEventsPropsandSavedSearchComponentProps, and declaring@kbn/core-execution-context-commoninkbn_referencesfor both affected packages.How to test
Verify the logs embeddable span label:
node scripts/synthtrace logs_traces_hosts).labels.kibana_meta_operation_id: fetch-trace-context-logsand confirm requests from the logs section appear with that label.Verify the fetch log span label and error capture:
labels.kibana_meta_operation_id: fetch-log-by-idand confirm the fetch span appears.labels.kibana_meta_operation_id: fetch-log-by-id, not two.