Added kafkaQueueProblems scenarios#267402

Merged

yuliia-fryshko merged 4 commits into

elastic:mainfrom

yuliia-fryshko:add-evals-kafka-scenarios-554

May 5, 2026

Contributor

yuliia-fryshko commented May 4, 2026 •

edited by kibanamachine

Loading

Closes https://github.com/elastic/obs-ai-team/issues/554

Add observability AI eval tests covering the Kafka queue problems flow.


          Added kafkaQueueProblems scenarios

4a28ebb

yuliia-fryshko self-assigned this

yuliia-fryshko added the release_note:skip label

yuliia-fryshko requested a review from a team as a code owner

May 4, 2026 08:13

yuliia-fryshko added backport:version v9.4.0 Team:obs-ai-team labels


          Merge branch 'main' into add-evals-kafka-scenarios-554

22c4c63

yuliia-fryshko added evals:observability-ai models:judge:eis/google-gemini-3.1-pro models:weekly-eis-models labels


          Merge branch 'main' into add-evals-kafka-scenarios-554

7c50bff

SrdjanLL approved these changes

View reviewed changes

Contributor

SrdjanLL left a comment

LGTM (code review only)!

Before merging, is there anything worth noting about evals results? Are all criteria consistently satisfied and right tools/skill invoked?

Asking because the related issue mentions:

Make improvements to the skill if necessary.


          corrected test message

de6f6f4

This comment was marked as outdated.

Sign in to view

Contributor Author

yuliia-fryshko commented May 5, 2026

LGTM (code review only)!

Before merging, is there anything worth noting about evals results? Are all criteria consistently satisfied and right tools/skill invoked?

Asking because the related issue mentions:

Make improvements to the skill if necessary.

Thank you, @SrdjanLL !
I post the results here.
As you can see, the Criteria results are low, and the explanation indicates that it couldn’t identify the issue in the upstream service. This should be addressed in this issue https://github.com/elastic/obs-ai-team/issues/569.

The output correctly identifies that the slowness is due to an artificial sleep in the fraud-detection service (the consumer) rather than a real Kafka delay, attributing the issue to the consumer side. "mjhq": The output does not identify the upstream producer (checkout) or mention it at all. "zudi": The output does not identify a throughput mismatch between checkout and fraud-detection. "exlp": The output does not mention checkout logs or excessive Kafka messages per order. "fuvk": The output only suggests actions for the fraud-detection service and does not mention investigating the checkout service for excessive duplicate messages.

Screenshot 2026-05-05 at 16 01 00

yuliia-fryshko merged commit b5db3b5 into elastic:main

24 checks passed

kibanamachine added the v9.5.0 label

Contributor

kibanamachine commented May 5, 2026

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/25381379199

Contributor

kibanamachine commented May 5, 2026

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 267402

Questions ?

Please refer to the Backport tool documentation

viduni94 added backport:skip backport:version v9.4.0 and removed backport:version v9.4.0 backport:skip labels

Contributor

kibanamachine commented May 5, 2026

Starting backport for target branches: 9.4

https://github.com/elastic/kibana/actions/runs/25386121464

Contributor

kibanamachine commented May 5, 2026

💔 All backports failed

Status	Branch	Result
❌	9.4	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 267402

Questions ?

Please refer to the Backport tool documentation

kibanamachine mentioned this pull request

feat(eslint): add no_sync_import_from_plugin rule (closes #171080) #266636

Merged

2 tasks

kibanamachine added the backport missing label

Contributor

kibanamachine commented May 7, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

7 similar comments

Contributor

kibanamachine commented May 8, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 11, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 12, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 13, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 14, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 15, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 18, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

yuliia-fryshko mentioned this pull request

Added kafkaQueueProblems scenarios #269901

Merged

yuliia-fryshko added a commit that referenced this pull request


          Add AI Insight Feedback Telemetry (#251124) (#269901)

e6478c5

### Backport
This will backport the following commits from main to 9.4:

[Added kafkaQueueProblems scenarios
#267402](#267402)


### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Viduni Wickramarachchi <viduni.ushanka@gmail.com>

Contributor

kibanamachine commented May 19, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

7 similar comments

Contributor

kibanamachine commented May 20, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 21, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 22, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 25, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 26, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 28, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Contributor

kibanamachine commented May 29, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

yuliia-fryshko removed backport missing backport:version v9.4.0 v9.5.0 labels

kibanamachine added the backport missing label

Contributor

kibanamachine commented Jun 1, 2026

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 267402 locally
cc: @yuliia-fryshko

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport missing evals:observability-ai models:judge:eis/google-gemini-3.1-pro models:weekly-eis-models release_note:skip Team:obs-ai-team