[onechat] add first set of base tools (elastic#223367)

## Summary

Fix elastic/search-team#10121

Add a base set of retrieval oriented tools, and expose them to the
default agent.

## Tools

**Warning: names are still TBD**

We're starting to see two different "layers" of tool appear: "simple"
tools, which are doing a simple programmatic (understand: no LLM) task,
and "smart" tools, which are more like workflows, with some of the steps
relying on a LLM.

This PR introduces the following tools:

### Simple tools

Simple tools (name TBD) are "programmatic" tools not relying on an LLM
for their execution.

This PR introduces this base set of tools:

- `get_document_by_id`: resolve a full document based on its id/index.
- `list_indices`: list the indices the current user has access to.
- `get_index_mappings`: retrieve the full mappings based for a given
index.
- `execute_esql`: executes a provided ES|QL query

### Smart tools

Smart tools can have multiple internal steps (even if it remains an
implementation detail), and are using LLM calls for some, or all, of
them.

*Note: there are huge potential areas of improvement in the current
implementation of all those smart tools. One of the intent of this work
is precisely to identify such areas of improvement*

#### `index_explorer`

Based on a natural language query, returns a list of indices that should
be searched, and their corresponding mappings.

<img width="984" alt="Screenshot 2025-06-17 at 16 44 32"
src="https://github.com/user-attachments/assets/edff3964-31e3-40ea-a761-adf7c45fcb17"
/>


#### `generate_esql`

Based on a natural language query, generates an ES|QL query.

- use the `nl-2-esql` task under the hood
- optional use `index-explorer` if `index` is not specified.

<img width="891" alt="Screenshot 2025-06-17 at 16 51 56"
src="https://github.com/user-attachments/assets/ce141d6b-dd4f-4eb9-ab32-823b81bc810b"
/>


#### `relevance_search`

Perform a "full-text search" based on given term and returns the most
relevant highlights.

<img width="1071" alt="Screenshot 2025-06-17 at 16 59 49"
src="https://github.com/user-attachments/assets/1f873e70-e277-424d-93e4-24b269a554e5"
/>

#### `natural_language_search`

Retrieve data based on a natural language query.

Converts a natural language query to an ES|QL one then executes it,
useing `generate_esql` and `execute_esql` under the hood.

<img width="768" alt="Screenshot 2025-06-18 at 08 31 52"
src="https://github.com/user-attachments/assets/cb319831-17ed-4ad7-9e1f-2fe90c2472fa"
/>

## Researcher assistant

The second part of this PR is implementing a researcher agent for deep
research tasks.

The researcher assistant is following a very classic
"act->process->reflect" cycle.

<img width="960" alt="Screenshot 2025-06-18 at 09 16 17"
src="https://github.com/user-attachments/assets/c24be323-ecf2-4c43-88fb-eaf874b18afc"
/>

The implementation of the cycle is currently as follow:

**1. Act**

Given a research topic, the research history and a list of tools, select
the tool best suited to search for this topic, and call it.

The tools exposed to the agent in this phase are:
- `index_explorer`
- `relevance_search`
- `nl_search`

*Note: later the whole `act` step could evolve to instead call sub
search agent with planning and multi-step execution.*

**2. Process**

Process the results from the latest `act` phase and create a
corresponding entry in the search log.

At the moment, we're simply storing the whole tool call + results to the
search log in a LLM-friendly format.

**3. Reflect**

Based on the main research query and the search log, identify where the
information collected are enough to answer the question. If not,
identify follow-up questions or sub-problems that it would be useful to
solve to gather more information

## What is out of scope of the current PR

- Figuring out which set of tools should be exposed by default to the
main agent (right now, all the tools listed in this PR are)

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

Author: pgayvallet

.github/CODEOWNERS

@@ -868,6 +868,7 @@ x-pack/platform/packages/shared/ml/runtime_field_utils @elastic/ml-ui
 x-pack/platform/packages/shared/ml/trained_models_utils @elastic/ml-ui
 x-pack/platform/packages/shared/onechat/onechat-browser @elastic/workchat-eng
 x-pack/platform/packages/shared/onechat/onechat-common @elastic/workchat-eng
+x-pack/platform/packages/shared/onechat/onechat-genai-utils @elastic/workchat-eng
 x-pack/platform/packages/shared/onechat/onechat-server @elastic/workchat-eng
 x-pack/platform/packages/shared/security/api_key_management @elastic/kibana-security
 x-pack/platform/packages/shared/security/form_components @elastic/kibana-security

package.json

@@ -739,6 +739,7 @@
     "@kbn/oidc-provider-plugin": "link:x-pack/test/security_api_integration/plugins/oidc_provider",
     "@kbn/onechat-browser": "link:x-pack/platform/packages/shared/onechat/onechat-browser",
     "@kbn/onechat-common": "link:x-pack/platform/packages/shared/onechat/onechat-common",
+    "@kbn/onechat-genai-utils": "link:x-pack/platform/packages/shared/onechat/onechat-genai-utils",
     "@kbn/onechat-plugin": "link:x-pack/platform/plugins/shared/onechat",
     "@kbn/onechat-server": "link:x-pack/platform/packages/shared/onechat/onechat-server",
     "@kbn/open-telemetry-instrumented-plugin": "link:src/platform/test/common/plugins/otel_metrics",

tsconfig.base.json

@@ -1406,6 +1406,8 @@
       "@kbn/onechat-browser/*": ["x-pack/platform/packages/shared/onechat/onechat-browser/*"],
       "@kbn/onechat-common": ["x-pack/platform/packages/shared/onechat/onechat-common"],
       "@kbn/onechat-common/*": ["x-pack/platform/packages/shared/onechat/onechat-common/*"],
+      "@kbn/onechat-genai-utils": ["x-pack/platform/packages/shared/onechat/onechat-genai-utils"],
+      "@kbn/onechat-genai-utils/*": ["x-pack/platform/packages/shared/onechat/onechat-genai-utils/*"],
       "@kbn/onechat-plugin": ["x-pack/platform/plugins/shared/onechat"],
       "@kbn/onechat-plugin/*": ["x-pack/platform/plugins/shared/onechat/*"],
       "@kbn/onechat-server": ["x-pack/platform/packages/shared/onechat/onechat-server"],

x-pack/platform/packages/shared/onechat/onechat-common/agents/events.ts

@@ -12,6 +12,7 @@ import type { StructuredToolIdentifier } from '../tools/tools';
 export enum ChatAgentEventType {
   toolCall = 'toolCall',
   toolResult = 'toolResult',
+  reasoning = 'reasoning',
   messageChunk = 'messageChunk',
   messageComplete = 'messageComplete',
   roundComplete = 'roundComplete',
@@ -53,6 +54,18 @@ export const isToolResultEvent = (event: OnechatEvent<string, any>): event is To
   return event.type === ChatAgentEventType.toolResult;
 };
 
+// reasoning
+
+export interface ReasoningEventData {
+  reasoning: string;
+}
+
+export type ReasoningEvent = ChatAgentEventBase<ChatAgentEventType.reasoning, ReasoningEventData>;
+
+export const isReasoningEvent = (event: OnechatEvent<string, any>): event is ReasoningEvent => {
+  return event.type === ChatAgentEventType.reasoning;
+};
+
 // Message chunk
 
 export interface MessageChunkEventData {
@@ -116,6 +129,7 @@ export const isRoundCompleteEvent = (
 export type ChatAgentEvent =
   | ToolCallEvent
   | ToolResultEvent
+  | ReasoningEvent
   | MessageChunkEvent
   | MessageCompleteEvent
   | RoundCompleteEvent;

x-pack/platform/packages/shared/onechat/onechat-common/agents/index.ts

@@ -19,6 +19,8 @@ export {
   type ToolResultEventData,
   type ToolCallEvent,
   type ToolCallEventData,
+  type ReasoningEvent,
+  type ReasoningEventData,
   type MessageChunkEventData,
   type MessageChunkEvent,
   type MessageCompleteEventData,
@@ -27,6 +29,7 @@ export {
   type RoundCompleteEvent,
   isToolCallEvent,
   isToolResultEvent,
+  isReasoningEvent,
   isMessageChunkEvent,
   isMessageCompleteEvent,
   isRoundCompleteEvent,

x-pack/platform/packages/shared/onechat/onechat-common/index.ts

@@ -23,6 +23,8 @@ export {
   createBuiltinToolId,
   builtinToolProviderId,
   unknownToolProviderId,
+  BuiltinToolIds,
+  BuiltinTags,
 } from './tools';
 export {
   OnechatErrorCode,

x-pack/platform/packages/shared/onechat/onechat-common/tools/constants.ts

@@ -0,0 +1,31 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+/**
+ * Ids of built-in onechat tools
+ */
+export const BuiltinToolIds = {
+  indexExplorer: 'index_explorer',
+  relevanceSearch: 'relevance_search',
+  naturalLanguageSearch: 'nl_search',
+  listIndices: 'list_indices',
+  getIndexMapping: 'get_index_mapping',
+  getDocumentById: 'get_document_by_id',
+  generateEsql: 'generate_esql',
+  executeEsql: 'execute_esql',
+  researcherAgent: 'researcher_agent',
+};
+
+/**
+ * Common set of tags used for platform tools.
+ */
+export const BuiltinTags = {
+  /**
+   * Tag associated to tools related to data retrieval
+   */
+  retrieval: 'retrieval',
+};

x-pack/platform/packages/shared/onechat/onechat-common/tools/index.ts

@@ -23,3 +23,4 @@ export {
   builtinToolProviderId,
   unknownToolProviderId,
 } from './tools';
+export { BuiltinToolIds, BuiltinTags } from './constants';

x-pack/platform/packages/shared/onechat/onechat-genai-utils/README.md

@@ -0,0 +1,3 @@
+# @kbn/onechat-genai-utils
+
+Empty package generated by @kbn/generate

x-pack/platform/packages/shared/onechat/onechat-genai-utils/framework/compose_provider.ts

@@ -0,0 +1,50 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { KibanaRequest } from '@kbn/core-http-server';
+import type { PlainIdToolIdentifier, ToolProviderId, ToolDescriptor } from '@kbn/onechat-common';
+import type { ToolProvider, ExecutableTool } from '@kbn/onechat-server';
+
+export interface ByToolIdRule {
+  type: 'by_tool_id';
+  providerId: ToolProviderId;
+  toolIds: PlainIdToolIdentifier[];
+}
+
+export interface ByProviderIdRule {
+  type: 'by_provider_id';
+  providerId: ToolProviderId;
+}
+
+export type ToolFilterRule = ByToolIdRule | ByProviderIdRule;
+
+const matches = (rule: ToolFilterRule, tool: ToolDescriptor): boolean => {
+  if (rule.type === 'by_tool_id') {
+    return tool.meta.providerId === rule.providerId && rule.toolIds.includes(tool.id);
+  } else if (rule.type === 'by_provider_id') {
+    return tool.meta.providerId === rule.providerId;
+  } else {
+    throw new Error('Unknown rule type');
+  }
+};
+
+const anyMatch = (rules: ToolFilterRule[], tool: ToolDescriptor): boolean => {
+  return rules.some((rule) => matches(rule, tool));
+};
+
+export const filterProviderTools = async ({
+  provider,
+  rules,
+  request,
+}: {
+  provider: ToolProvider;
+  rules: ToolFilterRule[];
+  request: KibanaRequest;
+}): Promise<ExecutableTool[]> => {
+  const tools = await provider.list({ request });
+  return tools.filter((tool) => anyMatch(rules, tool));
+};