Skip to content

Commit 99c57c2

Browse files
authored
[SecuritySolution] Create a Security Risk Scoring AI Assistant tool (elastic#233647)
## Summary * Introduces a new AI Assistant tool that retrieves entity risk score data. * Enhances the alert contribution and anonymises it. * Adds a button to the risk score contribution flyout tab that opens the assistant with a preconfigured context and suggested query. * Add an experimental flag: `riskScoreAssistantToolEnabled` <img width="600" alt="Screenshot 2025-09-03 at 14 23 00" src="https://github.com/user-attachments/assets/d8f93d4b-a058-403b-9d74-ee78a7abdd0e" /> <img width="600" alt="Screenshot 2025-09-03 at 14 41 24" src="https://github.com/user-attachments/assets/24bcad0d-65b4-4e29-94c7-320678519380" /> <img width="600" alt="Screenshot 2025-09-03 at 14 22 31" src="https://github.com/user-attachments/assets/f747ca1d-2c9a-4400-91ab-1d020b3e89d6" /> ### How to test it? **Basic scenario** * Kibana installed with alerts data and AI connectors (Please reach out if you need to configure a connector) * Experimental flag enabled `riskScoreAssistantToolEnabled` * Enable risk engine * Open an entity flyout and expand the risk contributions tab * Click "Explain with AI Assistant" * It should open the flyout with a pre-configure prompt and context * Send the prompt, and the Assistant should answer your question **Asset that fields are properly anonymised** * Kibana installed with alerts data and AI connectors (Please reach out if you need to configure a connector) * Experimental flag enabled `riskScoreAssistantToolEnabled` * Enable risk engine * Run Docker for Desktop * Run phoenix `node scripts/phoenix.js` * Open an entity flyout and expand the risk contributions tab * Click "Explain with AI Assistant" * It should open the flyout with a pre-configure prompt and context * Send the prompt, and the Assistant should answer your question * Open Phoenix web and assert that we don't send the entity data (user.name, host.name) to the LLM * You can also disable `user.name` and `host.name` fields anonymisation. Please test different scenarios **Disable scenario** * Kibana installed with alerts data and AI connectors (Please reach out if you need to configure a connector) * Enable risk engine * When the feature is disabled: * Option 1: Disable the AI assistant in the advanced settings * Option 2: Test with basic license * Option 3: Experimental flag is disabled `riskScoreAssistantToolEnabled` * It should not display the button *** To generate realistic risky alerts, you can use the Attack discovery datascript: `node x-pack/solutions/security/plugins/security_solution/scripts/load_attack_discovery_data.js` *** Phoenix local config ``` # OTelemetry # `node scripts/phoenix` from kibana root telemetry.enabled: true telemetry.tracing.enabled: true telemetry.tracing.exporters.phoenix.base_url: "http://0.0.0.0:6006" telemetry.tracing.exporters.phoenix.public_url: "http://0.0.0.0:6006" ``` ### Checklist Reviewers should verify this PR satisfies this list as well. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
1 parent 5c2d146 commit 99c57c2

File tree

16 files changed

+1168
-66
lines changed

16 files changed

+1168
-66
lines changed

x-pack/solutions/security/plugins/elastic_assistant/server/lib/prompt/tool_prompts.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,14 @@ If no relevant information is found, inform the user you could not locate the re
136136
'Call this for knowledge about the latest n open and acknowledged alerts (sorted by `kibana.alert.risk_score`) in the environment, or when answering questions about open alerts. Do not call this tool for alert count or quantity. The output is an array of the latest n open and acknowledged alerts.',
137137
},
138138
},
139+
{
140+
promptId: 'EntityRiskScoreTool',
141+
promptGroupId,
142+
prompt: {
143+
default:
144+
"Call this for knowledge about the latest entity risk score and the inputs that contributed to the calculation (sorted by 'kibana.alert.risk_score') in the environment, or when answering questions about how critical or risky an entity is. When informing the risk score value for a entity you must use the normalized field 'calculated_score_norm'.",
145+
},
146+
},
139147
{
140148
promptId: 'defendInsightsTool',
141149
promptGroupId,

x-pack/solutions/security/plugins/elastic_assistant/server/lib/telemetry/event_based_telemetry.ts

Lines changed: 19 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,14 @@ export const KNOWLEDGE_BASE_EXECUTION_ERROR_EVENT: EventTypeOpts<{
7272
},
7373
};
7474

75+
const toolCountSchema: SchemaValue<number | undefined> = {
76+
type: 'long',
77+
_meta: {
78+
description: 'Number of times tool was invoked.',
79+
optional: true,
80+
},
81+
};
82+
7583
export const INVOKE_ASSISTANT_SUCCESS_EVENT: EventTypeOpts<{
7684
assistantStreamingEnabled: boolean;
7785
actionTypeId: string;
@@ -87,6 +95,7 @@ export const INVOKE_ASSISTANT_SUCCESS_EVENT: EventTypeOpts<{
8795
SecurityLabsKnowledgeBaseTool?: number;
8896
ProductDocumentationTool?: number;
8997
CustomTool?: number;
98+
EntityRiskScoreTool?: number;
9099
};
91100
model?: string;
92101
isOssModel?: boolean;
@@ -133,69 +142,16 @@ export const INVOKE_ASSISTANT_SUCCESS_EVENT: EventTypeOpts<{
133142
},
134143
toolsInvoked: {
135144
properties: {
136-
AlertCountsTool: {
137-
type: 'long',
138-
_meta: {
139-
description: 'Number of times tool was invoked.',
140-
optional: true,
141-
},
142-
},
143-
GenerateESQLTool: {
144-
type: 'long',
145-
_meta: {
146-
description: 'Number of times tool was invoked.',
147-
optional: true,
148-
},
149-
},
150-
AskAboutESQLTool: {
151-
type: 'long',
152-
_meta: {
153-
description: 'Number of times tool was invoked.',
154-
optional: true,
155-
},
156-
},
157-
ProductDocumentationTool: {
158-
type: 'long',
159-
_meta: {
160-
description: 'Number of times tool was invoked.',
161-
optional: true,
162-
},
163-
},
164-
KnowledgeBaseRetrievalTool: {
165-
type: 'long',
166-
_meta: {
167-
description: 'Number of times tool was invoked.',
168-
optional: true,
169-
},
170-
},
171-
KnowledgeBaseWriteTool: {
172-
type: 'long',
173-
_meta: {
174-
description: 'Number of times tool was invoked.',
175-
optional: true,
176-
},
177-
},
178-
OpenAndAcknowledgedAlertsTool: {
179-
type: 'long',
180-
_meta: {
181-
description: 'Number of times tool was invoked.',
182-
optional: true,
183-
},
184-
},
185-
SecurityLabsKnowledgeBaseTool: {
186-
type: 'long',
187-
_meta: {
188-
description: 'Number of times tool was invoked.',
189-
optional: true,
190-
},
191-
},
192-
CustomTool: {
193-
type: 'long',
194-
_meta: {
195-
description: 'Number of times tool was invoked.',
196-
optional: true,
197-
},
198-
},
145+
AlertCountsTool: toolCountSchema,
146+
GenerateESQLTool: toolCountSchema,
147+
AskAboutESQLTool: toolCountSchema,
148+
ProductDocumentationTool: toolCountSchema,
149+
KnowledgeBaseRetrievalTool: toolCountSchema,
150+
KnowledgeBaseWriteTool: toolCountSchema,
151+
OpenAndAcknowledgedAlertsTool: toolCountSchema,
152+
SecurityLabsKnowledgeBaseTool: toolCountSchema,
153+
CustomTool: toolCountSchema,
154+
EntityRiskScoreTool: toolCountSchema,
199155
},
200156
},
201157
},

x-pack/solutions/security/plugins/security_solution/common/experimental_features.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,11 @@ export const allowedExperimentalValues = Object.freeze({
125125
*/
126126
riskScoringRoutesEnabled: true,
127127

128+
/**
129+
* Enables the Risk Score AI Assistant tool.
130+
*/
131+
riskScoreAssistantToolEnabled: false,
132+
128133
/**
129134
* disables ES|QL rules
130135
*/
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import React from 'react';
9+
import { render, screen, fireEvent } from '@testing-library/react';
10+
import { AskAiAssistant } from './ask_ai_assistant';
11+
import { TestProviders } from '../../../../../common/mock';
12+
import type { EntityType } from '../../../../../../common/search_strategy';
13+
14+
// Hard code the generated anonymized value for easier testing
15+
const ANONYMIZED_VALUE = 'anonymized-value';
16+
jest.mock('@kbn/elastic-assistant-common', () => {
17+
const actual = jest.requireActual('@kbn/elastic-assistant-common');
18+
return {
19+
...actual,
20+
getAnonymizedValue: () => ANONYMIZED_VALUE,
21+
};
22+
});
23+
24+
jest.mock('../../../../../common/hooks/use_experimental_features', () => {
25+
const actual = jest.requireActual('../../../../../common/hooks/use_experimental_features');
26+
return {
27+
...actual,
28+
useIsExperimentalFeatureEnabled: jest.fn().mockReturnValue(true),
29+
};
30+
});
31+
32+
const mockUseFetchAnonymizationFields = jest.fn().mockReturnValue({
33+
data: {
34+
data: [],
35+
},
36+
});
37+
38+
jest.mock(
39+
'@kbn/elastic-assistant/impl/assistant/api/anonymization_fields/use_fetch_anonymization_fields',
40+
() => ({
41+
useFetchAnonymizationFields: () => mockUseFetchAnonymizationFields(),
42+
})
43+
);
44+
45+
const mockUseAskInAiAssistant = jest.fn();
46+
jest.mock('./use_ask_ai_assistant', () => {
47+
const actual = jest.requireActual('./use_ask_ai_assistant');
48+
return {
49+
...actual,
50+
useAskAiAssistant: (params: unknown) => mockUseAskInAiAssistant(params),
51+
};
52+
});
53+
54+
describe('ExplainWithAiAssistant', () => {
55+
const defaultProps = {
56+
entityType: 'user' as EntityType,
57+
entityName: 'test-user',
58+
};
59+
60+
const mockShowAssistantOverlay = jest.fn();
61+
62+
beforeEach(() => {
63+
jest.clearAllMocks();
64+
mockUseFetchAnonymizationFields.mockReturnValue({
65+
data: {
66+
data: [{ field: 'user.name', allowed: true, anonymized: true }],
67+
},
68+
});
69+
mockUseAskInAiAssistant.mockReturnValue({
70+
showAssistantOverlay: mockShowAssistantOverlay,
71+
disabled: false,
72+
});
73+
});
74+
75+
it('should render the button when AI assistant is enabled', () => {
76+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
77+
78+
expect(screen.getByTestId('explain-with-ai-button')).toBeInTheDocument();
79+
expect(screen.getByText('Ask AI Assistant')).toBeInTheDocument();
80+
});
81+
82+
it('should not render the button when AI assistant is disabled', () => {
83+
mockUseAskInAiAssistant.mockReturnValue({
84+
showAssistantOverlay: mockShowAssistantOverlay,
85+
disabled: true,
86+
});
87+
88+
const { container } = render(<AskAiAssistant {...defaultProps} />, {
89+
wrapper: TestProviders,
90+
});
91+
92+
expect(container.firstChild).toBeNull();
93+
});
94+
95+
it('should call showAssistantOverlay when button is clicked', () => {
96+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
97+
98+
const button = screen.getByTestId('explain-with-ai-button');
99+
fireEvent.click(button);
100+
101+
expect(mockShowAssistantOverlay).toHaveBeenCalledTimes(1);
102+
});
103+
104+
it('should handle missing anonymization fields data', () => {
105+
mockUseFetchAnonymizationFields.mockReturnValue({
106+
data: {
107+
data: null,
108+
},
109+
});
110+
111+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
112+
113+
expect(screen.getByTestId('explain-with-ai-button')).toBeInTheDocument();
114+
});
115+
116+
it('should use original entity name when anonymized value is not available', () => {
117+
mockUseFetchAnonymizationFields.mockReturnValue({
118+
data: {
119+
data: [],
120+
},
121+
});
122+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
123+
124+
expect(mockUseAskInAiAssistant).toHaveBeenCalledWith(
125+
expect.objectContaining({
126+
title: "Explain user 'test-user' Risk Score",
127+
description: 'Entity: test-user',
128+
})
129+
);
130+
});
131+
132+
it('should pass correct props to useExplainInAiAssistant hook', () => {
133+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
134+
135+
expect(mockUseAskInAiAssistant).toHaveBeenCalledWith({
136+
title: "Explain user 'test-user' Risk Score",
137+
description: 'Entity: test-user',
138+
suggestedPrompt: expect.any(String),
139+
getPromptContext: expect.any(Function),
140+
replacements: expect.any(Object),
141+
});
142+
});
143+
144+
it('should generate prompt with anonymized field', async () => {
145+
render(<AskAiAssistant {...defaultProps} />, { wrapper: TestProviders });
146+
147+
const getPromptContext = mockUseAskInAiAssistant.mock.calls[0][0].getPromptContext;
148+
const promptContext = await getPromptContext();
149+
150+
expect(promptContext).toContain(`Identifier: \`${ANONYMIZED_VALUE}\``);
151+
});
152+
153+
it('should work with different entity types', () => {
154+
const hostProps = {
155+
entityType: 'host' as EntityType,
156+
entityName: 'test-host',
157+
};
158+
159+
render(<AskAiAssistant {...hostProps} />, { wrapper: TestProviders });
160+
161+
expect(mockUseAskInAiAssistant).toHaveBeenCalledWith(
162+
expect.objectContaining({
163+
title: "Explain host 'test-host' Risk Score",
164+
})
165+
);
166+
});
167+
});
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import React, { useCallback, useMemo } from 'react';
9+
import { EuiButton, EuiFlexGroup, EuiFlexItem, EuiSpacer } from '@elastic/eui';
10+
import { FormattedMessage } from '@kbn/i18n-react';
11+
import { AssistantIcon } from '@kbn/ai-assistant-icon';
12+
import { getAnonymizedValues } from '@kbn/elastic-assistant-common/impl/data_anonymization/get_anonymized_values';
13+
import { getAnonymizedValue } from '@kbn/elastic-assistant-common';
14+
import { useFetchAnonymizationFields } from '@kbn/elastic-assistant';
15+
import type { AnonymizedValues } from '@kbn/elastic-assistant-common/impl/data_anonymization/types';
16+
import { useIsExperimentalFeatureEnabled } from '../../../../../common/hooks/use_experimental_features';
17+
import { EntityTypeToIdentifierField } from '../../../../../../common/entity_analytics/types';
18+
import type { EntityType } from '../../../../../../common/search_strategy';
19+
import { useAskAiAssistant } from './use_ask_ai_assistant';
20+
21+
export interface ExplainWithAiAssistantProps<T extends EntityType> {
22+
entityType: T;
23+
entityName: string;
24+
}
25+
26+
const CURRENT_REPLACEMENTS = {} as const;
27+
28+
export const AskAiAssistant = <T extends EntityType>({
29+
entityType,
30+
entityName,
31+
}: ExplainWithAiAssistantProps<T>) => {
32+
const entityField = EntityTypeToIdentifierField[entityType];
33+
const { data: anonymizationFields } = useFetchAnonymizationFields();
34+
const isAssistantToolEnabled = useIsExperimentalFeatureEnabled('riskScoreAssistantToolEnabled');
35+
36+
const { anonymizedValues, replacements }: AnonymizedValues = useMemo(() => {
37+
if (!anonymizationFields.data) {
38+
return { anonymizedValues: [], replacements: CURRENT_REPLACEMENTS };
39+
}
40+
41+
return getAnonymizedValues({
42+
anonymizationFields: anonymizationFields.data,
43+
currentReplacements: CURRENT_REPLACEMENTS,
44+
field: entityField,
45+
getAnonymizedValue,
46+
rawData: { [entityField]: [entityName] },
47+
});
48+
}, [anonymizationFields, entityField, entityName]);
49+
50+
const anonymizedEntityName = anonymizedValues[0] ?? entityName;
51+
const getPromptContext = useCallback(
52+
async () =>
53+
`### The following entity is under investigation:\nType: ${entityType}\nIdentifier: ${`\`${anonymizedEntityName}\``}`,
54+
[anonymizedEntityName, entityType]
55+
);
56+
57+
const { showAssistantOverlay, disabled: aiAssistantDisable } = useAskAiAssistant({
58+
title: `Explain ${entityType} '${entityName}' Risk Score`,
59+
description: `Entity: ${entityName}`,
60+
suggestedPrompt: `Explain how inputs contributed to the risk score. Additionally, outline the recommended next steps for investigating or mitigating the risk if the entity is deemed risky.\nTo answer risk score questions, fetch the risk score information and take into consideration the risk score inputs.`,
61+
getPromptContext,
62+
replacements,
63+
});
64+
65+
if (aiAssistantDisable || !isAssistantToolEnabled) {
66+
return null;
67+
}
68+
69+
return (
70+
<>
71+
<EuiSpacer size="m" />
72+
<EuiFlexGroup justifyContent="flexEnd">
73+
<EuiFlexItem grow={false}>
74+
<EuiButton
75+
data-test-subj="explain-with-ai-button"
76+
iconType={AssistantIcon}
77+
iconSide="right"
78+
onClick={() => {
79+
showAssistantOverlay();
80+
}}
81+
>
82+
<FormattedMessage
83+
id="xpack.securitySolution.flyout.entityDetails.riskInputs.askAiAssistant"
84+
defaultMessage="Ask AI Assistant"
85+
/>
86+
</EuiButton>
87+
</EuiFlexItem>
88+
</EuiFlexGroup>
89+
</>
90+
);
91+
};
92+
93+
AskAiAssistant.displayName = 'ExplainWithAiAssistant';

0 commit comments

Comments
 (0)