Skip to content

Commit 16a5962

Browse files
logeekalcursoragent
logeekal
and
cursoragent
authored
[Security Solution] Add Sentinel watchlist lookup ingestion (elastic#269726)
## Summary Adds Microsoft Sentinel watchlist ingestion to the SIEM rule migration lookup upload flow. Sentinel watchlist ARM exports are validated in the UI, uploaded as raw JSON, normalized in the backend, and ingested through the existing generic processLookups path. This supports both direct watchlist ARM resources and one-resource ARM deployment templates, rejects unsupported non-CSV watchlists, denormalizes only the itemsSearchKey column, and stores itemsSearchKey in resource metadata. ## Sentinal Watchlist handling Sentinel watchlists are exported as ARM JSON resources, with the actual lookup data stored as CSV in `properties.rawContent` ( See [example](https://github.com/Azure/Azure-Sentinel/blob/840f0f490c3509335821e2030deca714d00c2760/Solutions/Network%20Session%20Essentials/Watchlists/NetworkSession_Monitor_Configuration.json#L13) ). During upload, Kibana validates the ARM JSON, extracts the CSV, parses it into rows, and converts the watchlist into a normal lookup resource. However, before creating the Elastic lookup index, it denormalizes the Sentinel `itemsSearchKey` column. In simple terms, if the search key cell contains multiple comma-separated values, Kibana creates one lookup row per value. For the Network Session Essentials example, the watchlist declares: ```json { "watchlistAlias": "NetworkSession_Monitor_Configuration", "itemsSearchKey": "Ports", "contentType": "text/csv" } ``` That means `Ports` is the field rules search against, so only `Ports` is denormalized. ### Watchlist before normalization | Ports | App | Name | |---|---|---| | `389,636` | `*` | `Suspicious LDAP Traffic` | | `3389` | `*` | `Suspicious Remote Desktop Network Traffic` | | `9150,9151` | `tor` | `Suspicious TOR Traffic` | | `139,445` | `smb` | `Suspicious SMB Traffic` | ### Watchlist after normalization | Ports | App | Name | |---|---|---| | `389` | `*` | `Suspicious LDAP Traffic` | | `636` | `*` | `Suspicious LDAP Traffic` | | `3389` | `*` | `Suspicious Remote Desktop Network Traffic` | | `9150` | `tor` | `Suspicious TOR Traffic` | | `9151` | `tor` | `Suspicious TOR Traffic` | | `139` | `smb` | `Suspicious SMB Traffic` | | `445` | `smb` | `Suspicious SMB Traffic` | ### Why denormalization is needed In the Sentinel rule export `network_session_essentials_rules_export.arm.json`, the KustoQL query loads the watchlist, splits comma-separated fields, expands them, and then joins event data against the expanded mapping: ```kusto let mapping = _GetWatchlist('NetworkSession_Monitor_Configuration') | where Type == "Detection" and ThresholdType == "Static" and Severity != "Disabled" | extend Ports = split(Ports, ",") | mv-expand Ports | extend Ports = tostring(Ports); ``` This works in KustoQL because the query can build a temporary `mapping` table, transform the watchlist inline, expand multi-value cells with `mv-expand`, and join using a flexible condition like `Ports has tostring(DstPortNumber)`. ES|QL lookup joins are more constrained. `LOOKUP JOIN` joins against a stored lookup index, not an inline transformed table, and lookup join fields with multi-valued entries do not match. So a lookup document where `Ports` is `"389,636"` will not behave like two separate values when joining against an event port such as `389`. Denormalizing during ingestion moves that transformation out of the query and into lookup index creation. The migrated ES|QL can then use a normal lookup join because each searchable port value exists as its own lookup row. ## Test plan ### Cypress coverage - ✅ TC1: Create a Microsoft Sentinel migration and identify missing watchlists in the `Upload watchlists` step. Steps: 1. Log in to Kibana and go to **Security** → **Automatic migrations** → **Manage Automatic Migrations**. 2. Open **Migrate your existing SIEM rules to Elastic** and click **Upload rules**. 3. In the **Upload SIEM rules** flyout, choose **Microsoft Sentinel** from **Select migration source**. 4. In **Upload rules**, select a Sentinel Analytics Rules ARM JSON export that references a watchlist. 5. Click **Upload**. Expectations: 1. The rules upload succeeds. 2. The flyout advances to **Upload watchlists**. 3. The watchlist step explains that watchlists were found in the uploaded rules. 4. The missing watchlist list includes the Sentinel watchlist name, for example `HighValueAccounts` or `NetworkSession_Monitor_Configuration` depending on the fixture. 5. The flow does not show Splunk-only macro wording for the Sentinel migration source. - ✅ TC2: Upload a valid Microsoft Sentinel watchlist ARM export from the UI. Steps: 1. Log in to Kibana and go to **Security** → **Automatic migrations** → **Manage Automatic Migrations**. 2. Create or open a Microsoft Sentinel rule migration that is waiting on **Upload watchlists**. 3. In the **Upload SIEM rules** flyout, go to **Upload watchlists**. 4. In the watchlist file picker labeled **Select or drag and drop the exported watchlist files**, select a valid Sentinel watchlist ARM export. 5. Click **Upload**. Expectations: 1. The watchlist upload succeeds. 2. The uploaded watchlist is accepted as a Sentinel watchlist resource. 3. The missing watchlist is marked as resolved in the list. 4. The **Translate** button is available so the user can continue the migration. 5. No watchlist parsing or unsupported content type error is shown. - ✅ TC4: Show a user-facing error for unsupported Sentinel watchlist content types. Steps: 1. Log in to Kibana and go to **Security** → **Automatic migrations** → **Manage Automatic Migrations**. 2. Click **Upload rules** and create a Microsoft Sentinel migration by uploading a Sentinel Analytics Rules ARM JSON export that references a watchlist. 3. Wait for the flyout to advance to **Upload watchlists**. 4. Select a Sentinel watchlist ARM JSON file where `properties.contentType` is `application/json` instead of `text/csv`. 5. Click **Upload**. Expectations: 1. The upload fails with a visible error for unsupported Sentinel watchlist content type. 2. The watchlist remains unresolved in the missing watchlist list. 3. The user stays in the **Upload watchlists** step. 4. The user can choose another file and retry. 5. No lookup index is created from the unsupported watchlist file. - ✅ TC5: Show a user-facing error for invalid Sentinel watchlist JSON. Steps: 1. Log in to Kibana and go to **Security** → **Automatic migrations** → **Manage Automatic Migrations**. 2. Click **Upload rules** and create a Microsoft Sentinel migration by uploading a Sentinel Analytics Rules ARM JSON export that references a watchlist. 3. Wait for the flyout to advance to **Upload watchlists**. 4. In the watchlist file picker, select a malformed JSON file. 5. Repeat with a valid JSON file that is not a Sentinel watchlist ARM resource or one-resource ARM deployment template. Expectations: 1. Malformed JSON shows **Sentinel watchlist must be valid JSON.** 2. Wrong-schema JSON shows the Sentinel watchlist ARM/template validation error. 3. No watchlist is marked as uploaded. 4. The user remains on **Upload watchlists**. 5. The user can remove the invalid file and retry with a valid Sentinel watchlist export. Cypress spec: `x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/investigations/siem_migrations/rules/sentinel_onboarding.cy.ts` ### Checklist - [x] Any text added follows EUI writing guidelines, uses sentence case text and includes i18n support - [ ] Documentation was added for features that require explanation or tutorials - [x] Unit or functional tests were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee - [ ] Flaky Test Runner was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct release_note label is applied per the guidelines - [ ] Review the backport guidelines and apply applicable backport labels. ### Identify risks - [ ] See the risk examples in RISK_MATRIX.mdx - [x] Low risk: Sentinel watchlist uploads are newly handled on the existing resources endpoint. Existing Splunk and QRadar lookup behavior remains on the existing generic lookup path. - [x] Low risk: malformed Sentinel ARM resources or unsupported contentType values are rejected with 400 responses rather than being ingested. - [x] Low risk: denormalization changes only the Sentinel itemsSearchKey column before lookup index creation and is covered by unit and FTR API tests. ### Release note > Adds Microsoft Sentinel watchlist upload support to SIEM rule migrations. Suggested label: release_note:enhancement --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 8816ba3 commit 16a5962

24 files changed

Lines changed: 1035 additions & 35 deletions

File tree

‎x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/common.gen.ts‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import { z, lazySchema } from '@kbn/zod/v4';
1919
import { NonEmptyString } from '../../api/model/primitives.gen';
2020
import { SplunkResourceType } from './vendor/common/splunk.gen';
2121
import { QradarResourceType } from './vendor/common/qradar.gen';
22+
import { SentinelResourceType } from './vendor/common/sentinel.gen';
2223

2324
/**
2425
* The GenAI connector id to use.
@@ -284,10 +285,13 @@ export const MigrationTaskStats = lazySchema(() =>
284285
);
285286
export type MigrationTaskStats = z.infer<typeof MigrationTaskStats>;
286287

287-
export const SiemMigrationResourceType = lazySchema(() =>
288-
z.union([SplunkResourceType, QradarResourceType])
288+
export const SiemMigrationResourceTypeInternal = lazySchema(() =>
289+
z.union([SplunkResourceType, QradarResourceType, SentinelResourceType])
289290
);
290-
export type SiemMigrationResourceType = z.infer<typeof SiemMigrationResourceType>;
291+
292+
export type SiemMigrationResourceType = z.infer<typeof SiemMigrationResourceTypeInternal>;
293+
export const SiemMigrationResourceType =
294+
SiemMigrationResourceTypeInternal as z.ZodType<SiemMigrationResourceType>;
291295

292296
/**
293297
* A resource of a migration

‎x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/common.schema.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,11 +239,12 @@ components:
239239
type: integer
240240
description: The number of items that have failed migration.
241241

242-
## should be combination of SplunkResourceType and QradarResourceType types in vendor folder
242+
## should be combination of SplunkResourceType, QradarResourceType and SentinelResourceType types in vendor folder
243243
SiemMigrationResourceType:
244244
oneOf:
245245
- $ref: './vendor/common/splunk.schema.yaml#/components/schemas/SplunkResourceType'
246246
- $ref: './vendor/common/qradar.schema.yaml#/components/schemas/QradarResourceType'
247+
- $ref: './vendor/common/sentinel.schema.yaml#/components/schemas/SentinelResourceType'
247248

248249
SiemMigrationResourceBase:
249250
type: object

‎x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/vendor/common/sentinel.gen.ts‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,75 @@ import { z, lazySchema } from '@kbn/zod/v4';
2121
*/
2222
export const SentinelResourceType = lazySchema(() => z.literal('watchlist'));
2323
export type SentinelResourceType = z.infer<typeof SentinelResourceType>;
24+
25+
/**
26+
* A Microsoft Sentinel watchlist ARM template resource
27+
*/
28+
export const SentinelWatchlistResource = lazySchema(() =>
29+
z.object({
30+
/**
31+
* The ARM resource identifier
32+
*/
33+
id: z.string().optional(),
34+
/**
35+
* The ARM resource name
36+
*/
37+
name: z.string().optional(),
38+
/**
39+
* The ARM resource type
40+
*/
41+
type: z.string().optional(),
42+
/**
43+
* The Sentinel watchlist properties
44+
*/
45+
properties: z.object({
46+
/**
47+
* The Sentinel watchlist alias
48+
*/
49+
watchlistAlias: z.string().min(1),
50+
/**
51+
* The raw CSV content for the watchlist
52+
*/
53+
rawContent: z.string(),
54+
/**
55+
* The watchlist search key column name
56+
*/
57+
itemsSearchKey: z.string().optional(),
58+
/**
59+
* The number of raw content lines to skip
60+
*/
61+
numberOfLinesToSkip: z.number().int().optional(),
62+
/**
63+
* The source file name
64+
*/
65+
source: z.string().optional(),
66+
/**
67+
* The source content type
68+
*/
69+
contentType: z.string().optional(),
70+
}),
71+
})
72+
);
73+
export type SentinelWatchlistResource = z.infer<typeof SentinelWatchlistResource>;
74+
75+
/**
76+
* A Microsoft Sentinel watchlist ARM deployment template
77+
*/
78+
export const SentinelWatchlistTemplate = lazySchema(() =>
79+
z.object({
80+
/**
81+
* The ARM deployment template schema
82+
*/
83+
$schema: z.string().optional(),
84+
/**
85+
* The ARM deployment template content version
86+
*/
87+
contentVersion: z.string().optional(),
88+
/**
89+
* The ARM deployment template parameters
90+
*/
91+
parameters: z.object({}).catchall(z.unknown()).optional(),
92+
resources: z.array(SentinelWatchlistResource).min(1).max(1),
93+
})
94+
);
95+
export type SentinelWatchlistTemplate = z.infer<typeof SentinelWatchlistTemplate>;

‎x-pack/solutions/security/plugins/security_solution/common/siem_migrations/model/vendor/common/sentinel.schema.yaml‎

Lines changed: 67 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
openapi: 3.0.3
22
info:
33
title: Sentinel components for both dashboards and rules
4-
version: 'not applicable'
4+
version: "not applicable"
55
paths: {}
66
components:
77
x-codegen-enabled: true
@@ -11,3 +11,69 @@ components:
1111
description: The type of the resource
1212
enum:
1313
- watchlist
14+
15+
SentinelWatchlistResource:
16+
type: object
17+
description: A Microsoft Sentinel watchlist ARM template resource
18+
required:
19+
- properties
20+
properties:
21+
id:
22+
type: string
23+
description: The ARM resource identifier
24+
name:
25+
type: string
26+
description: The ARM resource name
27+
type:
28+
type: string
29+
description: The ARM resource type
30+
example: Microsoft.OperationalInsights/workspaces/providers/Watchlists
31+
properties:
32+
type: object
33+
description: The Sentinel watchlist properties
34+
required:
35+
- watchlistAlias
36+
- rawContent
37+
properties:
38+
watchlistAlias:
39+
type: string
40+
minLength: 1
41+
description: The Sentinel watchlist alias
42+
rawContent:
43+
type: string
44+
description: The raw CSV content for the watchlist
45+
itemsSearchKey:
46+
type: string
47+
description: The watchlist search key column name
48+
numberOfLinesToSkip:
49+
type: integer
50+
description: The number of raw content lines to skip
51+
source:
52+
type: string
53+
description: The source file name
54+
contentType:
55+
type: string
56+
description: The source content type
57+
58+
SentinelWatchlistTemplate:
59+
type: object
60+
description: A Microsoft Sentinel watchlist ARM deployment template
61+
required:
62+
- resources
63+
properties:
64+
$schema:
65+
type: string
66+
description: The ARM deployment template schema
67+
contentVersion:
68+
type: string
69+
description: The ARM deployment template content version
70+
parameters:
71+
type: object
72+
additionalProperties: true
73+
description: The ARM deployment template parameters
74+
resources:
75+
type: array
76+
minItems: 1
77+
maxItems: 1
78+
items:
79+
$ref: "#/components/schemas/SentinelWatchlistResource"

‎x-pack/solutions/security/plugins/security_solution/public/siem_migrations/common/components/migration_steps/lookups/lookups_file_upload/index.tsx‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ import { UploadFileButton } from '../..';
2222
import { FILE_UPLOAD_ERROR } from '../../../../translations/file_upload_error';
2323
import type { SiemMigrationResourceData } from '../../../../../../../common/siem_migrations/model/common.gen';
2424
import * as i18n from './translations';
25-
import { convertQradarReferenceSetToLookup } from '../utils';
25+
import { convertQradarReferenceSetToLookup, convertSentinelWatchlistToResource } from '../utils';
2626
import { MigrationSource } from '../../../../types';
2727

2828
export interface LookupsFileUploadProps {
@@ -75,7 +75,7 @@ export const LookupsFileUpload = React.memo<LookupsFileUploadProps>(
7575

7676
const lookups = await Promise.all(
7777
Array.from(files).map((file) => {
78-
return new Promise<SiemMigrationResourceData>((resolve) => {
78+
return new Promise<SiemMigrationResourceData | undefined>((resolve) => {
7979
const reader = new FileReader();
8080

8181
reader.onloadstart = () => setIsParsing(true);
@@ -87,26 +87,48 @@ export const LookupsFileUpload = React.memo<LookupsFileUploadProps>(
8787

8888
if (content == null) {
8989
addError(FILE_UPLOAD_ERROR.CAN_NOT_READ);
90+
resolve(undefined);
9091
return;
9192
}
9293

9394
if (content === '' && e.loaded > 100000) {
9495
// V8-based browsers can't handle large files and return an empty string
9596
// instead of an error; see https://stackoverflow.com/a/61316641
9697
addError(FILE_UPLOAD_ERROR.TOO_LARGE_TO_PARSE);
98+
resolve(undefined);
9799
return;
98100
}
99101

100102
const name = file.name.replace(/\.[^/.]+$/, '').trim();
101103

102-
if (migrationSource === MigrationSource.QRADAR) {
103-
resolve(
104-
convertQradarReferenceSetToLookup({
105-
fileContent: content,
106-
fallbackName: name,
107-
})
104+
try {
105+
if (migrationSource === MigrationSource.QRADAR) {
106+
resolve(
107+
convertQradarReferenceSetToLookup({
108+
fileContent: content,
109+
fallbackName: name,
110+
})
111+
);
112+
return;
113+
}
114+
115+
if (migrationSource === MigrationSource.SENTINEL) {
116+
resolve(
117+
convertSentinelWatchlistToResource({
118+
fileContent: content,
119+
fallbackName: name,
120+
})
121+
);
122+
return;
123+
}
124+
} catch (error) {
125+
addError(
126+
error instanceof Error ? error.message : FILE_UPLOAD_ERROR.CAN_NOT_PARSE
108127
);
128+
resolve(undefined);
129+
return;
109130
}
131+
110132
resolve({ type: 'lookup', name, content });
111133
};
112134

@@ -117,6 +139,7 @@ export const LookupsFileUpload = React.memo<LookupsFileUploadProps>(
117139
} else {
118140
addError(FILE_UPLOAD_ERROR.CAN_NOT_READ);
119141
}
142+
resolve(undefined);
120143
};
121144

122145
reader.onerror = handleReaderError;
@@ -129,8 +152,11 @@ export const LookupsFileUpload = React.memo<LookupsFileUploadProps>(
129152
addError(e.message);
130153
return [];
131154
});
155+
const validLookups = lookups.flatMap((resource): SiemMigrationResourceData[] =>
156+
resource !== undefined ? [resource] : []
157+
);
132158
// Set the loaded lookups to the state
133-
setLookupResources((current) => [...current, ...lookups]);
159+
setLookupResources((current) => [...current, ...validLookups]);
134160
},
135161
[addError, migrationSource]
136162
);

‎x-pack/solutions/security/plugins/security_solution/public/siem_migrations/common/components/migration_steps/lookups/missing_lookups_list/index.tsx‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,13 @@ import {
1212
EuiCopy,
1313
EuiFlexGroup,
1414
EuiFlexItem,
15+
EuiCode,
1516
EuiIcon,
1617
EuiPanel,
1718
EuiText,
1819
EuiToolTip,
1920
useEuiTheme,
21+
EuiLink,
2022
} from '@elastic/eui';
2123
import { FormattedMessage } from '@kbn/i18n-react';
2224
import * as i18n from './translations';
@@ -50,6 +52,22 @@ export const MissingLookupsList = React.memo<MissingLookupsListProps>(
5052
<EuiText size="s">
5153
{migrationSource === MigrationSource.QRADAR ? (
5254
i18n.REFERENCE_SETS_QRADAR_APP
55+
) : migrationSource === MigrationSource.SENTINEL ? (
56+
<FormattedMessage
57+
id="xpack.securitySolution.siemMigrations.common.dataInputFlyout.lookups.missingWatchlistsList.sentinelAppSection"
58+
defaultMessage="Below are the watchlists found in your rules. Export them from Microsoft Sentinel and upload here. Exported Watchlist must be in <armlink>ARM Resource format</armlink>. CSV content should be included in the <code>rawContent</code> property of the watchlist."
59+
values={{
60+
armlink: (child) => (
61+
<EuiLink
62+
href="https://learn.microsoft.com/en-us/azure/templates/microsoft.securityinsights/watchlists?pivots=deployment-language-arm-template#resource-format-1"
63+
target="_blank"
64+
>
65+
{child}
66+
</EuiLink>
67+
),
68+
code: (child) => <EuiCode>{child}</EuiCode>,
69+
}}
70+
/>
5371
) : (
5472
<FormattedMessage
5573
id="xpack.securitySolution.siemMigrations.common.dataInputFlyout.lookups.copyExportQuery.splunk.description"
@@ -74,9 +92,13 @@ export const MissingLookupsList = React.memo<MissingLookupsListProps>(
7492
>
7593
<EuiFlexItem grow={false}>
7694
{uploadedLookups[lookupName] != null ? (
77-
<EuiIcon type="checkCircleFill" color={euiTheme.colors.success} />
95+
<EuiIcon
96+
type="checkCircleFill"
97+
color={euiTheme.colors.success}
98+
aria-hidden={true}
99+
/>
78100
) : (
79-
<EuiIcon type="dot" />
101+
<EuiIcon type="dot" aria-hidden={true} />
80102
)}
81103
</EuiFlexItem>
82104
<EuiFlexItem grow={false}>

‎x-pack/solutions/security/plugins/security_solution/public/siem_migrations/common/components/migration_steps/lookups/missing_lookups_list/translations.ts‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -40,14 +40,6 @@ export const CLEAR_EMPTY_REFERENCE_SET_TOOLTIP = i18n.translate(
4040
{ defaultMessage: 'Mark the reference set as empty' }
4141
);
4242

43-
export const WATCHLISTS_SENTINEL_APP = i18n.translate(
44-
'xpack.securitySolution.siemMigrations.common.dataInputFlyout.lookups.missingWatchlistsList.sentinelAppSection',
45-
{
46-
defaultMessage:
47-
'Below are the watchlists found in your rules. Export them from Microsoft Sentinel and upload here.',
48-
}
49-
);
50-
5143
export const COPY_WATCHLIST_NAME_TOOLTIP = i18n.translate(
5244
'xpack.securitySolution.siemMigrations.common.dataInputFlyout.lookups.missingWatchlistsList.copyWatchlistNameTooltip',
5345
{ defaultMessage: 'Copy watchlist name' }

‎x-pack/solutions/security/plugins/security_solution/public/siem_migrations/common/components/migration_steps/lookups/utils/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,5 @@
77

88
export { convertQradarReferenceSetToLookup } from './qradar_reference_set';
99
export type { ConvertQradarReferenceSetToLookupParams } from './qradar_reference_set';
10+
export { convertSentinelWatchlistToResource } from './sentinel_watchlist';
11+
export type { ConvertSentinelWatchlistToResourceParams } from './sentinel_watchlist';

0 commit comments

Comments
 (0)