Skip to content

Commit 6efb788

Browse files
mohitjha-elasticmohitjha-elastic
authored
aws, blacklens, github, microsoft_sentinel: fix conflicting event.kind values.
Fix conflicting event.kind mappings where pipelines set pipeline_error (or state) while the field was defined as constant_keyword, which Elasticsearch treats as a single immutable value per index. The securityhub_findings_full_posture data stream now references ECS for event.kind, matching the sibling Security Hub stream. For blacklens alerts, GitHub issues/code_scanning/dependabot/secret_scanning, and Microsoft Sentinel alert/event/incident streams, event.kind is mapped as keyword and the ingest pipelines set the normal ECS value (alert or event) so successful documents stay unchanged while failure handlers can still set pipeline_error.
1 parent 1cbfb94 commit 6efb788

44 files changed

Lines changed: 305 additions & 233 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/aws/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "6.4.3"
3+
changes:
4+
- description: Align `event.kind` with ECS in `securityhub_findings_full_posture` data stream to avoid conflicts with ingest pipeline values.
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/18143
27
- version: "6.4.2"
38
changes:
49
- description: Revert `event.kind` from `alert` to `event` on AWS WAF.

‎packages/aws/data_stream/securityhub_findings_full_posture/fields/ecs.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
- name: cloud.provider
33
type: constant_keyword
44
- name: event.kind
5-
type: constant_keyword
5+
external: ecs
66
- name: observer.vendor
77
type: constant_keyword

‎packages/aws/docs/securityhub.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1383,7 +1383,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
13831383
| data_stream.namespace | A user defined namespace. Namespaces are useful to allow grouping of data. Many users already organize their indices this way, and the data stream naming scheme now provides this best practice as a default. Many users will populate this field with `default`. If no value is used, it falls back to `default`. Beyond the Elasticsearch index naming criteria noted above, `namespace` value has the additional restrictions: \* Must not contain `-` \* No longer than 100 characters | constant_keyword |
13841384
| data_stream.type | An overarching type for the data stream. Currently allowed values are "logs" and "metrics". We expect to also add "traces" and "synthetics" in the near future. | constant_keyword |
13851385
| event.dataset | Name of the dataset. If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from. It's recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name. | constant_keyword |
1386-
| event.kind | | constant_keyword |
1386+
| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword |
13871387
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | constant_keyword |
13881388
| host.containerized | If the host is a container. | boolean |
13891389
| host.os.build | OS build information. | keyword |

‎packages/aws/manifest.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
format_version: 3.4.0
22
name: aws
33
title: AWS
4-
version: 6.4.2
4+
version: 6.4.3
55
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
66
type: integration
77
categories:

‎packages/blacklens/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.0.1"
3+
changes:
4+
- description: Change `event.kind` to `keyword` and set `alert` in the ingest pipeline so `pipeline_error` on failure does not conflict with constant_keyword mapping.
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/18143
27
- version: "1.0.0"
38
changes:
49
- description: Make integration GA.

‎packages/blacklens/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
"threat"
3232
],
3333
"id": "7ea10c5d-559a-4c55-8608-2e060956de68",
34+
"kind": "alert",
3435
"original": "{\"updated_date\":\"2025-12-31T16:10:56.155874Z\",\"created_date\":\"2025-12-30T16:11:57.194393Z\",\"id\":\"7ea10c5d-559a-4c55-8608-2e060956de68\",\"name\":\"External Vulnerability Detected\",\"type\":\"ExternalVulnerabilityDiscovered\",\"severity\":\"high\",\"status\":\"new\",\"analysis\":\"completed\",\"category\":\"vulnerability\",\"activities\":[{\"updated_date\":null,\"created_date\":\"2025-12-30T16:11:40.195989Z\",\"id\":\"73dcaa88-09e1-4c58-9fa5-5495f8dac2a4\",\"type\":\"ExternalVulnerabilityCreated\",\"description\":\"A Critical severity external vulnerability 'Blind SQL Injection via HTTP Header' has been detected on asset 'demo.example.com'\",\"category\":\"threat\",\"trace_id\":\"40eda190-83fd-4a1b-8155-3a1c7434b319\",\"data\":{}}]}",
3536
"type": [
3637
"indicator"

‎packages/blacklens/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ processors:
44
- set:
55
field: ecs.version
66
value: '8.11.0'
7+
- set:
8+
field: event.kind
9+
value: alert
710
- rename:
811
field: message
912
target_field: event.original

‎packages/blacklens/data_stream/alerts/fields/ecs.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# Define ECS constant fields as constant_keyword
22
- name: event.kind
3-
type: constant_keyword
4-
value: alert
3+
external: ecs
54
- name: observer.product
65
type: constant_keyword
76
value: blacklens.io

‎packages/blacklens/data_stream/alerts/sample_event.json‎

Lines changed: 64 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,64 +1,66 @@
11
{
2-
"@timestamp":"2024-11-12T09:39:58.489Z",
3-
"agent":{
4-
"ephemeral_id":"33939e93-54ef-4184-b92b-bc8f02e179a6",
5-
"id":"f98f4444-6fca-4500-83b6-a8c5e8f32bf1",
6-
"name":"elastic-agent-49577",
7-
"type":"filebeat",
8-
"version":"8.15.2"
9-
},
10-
"blacklens":{
11-
"alert":{
12-
"activities":[
13-
{
14-
"category":"threat",
15-
"created_date":"2025-12-30T16:11:40.195989Z",
16-
"description":"A Critical severity external vulnerability 'Blind SQL Injection via HTTP Header' has been detected on asset 'demo.example.com'",
17-
"id":"73dcaa88-09e1-4c58-9fa5-5495f8dac2a4",
18-
"trace_id":"40eda190-83fd-4a1b-8155-3a1c7434b319",
19-
"type":"ExternalVulnerabilityCreated"
20-
}
21-
],
22-
"analysis":"completed",
23-
"category":"vulnerability",
24-
"id":"7ea10c5d-559a-4c55-8608-2e060956de68",
25-
"name":"External Vulnerability Detected",
26-
"severity":"high",
27-
"status":"new",
28-
"updated_date":"2025-12-31T16:10:56.155Z"
29-
}
30-
},
31-
"data_stream":{
32-
"dataset":"blacklens.alerts",
33-
"namespace":"41265",
34-
"type":"logs"
35-
},
36-
"ecs":{
37-
"version":"8.11.0"
38-
},
39-
"elastic_agent":{
40-
"id":"f98f4444-6fca-4500-83b6-a8c5e8f32bf1",
41-
"snapshot":false,
42-
"version":"8.15.2"
43-
},
44-
"event":{
45-
"agent_id_status":"verified",
46-
"category":[
47-
"threat"
48-
],
49-
"created":"2025-12-09T05:45:05.855Z",
50-
"dataset":"blacklens.alerts",
51-
"id":"7ea10c5d-559a-4c55-8608-2e060956de68",
52-
"ingested":"2025-12-09T05:45:08Z",
53-
"type":[
54-
"indicator"
55-
]
56-
},
57-
"input":{
58-
"type":"httpjson"
59-
},
60-
"tags":[
61-
"forwarded",
62-
"blacklens-alert"
63-
]
2+
"@timestamp": "2026-02-03T06:17:57.260Z",
3+
"agent": {
4+
"ephemeral_id": "23a29ca3-cffa-4f55-9ef4-536308f22c95",
5+
"id": "6bcec12d-6281-4434-99e0-eb2f7c014fbf",
6+
"name": "elastic-agent-84667",
7+
"type": "filebeat",
8+
"version": "8.19.10"
9+
},
10+
"blacklens": {
11+
"alert": {
12+
"activities": [
13+
{
14+
"category": "threat",
15+
"created_date": "2026-02-03T06:17:57.260Z",
16+
"description": "A Critical severity external vulnerability 'Blind SQL Injection via HTTP Header' has been detected on asset 'demo.example.com'",
17+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa5",
18+
"trace_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
19+
"type": "ExternalVulnerabilityCreated",
20+
"updated_date": "2026-02-03T06:17:57.260Z"
21+
}
22+
],
23+
"analysis": "ongoing",
24+
"category": "vulnerability",
25+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
26+
"name": "External Vulnerability",
27+
"severity": "medium",
28+
"status": "new",
29+
"updated_date": "2026-02-03T06:17:57.260Z"
30+
}
31+
},
32+
"data_stream": {
33+
"dataset": "blacklens.alerts",
34+
"namespace": "43517",
35+
"type": "logs"
36+
},
37+
"ecs": {
38+
"version": "8.11.0"
39+
},
40+
"elastic_agent": {
41+
"id": "6bcec12d-6281-4434-99e0-eb2f7c014fbf",
42+
"snapshot": false,
43+
"version": "8.19.10"
44+
},
45+
"event": {
46+
"agent_id_status": "verified",
47+
"category": [
48+
"threat"
49+
],
50+
"created": "2026-03-31T08:58:43.557Z",
51+
"dataset": "blacklens.alerts",
52+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
53+
"ingested": "2026-03-31T08:58:46Z",
54+
"kind": "alert",
55+
"type": [
56+
"indicator"
57+
]
58+
},
59+
"input": {
60+
"type": "httpjson"
61+
},
62+
"tags": [
63+
"forwarded",
64+
"blacklens-alert"
65+
]
6466
}

‎packages/blacklens/docs/README.md‎

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -54,57 +54,59 @@ An example event for `alerts` looks as following:
5454

5555
```json
5656
{
57-
"@timestamp": "2024-11-12T09:39:58.489Z",
57+
"@timestamp": "2026-02-03T06:17:57.260Z",
5858
"agent": {
59-
"ephemeral_id": "33939e93-54ef-4184-b92b-bc8f02e179a6",
60-
"id": "f98f4444-6fca-4500-83b6-a8c5e8f32bf1",
61-
"name": "elastic-agent-49577",
59+
"ephemeral_id": "23a29ca3-cffa-4f55-9ef4-536308f22c95",
60+
"id": "6bcec12d-6281-4434-99e0-eb2f7c014fbf",
61+
"name": "elastic-agent-84667",
6262
"type": "filebeat",
63-
"version": "8.15.2"
63+
"version": "8.19.10"
6464
},
6565
"blacklens": {
6666
"alert": {
6767
"activities": [
6868
{
6969
"category": "threat",
70-
"created_date": "2025-12-30T16:11:40.195989Z",
70+
"created_date": "2026-02-03T06:17:57.260Z",
7171
"description": "A Critical severity external vulnerability 'Blind SQL Injection via HTTP Header' has been detected on asset 'demo.example.com'",
72-
"id": "73dcaa88-09e1-4c58-9fa5-5495f8dac2a4",
73-
"trace_id": "40eda190-83fd-4a1b-8155-3a1c7434b319",
74-
"type": "ExternalVulnerabilityCreated"
72+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa5",
73+
"trace_id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
74+
"type": "ExternalVulnerabilityCreated",
75+
"updated_date": "2026-02-03T06:17:57.260Z"
7576
}
7677
],
77-
"analysis": "completed",
78+
"analysis": "ongoing",
7879
"category": "vulnerability",
79-
"id": "7ea10c5d-559a-4c55-8608-2e060956de68",
80-
"name": "External Vulnerability Detected",
81-
"severity": "high",
80+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
81+
"name": "External Vulnerability",
82+
"severity": "medium",
8283
"status": "new",
83-
"updated_date": "2025-12-31T16:10:56.155Z"
84+
"updated_date": "2026-02-03T06:17:57.260Z"
8485
}
8586
},
8687
"data_stream": {
8788
"dataset": "blacklens.alerts",
88-
"namespace": "41265",
89+
"namespace": "43517",
8990
"type": "logs"
9091
},
9192
"ecs": {
9293
"version": "8.11.0"
9394
},
9495
"elastic_agent": {
95-
"id": "f98f4444-6fca-4500-83b6-a8c5e8f32bf1",
96+
"id": "6bcec12d-6281-4434-99e0-eb2f7c014fbf",
9697
"snapshot": false,
97-
"version": "8.15.2"
98+
"version": "8.19.10"
9899
},
99100
"event": {
100101
"agent_id_status": "verified",
101102
"category": [
102103
"threat"
103104
],
104-
"created": "2025-12-09T05:45:05.855Z",
105+
"created": "2026-03-31T08:58:43.557Z",
105106
"dataset": "blacklens.alerts",
106-
"id": "7ea10c5d-559a-4c55-8608-2e060956de68",
107-
"ingested": "2025-12-09T05:45:08Z",
107+
"id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
108+
"ingested": "2026-03-31T08:58:46Z",
109+
"kind": "alert",
108110
"type": [
109111
"indicator"
110112
]
@@ -143,7 +145,7 @@ An example event for `alerts` looks as following:
143145
| data_stream.namespace | Data stream namespace. | constant_keyword |
144146
| data_stream.type | Data stream type. | constant_keyword |
145147
| event.dataset | Event dataset. | constant_keyword |
146-
| event.kind | | constant_keyword |
148+
| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword |
147149
| event.module | Event module. | constant_keyword |
148150
| input.type | Type of filebeat input. | keyword |
149151
| observer.product | | constant_keyword |

0 commit comments

Comments
 (0)