Skip to content

[juniper_srx] extract IP from RPD_IFD_NOTIFICATION events #18927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ilyannn wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[juniper_srx] extract IP from RPD_IFD_NOTIFICATION events #18927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
db0a75e
[juniper_srx] extract IP from RPD_IFD_NOTIFICATION events
ilyannn May 9, 2026
2253eef
Merge branch 'main' into juniper_srx-system-pipeline-tweaks
ilyannn May 18, 2026
6a4507c
Merge remote-tracking branch 'origin/main' into juniper_srx-system-pi…
ilyannn Jul 1, 2026
6ee51ec
[juniper_srx] address review: drop redundant if_trace, map interface …
ilyannn Jul 1, 2026
e56617f
Merge branch 'main' into juniper_srx-system-pipeline-tweaks
ilyannn Jul 3, 2026
363ff7a
Merge branch 'main' into juniper_srx-system-pipeline-tweaks
ilyannn Jul 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/juniper_srx/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.28.0"
changes:
- description: Extract interface address from RPD_IFD_NOTIFICATION events into host.ip and related.ip, map the interface index to observer.ingress.interface.id, and avoid overwriting an already-populated message field.
type: enhancement
link: https://github.com/elastic/integrations/pull/18927
- version: "1.27.2"
changes:
- description: Fix Syslog Parsing without syslog version
Expand Down
3 changes: 2 additions & 1 deletion packages/juniper_srx/data_stream/log/_dev/test/pipeline/test-system.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,5 @@
<166>1 2023-05-08T10:54:24.756+10:00 AB1234-A-AB-AB01C-ABC - - - - AB1234-A-AB-AB01C-ABC nh_fabric_fill_jnhinfo: Storing nh_id as 0x2dd and jnh as 0x58e302
<167>1 2023-05-08T10:54:24.704+10:00 AB1234-A-AB-AB01C-ABC - - - - AB1234-A-AB-AB01C-ABC Copying remote chassis chassis 1, IP: 81.2.69.192
<166>1 2023-05-08T10:54:24.756+10:00 AB1234-A-AB-AB01C-ABC - - - - AB1234-A-AB-AB01C-ABC nh_fabric_fill_jnhinfo: ABCDE: Test default message 123456
<166> 2023-05-08T10:54:24.756+10:00 AB1234-A-AB-AB01C-ABC - - - - AB1234-A-AB-AB01C-ABC nh_fabric_fill_jnhinfo: ABCDE: Test default message 123456
<30>1 2026-05-07T12:35:17.448+02:00 AB1234-A-AB-AB01C-ABC rpd 20893 RPD_IFD_NOTIFICATION [junos@1111.1.1.1.1.111 state="UpDown" interface-name="st0.60" interface-index="508" message="<Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0"] IF_TRACE: EVENT <UpDown> st0.60 index 508 <Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0
<166> 2023-05-08T10:54:24.756+10:00 AB1234-A-AB-AB01C-ABC - - - - AB1234-A-AB-AB01C-ABC nh_fabric_fill_jnhinfo: ABCDE: Test default message 123456
57 changes: 57 additions & 0 deletions packages/juniper_srx/data_stream/log/_dev/test/pipeline/test-system.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1171,6 +1171,63 @@
"preserve_original_event"
]
},
{
"@timestamp": "2026-05-07T10:35:17.448Z",
"ecs": {
"version": "8.17.0"
},
"event": {
"category": [
"network"
],
"kind": "event",
"original": "<30>1 2026-05-07T12:35:17.448+02:00 AB1234-A-AB-AB01C-ABC rpd 20893 RPD_IFD_NOTIFICATION [junos@1111.1.1.1.1.111 state=\"UpDown\" interface-name=\"st0.60\" interface-index=\"508\" message=\"<Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0\"] IF_TRACE: EVENT <UpDown> st0.60 index 508 <Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0",
"severity": 30
},
"host": {
"ip": [
"175.16.199.0"
],
"name": "AB1234-A-AB-AB01C-ABC"
},
"juniper": {
"srx": {
"log_type": "system",
"message": "<Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0",
"process": "rpd",
"state": "UpDown",
"tag": "RPD_IFD_NOTIFICATION"
}
},
"log": {
"level": "informational"
},
"message": "IF_TRACE: EVENT <UpDown> st0.60 index 508 <Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0",
"observer": {
"ingress": {
"interface": {
"id": "508",
"name": "st0.60"
}
},
"name": "AB1234-A-AB-AB01C-ABC",
"product": "SRX",
"type": "firewall",
"vendor": "Juniper"
},
"process": {
"name": "rpd",
"pid": 20893
},
"related": {
"ip": [
"175.16.199.0"
]
},
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2023-05-08T00:54:24.756Z",
"ecs": {
Expand Down
7 changes: 7 additions & 0 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,13 @@ processors:
field: juniper.srx.interface_name
target_field: observer.ingress.interface.name
ignore_missing: true
# interface-index is the SNMP/IFD interface index, which maps to ECS observer.*.interface.id.
# Interface up/down notifications carry no flow direction, so it sits under ingress alongside the interface name.
- rename:
tag: rename_juniper_srx_interface_index_to_observer_ingress_interface_id
field: juniper.srx.interface_index
target_field: observer.ingress.interface.id
ignore_missing: true
- rename:
tag: rename_juniper_srx_source_zone_name_to_observer_ingress_zone_ebc8c681
field: juniper.srx.source_zone_name
Expand Down
35 changes: 33 additions & 2 deletions packages/juniper_srx/data_stream/log/elasticsearch/ingest_pipeline/system.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,28 @@ processors:
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

# RPD_IFD_NOTIFICATION
# Extract interface address (e.g. "address #0 175.16.199.0") into host.ip and related.ip
- grok:
tag: grok_rpd_ifd_notification_address
if: "ctx.juniper?.srx?.tag == 'RPD_IFD_NOTIFICATION' && ctx.juniper?.srx?.message != null"
field: juniper.srx.message
patterns:
- 'address #%{NONNEGINT} %{IP:_temp_.rpd_ifd.address}'
ignore_failure: true
- append:
tag: append_rpd_ifd_address_to_host_ip
if: "ctx._temp_?.rpd_ifd?.address != null"
field: host.ip
value: '{{{_temp_.rpd_ifd.address}}}'
allow_duplicates: false
- append:
tag: append_rpd_ifd_address_to_related_ip
if: "ctx._temp_?.rpd_ifd?.address != null"
field: related.ip
value: '{{{_temp_.rpd_ifd.address}}}'
allow_duplicates: false

# After System Structured, Structured-Brief, and Unstructured messages: there are optional key-value pairs seperated by ", ". These should now be in `message` field
# split k-v fields inside message field.
- kv:
Expand All @@ -329,6 +351,12 @@ processors:
source: >-
ctx.juniper.srx.system = ctx.juniper.srx.system.entrySet().stream().collect(Collectors.toMap(e -> e.getKey().replace(' ', '_').replace('-', '_').toLowerCase(), e -> e.getValue().trim()));

# Drop juniper.srx.system.if_trace: the k-v extractor above surfaces it from IF_TRACE messages, but it duplicates `message` (message minus the "IF_TRACE: " prefix).
- remove:
tag: remove_juniper_srx_system_if_trace
field: juniper.srx.system.if_trace
ignore_missing: true

#######################
## SRX System Fields ##
#######################
Expand Down Expand Up @@ -390,7 +418,10 @@ processors:
- "-"
- "Not-Available"
source: >-
ctx?.juniper?.srx?.system.entrySet().removeIf(entry -> params.values.contains(entry.getValue()));
ctx.juniper.srx.system.entrySet().removeIf(entry -> params.values.contains(entry.getValue()));
if (ctx.juniper.srx.system.isEmpty()) {
ctx.juniper.srx.remove('system');
}

# Copy `message_brief` into `message` if not already exists. This ensures unpased/unstructured data to be present in `message`.
- set:
Expand Down Expand Up @@ -901,7 +932,7 @@ processors:
field: juniper.srx.message
target_field: message
ignore_missing: true
if: "ctx.juniper?.srx?.message != null"
if: "ctx.juniper?.srx?.message != null && ctx.message == null"

@taylor-swanson taylor-swanson Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we getting complaints about overwriting an existing message?

@ilyannn ilyannn Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I just did not want to let juniper.srx.message field:

"message": "<Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0",

clobber the message field that is set in a different part:

"message": "IF_TRACE: EVENT <UpDown> st0.60 index 508 <Up Broadcast Multicast> agg_parent 0 address #0 175.16.199.0",

They come from two different places, one the key message and one the free text after the []:

<30>1 ... RPD_IFD_NOTIFICATION [junos@1111.1.1.1.1.111 ... message="<Up Broadcast Multicast> ..."] IF_TRACE: EVENT <UpDown> st0.60 index 508 <Up Broadcast Multicast> ...

Without the guard the rename just fails.

@ilyannn ilyannn Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can fix the rename failure (clear the message first), but since we do have juniper.srx.message field in fields.yml it looked like we might want to use that field as well. The description of that field is not very helpful:

    - name: message
      type: keyword
      description: |
        mesagge

This field is present in test-atp.log, for example:

            "juniper": {
                "srx": {
                    "message": "malware analysis detected host downloaded a malicious_file with score 9, sha256 ABC123",

and we don't rename it to message there.


- remove:
tag: remove_juniper_srx_process_f8fae6df
Expand Down
2 changes: 1 addition & 1 deletion packages/juniper_srx/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.3"
name: juniper_srx
title: Juniper SRX
version: "1.27.2"
version: "1.28.0"
description: Collect logs from Juniper SRX devices with Elastic Agent.
categories: ["network", "security", "firewall_security"]
type: integration
Expand Down
Loading