[Bug] Hashicorp KMIP events

[srilumpa]

Proposed commit message

The PR aims to add support for KMIP events to the HashiCorp Vault integration.

KMIP events are currently handled directly into a flattened field in Elasticsearch. Unfortunately, the recursive structure of the object triggers a JSON exception upon search, making the Elasticsearch query crash.

To fix this issue, I go through the KMIP request and the KMIP response objects to remove the nested arrays structure and to obtain a "clean" JSON object. Request and response headers and data are then stored in their respective fields. Events is finally normalised by processing request/response operation and result. Normalisation relies heavily on the Key Management Interoperability Protocol Specification Version 2.0.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

How to test this PR locally

Related issues

• Closes [HashiCorp Vault]: Support for KMIP audit logs #12498

Screenshots

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[srilumpa]

Current status: Data normalized and does not generate a crash anymore when searched.

Next step: normalize KMIP operation and status to dedicated fields

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[srilumpa]

Hello, respectfully pinging @elastic/integration-experience to consider this PR.

[vera-review-bot]

👀 I have started reviewing the PR

[vera-review-bot]

👀 I have started reviewing the PR

[srilumpa]

Hello, respectfully pinging @andrewkroh and @elastic/integration-experience to consider this PR

[vera-review-bot]

Severity: 🟡 Medium confidence: medium path: packages/hashicorp_vault/data_stream/audit/elasticsearch/ingest_pipeline/default.yml:681

The new remote_address dissect splits on the first colon, so it mangles IPv6 addresses; anchor the split (or use grok) so only a trailing host:port is separated.

Details

The dissect pattern %{...remote_address}:%{...remote_port} runs whenever remote_address contains a colon and splits on the FIRST colon. Vault emits bare IPv6 client addresses for HTTP audit events (e.g. 2001:db8::1) and bracketed IPv6 host:port for KMIP (e.g. [2001:db8::1]:36776), both of which contain multiple colons. In those cases dissect puts a fragment like 2001 into remote_address and db8::1 into remote_port. The following convert of remote_port to long (guarded by instanceof String) then throws a NumberFormatException, and/or the convert of remote_address to source.ip fails, routing the event to the pipeline on_failure handler. Since the pre-PR pipeline converted remote_address straight to source.ip without splitting, this is a regression for IPv6 clients.

Recommendation:

Use an anchored grok that understands IPv6 brackets and a bare address, instead of a first-colon dissect:

- grok:
    tag: split_source_address_if_contains_port
    field: hashicorp_vault.audit.request.remote_address
    patterns:
      - '^\[%{IP:hashicorp_vault.audit.request.remote_address}\]:%{POSINT:hashicorp_vault.audit.request.remote_port}$'
      - '^%{IPV4:hashicorp_vault.audit.request.remote_address}:%{POSINT:hashicorp_vault.audit.request.remote_port}$'
      - '^%{IP:hashicorp_vault.audit.request.remote_address}$'
    ignore_missing: true

This only strips a trailing port when one is actually present and leaves IPv6 literals intact, so the downstream convert/set for source.ip and source.port continue to work.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Review summary

Issues found across the latest commits 11ad485…f6306d0 (86 commits) — 1 medium

• 🟡 The new remote_address dissect splits on the first colon, so it mangles IPv6 addresses (link) (Unresolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.