gigamon: add CEF/UDP input to ami data stream

[sanjay-2307]

WHAT:
Adds a UDP input for CEF log ingestion to the existing Gigamon AMI
data stream alongside the existing HTTP Endpoint (JSON) input.

• Added UDP input in manifest.yml for CEF log collection
• Added cef-pipeline.yml for GigamonMdata* field normalization before rejoining the main default pipeline

WHY:
Gigamon AMI supports two output formats - JSON over HTTP and CEF over
UDP. The existing package only supported JSON. This change adds CEF
support within the same data stream so both inputs share the same
index, ECS mappings, and dashboards.

[cla-checker-service]

💚 CLA has been signed

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[kcreddy]

/test

[elasticmachine]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[sanjay-2307]

@kcreddy could you please drop a comment to initiate the test ?

[kcreddy]

/test

[sanjay-2307]

@kcreddy we noticed a few fields that still need to be renamed or dropped. I’ll update those early next week, and then you can proceed with merging the PR.

[kcreddy]

@kcreddy we noticed a few fields that still need to be renamed or dropped. I’ll update those early next week, and then you can proceed with merging the PR.

@sanjay-2307, I've moved this PR to draft til then. You can mark it Ready for review when your changes are made.

and then you can proceed with merging the PR.

GIgamon used to be owned by our team, but recently there was a change in ownership. Once the owned team approves the PR, then only it can be merged.

[sanjay-2307]

@kcreddy PR is now ready for review. Could you please trigger the Buildkite test with /test to kick off the CI pipeline?

[kcreddy]

/test

[kcreddy]

@sanjay-2307 , there is an error in the CI:
Error: checking package failed: checking readme files are up-to-date failed: files do not match

You will need to run elastic-package build and commit the generated README.md file.

[sanjay-2307]

@kcreddy I have added the latest README.md file.

[kcreddy]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c78a3af

History

• 💔 Build #42092 failed 79a01d8
• 💔 Build #41381 failed df66721

[kcreddy]

@elastic/integration-experience PTAL at this PR. This integration recently changed ownership from Service Integrations team to IE.

[sanjay-2307]

@elastic/integration-experience / @andrewkroh Gentle follow-up on this PR, could you please share the current status when you have a moment?

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[vera-review-bot]

👀 I have started reviewing the PR

[vera-review-bot]

🟡 MEDIUM data_stream/ami/.../default.yml:12

New CEF/UDP processing path ships without pipeline tests

This PR adds a brand-new CEF processing branch here (- pipeline: guarded by if: ctx.cef != null) that calls the new 730-line cef-pipeline.yml. None of it is exercised by a test: the only pipeline fixture, _dev/test/pipeline/test-ami.json, contains JSON-input events only, so every rename in cef-pipeline.yml and the entire cef != null branch run untested. This is a newly introduced code path (not a pre-existing gap), so a regression in CEF field renaming, the cef.extensions -> gigamon.ami mapping, or the new TcpRtt* converts would not be caught.

Recommendation:

Add a CEF pipeline test fixture and expected output covering the cef != null branch, e.g. _dev/test/pipeline/test-cef.log with a matching test-cef.log-expected.json. Configure the CEF test in _dev/test/pipeline/test-common-config.yml so the decode happens correctly:

fields:
  tags:
    - preserve_original_event
dynamic_fields:
  "event.ingested": ".*"

The fixture should include at least one raw CEF message that populates gigamon.ami.* extension fields (src/dst IP, ports, the new TcpRtt* and app_tags fields) so the new pipeline and converts are validated.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[sanjay-2307]

I wanted to confirm the right approach, as I hit a constraint while implementing it.

The recommendation was to add a raw test-cef.log fixture and configure decoding in test-common-config.yml. However, in this package CEF decoding is done by the agent-side decode_cef processor (in udp.yml.hbs), and decode_cef is not available as an Elasticsearch ingest processor. When I tried adding decode_cef to the ingest pipeline (default.yml), Put Pipeline fails with:

No processor type exists with name [decode_cef]

Because pipeline tests only exercise the ingest pipeline (not agent processors), a raw test-cef.log never gets decoded during the test, so ctx.cef stays null and the cef != null branch isn’t actually exercised (the generated expected doc only contains the raw message and no gigamon.ami.* fields).

To genuinely cover the branch, I added a pre-decoded CEF fixture (test-cef.json) that mirrors the agent’s decode_cef output (populated cef.extensions), which correctly triggers the cef != null path and validates the cef.extensions → gigamon.ami renames, the new TcpRtt* converts, and app_tags. Both tests pass:

test-ami.json PASS
test-cef.json PASS

Could you please confirm this approach is acceptable, i.e. using a pre-decoded test-cef.json fixture instead of a raw test-cef.log, given that decode_cef can only run agent-side? If you’d prefer a different pattern for covering the CEF branch, happy to adjust.

[vera-review-bot]

No issues across the latest commits bfb883f.

Review summary

Issues found across earlier commits 591860c — 1 medium, 2 low

• 🟡 The new UDP stream declares a preserve_duplicate_custom_fields variable, but udp.yml.hbs never renders it into tags, so the toggle is inert — enabling it has no effect. Drop the variable, or wire it into the emitted tags so the pipeline's tags.contains('preserve_duplicate_custom_fields') guard can see it. (link) (Resolved)
• 🔵 The new app_tags field definition has trailing whitespace after type: keyword, which elastic-package format will normalise. Strip the trailing spaces to keep the file clean and avoid a format-check delta. (link) (Resolved)
• 🔵 The updated README.md ends without a trailing newline on the final tags table row. Restore the trailing newline so the file matches standard generated docs formatting. (link) (Resolved)

Issues found across earlier commits 05bbead — 1 low

• 🔵 The new cef-pipeline dispatch processor uses tag: cef pipeline, which contains a space and breaks the snake_case convention every other tag in default.yml follows (link) (Resolved)

Issues found across earlier commits 4b389d5 — 2 high

• 🟠 The expected fixture asserts PascalCase gigamon.ami.TcpRtt* keys, but the renamed cef-pipeline now emits snake_case tcp_rtt_* keys, so the pipeline test will fail (link) (Resolved)
• 🟠 The README field table lists PascalCase gigamon.ami.TcpRtt* fields that no longer exist after the snake_case rename, so it is out of date and the readme-up-to-date check will fail (link) (Resolved)

Issues found across earlier commits 087d7ab — 1 medium, 1 low

• 🟡 The new TcpRtt* fields use PascalCase, breaking the snake_case naming convention every other gigamon.ami field follows (link) (Resolved)
• 🔵 The rename/remove processors in the new cef-pipeline.yml have no tag fields, so on_failure error.message cannot identify which processor failed (link) (Resolved)

Issues found across earlier commits 2a55beb…5c3e6ee (250 commits) — 1 low

• 🔵 The new cef-pipeline.yml on_failure omits set event.kind: pipeline_error (link) (Resolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.