LOG-6860: Support Splunk Metadata keys in ClusterLogForwarder #2999

vparfonov · 2025-03-26T17:55:16Z

Description

This PR addressed to extend the ClusterLogForwarder to support Splunk metadata fields when forwarding logs
so that we enable better integration with Splunk.

/cc @Clee2691 @cahartma
/assign @jcantrill

/cherry-pick

Links

Depending on PR(s):
Bugzilla:
Github issue:
JIRA: https://issues.redhat.com/browse/LOG-6860
Enhancement proposal: LOG-6734: Splunk Event Metadata: Enhancements Doc #2979

openshift-ci-robot · 2025-03-26T17:55:21Z

@vparfonov: This pull request references LOG-6860 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.8.0" version, but no target version was set.

In response to this:

Description

This PR addressed to extend the ClusterLogForwarder to support Splunk metadata fields when forwarding logs
so that we enable better integration with Splunk.

/cc @Clee2691 @cahartma
/assign @jcantrill

/cherry-pick

Links

Depending on PR(s):

Bugzilla:

Github issue:

JIRA: https://issues.redhat.com/browse/LOG-6860

Enhancement proposal: LOG-6734: Splunk Event Metadata: Enhancements Doc #2979

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

cahartma · 2025-03-27T13:20:56Z

/lgtm

api/observability/v1/clusterlogforwarder_types.go

api/observability/v1/output_types.go

internal/generator/vector/helpers/helpers.go

jcantrill · 2025-03-27T15:28:25Z

internal/validations/observability/outputs/splunk.go

+		payload := string(spec.Splunk.PayloadKey)
+		for _, v := range spec.Splunk.IndexedFields {
+			if !strings.HasPrefix(string(v), payload) {
+				results = append(results, fmt.Sprintf("Indexed field: %s not part of payload: %s", v, payload))


are we trying to restrict they payload field from being part of indexed fields? If so then the error should read that it can not or is not allowed

Indexed fields can't be outside of a given payload

I don't believe this to be a reasonable restriction even though it is the behavior of the encoder. Use the audit event example from the feature request where they are indexing annotations but the "event"/payload field comes from a sibling node in the original message. We should be able to copy the fields in question to the "root" of the message so the encoder picks them up while setting the payload to something else.

Maybe you are right, this checks are overhead and not clear. I will remove this

internal/validations/observability/outputs/splunk_test.go

test/framework/functional/output_splunk.go

test/functional/outputs/splunk/forward_to_splunk_test.go

jcantrill · 2025-03-27T15:42:57Z

/hold

vparfonov · 2025-03-28T14:24:44Z

/test functional-target

vparfonov · 2025-03-28T16:35:11Z

/test functional-target

vparfonov · 2025-03-28T20:32:20Z

/test functional-target

vparfonov · 2025-03-28T22:18:58Z

/test functional-target

vparfonov · 2025-03-29T08:26:44Z

/test functional-target

vparfonov · 2025-03-29T11:27:09Z

/test functional-target

vparfonov · 2025-03-29T13:45:13Z

/test functional-target

vparfonov · 2025-03-29T17:21:04Z

/test functional-target

vparfonov · 2025-03-29T19:14:50Z

/test functional-target

vparfonov · 2025-03-29T22:01:29Z

/test functional-target

vparfonov · 2025-04-03T13:40:35Z

/test e2e-target

jcantrill · 2025-03-28T13:05:37Z

api/observability/v1/output_types.go

+	// A dynamic value is encased in single curly brackets `{}` and MUST end with a static fallback value separated with `||`.
+	// Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes.
+	// If not specified will be detected according to .log_source and .log_type value.
+	// Details see in: docs/features/logforwarding/outputs/splunk-forwarding.adoc


This will have to be a full link URL to be useful. Ideally it would be to the released branch instead of 'master' but that will have to do until we release

yes, i will change this after we will release

jcantrill · 2025-04-03T14:18:57Z

internal/validations/observability/outputs/splunk.go

+		payload := string(spec.Splunk.PayloadKey)
+		for _, v := range spec.Splunk.IndexedFields {
+			if !strings.HasPrefix(string(v), payload) {
+				results = append(results, fmt.Sprintf("Indexed field: %s not part of payload: %s", v, payload))


I don't believe this to be a reasonable restriction even though it is the behavior of the encoder. Use the audit event example from the feature request where they are indexing annotations but the "event"/payload field comes from a sibling node in the original message. We should be able to copy the fields in question to the "root" of the message so the encoder picks them up while setting the payload to something else.

internal/validations/observability/outputs/splunk_test.go

internal/generator/vector/output/splunk/splunk_sink_with_indexed_fields.toml

jcantrill · 2025-04-09T17:28:31Z

/approve

openshift-ci · 2025-04-09T17:36:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcantrill, vparfonov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jcantrill · 2025-04-09T17:37:56Z

/lgtm

vparfonov · 2025-04-12T17:22:09Z

/retest

Signed-off-by: Vitalii Parfonov <[email protected]>

jcantrill · 2025-04-21T14:20:24Z

/hold cancel
/lgtm

jcantrill · 2025-04-21T14:20:45Z

/lgtm

openshift-cherrypick-robot · 2025-04-21T18:27:18Z

@vparfonov: cannot checkout <!--: error checking out "<!--": exit status 1 error: pathspec '<!--' did not match any file(s) known to git

In response to this:

Description

This PR addressed to extend the ClusterLogForwarder to support Splunk metadata fields when forwarding logs
so that we enable better integration with Splunk.

/cc @Clee2691 @cahartma
/assign @jcantrill

/cherry-pick

Links

Depending on PR(s):

Bugzilla:

Github issue:

JIRA: https://issues.redhat.com/browse/LOG-6860

Enhancement proposal: LOG-6734: Splunk Event Metadata: Enhancements Doc #2979

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 26, 2025

openshift-ci bot requested review from Clee2691 and jcantrill March 26, 2025 17:56

vparfonov force-pushed the log6860 branch from 8dddf4d to 6134ab6 Compare March 26, 2025 20:46

openshift-ci bot assigned cahartma Mar 27, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

jcantrill added the release/6.3 label Mar 27, 2025

vparfonov force-pushed the log6860 branch from 6134ab6 to dae40d4 Compare March 27, 2025 13:34

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2025

vparfonov force-pushed the log6860 branch from dae40d4 to bf491b5 Compare March 27, 2025 14:25

jcantrill reviewed Mar 27, 2025

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 27, 2025

vparfonov requested a review from jcantrill March 28, 2025 12:18

vparfonov force-pushed the log6860 branch from 0f8c854 to 248268e Compare March 28, 2025 12:26

vparfonov marked this pull request as draft March 28, 2025 20:31

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 28, 2025

vparfonov force-pushed the log6860 branch from 06c3d18 to e9b8696 Compare March 31, 2025 10:54

vparfonov marked this pull request as ready for review March 31, 2025 11:12

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 31, 2025

jcantrill reviewed Apr 3, 2025

View reviewed changes

vparfonov requested a review from jcantrill April 3, 2025 14:37

vparfonov force-pushed the log6860 branch from 32b09c7 to 5b2256c Compare April 3, 2025 14:49

jcantrill reviewed Apr 9, 2025

View reviewed changes

internal/generator/vector/output/splunk/splunk_sink_with_indexed_fields.toml Show resolved Hide resolved

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 9, 2025

openshift-ci bot assigned jcantrill Apr 9, 2025

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Apr 9, 2025

vparfonov force-pushed the log6860 branch from dde4f6b to cea8db6 Compare April 12, 2025 14:30

LOG-6860: Support Splunk Metadata keys in ClusterLogForwarder

3df3daa

Signed-off-by: Vitalii Parfonov <[email protected]>

vparfonov force-pushed the log6860 branch from cea8db6 to 3df3daa Compare April 13, 2025 13:51

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 21, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 21, 2025

openshift-merge-bot bot merged commit 9c6f260 into openshift:master Apr 21, 2025
6 of 8 checks passed

vparfonov deleted the log6860 branch April 22, 2025 10:11

vparfonov mentioned this pull request Apr 22, 2025

LOG-6860: fixup: changes in generated CRD and CSV #3030

Merged

LOG-6860: Support Splunk Metadata keys in ClusterLogForwarder #2999

LOG-6860: Support Splunk Metadata keys in ClusterLogForwarder #2999

Uh oh!

Conversation

vparfonov commented Mar 26, 2025

Description

Links

Uh oh!

openshift-ci-robot commented Mar 26, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

cahartma commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jcantrill Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

vparfonov Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

jcantrill Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

vparfonov Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jcantrill commented Mar 27, 2025

Uh oh!

vparfonov commented Mar 28, 2025

Uh oh!

vparfonov commented Mar 28, 2025

Uh oh!

vparfonov commented Mar 28, 2025

Uh oh!

vparfonov commented Mar 28, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Mar 29, 2025

Uh oh!

vparfonov commented Apr 3, 2025

Uh oh!

jcantrill Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

vparfonov Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

jcantrill Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jcantrill commented Apr 9, 2025

Uh oh!

openshift-ci bot commented Apr 9, 2025

Uh oh!

jcantrill commented Apr 9, 2025

Uh oh!

vparfonov commented Apr 12, 2025

Uh oh!

jcantrill commented Apr 21, 2025

openshift-ci-robot commented Mar 26, 2025 •

edited by openshift-ci bot

Loading