tetragon: Add support to retrieve environment variables #4184

olsajiri · 2025-10-13T16:18:17Z

storing environment variables into process event data, enabled with --enable-process-environment-variables option,
and we allow to redact info from the variables

reworked a bit the exec data parsing and added the environment variables parsing contributed by @slntopp,
I left the SOB info in some changes, @slntopp please shout if you want to make some changes

netlify · 2025-10-13T16:18:52Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`b546118`
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/691cf028ee2a1e00081b3cdd
😎 Deploy Preview	https://deploy-preview-4184--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

FedeDP

Left some comments but the PR is egregious :) Great job!

bpf/process/bpf_execve_event.c

As the comment suggested the filename could be sent through data event, let's read the filename instead. Signed-off-by: Jiri Olsa <[email protected]>

We can return just proc object and error without the bool state. Making nopMsgProcess part of execParse init code so there's no longer need for the function. Signed-off-by: Jiri Olsa <[email protected]>

We are about to add another extra info to the exec event (environment variables), so instead of relying on separators lets add size for each data and allow the user space easier parsing. Signed-off-by: Jiri Olsa <[email protected]>

Let's share the functionality in single function. Signed-off-by: Jiri Olsa <[email protected]>

Adding test for execParse with api.EventErrorFilename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Add test for execParse with just filename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Adding support to read environment variables. It's guarded with tg_conf_map::env_vars_enabled variable and same as for arguments we either use space in the exec event or send it through separate data event. Signed-off-by: Mikita Iwanowski <[email protected]> Signed-off-by: Jiri Olsa <[email protected]>

It will store environment variables for the process. Signed-off-by: Jiri Olsa <[email protected]>

Signed-off-by: Jiri Olsa <[email protected]>

Adding parsing of the environment variables data from the exec event and the --enable-process-environment-variables option to enable this feature. Signed-off-by: Mikita Iwanowski <[email protected]> Signed-off-by: Jiri Olsa <[email protected]>

Add test for execParse with filename, args, cwd, envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

Add support to redact data from environment variables. Signed-off-by: Jiri Olsa <[email protected]>

Signed-off-by: Jiri Olsa <[email protected]>

Adding docs for environment variables redaction. Signed-off-by: Jiri Olsa <[email protected]>

kkourt

Thanks!

Please find some comments below.

bpf/process/bpf_execve_event.c

kkourt · 2025-11-20T12:51:41Z

pkg/sensors/exec/exec_linux.go

-		var desc dataapi.DataEventDesc
-
-		dr := bytes.NewReader(args)
+	if exec.SizePath != 0 {


Are there legitimate cases where this can be zero? (similarly for the other added sizes)

yes, if probe read fails on the envs data, then EVENT_ERROR_ARGS should be set

But we continue reading everything else if I understand correctly from the code?

kkourt · 2025-11-20T12:55:06Z

bpf/process/bpf_execve_event.c

 }

+#ifdef __LARGE_BPF_PROG
+FUNC_INLINE __u32 read_envs(void *ctx, struct msg_execve_event *event)


Will this work correctly for execs happening from inside the kernel (e.g, call_usrmodehelper and friends).

yep, environment is stored the same way, so we get it.. like:

{"process_exec":{"process":{"exec_id":"cWVtdS0xOjUyNzE3MDc3ODA3MjoxMjYxOA==","pid":12618,"uid":0,"cwd":"/","binary":"/usr/lib/systemd/systemd-coredump","arguments":"12617 1000 1000 11 1763675427 102400000000 qemu-1 1 3","flags":"execve rootcwd clone","start_time":"2025-11-20T21:50:27.329868374Z","auid":4294967295,"parent_exec_id":"cWVtdS0xOjI5NzAwMDAwMDA6Mg==","tid":12618,"in_init_tree":false,"environment_variables":"HOME=/ TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin"},"parent":{"exec_id":"cWVtdS0xOjI5NzAwMDAwMDA6Mg==","pid":2,"uid":0,"binary":"[kthreadd]","flags":"procFS","start_time":"2025-11-20T21:41:43.129089721Z","auid":4294967295,"parent_exec_id":"cWVtdS0xOjE6MA==","tid":2,"in_init_tree":false}},"node_name":"qemu-1","time":"2025-11-20T21:50:27.329867762Z"}

kkourt · 2025-11-20T12:57:41Z

api/v1/tetragon/tetragon.proto

  // tool like nsenter or kubectl exec.
  google.protobuf.BoolValue in_init_tree = 20;
+  // Environment variables passed to the binary at execution.
+  string environment_variables = 21;


Question: Should we parse this in user-space and offer an array to the user?
E.g.,

message EnvEntry { string Key = 1; string Value = 2; }

And return:

repeated EnvEntry environment_variables = 21

not sure.. as you already mentioned catching this impacts performance already,
also it's sort of similar to args string which we handle the same way together with the redaction support

Yeah, but splitting a string in userspace (I'm guessing based on a ) character shouldn't have a big impact on performance, no?

Yeah I'm wondering too if it's not a nice UX improvement that is pretty cheap considering everything that we already do in this gRPC conversion. No strong feelings.

ok, will add that

kkourt · 2025-11-20T12:59:53Z

pkg/option/flags.go

+	flags.Bool(KeyEnableProcessEnvironmentVariables, false, "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance")
+


Let's add a note about sensitive information in the flag.

E.g., "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance, as well as capture sensitive information such as passwords in the events"

"as well as capture sensitive information such as passwords in the events" might be super verbose but we can also mention the feature we have for redacting such content here.

mtardy

Thanks, here are a few comments, I think using size based parsing is cleaner indeed and easier to understand than searching for those NULL bytes.

mtardy · 2025-11-20T18:56:31Z

bpf/process/bpf_execve_event.c

 	return size;
 }

+#ifdef __LARGE_BPF_PROG


if we are guarding this with __LARGE_BPF_PROG anyway, why can't we stop using tetragon_conf and use proper runtime constants https://ebpf-go.dev/concepts/global-variables/#runtime-constants ? (so that we benefit from dead code elimination etc.)

I think we wanted to switch existing code so that would be cool to not add more code using the legacy style if we can? wdyt?

probably good idea :) will check, thanks

mtardy · 2025-11-20T19:01:39Z

bpf/process/bpf_execve_event.c

-		// reuse p->args first string that contains the filename, this can't be
-		// above 256 in size (otherwise the complete will be send via data msg)
-		// which is okay because we need the 256 first bytes.
-		curr->bin.path_length = probe_read_str(curr->bin.path, BINARY_PATH_MAX_LEN, &p->args);
+		char *filename = (char *)ctx + (_(ctx->__data_loc_filename) & 0xFFFF);
+
+		curr->bin.path_length = probe_read_str(curr->bin.path, BINARY_PATH_MAX_LEN, (void *)filename);


ah I remember seeing this change in another patch set and couldn't get why doing that. So here I understand that it simplifies your changes from the next commit!

mtardy · 2025-11-20T19:05:20Z

api/v1/tetragon/tetragon.proto

  // tool like nsenter or kubectl exec.
  google.protobuf.BoolValue in_init_tree = 20;
+  // Environment variables passed to the binary at execution.
+  string environment_variables = 21;


Yeah I'm wondering too if it's not a nice UX improvement that is pretty cheap considering everything that we already do in this gRPC conversion. No strong feelings.

olsajiri added the release-note/minor This PR introduces a minor user-visible change label Oct 13, 2025

olsajiri force-pushed the pr/olsajiri/exec branch 3 times, most recently from 0fbb328 to 9dd22a4 Compare October 13, 2025 22:08

olsajiri mentioned this pull request Oct 14, 2025

tetragon: assorted fixes #4131

Merged

olsajiri force-pushed the pr/olsajiri/exec branch 2 times, most recently from e35bac7 to 310e337 Compare October 17, 2025 11:42

olsajiri force-pushed the pr/olsajiri/exec branch 3 times, most recently from f906c12 to 6a4eec3 Compare November 11, 2025 20:59

olsajiri changed the title ~~Pr/olsajiri/exec~~ tetragon: Add support to retrieve environment variables Nov 11, 2025

olsajiri marked this pull request as ready for review November 11, 2025 21:06

olsajiri requested review from a team and mtardy as code owners November 11, 2025 21:06

olsajiri requested review from jrfastab and kkourt November 11, 2025 21:06

olsajiri force-pushed the pr/olsajiri/exec branch from 6a4eec3 to 8b4eaf6 Compare November 15, 2025 21:07

FedeDP reviewed Nov 18, 2025

View reviewed changes

bpf/process/bpf_execve_event.c Outdated Show resolved Hide resolved

bpf/process/bpf_execve_event.c Show resolved Hide resolved

olsajiri and others added 9 commits November 18, 2025 22:04

tetragon: Read 4.19 curr->bin.path from tracepoint filename

66014af

As the comment suggested the filename could be sent through data event, let's read the filename instead. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Simplify execParse error value handling

efd2153

We can return just proc object and error without the bool state. Making nopMsgProcess part of execParse init code so there's no longer need for the function. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Factor data read from exec event into single function

2e7064b

Let's share the functionality in single function. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for execParse with api.EventErrorFilename

b8f4e58

Adding test for execParse with api.EventErrorFilename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add test for execParse with just filename

821340d

Add test for execParse with just filename and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add environment_variables string to Process message

eabb726

It will store environment variables for the process. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add generated code for previous changes

6d03814

Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/exec branch from 8b4eaf6 to f4de6a0 Compare November 18, 2025 22:04

olsajiri added 4 commits November 18, 2025 22:15

tetragon: Add test for execParse with environment variables

2c88fb3

Add test for execParse with filename, args, cwd, envs and make sure we get proper results. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Apply redaction filters also on environment variables

da81816

Add support to redact data from environment variables. Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add redact test for environment variables

1a8ae56

Signed-off-by: Jiri Olsa <[email protected]>

tetragon: Add docs for environment variables redaction

b546118

Adding docs for environment variables redaction. Signed-off-by: Jiri Olsa <[email protected]>

olsajiri force-pushed the pr/olsajiri/exec branch from f4de6a0 to b546118 Compare November 18, 2025 22:16

FedeDP approved these changes Nov 19, 2025

View reviewed changes

kkourt reviewed Nov 20, 2025

View reviewed changes

mtardy reviewed Nov 20, 2025

View reviewed changes

mtardy added release-note/major This PR introduces major new functionality and removed release-note/minor This PR introduces a minor user-visible change labels Nov 20, 2025

		flags.Bool(KeyEnableProcessEnvironmentVariables, false, "Include environment variables in process_exec events. Disabled by default. Note that this option can significantly increase the size of the events and may impact performance")

tetragon: Add support to retrieve environment variables #4184

Are you sure you want to change the base?

tetragon: Add support to retrieve environment variables #4184

Uh oh!

Conversation

olsajiri commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for tetragon ready!

Uh oh!

FedeDP left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kkourt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtardy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

olsajiri commented Oct 13, 2025 •

edited

Loading

netlify bot commented Oct 13, 2025 •

edited

Loading