wip: feat(sensors): support 38+ override on lsm funcs #4244

holyspectral · 2025-10-23T20:08:59Z

Description

This draft PR is to get early feedback about the potential fix for #4204. Any comments will be appreciated.

As mentioned in #4204, this PR makes multiple kprobe sensors to share their fmod_ret programs, as long as they attach to the same function.

The changes include:

A new probe type generic_fmod_ret is created, so we can manage the lifecycle of shared fmod_ret separately from the original kprobe sensors.
program.Program and program.Map's LoadState fields are used to track the reference count of these data, so they can be automatically cleaned up.
The pinPath of these programs/maps are moved to a global path under /tetragon/fmod_ret/<func>/. For example,
- /.../tetragon/fmod_ret/security_bprm_creds_for_exec/prog
- /.../tetragon/fmod_ret/security_bprm_creds_for_exec/override_tasks
A unload() function is injected into unloader.UnloadProg(), so we can clean up fmodretMap.

There might be a better way to implement these, so any comments will be appreciated.

Changelog

holyspectral · 2025-10-23T20:09:44Z

@olsajiri may I have your feedback on this one? Thanks a lot.

olsajiri

looks good, left few comments, thanks

olsajiri · 2025-10-24T10:08:16Z

pkg/sensors/tracing/genericfmod_ret.go

+	if fmodret, ok = fmodretMap[attachFunc]; !ok {
+
+		fmodret = program.Builder(
+			path.Join(option.Config.HubbleLib, loadProgName),


since we already add new program for this, could we move override programs into separated object?
that might speed up the load

Sure. I think it makes a lot of sense to do this. Thanks!

olsajiri · 2025-10-24T10:09:34Z

pkg/sensors/tracing/genericfmod_ret.go

+	"github.com/cilium/tetragon/pkg/sensors/program"
+)
+
+var fmodretMap map[string]*program.Program


I was thinking about something similar but bit more generic to cover also standard kprobe override programs.. there's no limitation for attached kprobe programs, but it would at least benefit from having just single copy of the program and reduce the footprint

I think that's a good idea. Let me look into this.

One issue I noticed is that, if I'm not mistaken, for kprobe override programs, when multi-kprobe is enabled, each kprobe attach points actually share the same override_maps. Do you think we should change that and use per-hookpoint map instead?

hum not sure what you mean..

Ah, I wasn't aware that the creation and loading of maps and programs in multi kprobe flows are based on the data type of load.LoaderData. That makes sense now. Thanks a lot!

olsajiri · 2025-10-24T10:13:33Z

pkg/sensors/tracing/genericfmod_ret.go

+		fmodret.PinPath = "fmod_ret/" + attachFunc
+
+		fmodretmap := program.MapBuilder("override_tasks", fmodret)
+		fmodretmap.PinPath = path.Join("fmod_ret/", attachFunc, "override_tasks")


i'd rather place it in some visibly special place like we do for base sensor, perhaps something like:

__override__/fmodret/security_xxx __override__/fmodret/security_yyy ... __override__/kprobe/ksys_read __override__/kprobe/ksys_write ... __override__/override_tasks

olsajiri · 2025-10-24T10:14:31Z

pkg/sensors/load.go

-		// setup sensor based program pin path
-		p.PinPath = filepath.Join(s.policyDir(), s.Name, p.PinName)
+		// setup sensor based program pin path if it's not specified
+		if p.PinPath == "" {


is this needed, or it's just a leftover? PinPath should be always defined even with your change no?

yeah there might be a better way to implement this, but the code here is basically a lazy initialization. When the PinPath is already given, it uses the existing one. For override program, we prefill the PinPath so it falls under this category.

For other programs, the default PinPath under s.policyDir() will be used in Ln70.

olsajiri · 2025-10-24T10:16:13Z

pkg/sensors/tracing/generickprobe.go

+	logger.GetLogger().Info("loading generic fmod ret program", "prog", load)
+
+	unload := func() {
+		deleteFmodRetProg(load.Attach)


could we use program.Program.unloaderOverride or add something in there so it gets removed during program's unload?

Ah I didn't notice this function exists when I wrote the code. From its name it's definitely more suitable. Let me look into this.

netlify · 2025-11-13T14:42:43Z

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	`acfea29`
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/6915ee392505df00076c3f96
😎 Deploy Preview	https://deploy-preview-4244--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

holyspectral · 2025-11-13T14:51:48Z

Changes since v1:

Allow kprobe to use shared override programs.
Instead of having one override_tasks map for each hook point, in v2 each override program shares the same override_tasks map. This helps to support multi-kprobe scenario.
The pinned path of ebpf programs and maps will be as below:

/bpffs/tetragon/__override__
/bpffs/tetragon/__override__/kprobe
/bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat
/bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat/link_override
/bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat/prog_override
/bpffs/tetragon/__override__/kprobe/__x64_sys_execve
/bpffs/tetragon/__override__/kprobe/__x64_sys_execve/link_override
/bpffs/tetragon/__override__/kprobe/__x64_sys_execve/prog_override
/bpffs/tetragon/__override__/override_tasks
/bpffs/tetragon/__override__/fmod_ret
/bpffs/tetragon/__override__/fmod_ret/security_bprm_creds_for_exec
/bpffs/tetragon/__override__/fmod_ret/security_bprm_creds_for_exec/prog

Make sure that unused override programs are cleaned up via
unloaderOverride().
Move bpf functions regarding overrides into another object file.

holyspectral · 2025-11-13T15:20:37Z

I still keep this as draft because of an item that I'd like to discuss first. Say we have a scenario is like the below (keep in mind that in v2 we use a shared override_tasks map):

We have two tracing policies A & B.
- PolicyA is hooked on sys_execve and policyB is on sys_symlinkat.
- They both have their own override action in matchActions.
A process triggers the override action of policy A via sys_execve, and has an item inserted into the override_tasks map.
- override_tasks[<pid_tgid>] = -EPERM
Before the program triggers the override program attached to sys_execve, a policy change on A happens and the override program on sys_execve is removed from another core.
- the content of the map stays the same, override_tasks[<pid_tgid>] = -EPERM
- but the override program on sys_execve is removed.
When the program triggers policy B via sys_symlinkat later, because override_tasks[<pid_tgid>] = -EPERM, the action will be denied unexpectedly, even if nothing is matched.

I can think of a few directions:

Maybe this is not a real issue, the window is too short, or Tetragon has already handled this, so we don't have to care about it.
Change the override_tasks to be a BPF_MAP_TYPE_HASH_OF_MAPS and let its inner map be associated to each hook points, so we can clean up the content of override_tasks map when we remove an override program. For example,

    override_tasks: {
        "syscall:sys_execve": <inner map like the current override_tasks map>,
        "syscall:sys_symlinkat": <inner map like the current override_tasks map>,
        "fmod_ret:security_bprm_creds_for_exec": <inner map like the current override_tasks map>,
    }

We don't delete the override programs immediately when its policy is removed. This will give the override programs some time to remove the items.

@olsajiri do you think this is a real issue that we should address? I'd love to know your thoughts on this.

olsajiri · 2025-11-16T22:55:41Z

I can think of a few directions:

1. Maybe this is not a real issue, the window is too short, or Tetragon has already handled this, so we don't have to care about it.

2. Change the `override_tasks` to be a BPF_MAP_TYPE_HASH_OF_MAPS and let its inner map be associated to each hook points, so we can clean up the content of override_tasks map when we remove an override program.  For example,

    override_tasks: {
        "syscall:sys_execve": <inner map like the current override_tasks map>,
        "syscall:sys_symlinkat": <inner map like the current override_tasks map>,
        "fmod_ret:security_bprm_creds_for_exec": <inner map like the current override_tasks map>,
    }

3. We don't delete the override programs immediately when its policy is removed.  This will give the override programs some time to remove the items.

@olsajiri do you think this is a real issue that we should address? I'd love to know your thoughts on this.

yea, that seems like a problem.. so at the moment override_task is program's map, so each override program has its own copy, I'd suggest to have some state of this change doing the same, and adding a change to single map on top of that

perhaps we could have policy id as part of the override_task value and have sensor unload to cleanup its records before it unloads the override program.. something like you suggest in 2) but not sure what's benefit of inner map

I'll check on your change in more detail but on first glance please try to split the change into more logical changes/commits, it's easier to review, thanks

Move bpf functions regarding overrides into bpf_generic_override.o Signed-off-by: Sam Wang (holyspectral) <[email protected]>

Provide a custom CustomUnloader, so custom actions can be performed when a ebpf program is unloaded and unloaderOverride is called. Signed-off-by: Sam Wang (holyspectral) <[email protected]>

1. Allow policies to share override programs as long as they share the same attached functions. This includes both programs based on kprobe and fmod_ret. 2. The pinned path of ebpf programs and maps are moved to below locations: /bpffs/tetragon/__override__ /bpffs/tetragon/__override__/kprobe /bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat /bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat/link_override /bpffs/tetragon/__override__/kprobe/__x64_sys_symlinkat/prog_override /bpffs/tetragon/__override__/kprobe/__x64_sys_execve /bpffs/tetragon/__override__/kprobe/__x64_sys_execve/link_override /bpffs/tetragon/__override__/kprobe/__x64_sys_execve/prog_override /bpffs/tetragon/__override__/override_tasks /bpffs/tetragon/__override__/fmod_ret /bpffs/tetragon/__override__/fmod_ret/security_bprm_creds_for_exec /bpffs/tetragon/__override__/fmod_ret/security_bprm_creds_for_exec/prog Signed-off-by: Sam Wang (holyspectral) <[email protected]>

Signed-off-by: Sam Wang (holyspectral) <[email protected]>

holyspectral · 2025-11-19T20:26:27Z

perhaps we could have policy id as part of the override_task value and have sensor unload to cleanup its records before it unloads the override program.. something like you suggest in 2) but not sure what's benefit of inner map

Thanks. I think that makes sense. Let me see if I can add an extra field to the key or value of override_tasks map.

I'll check on your change in more detail but on first glance please try to split the change into more logical changes/commits, it's easier to review, thanks

I've split them into multiple commits. Please feel free to let me know if anything is not clear!

olsajiri

thanks, left few comments, please split the change into more commits in next version

olsajiri · 2025-11-19T21:08:33Z

pkg/sensors/program/loader_linux.go

+			spec *ebpf.ProgramSpec,
+		) (unloader.Unloader, error) {
+
+			pinPath := filepath.Join(bpfDir, load.PinPath, "prog_override")


perhaps no need for the override suffix now? also for link

olsajiri · 2025-11-19T21:10:43Z

pkg/sensors/program/loader_linux.go

-func LoadFmodRetProgram(bpfDir string, load *Program, maps []*Map, progName string, verbose int) error {
+func LoadKProbeOverrideProgram(bpfDir string, load *Program, maps []*Map, progName string, verbose int, unloadFunc func(bool) error) error {
+	opts := &LoadOpts{
+		Attach: func(


we tend to put callbacks code in functions, please add override attach and open fnuctions

olsajiri · 2025-11-19T21:12:53Z

pkg/sensors/program/loader_linux.go

+			if !ok {
+				return fmt.Errorf("progName %s not in collecition spec programs: %+v", progName, coll.Programs)
+			}
+			progSpec.AttachTo = load.Attach


is this needed for kprobe?

olsajiri · 2025-11-19T21:14:20Z

pkg/sensors/tracing/genericoverride.go

+		"fmod_ret/"+attachFunc,
+		"generic_fmod_ret")
+
+	overrideProg.PinPath = path.Join("__override__", "fmod_ret", attachFunc)


I think it's probably ok, but could you please check tetrag debug progs/maps works properly with this?

olsajiri · 2025-11-19T21:24:28Z

pkg/sensors/tracing/generickprobe.go

+}
+
+func createFmodRetOverrideProgramFromEntry(load *program.Program, attachFunc string, progs []*program.Program, maps []*program.Map) ([]*program.Program, []*program.Map) {
+	// setup fmodret program and its input


looks like this function and the above createKProbeOverrideProgramFromEntry could be unified in just one function with load.OverrideFmodRet as bool argument

olsajiri · 2025-11-19T21:30:16Z

pkg/sensors/program/loader_linux.go

 			}
+			if unloadFunc != nil {
+				load.unloaderOverride = &unloader.CustomUnloader{
+					UnloadFunc: unloadFunc,


could we just set load.unloaderOverride in getOverrideProg ? then we could leave LoadFmodRetProgram as is

olsajiri · 2025-11-19T21:32:23Z

pkg/sensors/program/loader_linux.go

 }

-func LoadFmodRetProgram(bpfDir string, load *Program, maps []*Map, progName string, verbose int) error {
+func LoadKProbeOverrideProgram(bpfDir string, load *Program, maps []*Map, progName string, verbose int, unloadFunc func(bool) error) error {


hm why do we need special override function for kprobe and not for fmodret? can't we just use LoadKprobeProgram?

olsajiri · 2025-11-19T21:36:49Z

pkg/sensors/tracing/generickprobe.go


-func (k *observerKprobeSensor) LoadProbe(args sensors.LoadProbeArgs) error {
-	return loadGenericKprobeSensor(args.BPFDir, args.Load, args.Maps, args.Verbose)
+func (k *kprobeOverrideProgram) LoadProbe(args sensors.LoadProbeArgs) error {


so the kprobe override program depends on being loaded 'before' the actual program so it gets in the kprobe list behind the actual program, which is laded after.. tricky ;-) I guess if tests are passing we're doing it somehow, but I wonder we specifically ensure this

holyspectral changed the title ~~feat(sensors): support 38+ override on lsm funcs~~ wip: feat(sensors): support 38+ override on lsm funcs Oct 23, 2025

holyspectral force-pushed the remove-38-limits branch from c0067a5 to 23c75cf Compare October 23, 2025 21:04

mtardy requested a review from olsajiri October 24, 2025 08:44

mtardy added the release-note/minor This PR introduces a minor user-visible change label Oct 24, 2025

olsajiri reviewed Oct 24, 2025

View reviewed changes

holyspectral mentioned this pull request Oct 28, 2025

Support more than 38 override matchActions on LSM function #4204

Open

2 tasks

holyspectral force-pushed the remove-38-limits branch from 23c75cf to acfea29 Compare November 13, 2025 14:41

holyspectral added 4 commits November 19, 2025 14:51

kprobe: support 38+ override on lsm funcs

09d21bd

Move bpf functions regarding overrides into bpf_generic_override.o Signed-off-by: Sam Wang (holyspectral) <[email protected]>

kprobe: custom ebpf program unloader

f392065

Provide a custom CustomUnloader, so custom actions can be performed when a ebpf program is unloaded and unloaderOverride is called. Signed-off-by: Sam Wang (holyspectral) <[email protected]>

kprobe: cleanup obsolete code.

99bf162

Signed-off-by: Sam Wang (holyspectral) <[email protected]>

holyspectral force-pushed the remove-38-limits branch from acfea29 to 99bf162 Compare November 19, 2025 20:22

olsajiri reviewed Nov 19, 2025

View reviewed changes

wip: feat(sensors): support 38+ override on lsm funcs #4244

Are you sure you want to change the base?

wip: feat(sensors): support 38+ override on lsm funcs #4244

Uh oh!

Conversation

holyspectral commented Oct 23, 2025

Description

Changelog

Uh oh!

holyspectral commented Oct 23, 2025

Uh oh!

olsajiri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

netlify bot commented Nov 13, 2025

✅ Deploy Preview for tetragon ready!

Uh oh!

holyspectral commented Nov 13, 2025

Uh oh!

holyspectral commented Nov 13, 2025

Uh oh!

olsajiri commented Nov 16, 2025

Uh oh!

holyspectral commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

olsajiri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

holyspectral commented Nov 19, 2025 •

edited

Loading