Handle processes whose main thread has exited #376

christos68k · 2025-02-27T22:02:00Z

Summary

This PR implements both steps described in #365 (comment).

Thanks to @korniltsev for suggesting disassociate_ctty, I ended up using another tracepoint sched_process_free instead as it makes fewer assumptions and is more stable (see this comment for more context). It also allows us to simplify cleanup logic (no need for the extra periodic cleanups I had in the first prototype solution), as userspace will get a final PID notification when the process gets freed by the kernel.

Essentially, whenever the main thread exits, we do not unload process information thus allowing profiling the remaining threads to continue. Processmanager can also track mapping changes triggered by one of the remaining threads.

I added some debug warning statements to ease review, I will remove the commit that introduced them before merging. I also added a C program that you can compile and run as a testing workload with the profiling agent also running, that should exercise all the corner cases that this PR addresses. Looking at the warning logs I added and the generated flamegraph in devfiler should make the timeline of processmanager operations very clear.

It's probably easier to review this commit-by-commit.

TODO:

DONE ~~Add test program~~
More testing

processmanager/processinfo.go

korniltsev · 2025-03-04T12:45:14Z

Thanks for looking into this.

This looks OK overall and should solve the issue from the user perspective.

One of the downsides I see is that while we do not unload the old mappings, we re also not loading new mappings, which may degrade profiling of such processes ( I am still not sure if there are legit applications with dead main thread, or is it a highly infrequent corner case)

I personally would prefer if the processmanager "re-elected" a main thread by looking into the process threads, although I realize it may require more work and we may do this later.

Another thing to consider is to hook a kprobe on disassociate_ctty which is called when the process group is dead
https://github.com/torvalds/linux/blame/master/kernel/exit.c#L935-L936 this may help avoiding a separate timer for this case.

It would be nice to have a unit test for this case regardless of the solution we chose.

christos68k · 2025-03-04T14:35:43Z

One of the downsides I see is that while we do not unload the old mappings, we re also not loading new mappings, which may degrade profiling of such processes ( I am still not sure if there are legit applications with dead main thread, or is it a highly infrequent corner case)
I personally would prefer if the processmanager "re-elected" a main thread by looking into the process threads, although I realize it may require more work and we may do this later.

I'm currently working on this, will push new commits (implementing part 2 of the proposed solution in #365) today.

Another thing to consider is to hook a kprobe on disassociate_ctty which is called when the process group is dead https://github.com/torvalds/linux/blame/master/kernel/exit.c#L935-L936 this may help avoiding a separate timer for this case.

I think we can switch to sched_process_free tracepoint (instead of sched_process_exit) which should be more performant than a kprobe. I'll verify.

EDIT: sched_process_free fires for every kernel task so it's not suitable if we want to avoid notifying userspace of every thread exit. On the other hand, disassociate_ctty seemingly (no in-depth investigation done on my part) does what we want and also seemingly executes after task has been removed from /proc which eliminates a possible race in userspace that would otherwise be a (probably unlikely) concern.

EDIT2 Went back to sched_process_free which we can make work by checking whether PID is something we track or not.

support/ebpf/process_monitor.ebpf.c

sched_process_free is called when the task is freed by the kernel, which allows for simpler cleanup of processes whose main thread has exited.

Making TID available to processmanager allows the agent to keep profiling a process whose main thread calls pthread_exit while other threads continue to run.

This allows the agent to continue profiling a process whose main thread has exited, but other threads continue to run. Mapping changes triggered by one of the remaining threads are also tracked.

christos68k · 2025-03-20T18:12:18Z

process/process.go

-			} else if path != "" {
-				// Ignore [vsyscall] and similar executable kernel
-				// pages we don't care about
+			} else {


No semantic change, I just inlined the logic from GetMappings here as this is the more appropriate place.

christos68k · 2025-03-20T18:19:07Z

processmanager/processinfo.go

@@ -538,7 +538,7 @@ func (pm *ProcessManager) synchronizeMappings(pr process.Process,
 // fast enough and this particular pid is reused again by the system.
 func (pm *ProcessManager) processPIDExit(pid libpf.PID) {
 	exitKTime := times.GetKTime()
-	log.Debugf("- PID: %v", pid)
+	log.Warnf("- PID: %v", pid)


I'll remove these newly added warnings before merging, they should help with reviewing the PR as you don't need to run the agent with debug logs enabled and sort through a lot of irrelevant noise.

christos68k · 2025-03-20T18:20:14Z

processmanager/processinfo.go

@@ -626,22 +633,7 @@ func (pm *ProcessManager) SynchronizeProcess(pr process.Process) {
 			// return ESRCH. Handle it as if the process did not exist.
 			pm.mappingStats.errProcESRCH.Add(1)
 		}
-		return
-	}
-	if len(mappings) == 0 {


These comments are no longer relevant.

christos68k · 2025-03-20T18:54:15Z

I added some more information and notes on how to review/test to the description.

@korniltsev please take another look and review/test.

korniltsev · 2025-03-21T15:38:19Z

Great job. Thank you for looking into this.
I like the trick with sched_process_free and that we have no extra timers in userspace and the logic of the PM did not complicate.
I've run both my repro and your repro with libcrypto and the profiler works as expected. It keeps profiling remaining threads including new libraries (libcrypto)
I wish we could somehow create an integration test for it from the repro you've added so that it is run with every testruns instead of hoping I don't forget to run it. But I understand writing a test may be hard / time consuming so we may do this later.
LGTM

support/tests/main_thread_exit.c

support/ebpf/sched_monitor.ebpf.c

florianl · 2025-03-21T16:37:57Z

tracer/tracer.go

 	// It needs to be buffered to avoid locking the writers and stacking up resources when we
 	// read new PIDs at startup or notified via eBPF.
-	pidEvents chan libpf.PID
+	pidEvents chan uint64


Should a new type, maybe libpf.PidTidg, be used here to make clear that these are not ordinary uint64 numbers?

Added libpf.PIDTID

christos68k · 2025-03-21T20:58:52Z

process/process.go

@@ -53,9 +58,10 @@ func init() {
 }

 // New returns an object with Process interface accessing it
-func New(pid libpf.PID) Process {
+func New(pid, tid libpf.PID) Process {


I didn't switch Process to accept libpf.PIDTID as the latter is only used with PID events, and I'd rather not couple it here too.

christos68k requested review from a team as code owners February 27, 2025 22:02

christos68k mentioned this pull request Feb 27, 2025

Profiler incorrectly handles process exit when non-main threads are still running #365

Open

christos68k commented Feb 27, 2025

View reviewed changes

processmanager/processinfo.go Show resolved Hide resolved

processmanager/processinfo.go Outdated Show resolved Hide resolved

christos68k marked this pull request as draft February 27, 2025 22:07

fabled reviewed Feb 28, 2025

View reviewed changes

processmanager/processinfo.go Outdated Show resolved Hide resolved

christos68k force-pushed the ck/process-exit branch 3 times, most recently from 093a15f to 38f6e51 Compare March 5, 2025 04:18

christos68k mentioned this pull request Mar 5, 2025

processmanager: Don't synchronize a process that's waiting cleanup #379

Merged

fabled reviewed Mar 5, 2025

View reviewed changes

support/ebpf/process_monitor.ebpf.c Outdated Show resolved Hide resolved

christos68k force-pushed the ck/process-exit branch from 38f6e51 to 62d82c6 Compare March 20, 2025 17:22

christos68k added 3 commits March 20, 2025 14:36

Hook sched_process_free instead of sched_process_exit

dfaf278

sched_process_free is called when the task is freed by the kernel, which allows for simpler cleanup of processes whose main thread has exited.

ebpf: report_pid now sends both pid and tid to userspace

2835340

Making TID available to processmanager allows the agent to keep profiling a process whose main thread calls pthread_exit while other threads continue to run.

Handle processes whose main thread has called pthread_exit

eb4e909

This allows the agent to continue profiling a process whose main thread has exited, but other threads continue to run. Mapping changes triggered by one of the remaining threads are also tracked.

christos68k force-pushed the ck/process-exit branch from 57c0ebc to e11a0dc Compare March 20, 2025 18:36

processmanager: Better handling of missing mappings

068322f

christos68k force-pushed the ck/process-exit branch from e11a0dc to 87e351e Compare March 20, 2025 18:46

christos68k marked this pull request as ready for review March 20, 2025 18:53

christos68k commented Mar 20, 2025

View reviewed changes

christos68k requested a review from fabled March 20, 2025 18:55

christos68k self-assigned this Mar 20, 2025

christos68k linked an issue Mar 20, 2025 that may be closed by this pull request

Profiler incorrectly handles process exit when non-main threads are still running #365

Open

christos68k added 4 commits March 20, 2025 20:45

support: Add main thread exit C test

b97ae39

Minor doc update

0da64dc

Update eBPF tracers

01392e6

[Remove] Add debug warn logging to help with review

48698d5

christos68k force-pushed the ck/process-exit branch from 87e351e to 48698d5 Compare March 21, 2025 00:46

korniltsev reviewed Mar 21, 2025

View reviewed changes

support/tests/main_thread_exit.c Show resolved Hide resolved

Add copyright

5905d6a

christos68k force-pushed the ck/process-exit branch from 94a86eb to 5905d6a Compare March 21, 2025 16:01

florianl reviewed Mar 21, 2025

View reviewed changes

support/ebpf/sched_monitor.ebpf.c Show resolved Hide resolved

florianl reviewed Mar 21, 2025

View reviewed changes

christos68k added 2 commits March 21, 2025 16:30

sched_monitor: Add struct layout comment

1eeb764

Add libpf.PIDTID

78b9208

christos68k commented Mar 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle processes whose main thread has exited #376

Handle processes whose main thread has exited #376

christos68k commented Feb 27, 2025 •

edited

Loading

korniltsev commented Mar 4, 2025

christos68k commented Mar 4, 2025 •

edited

Loading

christos68k Mar 20, 2025

christos68k Mar 20, 2025

christos68k Mar 20, 2025

christos68k commented Mar 20, 2025 •

edited

Loading

korniltsev commented Mar 21, 2025

florianl Mar 21, 2025 •

edited

Loading

christos68k Mar 21, 2025

christos68k Mar 21, 2025

Handle processes whose main thread has exited #376

Are you sure you want to change the base?

Handle processes whose main thread has exited #376

Conversation

christos68k commented Feb 27, 2025 • edited Loading

Summary

korniltsev commented Mar 4, 2025

christos68k commented Mar 4, 2025 • edited Loading

christos68k Mar 20, 2025

Choose a reason for hiding this comment

christos68k Mar 20, 2025

Choose a reason for hiding this comment

christos68k Mar 20, 2025

Choose a reason for hiding this comment

christos68k commented Mar 20, 2025 • edited Loading

korniltsev commented Mar 21, 2025

florianl Mar 21, 2025 • edited Loading

Choose a reason for hiding this comment

christos68k Mar 21, 2025

Choose a reason for hiding this comment

christos68k Mar 21, 2025

Choose a reason for hiding this comment

christos68k commented Feb 27, 2025 •

edited

Loading

christos68k commented Mar 4, 2025 •

edited

Loading

christos68k commented Mar 20, 2025 •

edited

Loading

florianl Mar 21, 2025 •

edited

Loading