How to exit middleware gracefully？

[houxul]

Start goreplay and middleware with goreplay --middleware /app/middleware, which handles signals like syscall.SIGINT, syscall.SIGTERM, etc.​ After the goreplay process is terminated, middleware also exits, but a middleware zombie process is formed. How to implement middleware correctly with golang?

[buger]

/probe can you confirm?

[github-actions]

Yes, I can confirm the issue with middleware zombie processes in GoReplay. After analyzing the code, I've identified the root cause and can provide a solution.

The Issue

When GoReplay runs with a middleware (--middleware /app/middleware), it creates a child process to execute the middleware command. When GoReplay receives a termination signal (SIGINT, SIGTERM), it attempts to gracefully shut down by:

1. Calling e.Close() in the main process
2. Which calls middleware.Close()
3. Which calls m.commandCancel() to cancel the context

However, this doesn't always properly terminate the middleware process, resulting in a zombie process.

Root Cause

The current implementation in middleware.go uses exec.CommandContext() to run the middleware process, which should terminate the process when the context is canceled. However, there are two issues:

1. The middleware process might not be properly handling the signal sent when the context is canceled
2. There's no explicit cleanup of the process resources after termination

Solution

To fix this issue, middleware implementations in Go should:

1. Properly handle termination signals (SIGINT, SIGTERM)
2. Ensure all resources are cleaned up on exit
3. Use a more robust process termination approach

Here's an example of a properly implemented middleware in Go:

package main

import (
	"bufio"
	"encoding/hex"
	"fmt"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	// Set up signal handling
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	
	// Create a done channel to coordinate shutdown
	done := make(chan struct{})
	
	// Start processing in a goroutine
	go func() {
		scanner := bufio.NewScanner(os.Stdin)
		for scanner.Scan() {
			// Process input and write output
			line := scanner.Bytes()
			// Your middleware logic here
			fmt.Fprintf(os.Stdout, "%s\n", line)
		}
		close(done)
	}()
	
	// Wait for either a signal or processing to complete
	select {
	case <-sigChan:
		fmt.Fprintln(os.Stderr, "Signal received, shutting down...")
	case <-done:
		fmt.Fprintln(os.Stderr, "Processing complete")
	}
	
	// Perform any cleanup needed
	// No need to explicitly exit - the program will exit naturally
}

This implementation ensures that:

1. Signals are properly caught and handled
2. The middleware exits cleanly when stdin is closed (which happens when GoReplay terminates)
3. Resources are properly cleaned up

With this approach, the middleware process will terminate gracefully when GoReplay exits, preventing zombie processes.

---

Tip: Start comment with /probe <request>.
Built by probe