Allow sidecars to terminate gracefully

[pierretasci]

Expected Behavior

A sidecar might need to complete any pending work before being torn down (eg, uploading metadata).

Actual Behavior

Sidecars are replaced with a nop image as soon as a TaskRun completes preventing any cleanup work.

Steps to Reproduce the Problem

1. Create a TaskRun
2. Using an AdmissionsController to inject a sidecar that streams output from the Task to another service.

We would expect all of the output to be uploaded as the task runs but whatever chunk isn't uploaded before the sidecar is torn down is presumably lost.

Additional Info

There are ways around this but it seems to me that the very least, a sidecar should receive the kill signal so it can handle shutdown however it needs to. A timeout can always been inserted to ensure it doesn't hang forever.

[dibyom]

/kind bug
/cc @sbwsg

There was some discussion of this problem on the original design proposal. I think the main issue raised was that you can't guarantee an arbitrary sidecar has a kill binary available, so exec'ing into a container and running kill on a pid won't always work. I'll do some reading on how kubernetes kills containers in a pod and see what it would take to do same for Tekton.

@pierretasci Do you ideas for ways this could be implemented that would support arbitrary containers?

[pierretasci]

Yeah, I read through the discussion on the original proposal and I think it makes sense. I do think that it wouldn't be particularly harmful to send a SIGTERM or SIGKILL into the container arbitrarily with the contract that if you don't handle shutdown in XX seconds then we will perform our nop container swap.

If the container doesn't have shutdown logic, no harm done it seems. At the worst, it would introduce an XX second delay to completing each task.

I am of course open to other ideas but I do think stateful sidecars (logging is another use case that comes to mind) will want a shutdown mechanism.

I do think that it wouldn't be particularly harmful to send a SIGTERM or SIGKILL into the container

I totally agree, I'm just not entirely sure yet how this would be implemented if the container doesn't include a kill binary.

I've been skimming through the k8s codebase looking for the spot where the container termination flow is implemented and it looks as though the precise implementation of the grace period is delegated to the container runtime.

Another alternative here would be to introduce an optional contract for sidecars: If you want to receive a SIGTERM / shutdown warning then your sidecar must contain a kill binary on $PATH. If a sidecar provides this then Tekton can execute a similar container shutdown procedure to Kubernetes' before switching out the sidecar container's image to nop. Sidecars that don't implement the contract will simply be stopped without warning.

This would provide enough of a stopgap until the sidecars KEP lands I think.

[pierretasci]

Yeah, that is sort-of what I was getting at above. If there is a kill binary, we provide graceful shutdown, otherwise, the timeout and hot-swap seems more than a reasonable contract.

I suspect the timeout will actually be enough for most sidecars but having a dedicated way to shut down if it is required will help cover edge cases better.

OK cool so then the steps to implement are something like:

1. Exec kill in sidecar to send SIGTERM to PID 1
2. After termination grace period (30s default) exec kill again to send SIGKILL to PID 1
3. If exec'ing kill doesn't work in step 1 or the sidecar doesn't stop after step 2 do a "nop swap" of the sidecar's image field and forcefully shut it down.

Sounds doable! I'll try to find some time this week or next to implement.

[Mohjive]

There’s no need for a ‘kill’ binary in the container. It’s possible to send SIGTERM to the container process 1: https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/
It’s up to that process to handle the signal, e.g https://gobyexample.com/signals if image is just a Golang binary.
If the container doesn’t exit within the grace period it should be killed, and as last resort removed (and replaced with nop image)

[Mohjive]

Also consider new sidecar lifecycle in k8s future version (maybe 1.19): kubernetes/enhancements#753

Many thanks @Mohjive . You mention that there's no need for a kill binary in the Pod's container but then you forgot to demonstrate how to send SIGTERM without it 😆 . Assume I'm a Linux noob. How do I send SIGTERM to a process without kill? I can't run docker stop using the kubernetes controller, or can I?

Thanks for the pointer to the sidecar lifecycle KEP. It's definitely on our radar. We'd be very happy to remove the nop image hack and replace it with that feature once it's merged and k8s have committed to its support.