Command and args

[travisghansen]

Question

None of the examples show a usage of command and/args in the pod spec. Is it possible to set a script to run with proper env per the pod spec and then when it exits the vm is destroyed by nature of the ‘entrypoint’ exiting?

Suggestion (Optional)

Should I simply be doing this in the postStart script and make sure it runs shutdown or similar at the end? If this is the correct approach are env vars flowing down as expected?

[Raikerian]

Hi. What kind of commands you are looking for? You can execute anything under macOS in both postStart and preStop as per example here https://github.com/agoda-com/macOS-vz-kubelet/blob/main/example/pod-gitlab-sidecar.yml#L48
The script is also supported, of course. If you want to just cleanup after script is finished, perhaps k8s Job is something you can use here?

In any case, you might want to show some examples in what you are trying to achieve here.

[travisghansen]

Yeah, I guess is command/args used/supported at all, or is it simply ignored? Or only the lifecycle hooks can be used to execute something on start?

I'm currently predominantly looking at running ephemeral ci jobs (so I think slightly different than the gitlab approach in the examples) where each job (pod) runs (either a postStart or command + args) and then is tore down after the single build. From what I can tell the gitlab approach looks to be running as a more standard long-running agent process which spans multiple builds/jobs.

Also the supported scope of env vars is critical to me as each job (ie: pod) has data embedded via env vars to tell the bamboo agent which specific job it should pick up etc. Particularly are the env vars on the pod injected into the pre/post scripts and/or the command if possible?

Ideally I can set the specific command and args as a normal pod would do and when that command exits the pod/vm is destroyed.

Thanks!

[travisghansen]

Essentially I want to copy/utilize some of the scripts available here: https://bitbucket.org/atlassian-docker/docker-bamboo-agent-base/src/master/ and bundle them into the VM. I want the pod to declare a command which would be effectively my own entrypoint, and when that command exits I would expect standard pod behavior to kick in meaning...the VM is killed and the pod is updated in the k8s api etc.

[Raikerian]

If I understood correctly, you want to spawn pods that will connect to your bamboo instance as an image.
So, the command is indeed not currently supported by macOS container implementation. What you can do here is either utilize postStart script, which is fully supported, or (ideally) build your macOS image with all the binaries you need. And then just mount configuration or set it up using postStart script.

[Raikerian]

Similar to how they set up their docker image in that example (atlassian/bamboo-agent-base) you want to set up your MacOS image (pre install all the binaries, set them to auto start up, etc. etc.).
Based on their example, however, I am not sure why you would need a script. As it seems like the remote server decides what script to execute and executes it through agent running in the image rather than k8s script field?

[travisghansen]

Yeah sorry I am not defining the problem clearly. Really it’s about vm lifecycle. I could indeed include the binaries needed (and I will frankly). The question is more of how to handle a oneshot container. I want the container (vm) to shut itself down when the build is finished.

If I understand your proposal I would likely use postStart as a sort of entrypoint and use it to launch a 2nd script which is the bamboo agent etc. As the parent script needs to act as a sort of monitor of the agent, I wait for the sub script to finish execution and then make sure to execute shutdown to ensure the pod goes to a completed/stopped state?

Would it be possible to implement something like this: s-u/macosvm@0a34c52

vz-kubelet would launch the script and wait for it to exit, when it exits vz-kubelet automatically stops/shutdowns the vm? Effectively an implementation of command/args..

Are the pod env vars passed to the postStart script?

[Raikerian]

Yes, pod env vars are passed to postStart, preStop and exec scripts. All of those are essentially just an ssh to the VM under the hood.
The Command part of a container itself (which I think is what you are referring to as a script) is currently not implemented indeed, and yes, it would definitely be possible to implement. I can backlog this feature and add support in the future.

To suggest what's possible to do now I would like to understand how your bamboo agent works, so I have a few questions:

1. Who will spawn this pods that connect to the bamboo server? Is it server itself, or some kind of controller managed by them running in the same cluster?
2. Where will the build script be located? Last time I had an opportunity to work with bamboo 10 years ago it was all server side, is it the same here or do you plan to have the whole build script in this Command part of the pod container?
3. If the answer to 2 is server side script and If you want to handle Pod exit in the Command of the pod container as a supplementary script, how will you know when the server side script is finished?

[travisghansen]

Thanks for the info.

1. Bamboo has a kubeconfig that allows it to create pods in the target cluster. Essentially a pod template is created. Bamboo will submit the pod to the cluster and it is expected the pod will launch the agent with very specific args etc and, critically, when the build is finished the pod will die (implicitly because pid 1 exits)…bamboo does not stop or kill the pod directly.
2. Indeed the build script is located server-side.
3. The agent knows when the script is finished and then itself exits understanding that it is a onehsot and that the agent should no longer continue to run.

I am not trying to bundle the server-side logic into the postStart script, effectively I don’t want to exec the shutdown command manually, nor in the server-side build script. I want the kubelet to monitor ‘pid 1’ and when it exits I want the kubelet to kill the vm…to mimic actual k8s semantics.

Unless I am misunderstanding something currently I have 2 options for killing the pod/VM:

• manually monitor my own entrypoint style script and literally execute a vm shutdown command in postStart
• have some external actor use the kubernetes api to delete the pod

What I am after is 3 option which is: ‘kubelet’ launches command/args for me and it monitors the pid, when the pid exits kubelet issues a vm shutdown and the pod is dead (completed).

In such a scenario kubelet starts the command (in my case the agent), the agent runs, gets the server-side script and executes it etc, etc, and then eventually the agent after running the script exits. kubelet detects the command/process/agent pid is no longer running and begins the tear down sequence.

[Raikerian]

The agent knows when the script is finished and then itself exits understanding that it is a onehsot and that the agent should no longer continue to run.

I see, in that case it is indeed would be ideal to use script here. But for the time being you can just execute shutdown command at the end of your script and that will turn off the VM and send pod in the succeeded state. After that just have some GC rule to clean up such pods at some point, but as long as they are not in the Running state - there is no VM or resource being used.

[travisghansen]

So I have this fully functional with a kind cluster now. I would like to formally request the behavior described above. Ideally it launches and runs as described and can subsequently be used to attach to the stdio bit for handling kubectl logs requests as well. Given the nature how this project works it may make sense to exec the command over ssh with nohup/disown/whatever and write a pid file to disk on the VM and then have the kubelet periodically poll the status of the pid to make sure it's still alive. If dead shut down the VM. If not dead, attach to it during logs requests etc.

Thanks for the consideration!

[Raikerian]

Yea, I will look into command and args as soon as I have some time for this project.

[travisghansen]

As an FYI even using postStart requires pretty special handling because of the 10s timeout in place. I will essentially need to nohup/disown an entrypoint-like script and then let it handle all the things we mentioned above.

[Raikerian]

Yeah, the only solution for now would probably use shutdown command at the end of the script on your cicd server.