Podman Machine GRUB timeout

[inknos]

Hey,

As mentioned in our Office Hours yesterday I am currently investigating ways to cut the startup time for podman machine. One of the corners to cut is the 1 second GRUB timeout that could be set to 0 seconds. This is done by patching either of the files /boot/grub2/grub.cfg or /boot/grub2/grubenv in c/podman-machine-os.

Doing that will prevent debugging at boot time by adding kernel parameters from GRUB, so now the question becomes: "how to set a custom timeout to GRUB?"

I would like to propose three solutions for modifying the startup behavior of podman machine.

Prerequisite:

Assume the boot timeout is set to 0 having a patched FCOS image within the podman-machine-os repository.

Problem:

Provide an option to change the timeout for debugging purposes.

Analysis:

The GRUB timeout is defined in /boot/grub2/grub.cfg or /boot/grub2/grubenv. To boot with a different timeout the files need to have an option that needs to be set BEFORE the boot. As far as I know this can be done in two ways:

1. during podman machine init (for the current boot)
2. via ssh when the machine is on (for the next boot)

Considering that:

• machine init is an injection process that will force to reinitialize the machine
• ssh implies that the machine is healthy and running. changing GRUB time would be needed for debugging purposes, therefore we can't really assume that a machine would be reachable via ssh
• patching the file with either strategy will make the GRUB timeout sticky, meaning that all subsequent reboots will have the same configuration if not changed, so revert to default can be an issue

---

I propose some solutions, with pros and cons:

Solution A:

Add a new flag to the initialization command: podman machine init --grub-timeout=N

Pros:

• Easy to implement with injection
• Effective immediately
• Adjusting the timeout is easy (a warning can be printed with a command suggestion if timeout is not default)

Cons:

• Changing the timeout requires re-creating the machine images

Solution B:

Change the behavior of podman machine start --loglevel=debug to automatically set the timeout. This would be handled via an SSH command. However, since this change persists, a reset mechanism is required.

Pros:

• The option to increase timeout is incorporated in an existing command that will likely show more logs and don't skip the GRUB screen

Cons:

• The --loglevel flag should not modify persistent system settings like timeout. If the change is not ephemeral, it may lead to unexpected behavior for the user.
• Not immediate change. Requires 1 restart to take effect
• Resetting to default timeout is not obvious to the user

Solution C:

Introduce a dedicated configuration command: podman machine set --grub-timeout=N. This setting would be applied via SSH.

Pros:

• The functionality is explicit and easy for users to understand or revert
• Adjusting the timeout is easy (a warning can be printed with a command suggestion if timeout is not default)

Cons:

• Requires adding a new configuration option to the CLI
• Not immediate change. Requires 1 restart to take effect

---

I'll post here two questions and observations that came from an internal discussion prior to this.

Q1:

Regardless of the solution chosen, will there be a default timeout, or is there one currently? It might be nice to specify that somewhere in here. Personally, I like KISS, and option 1. However, my vote can easily be ignored if folks with more machine experience think that's not a good option.

The timeout is 1 second, currently, and that's the one set by FCOS afaik. we don't override it and fyi, if we do we have to do it from podman-machine-os, not podman

Q2:

How common would a need for increasing the timeout for debugging actually be?

Solution C sounds nice and the reboot should be only a minor annoyance, unless it's a common need. IMHO If is it's niche enough, I could even understand a manual change needed (no option) to debug.

Yeah, looking at it again A probably sounds better than C, but both sounds much better than B (because that's an unexpected side-effect like the Cons says) :)

[Luap99]

Is there any way to have a temporary timeout without configuring the grub cfg file?

By default users will never see the console so the timeout should be 0 as default, allowing the user to set a different timeout there via specific cli argument feels not useful to me?

The only way to get to the boot console is with --log-level debug AFAIK so it does make sense to me to use this option to set a longer timeout, however if that timeout cannot be set for the current boot then it is kinda useless and I agree with your statement about not making persistent changes with --log-level.

IMO I am not a huge fan of a specific option to set something most users will not even need to care about. If you plan is to ssh in and edit the file than this feels not super useful compared to just documenting this what a user has to do manually.
That means we do not have to carry the maintenance burden of an extra option.

For a) effective immediately would also not be true is it? If you want to change the file you would still need to first boot once, no?

[simek-m]

I think it's not possible without editing the GRUB config (allegedly, it can only be only overridden by holding specific keys at the right moment together with the right settings for GRUB_TIMEOUT_STYLE).

[Honny1]

I think the best approach is to set grub_timeout when --log-level=debug is used. That change would need to happen before the machine starts. It might be possible to modify config files inside the disk image before boot, but I am not sure how complex that would be. Also, the timeout should be reset when the machine shuts down.

Another idea is to use a non-zero timeout only on the first boot.

Or we could set the timeout to 0 by default and document how to change it when needed. In my experience, I have only touched GRUB configuration once or twice, when setting up dual boot between Linux and Windows.

[baude]

@inknos thanks for putting this together. I agree with the sediment here (or at least what i think it is). In an ideal scenario, you could make the change to remove the timeout in our images. And when --log-level=debug is used, we somehow set the grub timeout before the machine is booted (as @Honny1 said). Given the nature of this sort of being a black box, if for any reason it cannot boot, we need to see why in its current condition. So right now, if we cannot figure out how to change the grub timeout temporarily while the machine is off, i personally would not be in favor of changing the timeout defaults bc it makes supporting this thing impossible?

Is this kind of what others are thinking ?

[inknos]

I put together a POC with Claude, just to see if it would work, and it works. It's another idea of how to use debug to set the timeout at boot.

In short, it mounts the qcow2 (or raw) volume, patches it with a file called /boot/grub2/user.cfg (it's a grub override) and marks the fs with a file called .grub-debug-timeout.

On the next boot, if the marker is found, the same operation of mounting + patching is done.

Skipped otherwise for normal boots, which can have boot set to 0 in machine-os.

It's really badly implemented/tested, but hey, it gives us a direction if you like it

inknos@0f3e9c4

[Luap99]

I rather not maintain custom code that relies any many assumptions to mess with the raw image contents. This puts us in a bad spot where we cannot update the image format without updating this code, but then old machine will still be on the old format, etc...

Converting the image on start to raw then back to qcow also seems inefficient to the point where you need to copy the whole thing which can cause you to run out of space? Would that whole process not be much slower than whatever timeout we are trying to remove?

Also the code does not look portable either, the majority of machine users are on macos and windows, does debugfs really work there? And if it is not installed as default is practically useless as we cannot expect users to have this.

[inknos]

Fair point, for --loglevel=debug I see no other option than inject the timeout via ssh though. I'll dig more into that since ssh might not be available in a debug situation, but it's more sane than custom code for sure