FreeBSD guest: port init.c, make libkrun adjustments for processing kernel cmdline

[nohajc]

This PR makes it possible to use init on FreeBSD.
It also deals with the differences between BSD and Linux processing of the kernel command-line.

For one thing, FreeBSD kernel parameters are strictly key=value.
There's also no such thing as injecting arguments after -- into user space environment.
All key/value pairs are available from kenv instead.

If you try to pass an argument without =, the kernel will append ="1" to it.
In order to properly pass any space-delimited command-line, I store it as a sequence of null-terminated byte strings, apply base64 encoding, split it into chunks so that I don't exceed the maximum length of a kernel param value and pass it like KRUN_INIT_ARGV0="<chunk0>", KRUN_INIT_ARGV1="<chunk1>", etc.

Maximum length of the entire command-line is 512 bytes (LBABI_MAX_COMMAND_LINE) unlike 2048 on Linux.

[nohajc]

Marked as draft until I squash the commits and make them prettier.

[nohajc]

Fixing a potential issue here:

The string pointed to by the return value of getenv() may be statically allocated, and can be modified by a subsequent call to getenv(), putenv(3), setenv(3), or unsetenv(3).

https://linux.die.net/man/3/getenv

[nohajc]

These are needed to process command-line in a FreeBSD VM but I've been thinking they might be useful in general. The current method of passing arguments (concat everything into one string) isn't exactly lossless.

[slp]

@nohajc do you need this one to be present in the next release? Since it changes the way in which the kernel command line is generated, I'd like to give it more time to mature.

[nohajc]

It shouldn't change anything for Linux guests. Apart from the extra environment variables which are currently unused by that version of init.

It would be nice to include it in the release but of course I don't want to cause any trouble.

[nohajc]

@slp Maybe my original description was a bit misleading. I only use the base64 mechanism for BSD. The old plain-text encoding is still applied to Linux.

[slp]

@nohajc OK, I'll start reviewing it now then.

[jakecorrenti]

code seems fine to me, but I also know nothing about BSD systems

[slp]

@nohajc I would like to deprecate passing arguments to the entry point using the kernel command line because, as you noticed, it breaks easily. On Linux we favor putting the arguments in /.krun_config.json. Even though that's following the OCI format, I think it should work just fine for passing your arguments to the entry point in FreeBSD too.

The parser is very flexible. You basically just need to put Env and Cwd into a valid JSON file, and save it to /.krun_config in your root file system.

[nohajc]

Sounds good, except (Free)BSD still doesn't properly support virtiofs. It has 9p support but that's missing in libkrun.

What I'm doing to pass any files into a VM (short of regenerating the entire root image) is to generate ad hoc iso images using bsdtar and attaching them as secondary drives. Works for my needs but I'm not sure I'd like it for passing arguments.

[slp]

Sounds good, except (Free)BSD still doesn't properly support virtiofs. It has 9p support but that's missing in libkrun.

What I'm doing to pass any files into a VM (short of regenerating the entire root image) is to generate ad hoc iso images using bsdtar and attaching them as secondary drives. Works for my needs but I'm not sure I'd like it for passing arguments.

Ah, yes, lack of virtio-fs support is a PITA. Okay, let's rely on the kernel command line until there's a better alternative in BSDs.

[nohajc]

Anyway, this should solve #454. Along with the console fixes which are already merged.

[nohajc]

I was thinking about maintenance once this is merged.

To be clear, I don't expect feature parity between Linux and BSD. Sometimes it's not even possible.

What I would propose is to have a test build so we make sure we don't break the port. Other than that, any time a feature is added which is Linux-specific, I'd just disable it in the BSD version.

One example is the various mount calls which are currently only built on Linux. FreeBSD mounts the bare minimum before executing init, so I didn't have to deal with that.

Otherwise porting would mean abstracting the differences between mount APIs which are a bit different.

[valpackett]

hm, why are all of them just if and not ifdef?

[nohajc]

No particular reason. Both should work. This way the variable must evaluate to true not just "be defined".

[slp]

Given than most of the VMM context works fine with both Linux and FreeBSD without any changes, it's kind of a bummer having to make a difference for argument passing. Please consider implementing this unconditionally, so it's used on Linux too.

Most use cases should be using with /.krun_config.json, but for those that aren't, I think it's better to have a single path.

[nohajc]

I agree it would be better to unify this. At the same time, I didn't want to do any disruptive changes for Linux, especially before release.

So, would it be OK to merge it like this (as phase 1) and then unify in a separate PR?

Or do you prefer to do it properly in one go? I can also live with this not being included in 1.17.

[slp]

If you go for the unified solution for both Linux and FreeBSD, please make this a value that can be "reallocatable" so we aren't limited to 512 bytes.

[slp]

In addition to the other comments, please limit the commit title in the second commit to 50 characters.

[slp]

As stated above, I'd prefer to avoid this, having the same behavior for both Linux and FreeBSD.

[nohajc]

I agree. Especially this if is kind of ugly.

[mtjhrc]

Does FreeBSD support multiport virtio console?
If yes a nice alternative (and also usable for Linux and better then the current state), could be to just pipe the existing krun_config.json format using a virtio console port into the guest's init process. Basically instead of using a file (since there is no virtio-fs), just use a pipe.

This also wouldn't put any limit on the maximum length of the arguments.

[nohajc]

I'm not really sure what the extent of virtio console support is. There is a kernel module but as @slp already found out, for example, it is not possible to use virtio console as the primary output for the VM (it won't boot).

[slp]

Does FreeBSD support multiport virtio console? If yes a nice alternative (and also usable for Linux and better then the current state), could be to just pipe the existing krun_config.json format using a virtio console port into the guest's init process. Basically instead of using a file (since there is no virtio-fs), just use a pipe.

This also wouldn't put any limit on the maximum length of the arguments.

This is a pretty cool idea, indeed.

I'm not really sure what the extent of virtio console support is. There is a kernel module but as @slp already found out, for example, it is not possible to use virtio console as the primary output for the VM (it won't boot).

There's multiport support and, while can't be used as kernel console, should work great for this kind of thing. I can help you prototype something on init.c. If it turns out to be too cumbersome, we can always fall back to using the kernel command line. Do you have a prebuilt FreeBSD kernel somewhere?

[nohajc]

@slp I use this with libkrun:

https://github.com/nohajc/freebsd/releases/download/alfs%2F15.0/kernel.txz

If you also need a working VM image, I can provide that too.

[nohajc]

@slp Oh, forgot the console driver is not built into the kernel by default. It is loaded from .ko module. I can probably adjust the config and make another build.

EDIT: use this one

[nohajc]

@mtjhrc Love the idea. If it works reasonably well, we can ditch all the hacky b64 encoding. I didn't like it in the first place. Just needed to pass the arguments somehow.

[nohajc]

@slp Ok, actually, for the prototyping, you could use my launcher which is able to inject a newly built init into an existing FreeBSD VM image. Also that image is automatically bootstrapped from a few URLs. Note that this is strictly a macOS setup.

git clone https://github.com/nohajc/anylinuxfs.git
cd anylinuxfs
git checkout freebsd-guest

rustup target add aarch64-unknown-linux-musl
rustup +nightly component add rust-src

Edit etc/anylinuxfs.toml in the [images."freebsd-15.0"] section:

-kernel.bundle_url = "https://github.com/nohajc/freebsd/releases/download/alfs%2F15.0/kernel.txz"
+kernel.bundle_url = "https://github.com/nohajc/freebsd/releases/download/alfs%2F15.0/kernel-virtio-console.txz"

FREEBSD=1 ./download-dependencies.sh
# this will link against /usr/local/lib/libkrun.1.dylib (it needs to have the `install_name_tool -id` correctly set)
FREEBSD=1 ./build-app.sh --release

bin/anylinuxfs shell -i freebsd-15   # will bootstrap a VM on the first run

Now, every time you replace libexec/init-freebsd, the VM shell should use it on the next run.
You can pass commands to the VM using the -c flag (like sh -c).

[slp]

@slp Oh, forgot the console driver is not built into the kernel by default. It is loaded from .ko module. I can probably adjust the config and make another build.

EDIT: use this one

For some reason, this kernel hangs on boot for me when there's a virtio-console configured. Using a generic kernel from a FreeBSD VM image works fine though.

As a PoC, I've extended boot_efi.c with this:

    char guest_config[] =
        "{\"process\": {"
        "   \"args\": ["
        "       \"/bin/sh\","
        "        \"-x\""
        "   ],"
        "   \"env\": ["
        "      "
        "\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\","
        "       \"TERM=xterm\","
        "       \"HOME=/root\","
        "       \"HOSTNAME=f771156e4273\""
        "   ],"
        "   \"cwd\": \"/\""
        "}}";

    int pipefd_output[2];
    err = pipe(pipefd_output);
    if (err) {
        perror("Error creating pipe");
        return -1;
    }
    int pipefd_input[2];
    err = pipe(pipefd_input);
    if (err) {
        perror("Error creating pipe");
        return -1;
    }
    krun_disable_implicit_console(ctx_id);
    krun_add_serial_console_default(ctx_id, 0, 1);
    int console_id = krun_add_virtio_console_multiport(ctx_id);
    krun_add_console_port_inout(ctx_id, console_id, "krun-config",
                                pipefd_output[0], pipefd_input[1]);

    pid_t pid = fork();
    if (pid == 0) {
        // Wait for handshake ("READY")
        char buf[6];
        err = read(pipefd_input[0], &buf[0], 6);
        fprintf(stderr, "Received: %d bytes from the guest\n", err);

        // Send guest config
        write(pipefd_output[1], guest_config, sizeof(guest_config));

        // Send EOF
        buf[0] = 0x04;
        write(pipefd_output[1], &buf[0], 1);
        return 0;
    }

Paired with this on the guest:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <termios.h>

// --- UPDATED SERIAL PORT DEFINITION ---
#define KRUN_CONFIG_PORT "/dev/vtcon/krun-config"
#define BUFFER_SIZE 256
#define EOF_BYTE 0x04 // ASCII EOT (End of Transmission) / POSIX EOF (Ctrl+D)
#define READY_MESSAGE "READY\n" // Message to send

// Global flag to indicate if EOF was found
static int eof_found = 0;

/**
 * @brief Processes the received data, checking for the EOF byte.
 *
 * @param buffer The received data buffer.
 * @param bytes_read The number of bytes read.
 * @return 1 if EOF_BYTE was found, 0 otherwise.
 */
int process_data(const char *buffer, ssize_t bytes_read) {
    for (ssize_t i = 0; i < bytes_read; i++) {
        if (buffer[i] == EOF_BYTE) {
            // EOF byte found! Print preceding data and exit indicator
            ssize_t data_length = i;
            if (data_length > 0) {
                // Print only the data before the EOF byte
                // Use '%.*s' format for printing a specific length of a string
                printf("Data before EOF: %.*s\n", (int)data_length, buffer);
            }
            printf("--- Detected EOF (0x04) byte. Exiting. ---\n");
            return 1; // Signal EOF found
        }
    }

    // If EOF was not found, print all the data received in this chunk
    printf("Received: %.*s", (int)bytes_read, buffer);
    fflush(stdout);

    return 0; // EOF not found
}

int main(void) {
    int fd;
    char buffer[BUFFER_SIZE];
    ssize_t bytes_read;

    // 1. Open the device file
    fd = open(KRUN_CONFIG_PORT, O_RDWR | O_NOCTTY);
    if (fd < 0) {
        fprintf(stderr, "Error opening %s: %s\n", KRUN_CONFIG_PORT, strerror(errno));
        fprintf(stderr, "Ensure the device file exists and you have permissions.\n");
        return EXIT_FAILURE;
    }

    printf("Successfully opened device %s.\n", KRUN_CONFIG_PORT);

    // 2. Write "READY" to the device
    ssize_t bytes_written = write(fd, READY_MESSAGE, strlen(READY_MESSAGE));
    if (bytes_written < 0) {
        perror("Error writing 'READY' message");
        close(fd);
        return EXIT_FAILURE;
    }
    printf("Wrote '%s' (%zd bytes) to %s. Starting read loop...\n",
           READY_MESSAGE, bytes_written, KRUN_CONFIG_PORT);

    // 3. Read loop
    while (!eof_found) {
        bytes_read = read(fd, buffer, BUFFER_SIZE);

        if (bytes_read > 0) {
            eof_found = process_data(buffer, bytes_read);
        } else if (bytes_read == 0) {
            // Read returns 0 on VMIN=1/VTIME=0 settings only if the device is closed
            // from the other end or disconnected. This is equivalent to EOD/EOF for a stream.
            printf("\n--- Read returned 0. Device stream closed. ---\n");
            break;
        } else {
            // Error handling
            if (errno == EINTR) {
                continue;
            }
            perror("\nError reading from device");
            break;
        }
    }

    // 4. Close the device
    close(fd);
    printf("Device closed. Program terminated.\n");

    return EXIT_SUCCESS;
}

This works fine here. It should be a matter of putting that in init.c, triggered by the presence of /dev/vtcon/krun-config, and refactor config_parse_file to be able to tell it to parse data from a buffer instead of opening a file.

[nohajc]

Thanks! I should be able to test that. It doesn't depend on EFI though, does it? I use direct kernel boot in my setup.

[nohajc]

Btw, what about auto-generating the json config based on configured command-line?

As a future usability improvement...

[mtjhrc]

@slp

As a PoC, I've extended boot_efi.c with this:

As a PoC it doesn't matter, but can't you just close(pipefd_output[1]) instead of sending the ASCII EOF / 0x04 ? I think it should work...

Btw, what about auto-generating the json config based on configured command-line?

Yes, I think we should do this, in fact I think we should replace the krun_set_exec to do this behind the scenes on Linux too.

In addition to that we might want to consider krun_set_payload_json that you can pass in a json string. (currently it would use the virtio-console, but eventually we might make it use virtio-fs with a magic virtual file like we do for the init.

[slp]

As a PoC, I've extended boot_efi.c with this:

As a PoC it doesn't matter, but can't you just close(pipefd_output[1]) instead of sending the ASCII EOF / 0x04 ? I think it should work...

I've tried that but doesn't seem to get propagated to the guest. Sending EOF works fine though.

[nohajc]

Ok, I am going to rework this PR soon but since I don't need any changes in libkrun itself to actually pass arguments to the VM (just the init, right?), this really doesn't have the priority to be included in the upcoming 1.17 release.

I didn't count with any binary distribution of FreeBSD init along with libkrun anyway. I can easily build it separately.

[mtjhrc]

I don't need any changes in libkrun itself to actually pass arguments to the VM (just the init, right?)

It's possible to do this in 2 stages, here you add support for for this in the init and you can manually set up the host side to pass in the arguments (like Sergio's PoC).

For a second part, it would be nice to make krun_set_exec do this automatically and use this mechanism by default in Linux too. (the current way it works is also not great on Linux, there is a length limit, possibly some parsing weirdness too...).

I’m happy to handle the second part, but if you're inclined to work on that as well, just let me know! 👍

[mtjhrc]

Also, I would make the init behavior be enabled by passing in a kernel arg (kenv on freebsd like you mentioned).
e. g. KRUN_USE_CONSOLE_CONFIG=1 or even KRUN_CONSOLE_CONFIG=name_of_console_port.

[nohajc]

@mtjhrc I'd also split the task as you say. Anyway, I was thinking I could handle both parts.

About the release, @slp originally asked if I needed this in 1.17. Since there's no need for the base64 handling, all the pieces for constructing the json config are already in place so we just need the init support (and I'm building bsd init separately for my project anyway).

If I'm able to make it work with 1.17 libkrun and a custom build of init, I'm happy. All the library changes are just automation.