cannot setuid and setgid files via ignition

[bexelbie]

Bug

Operating System Version

Reproduced on Flatcar and Fedora CoreOS

Ignition Version

Environment

BareMetal and Azure

Expected Behavior

Files setgid and setuid

Actual Behavior

Files not setgid or setuid

Reproduction Steps

Flatcar:

$ ls -l /opt/setuid-setgid-test/
total 97632
-rwxr-xr-x. 1 op op 24990551 Mar 16 16:49 normal
-rwxr-xr-x. 1 op op 24990551 Mar 16 16:49 setgid
-rwxr-xr-x. 1 op op 24990551 Mar 16 16:49 setuid
-rwxr-xr-x. 1 op op 24990551 Mar 16 16:49 setuid+setgid

Using:

variant: flatcar
version: 1.1.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC1N+xhi9y/rHURF3P0c6TiEGizwFnTBKH5GbQI46uyb
    - name: op
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC1N+xhi9y/rHURF3P0c6TiEGizwFnTBKH5GbQI46uyb

storage:
  files:
    - path: /opt/setuid-setgid-test/setuid+setgid
      mode: 06755 # Set UID and GID
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/setgid
      mode: 02755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/setuid
      mode: 04755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/normal
      mode: 0755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64

Fedora CoreOS:

core@localhost:~$ stat /opt/setuid-setgid-test/setuid+setgid
  File: /opt/setuid-setgid-test/setuid+setgid
  Size: 24990551  	Blocks: 48816      IO Block: 4096   regular file
Device: 252,4	Inode: 29360257    Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1001/      op)   Gid: ( 1001/      op)
Context: system_u:object_r:var_t:s0
Access: 2025-03-20 09:20:34.188000000 +0000
Modify: 2025-03-20 09:20:36.847000000 +0000
Change: 2025-03-20 09:20:43.430000000 +0000
 Birth: 2025-03-20 09:20:34.188000000 +0000
core@localhost:~$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="41.20250302.3.2 (CoreOS)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora CoreOS 41.20250302.3.2"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='41.20250302.3.2'

Using

variant: fcos
version: 1.6.0

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPKIbw6kZVnn7mih7Av2wk+eNZNIag8yZkKT54vICh1tcUro/PtCin+xlyJeN5yrokF3n7UMIMcwPe+QGEaoAPImSGxstpXzSXSxFo9npmcsA3cbJQReiwjVU6NDRBbYGrCgYdPtshHOMN19J+vIbRPCBZtZZbguQTBO9gDn/5uhzikZ3b7J2VWy2cOtkfBP6AwVM+9ZmKwLaF6CWIEx23BLJjpLtkuNHOjtg0SuqAF1D+lEEf7Ld77zVDMst1waCz2HfSEIr+a7J1/VaBMvxqG8iHwnVV3JcxuYZpYy7lGLAVb9V1SHbAbV54rKdqS5Eetqk9Woo2ABdaCZB//bn9Qs0q5kjw6cUlGleqKZzEFtxMZr6fJIzcd8TGLX+QOBlSMfszCcsdo6/LG8X3ySvDE+vqGTlHPXpeaHJv9Z21rZTnXIB/toVo/6iMrJseBrb6FUkp3oIqSWVTivEoKprTM2cc1ft4mRsmHWLCo1yonX9WfA8etgp/bAiAb4fwpG2oE7E/Mf5y+hS0EH5M88LBfxJMpNkjTkaWxcYfHUQHJGVv//fN5r4JC4ZNVc/PyU5WqmHoTBtQeCZ1p8o7VEwOJCRDl+Np/ykrPB1AobnSsNeiuUD1uCPs3KjzggTt3AeNRXKX2q0N9/dfsFEBTmJFZuSfTcoGUSBEUadfWucIww== mathieu.tortuyaux@gmail.com
    - name: op
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC1N+xhi9y/rHURF3P0c6TiEGizwFnTBKH5GbQI46uyb

storage:
  files:
    - path: /opt/setuid-setgid-test/setuid+setgid
      mode: 06755 # Set UID and GID
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/setgid
      mode: 02755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/setuid
      mode: 04755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64
    - path: /opt/setuid-setgid-test/normal
      mode: 0755
      user:
        name: op
      group:
        name: op
      contents:
        source: https://github.com/bexelbie/op-secret-manager/releases/latest/download/op-secret-manager-linux-amd64

Other Information

See also flatcar/Flatcar#1688

[bexelbie]

@tormath1

[prestist]

Hey @bexelbie Is this specifically in the opt/ dir? did you get a chance to try home/ or var ?

[bexelbie]

@prestist I tested only in /opt.

[travier]

This should have been fixed with #1325 but I can not find any tests for it so maybe it wasn't. I could not find a version of FCOS with a version of Ignition where that works either. Tests in #2049.

[travier]

Trying to reproduce that with a smaller program, which is giving me weird results:

$ cat test.go
package main

import (
        "fmt"
        "os"
        "syscall"

        "golang.org/x/sys/unix"
)

func check(e error) {
        if e != nil {
                panic(e)
        }
}

func main() {

        d1 := []byte("hello\ngo\n")
        err := os.WriteFile("setuidtest", d1, 0644)
        check(err)

        mode := uint32(3565)
        res := os.FileMode(mode & 0777)

        if mode&syscall.S_ISGID != 0 {
                res |= os.ModeSetgid
        }
        if mode&syscall.S_ISUID != 0 {
                res |= os.ModeSetuid
        }
        if mode&syscall.S_ISVTX != 0 {
                res |= os.ModeSticky
        }
        fmt.Printf("Setting mode of %s: %v\n", "setuidtest", res)
        if err := os.Chmod("setuidtest", res); err != nil {
                fmt.Printf("failed to change mode of %s: %v", "setuidtest", err)
        }

        info := unix.Stat_t{}
        if err := unix.Stat("setuidtest", &info); err != nil {
                fmt.Printf("error getting file permissions for %s: %v\n", "setuidtest", err)
        }
        fmt.Printf("Effective permissions %s: %vc\n", "setuidtest", os.FileMode(info.Mode))

}

$ go run test.go
Setting mode of setuidtest: ugrwxr-xr-x
Effective permissions setuidtest: -rwxr-xr-xc
$ ls -alh setuidtest
-rwsr-sr-x. 1 tim tim 9 Apr  7 18:21 setuidtest

[bexelbie]

@travier I am not following your reproducer. I was trying to use ignition to lay down a binary that was setuid at install not have the binary set itself setuid. Did I misunderstand your goal?

[travier]

I'm trying to reproduce what Ignition does with a shorter Go program to see where the issue might be. With the standalone reproducer, the Chmod call does set the setuid/gid bits but the following Stat call does not "see" them, and I don't know why. I think the stat call is what's used to check the mode in the blackbox test.

If the Chmod call is correct, then this would point to the issue happening after the file is first written to the filesystem.

[travier]

Just in case, it should be possible to workaround this issue using a service unit that does the chmod again as needed on first boot.

[prestist]

Trying to reproduce that with a smaller program, which is giving me weird results:

$ cat test.go
package main

import (
        "fmt"
        "os"
        "syscall"

        "golang.org/x/sys/unix"
)

func check(e error) {
        if e != nil {
                panic(e)
        }
}

func main() {

        d1 := []byte("hello\ngo\n")
        err := os.WriteFile("setuidtest", d1, 0644)
        check(err)

        mode := uint32(3565)
        res := os.FileMode(mode & 0777)

        if mode&syscall.S_ISGID != 0 {
                res |= os.ModeSetgid
        }
        if mode&syscall.S_ISUID != 0 {
                res |= os.ModeSetuid
        }
        if mode&syscall.S_ISVTX != 0 {
                res |= os.ModeSticky
        }
        fmt.Printf("Setting mode of %s: %v\n", "setuidtest", res)
        if err := os.Chmod("setuidtest", res); err != nil {
                fmt.Printf("failed to change mode of %s: %v", "setuidtest", err)
        }

        info := unix.Stat_t{}
        if err := unix.Stat("setuidtest", &info); err != nil {
                fmt.Printf("error getting file permissions for %s: %v\n", "setuidtest", err)
        }
        fmt.Printf("Effective permissions %s: %vc\n", "setuidtest", os.FileMode(info.Mode))

}

$ go run test.go
Setting mode of setuidtest: ugrwxr-xr-x
Effective permissions setuidtest: -rwxr-xr-xc
$ ls -alh setuidtest
-rwsr-sr-x. 1 tim tim 9 Apr  7 18:21 setuidtest

I dont understand how that is happening, could it be something with how ls is displaying perms or is this maybe a bug in go?

[bexelbie]

I 'slapped some AI on it' and got back a seemingly good answer. The short version is that stat returns more than just the mode, it returns file type information in the high bits. See "The file type and mode" at https://www.man7.org/linux/man-pages/man7/inode.7.html.

The AI wrote this:

The original program is broken because it incorrectly compares the requested file mode (3565 or 06755 in octal) directly with the raw mode value returned by Unix's Stat system call. This comparison fails because Unix systems store file type information in the high-order bits of the mode value, which the program isn't accounting for. Specifically, the Stat call returns mode 106755, where the leading "10" indicates a regular file type, while the program is only setting and expecting the permission bits (6755). The fix is to apply a bitmask (07777) to isolate just the permission bits when comparing the requested mode with the actual mode. This mask captures all the permission bits including standard rwx permissions (0777) plus the setuid, setgid, and sticky bits (7000), while filtering out the file type information that would otherwise cause the comparison to fail.

and offered this alternative program as a test:

package main

import (
	"fmt"
	"os"
	"syscall"
	"golang.org/x/sys/unix"
)

func check(e error) {
	if e != nil {
		panic(e)
	}
}

func main() {
	// Create a test file with some content
	d1 := []byte("hello\ngo\n")
	err := os.WriteFile("setuidtest", d1, 0644)
	check(err)

	// Define the permissions we want to set (octal 3565)
	mode := uint32(3565)
	
	// Convert to FileMode for chmod, handling special bits
	res := os.FileMode(mode & 0777)
	if mode&syscall.S_ISGID != 0 {
		res |= os.ModeSetgid
	}
	if mode&syscall.S_ISUID != 0 {
		res |= os.ModeSetuid
	}
	if mode&syscall.S_ISVTX != 0 {
		res |= os.ModeSticky
	}

	fmt.Printf("Setting mode of %s: %v (octal %04o)\n", "setuidtest", res, mode)
	
	// Apply the permissions
	if err := os.Chmod("setuidtest", res); err != nil {
		fmt.Printf("Failed to change mode of %s: %v\n", "setuidtest", err)
		return
	}

	// Get actual file stats to verify permissions
	info := unix.Stat_t{}
	if err := unix.Stat("setuidtest", &info); err != nil {
		fmt.Printf("Error getting file permissions for %s: %v\n", "setuidtest", err)
		return
	}

	// Display raw mode and check each special bit
	fmt.Printf("Raw mode: %o\n", info.Mode)
	fmt.Printf("Standard permissions: %o\n", info.Mode&0777)
	fmt.Printf("SetUID bit set: %v\n", info.Mode&syscall.S_ISUID != 0)
	fmt.Printf("SetGID bit set: %v\n", info.Mode&syscall.S_ISGID != 0)
	fmt.Printf("Sticky bit set: %v\n", info.Mode&syscall.S_ISVTX != 0)

	// Convert raw mode to os.FileMode for comparison
	osMode := os.FileMode(info.Mode & 0777)
	if info.Mode&syscall.S_ISUID != 0 {
		osMode |= os.ModeSetuid
	}
	if info.Mode&syscall.S_ISGID != 0 {
		osMode |= os.ModeSetgid
	}
	if info.Mode&syscall.S_ISVTX != 0 {
		osMode |= os.ModeSticky
	}
	fmt.Printf("Converted FileMode: %v\n", osMode)

        // Report summary of whether it worked as expected
        fmt.Println("\nSUMMARY:")
        fmt.Printf("Requested mode: %04o\n", mode)
        fmt.Printf("Actual mode (raw): %06o\n", info.Mode)
        fmt.Printf("Actual mode (permission bits only): %04o\n", info.Mode & 07777)
        fmt.Printf("Success: %v\n", mode == (info.Mode & 07777))
}

I am not sure this helps, but the inode (7) page feels like part of the right trail.

[travier]

There is indeed two different representation for the mode so we can not just use one for the other. Tried some fix in c992889, but it does not appear to be enough.