Unable to run mssql image as a LBI

[ckyrouac]

Here's an example Containerfile:

FROM registry.redhat.io/rhel9/rhel-bootc

COPY usr usr

RUN <<EOF
    set -euxo pipefail

    # open the mssql port
    dnf -y install firewalld
    firewall-offline-cmd --zone=public --add-port=1433/tcp

    # bind the mssql image to the bootc image
    ln -s /usr/share/containers/systemd/mssql.container /usr/lib/bootc/bound-images.d/mssql.container
EOF

and example mssql.container:

[Unit]
Description=mssql server

[Container]
Image=mcr.microsoft.com/mssql/rhel/server:2022-latest
GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage
ContainerName=mssql
HostName=mssql
PublishPort=1433:1433
Environment=ACCEPT_EULA=Y

# Hardcoding this for simplicity.
# In a production environment, this should be set by the build system.
Environment=MSSQL_SA_PASSWORD=Pizzaisg00d!

[Service]
Restart=never

[Install]
WantedBy=multi-user.target

To reproduce:

• podman build the image, bootc install it to a system
• login to the bootc system
• systemctl status mssql.service and notice it failed
• journalctl -u mssql.service shows the following error:

Mar 06 21:02:18 rhel9 systemd[1]: Started mssql server.
Mar 06 21:02:18 rhel9 podman[4937]: 2025-03-06 21:02:18.557640904 +0000 UTC m=+0.198541908 container start d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name
=mssql, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, io.openshift.tags=base rhel9, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=16.0.4175.1
, io.k8s.display-name=Microsoft SQL Server, vendor=Microsoft, name=mcr.microsoft.com/mssql/rhel9/server, build-date=2023-04-04T12:59:33, release=1, com.microsoft.product=Microsoft SQL Server, PODMAN_SYSTEMD_UNIT=mssql.service, vcs-t
ype=git, io.k8s.description=Microsoft SQL Server, url=https://www.microsoft.com/en-us/sql-server/, io.buildah.version=1.37.5, architecture=x86_64, io.openshift.expose-services=, maintainer=Red Hat, Inc., summary=Microsoft SQL Server
, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> , description=Microsoft SQL Server, com.redhat.component=ubi9-container, com.microsoft.version=16.0.4
175.1)
Mar 06 21:02:18 rhel9 mssql[4995]: SQL Server 2022 will run as non-root by default.
Mar 06 21:02:18 rhel9 mssql[4995]: This container is running as user mssql.
Mar 06 21:02:18 rhel9 mssql[4995]: To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
Mar 06 21:02:18 rhel9 mssql[4937]: d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4
Mar 06 21:02:18 rhel9 mssql[4995]: This program has encountered a fatal error and cannot continue running at Thu Mar  6 21:02:18 2025
Mar 06 21:02:18 rhel9 mssql[4995]: The following diagnostic information is available:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:          Reason: 0x00000004
Mar 06 21:02:18 rhel9 mssql[4995]:         Message: RETAIL ASSERT: Expression=(NT_SUCCESS(status)) File=drtl.cpp Line=1552
Mar 06 21:02:18 rhel9 mssql[4995]:         Process: 4 - sqlservr
Mar 06 21:02:18 rhel9 mssql[4995]:          Thread: 9 (application thread 0x8)
Mar 06 21:02:18 rhel9 mssql[4995]:     Instance Id: 25fe55ef-e97a-47db-9d3c-76aceea82e9e
Mar 06 21:02:18 rhel9 mssql[4995]:        Crash Id: dfe2889b-ca74-43ed-a52f-cbe0979b9ad5
Mar 06 21:02:18 rhel9 mssql[4995]:     Build stamp: b395f0d1e33e18f38e89ef83d159436b6140891e0b30bcb265759bb58039f7fd
Mar 06 21:02:18 rhel9 mssql[4995]:    Distribution: Red Hat Enterprise Linux 9.1 (Plow)
Mar 06 21:02:18 rhel9 mssql[4995]:      Processors: 4
Mar 06 21:02:18 rhel9 mssql[4995]:    Total Memory: 8325775360 bytes
Mar 06 21:02:18 rhel9 mssql[4995]:       Timestamp: Thu Mar  6 21:02:18 2025
Mar 06 21:02:18 rhel9 mssql[4995]:      Last errno: 13
Mar 06 21:02:18 rhel9 mssql[4995]: Last errno text: Permission denied
Mar 06 21:02:18 rhel9 mssql[4995]: Capturing a dump of 4
Mar 06 21:02:18 rhel9 mssql[4995]: FAILED to capture a dump. Details in paldumper log.
Mar 06 21:02:18 rhel9 mssql[4995]: Executing: /opt/mssql/bin/handle-crash.sh with parameters
Mar 06 21:02:18 rhel9 mssql[4995]:      handle-crash.sh
Mar 06 21:02:18 rhel9 mssql[4995]:      /opt/mssql/bin/sqlservr
Mar 06 21:02:18 rhel9 mssql[4995]:      4
Mar 06 21:02:18 rhel9 mssql[4995]:      /opt/mssql/bin
Mar 06 21:02:18 rhel9 mssql[4995]:      /var/opt/mssql/log/
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:      25fe55ef-e97a-47db-9d3c-76aceea82e9e
Mar 06 21:02:18 rhel9 mssql[4995]:      dfe2889b-ca74-43ed-a52f-cbe0979b9ad5
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]: Red Hat Enterprise Linux 9.1 (Plow)
Mar 06 21:02:18 rhel9 mssql[4995]: Capturing core dump and information to /var/opt/mssql/log...
Mar 06 21:02:18 rhel9 mssql[4995]: cat: /proc/4/maps: Permission denied
Mar 06 21:02:18 rhel9 mssql[4995]: /opt/mssql/bin/crash-support-functions.sh: line 460: hash: lsof: not found
Mar 06 21:02:18 rhel9 mssql[4995]: cat: /proc/4/environ: Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: dmesg: read kernel buffer failed: Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: No journal files were found.
Mar 06 21:02:19 rhel9 mssql[4995]: No journal files were found.
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Capturing program information
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Attempting to capture a dump with paldumper for pid 4
Mar 06 21:02:19 rhel9 mssql[4995]: WARNING: Capture attempt failure detected
Mar 06 21:02:19 rhel9 mssql[4995]: Attempting to capture a filtered dump with paldumper for pid 4
Mar 06 21:02:19 rhel9 mssql[4995]: WARNING: Attempt to capture dump failed.  Reference /var/opt/mssql/log/core.sqlservr.4.temp/log/paldumper-debug.log for details
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Attempting to capture a dump with gdb
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Captured a dump with gdb
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Capturing program binaries
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Compressing the dump files
Mar 06 21:02:19 rhel9 mssql[4995]: Core dump and information are being compressed in the background. When
Mar 06 21:02:19 rhel9 mssql[4995]: complete, they can be found in the following location:
Mar 06 21:02:19 rhel9 mssql[4995]:   /var/opt/mssql/log/core.sqlservr.03_06_2025_21_02_18.4.tbz2
Mar 06 21:02:19 rhel9 podman[5245]: 2025-03-06 21:02:19.316135813 +0000 UTC m=+0.011743492 container died d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name=mssql, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, distribution-scope=public, vcs-type=git, PODMAN_SYSTEMD_UNIT=mssql.service, build-date=2023-04-04T12:59:33, com.redhat.component=ubi9-container, io.k8s.display-name=Microsoft SQL Server, url=https://www.microsoft.com/en-us/sql-server/, description=Microsoft SQL Server, architecture=x86_64, io.k8s.description=Microsoft SQL Server, io.openshift.expose-services=, release=1, vendor=Microsoft, io.openshift.tags=base rhel9, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Microsoft SQL Server, version=16.0.4175.1, name=mcr.microsoft.com/mssql/rhel9/server, com.microsoft.version=16.0.4175.1, io.buildah.version=1.37.5, maintainer=Red Hat, Inc., com.microsoft.product=Microsoft SQL Server, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> )
Mar 06 21:02:19 rhel9 podman[5245]: 2025-03-06 21:02:19.674982611 +0000 UTC m=+0.370590273 container remove d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name=mssql, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=Microsoft SQL Server, io.openshift.tags=base rhel9, maintainer=Red Hat, Inc., version=16.0.4175.1, com.microsoft.version=16.0.4175.1, description=Microsoft SQL Server, distribution-scope=public, name=mcr.microsoft.com/mssql/rhel9/server, summary=Microsoft SQL Server, release=1, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, io.buildah.version=1.37.5, com.microsoft.product=Microsoft SQL Server, io.openshift.expose-services=, PODMAN_SYSTEMD_UNIT=mssql.service, build-date=2023-04-04T12:59:33, url=https://www.microsoft.com/en-us/sql-server/, vendor=Microsoft, architecture=x86_64, com.redhat.component=ubi9-container, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> , io.k8s.display-name=Microsoft SQL Server, vcs-ty
pe=git)
Mar 06 21:02:19 rhel9 systemd[1]: mssql.service: Main process exited, code=exited, status=1/FAILURE
Mar 06 21:02:19 rhel9 systemd[1]: mssql.service: Failed with result 'exit-code'.

---

• Disabling selinux via setenforce Permissive then systemctl restart mssql.service makes the service start
• Also pulling the image into the primary storage (/var/lib/containers/storage) and running it from there works
• A key difference I noticed is the labels on the bootc additional storage (/sysroot/ostree/bootc/storage/) is system_u:object_r:usr_t:s0
  • The primary image storage (/var/lib/containers/storage) has the container labels, e.g system_u:object_r:container_var_lib_t:s0
• One last note podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage run -e ACCEPT_EULA=Y mcr.microsoft.com/mssql/rhel/server is a quick way to reproduce the error outside of the systemd unit

---

selinux logs:

Mar 06 21:18:37 rhel9 kernel: audit: type=1400 audit(1741295917.710:25): avc:  denied  { execmod } for  pid=5789 comm="Wt-8" path="/opt/mssql/lib/system.sfp" dev="dm-0" ino=72963579 scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:object_r:usr_t:s0 tclass=file permissive=0
Mar 06 21:18:38 rhel9 kernel: audit: type=1400 audit(1741295918.039:26): avc:  denied  { syslog_read } for  pid=5899 comm="dmesg" scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0
Mar 06 21:18:38 rhel9 kernel: audit: type=1400 audit(1741295918.039:27): avc:  denied  { syslog_read } for  pid=5899 comm="dmesg" scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0

[cgwalters]

cc @mheon any ideas here?

[cgwalters]

@ckyrouac can you also get the selinux logs which will be in /var/log/audit.log or journalctl |grep avc?

[ckyrouac]

yep, put them in the description

[cgwalters]

Hmm so presumably the fatal problem is the execmod. That's a funky one...web searches turn up ~19 year old docs like https://www.akkadia.org/drepper/selinux-mem.html

Offhand what I'm wondering here is...in the case where the container overlayfs is on top of an immutable lower store that's also labeled usr_t as it is in this case, I wonder if the overlayfs copy-up operation here is checking the execmod operation against the lower filesystem somehow. Maybe we could try reproducing this with that minimal C program he pasted?

AIUI though again the argument in that page is usually you want to avoid needing execmod and to do the "dual mapping" trick...so that's something we could ask them to do.

Does it fix it to do e.g. unshare -m /bin/bash -c 'mount -o remount,rw /sysroot && chcon -R -h -t container_var_lib_t /sysroot/ostree/bootc/storage' or so?

[cgwalters]

Also cc @rhatdan

[ckyrouac]

so if I recursively set the label of /sysroot/ostree/bootc/storage to container_var_lib_t, then try to run the container, I get a single selinux error which is for a read:

Mar 11 14:34:53 rhel9 kernel: audit: type=1400 audit(1741703693.215:8): avc:  denied  { read } for  pid=2263 comm="permissions_che" path="/usr/lib64/libc.so.6" dev="dm-0" ino=107034670 scontext=system_u:system_r:container_t:s0:c214,c425 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=file permissive=0

in the default storage that file is labeled as container_ro_file_t. If I set /sysroot/ostree/bootc/storage/overlay to container_ro_file_t, the mssql container starts successfully.

[ckyrouac]

so I guess the correct thing to do would be to set the labels when creating the storage as described in the containers-storage.conf man page?

[cgwalters]

so I guess the correct thing to do would be to set the labels when creating the storage as described in the containers-storage.conf man page?

Hmm, yes. We should do that indeed. Though note that one problem we'll have is that for existing systems the labels won't change by default, so we'll probably need to add logic in the upgrade flow that automatically relabels as well.

---

(This said...I would probably say that the value of SELinux label separation for storage components matters a lot less when it's logically read-only and so it should be allowed to just be usr_t or whatever but...eh, let's go with the flow here)

[cgwalters]

Though note that one problem we'll have is that for existing systems the labels won't change by default, so we'll probably need to add logic in the upgrade flow that automatically relabels as well.

But still it's OK by me to break this into two parts; first a PR for the install flow which sets the label, and then a secondary upgrade PR.

[vrothberg]

But still it's OK by me to break this into two parts; first a PR for the install flow which sets the label, and then a secondary upgrade PR.

+1

Sounds like another 0day?

[cgwalters]

@ckyrouac can you try starting from something like this?

diff --git a/lib/src/install.rs b/lib/src/install.rs
index 2495f094..cb4b23bd 100644
--- a/lib/src/install.rs
+++ b/lib/src/install.rs
@@ -45,7 +45,7 @@ use ostree_ext::sysroot::SysrootLock;
 use ostree_ext::{container as ostree_container, ostree_prepareroot};
 #[cfg(feature = "install-to-disk")]
 use rustix::fs::FileTypeExt;
-use rustix::fs::MetadataExt as _;
+use rustix::fs::{MetadataExt as _, Mode};
 use serde::{Deserialize, Serialize};
 
 #[cfg(feature = "install-to-disk")]
@@ -671,9 +671,19 @@ async fn initialize_ostree_root(state: &State, root_setup: &RootSetup) -> Result
 
     state.tempdir.create_dir("temp-run")?;
     let temp_run = state.tempdir.open_dir("temp-run")?;
-    sysroot_dir
-        .create_dir_all(Utf8Path::new(crate::imgstorage::SUBPATH).parent().unwrap())
-        .context("creating bootc dir")?;
+    {
+        let storage_parent = Utf8Path::new(crate::imgstorage::SUBPATH).parent().unwrap();
+        // Ensure we label this as if it's the default container store
+        // https://github.com/containers/storage/blob/main/docs/containers-storage.conf.5.md#storage-table
+        let as_path = Some("/var/lib/containers/storage".into());
+        crate::lsm::ensure_dir_labeled(
+            &sysroot_dir,
+            storage_parent,
+            as_path,
+            Mode::from_raw_mode(0o755),
+            sepolicy,
+        )?;
+    }
     let imgstore = crate::imgstorage::Storage::create(&sysroot_dir, &temp_run)?;
     // And drop it again - we'll reopen it after this
     drop(imgstore);

(only compile tested)

[ckyrouac]

I tried something similar to this earlier. This only sets the top level label to container_var_lib_t since in the end it's just calling fsetxattr. All the sub directories are still usr_t. I think we need to handle this at the policy level, otherwise it'll get messy to properly label everything per these rules:

/var/lib/containers(/.*)?                          all files          system_u:object_r:container_var_lib_t:s0
/var/lib/containers/atomic(/.*)?                   all files          <<None>>
/var/lib/containers/home(/.*)?                     all files          system_u:object_r:openshift_var_lib_t:s0
/var/lib/containers/overlay(/.*)?                  all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/overlay-images(/.*)?           all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/overlay-layers(/.*)?           all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/overlay2(/.*)?                 all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/overlay2-images(/.*)?          all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/overlay2-layers(/.*)?          all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay(/.*)?          all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay-images(/.*)?   all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay-layers(/.*)?   all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay2(/.*)?         all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay2-images(/.*)?  all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/overlay2-layers(/.*)?  all files          system_u:object_r:container_ro_file_t:s0
/var/lib/containers/storage/volumes/[^/]*/.*       all files          system_u:object_r:container_file_t:s0

The two options I can think of are:

Run semanage fcontext ... in a chroot in the target (not sure how)

or

• Parse /etc/selinux/config for SELINUXTYPE=
• Append /sysroot/ostree/bootc/storage /var/lib/containers to /etc/selinux/<type>/contexts/files/file_contexts.subs
• Run restorecon -R -v /sysroot/ostree/bootc/storage