Unable to run mssql image as a LBI

[ckyrouac]

Here's an example Containerfile:

FROM registry.redhat.io/rhel9/rhel-bootc

COPY usr usr

RUN <<EOF
    set -euxo pipefail

    # open the mssql port
    dnf -y install firewalld
    firewall-offline-cmd --zone=public --add-port=1433/tcp

    # bind the mssql image to the bootc image
    ln -s /usr/share/containers/systemd/mssql.container /usr/lib/bootc/bound-images.d/mssql.container
EOF

and example mssql.container:

[Unit]
Description=mssql server

[Container]
Image=mcr.microsoft.com/mssql/rhel/server:2022-latest
GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage
ContainerName=mssql
HostName=mssql
PublishPort=1433:1433
Environment=ACCEPT_EULA=Y

# Hardcoding this for simplicity.
# In a production environment, this should be set by the build system.
Environment=MSSQL_SA_PASSWORD=Pizzaisg00d!

[Service]
Restart=never

[Install]
WantedBy=multi-user.target

To reproduce:

• podman build the image, bootc install it to a system
• login to the bootc system
• systemctl status mssql.service and notice it failed
• journalctl -u mssql.service shows the following error:

Mar 06 21:02:18 rhel9 systemd[1]: Started mssql server.
Mar 06 21:02:18 rhel9 podman[4937]: 2025-03-06 21:02:18.557640904 +0000 UTC m=+0.198541908 container start d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name
=mssql, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, io.openshift.tags=base rhel9, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=16.0.4175.1
, io.k8s.display-name=Microsoft SQL Server, vendor=Microsoft, name=mcr.microsoft.com/mssql/rhel9/server, build-date=2023-04-04T12:59:33, release=1, com.microsoft.product=Microsoft SQL Server, PODMAN_SYSTEMD_UNIT=mssql.service, vcs-t
ype=git, io.k8s.description=Microsoft SQL Server, url=https://www.microsoft.com/en-us/sql-server/, io.buildah.version=1.37.5, architecture=x86_64, io.openshift.expose-services=, maintainer=Red Hat, Inc., summary=Microsoft SQL Server
, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> , description=Microsoft SQL Server, com.redhat.component=ubi9-container, com.microsoft.version=16.0.4
175.1)
Mar 06 21:02:18 rhel9 mssql[4995]: SQL Server 2022 will run as non-root by default.
Mar 06 21:02:18 rhel9 mssql[4995]: This container is running as user mssql.
Mar 06 21:02:18 rhel9 mssql[4995]: To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
Mar 06 21:02:18 rhel9 mssql[4937]: d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4
Mar 06 21:02:18 rhel9 mssql[4995]: This program has encountered a fatal error and cannot continue running at Thu Mar  6 21:02:18 2025
Mar 06 21:02:18 rhel9 mssql[4995]: The following diagnostic information is available:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:          Reason: 0x00000004
Mar 06 21:02:18 rhel9 mssql[4995]:         Message: RETAIL ASSERT: Expression=(NT_SUCCESS(status)) File=drtl.cpp Line=1552
Mar 06 21:02:18 rhel9 mssql[4995]:         Process: 4 - sqlservr
Mar 06 21:02:18 rhel9 mssql[4995]:          Thread: 9 (application thread 0x8)
Mar 06 21:02:18 rhel9 mssql[4995]:     Instance Id: 25fe55ef-e97a-47db-9d3c-76aceea82e9e
Mar 06 21:02:18 rhel9 mssql[4995]:        Crash Id: dfe2889b-ca74-43ed-a52f-cbe0979b9ad5
Mar 06 21:02:18 rhel9 mssql[4995]:     Build stamp: b395f0d1e33e18f38e89ef83d159436b6140891e0b30bcb265759bb58039f7fd
Mar 06 21:02:18 rhel9 mssql[4995]:    Distribution: Red Hat Enterprise Linux 9.1 (Plow)
Mar 06 21:02:18 rhel9 mssql[4995]:      Processors: 4
Mar 06 21:02:18 rhel9 mssql[4995]:    Total Memory: 8325775360 bytes
Mar 06 21:02:18 rhel9 mssql[4995]:       Timestamp: Thu Mar  6 21:02:18 2025
Mar 06 21:02:18 rhel9 mssql[4995]:      Last errno: 13
Mar 06 21:02:18 rhel9 mssql[4995]: Last errno text: Permission denied
Mar 06 21:02:18 rhel9 mssql[4995]: Capturing a dump of 4
Mar 06 21:02:18 rhel9 mssql[4995]: FAILED to capture a dump. Details in paldumper log.
Mar 06 21:02:18 rhel9 mssql[4995]: Executing: /opt/mssql/bin/handle-crash.sh with parameters
Mar 06 21:02:18 rhel9 mssql[4995]:      handle-crash.sh
Mar 06 21:02:18 rhel9 mssql[4995]:      /opt/mssql/bin/sqlservr
Mar 06 21:02:18 rhel9 mssql[4995]:      4
Mar 06 21:02:18 rhel9 mssql[4995]:      /opt/mssql/bin
Mar 06 21:02:18 rhel9 mssql[4995]:      /var/opt/mssql/log/
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:      25fe55ef-e97a-47db-9d3c-76aceea82e9e
Mar 06 21:02:18 rhel9 mssql[4995]:      dfe2889b-ca74-43ed-a52f-cbe0979b9ad5
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]:
Mar 06 21:02:18 rhel9 mssql[4995]: Red Hat Enterprise Linux 9.1 (Plow)
Mar 06 21:02:18 rhel9 mssql[4995]: Capturing core dump and information to /var/opt/mssql/log...
Mar 06 21:02:18 rhel9 mssql[4995]: cat: /proc/4/maps: Permission denied
Mar 06 21:02:18 rhel9 mssql[4995]: /opt/mssql/bin/crash-support-functions.sh: line 460: hash: lsof: not found
Mar 06 21:02:18 rhel9 mssql[4995]: cat: /proc/4/environ: Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: find: '/proc/4/map_files': Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: dmesg: read kernel buffer failed: Permission denied
Mar 06 21:02:19 rhel9 mssql[4995]: No journal files were found.
Mar 06 21:02:19 rhel9 mssql[4995]: No journal files were found.
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Capturing program information
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Attempting to capture a dump with paldumper for pid 4
Mar 06 21:02:19 rhel9 mssql[4995]: WARNING: Capture attempt failure detected
Mar 06 21:02:19 rhel9 mssql[4995]: Attempting to capture a filtered dump with paldumper for pid 4
Mar 06 21:02:19 rhel9 mssql[4995]: WARNING: Attempt to capture dump failed.  Reference /var/opt/mssql/log/core.sqlservr.4.temp/log/paldumper-debug.log for details
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Attempting to capture a dump with gdb
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Captured a dump with gdb
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Capturing program binaries
Mar 06 21:02:19 rhel9 mssql[4995]: Thu Mar  6 21:02:19 UTC 2025 Compressing the dump files
Mar 06 21:02:19 rhel9 mssql[4995]: Core dump and information are being compressed in the background. When
Mar 06 21:02:19 rhel9 mssql[4995]: complete, they can be found in the following location:
Mar 06 21:02:19 rhel9 mssql[4995]:   /var/opt/mssql/log/core.sqlservr.03_06_2025_21_02_18.4.tbz2
Mar 06 21:02:19 rhel9 podman[5245]: 2025-03-06 21:02:19.316135813 +0000 UTC m=+0.011743492 container died d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name=mssql, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, distribution-scope=public, vcs-type=git, PODMAN_SYSTEMD_UNIT=mssql.service, build-date=2023-04-04T12:59:33, com.redhat.component=ubi9-container, io.k8s.display-name=Microsoft SQL Server, url=https://www.microsoft.com/en-us/sql-server/, description=Microsoft SQL Server, architecture=x86_64, io.k8s.description=Microsoft SQL Server, io.openshift.expose-services=, release=1, vendor=Microsoft, io.openshift.tags=base rhel9, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Microsoft SQL Server, version=16.0.4175.1, name=mcr.microsoft.com/mssql/rhel9/server, com.microsoft.version=16.0.4175.1, io.buildah.version=1.37.5, maintainer=Red Hat, Inc., com.microsoft.product=Microsoft SQL Server, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> )
Mar 06 21:02:19 rhel9 podman[5245]: 2025-03-06 21:02:19.674982611 +0000 UTC m=+0.370590273 container remove d21ef3d90483bd9b96fae2be65efbb67023e576bcab28dbc2937fa6c45a5d7e4 (image=mcr.microsoft.com/mssql/rhel/server:2022-latest, name=mssql, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=Microsoft SQL Server, io.openshift.tags=base rhel9, maintainer=Red Hat, Inc., version=16.0.4175.1, com.microsoft.version=16.0.4175.1, description=Microsoft SQL Server, distribution-scope=public, name=mcr.microsoft.com/mssql/rhel9/server, summary=Microsoft SQL Server, release=1, vcs-ref=cf87ad00feaef3d9d7a442dad55ab6a14f6a3f81, io.buildah.version=1.37.5, com.microsoft.product=Microsoft SQL Server, io.openshift.expose-services=, PODMAN_SYSTEMD_UNIT=mssql.service, build-date=2023-04-04T12:59:33, url=https://www.microsoft.com/en-us/sql-server/, vendor=Microsoft, architecture=x86_64, com.redhat.component=ubi9-container, run=docker run --name mssql-server         -e ACCEPT_EULA=Y -e SA_PASSWORD=<password>         -p 1433:1433         -d <IMAGE ID> , io.k8s.display-name=Microsoft SQL Server, vcs-ty
pe=git)
Mar 06 21:02:19 rhel9 systemd[1]: mssql.service: Main process exited, code=exited, status=1/FAILURE
Mar 06 21:02:19 rhel9 systemd[1]: mssql.service: Failed with result 'exit-code'.

---

• Disabling selinux via setenforce Permissive then systemctl restart mssql.service makes the service start
• Also pulling the image into the primary storage (/var/lib/containers/storage) and running it from there works
• A key difference I noticed is the labels on the bootc additional storage (/sysroot/ostree/bootc/storage/) is system_u:object_r:usr_t:s0
  • The primary image storage (/var/lib/containers/storage) has the container labels, e.g system_u:object_r:container_var_lib_t:s0
• One last note podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage run -e ACCEPT_EULA=Y mcr.microsoft.com/mssql/rhel/server is a quick way to reproduce the error outside of the systemd unit

---

selinux logs:

Mar 06 21:18:37 rhel9 kernel: audit: type=1400 audit(1741295917.710:25): avc:  denied  { execmod } for  pid=5789 comm="Wt-8" path="/opt/mssql/lib/system.sfp" dev="dm-0" ino=72963579 scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:object_r:usr_t:s0 tclass=file permissive=0
Mar 06 21:18:38 rhel9 kernel: audit: type=1400 audit(1741295918.039:26): avc:  denied  { syslog_read } for  pid=5899 comm="dmesg" scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0
Mar 06 21:18:38 rhel9 kernel: audit: type=1400 audit(1741295918.039:27): avc:  denied  { syslog_read } for  pid=5899 comm="dmesg" scontext=system_u:system_r:container_t:s0:c509,c1007 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0

[cgwalters]

cc @mheon any ideas here?

[cgwalters]

@ckyrouac can you also get the selinux logs which will be in /var/log/audit.log or journalctl |grep avc?

[ckyrouac]

yep, put them in the description

[cgwalters]

Hmm so presumably the fatal problem is the execmod. That's a funky one...web searches turn up ~19 year old docs like https://www.akkadia.org/drepper/selinux-mem.html

Offhand what I'm wondering here is...in the case where the container overlayfs is on top of an immutable lower store that's also labeled usr_t as it is in this case, I wonder if the overlayfs copy-up operation here is checking the execmod operation against the lower filesystem somehow. Maybe we could try reproducing this with that minimal C program he pasted?

AIUI though again the argument in that page is usually you want to avoid needing execmod and to do the "dual mapping" trick...so that's something we could ask them to do.

Does it fix it to do e.g. unshare -m /bin/bash -c 'mount -o remount,rw /sysroot && chcon -R -h -t container_var_lib_t /sysroot/ostree/bootc/storage' or so?

[cgwalters]

Also cc @rhatdan