GSC failed to run signed Docker image

[NandiniKJ]

Description of the problem

We are trying to run a Docker image using Gramine shielded containers.
We are successfully able to create the signed image but the container is throwing the below error when we try to run the signed docker image.

We have been following this link: https://gramine.readthedocs.io/projects/gsc/en/latest/

We were able to bring this application up with normal container deployment and were able to verify that the start-baffle-shiled.sh file exists. After converting to and running inside Gramine container I found that the opt folder is empty.

Could you help us with this, are we missing something here.

Steps to reproduce

Expected results

Actual results

[mkow]

Please don't paste screenshots of text, instead just copy and paste the text itself. It's hard to read and makes it impossible to copy and search in it.

[dimakuv]

After converting to and running inside Gramine container I found that the opt folder is empty.

This doesn't sound right. GSC doesn't do anything with the /opt directory. So I don't know why it becomes empty.

[NandiniKJ]

I am able to run the docker image without Gramine and it works fine. Please find the below output. After converting it to Gramine container should we mount the /opt path or change the entrypoint.sh. I'm I missing something here.

`SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/baffle/Release-Baffle.1.7.0.161/shield/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/baffle/Release-Baffle.1.7.0.161/shield/bm-connector.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-07-17 10:49:08,086 1 INFO CommonConfig:508 - baffle.config.bmShieldSyncID is defined, using value from system parameter
2023-07-17 10:49:08,090 1 INFO BMShieldClient:69 - Initializing BMShieldClient
2023-07-17 10:49:08,091 1 INFO BMShieldClient:760 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield
2023-07-17 10:49:08,091 1 INFO BMShieldClient:756 - baffle.ssl.path undefined, loading from default value at /opt/baffle/ssl
2023-07-17 10:49:08,091 1 INFO BMShieldClient:761 - baffle.config.bmShieldSyncID is defined, using value from system parameter
2023-07-17 10:49:08,201 1 INFO BMShieldClient:756 - baffle.config.shieldTag undefined, loading from default value at
2023-07-17 10:49:08,201 1 INFO BMShieldClient:760 - baffle.config.bm.ip is defined, using value dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud
2023-07-17 10:49:08,201 1 INFO BMShieldClient:756 - baffle.config.bm.port undefined, loading from default value at 443
2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.tenant undefined, loading from default value at ibm
2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.appID undefined, loading from default value at 64acfabd65fab6124f7872fa
2023-07-17 10:49:08,202 1 INFO BMShieldClient:756 - baffle.config.bm.shieldID undefined, loading from default value at null
2023-07-17 10:49:08,202 1 INFO BMRestClient:58 - Checking BM Connectivity using URL - https://dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud:443/
2023-07-17 10:49:09,225 1 INFO BMShieldClient:795 - baffle.shield.ssl is defined, using value true
2023-07-17 10:49:09,376 1 INFO BMShieldClient:191 - Registration Successful ShieldID =64b3f41765fab6124f78731f
2023-07-17 10:49:09,377 1 INFO BMShieldClient:205 - Getting Shield Initial Configuration
2023-07-17 10:49:09,522 1 INFO BMShieldClient:318 - Downloading Config File : BafflePrivacySchema
2023-07-17 10:49:09,664 1 INFO BMShieldClient:318 - Downloading Config File : BaffleEntitySchema
2023-07-17 10:49:09,803 1 INFO BMShieldClient:318 - Downloading Config File : KmsConfig.properties
2023-07-17 10:49:09,955 1 INFO BMShieldClient:318 - Downloading Config File : BaffleCommonConfig
2023-07-17 10:49:10,101 1 INFO CommonConfig:1156 - Tier: 0
2023-07-17 10:49:10,102 1 INFO CommonConfig:1067 - Nothing to do for the deployment type local
2023-07-17 10:49:10,136 1 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,136 1 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,136 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,139 1 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,140 1 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,140 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,141 1 INFO BaffleSecretStoreUtil:587 - No SecretStores configured
2023-07-17 10:49:10,142 1 WARN CommonConfig:1324 - defaulting value of encMode as M_CTR
2023-07-17 10:49:10,142 1 INFO CommonConfig:465 - baffle.shield.ssl is defined, using value true
2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.keystore is defined, using value /opt/sslconfig/baffleshield-keystore.jks
2023-07-17 10:49:10,143 1 INFO CommonConfig:508 - baffle.shield.keystore.password is defined, using value from system parameter
2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.truststore is defined, using value /opt/sslconfig/baffleshield-keystore.jks
2023-07-17 10:49:10,143 1 INFO CommonConfig:508 - baffle.shield.truststore.password is defined, using value from system parameter
2023-07-17 10:49:10,143 1 INFO CommonConfig:507 - baffle.shield.ssl.tlsVersion is defined, using value TLSv1.2
2023-07-17 10:49:10,145 1 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/BaffleCommonConfig
baffleHeartbeatInterval=30000
....
2023-07-17 10:49:10,146 1 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,146 1 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,146 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,146 1 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,147 1 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,147 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,147 1 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,147 1 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,147 1 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,149 1 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/KmsConfig.properties
baffle_secret=***
kmsType=local

2023-07-17 10:49:10,175 1 INFO CommonConfig:507 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield
2023-07-17 10:49:10,176 1 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema
2023-07-17 10:49:10,187 1 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema
2023-07-17 10:49:10,189 1 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema
2023-07-17 10:49:10,189 1 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema
2023-07-17 10:49:10,190 1 INFO PrivacySchemaReaderToml:194 - Copying BafflePrivacySchema to .BafflePrivacySchema.Verified
2023-07-17 10:49:10,192 1 INFO BaffleMonitorMetric:42 - Baffle Health Metrics bean registration complete
2023-07-17 10:49:10,288 15 INFO BaffleShield:392 - ************************************************************
2023-07-17 10:49:10,289 15 INFO BaffleShield:393 - ****** BAFFLESHIELD ******
2023-07-17 10:49:10,289 15 INFO BaffleShield:394 - ************************************************************
2023-07-17 10:49:10,289 15 INFO BaffleShield:395 - Deployment type: local
2023-07-17 10:49:10,295 15 INFO CommonConfig:756 - Baffle Release Version: Release-Baffle.1.7.0.161
2023-07-17 10:49:10,295 15 INFO BaffleShield:397 - Parser version: 2.6.5.4-baffle2
2023-07-17 10:49:10,296 15 INFO BaffleShield:398 - JVM bit size: 64
2023-07-17 10:49:10,296 15 INFO BaffleShield:399 - Free memory: 179432360 Total memory: 204472320 Max memory: 3674210304
2023-07-17 10:49:10,296 15 INFO BaffleShield:401 - nofile limit : 1048576
2023-07-17 10:49:10,296 15 INFO BaffleShield:403 - Using OpenSSL version: BoringSSL
2023-07-17 10:49:10,296 15 INFO BaffleShield:412 - Using BaffleCommonConfig at: /opt/baffle/Release-Baffle.1.7.0.161/shield
2023-07-17 10:49:10,296 15 INFO BaffleShield:413 - Using Credential store configuration at: /opt/baffle/Release-Baffle.1.7.0.161/shield/credstore
2023-07-17 10:49:10,297 15 INFO BaffleShield:417 - Proxying *:8444 to f637666e-dc0a-48ac-806d-8e6af4046a2c.c9v3nahd0oekcvsra2t0.databases.appdomain.cloud:31841
2023-07-17 10:49:10,297 15 INFO BaffleShield:437 - BaffleShutdownHook Thread Started
2023-07-17 10:49:10,428 15 INFO CommonConfig:507 - baffle.config.path is defined, using value /opt/baffle/Release-Baffle.1.7.0.161/shield
2023-07-17 10:49:10,428 15 INFO KmsConfig:59 - KeyStore type - local
2023-07-17 10:49:10,428 15 INFO KmsConfig:63 - local : Configured application namespace - null
2023-07-17 10:49:10,428 15 INFO KmsConfig:75 - No dek storage module specified. The keys will be in the cache until revoked
2023-07-17 10:49:10,429 15 INFO ConfigReaderUtil:398 - File: /opt/baffle/Release-Baffle.1.7.0.161/shield/BafflePrivacySchema
format = "TOML"

2023-07-17 10:49:10,429 15 INFO PrivacySchemaReaderToml:106 - Reading from BafflePrivacySchema
2023-07-17 10:49:10,430 15 INFO TomlPrivacySchemaReader:60 - Nothing to encrypt. Empty BafflePrivacySchema
2023-07-17 10:49:10,430 15 INFO TransformDB:460 - Privacy schema update:
2023-07-17 10:49:10,432 15 INFO PrivacySchemaHandler:92 - Starting BafflePrivacySchema Watcher on /opt/baffle/Release-Baffle.1.7.0.161/shield/BafflePrivacySchema
`

[dimakuv]

Can you show the docker run command with which you run:

1. the original app image,
2. the GSC-generated image

[NandiniKJ]

Original app image command:
docker run --rm -it -e BM_IP=dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud -e BM_SHIELD_SYNC_ID="*****" -e BM_SHIELD_TAG=dsb-shield-app1 -e BS_SSL=true -e BS_SSL_KEYSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_KEYSTORE_PASSWORD=keystore -e BS_SSL_TRUSTSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_TRUSTSTORE_PASSWORD=keystore -e BS_SSL_TLS_VERSION=TLSv1.2 -e KMS_CONFIG_PROPERTIES="{'baffle_secret':'*****','kmsType': 'local'}" icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

GSC generated image:

docker run --rm -it -e BM_IP=dsb-manager-dsb-for-nandini.dsb-roks-vpc-412-c9b7119538b194dae4a1958742b244b0-0000.eu-de.containers.appdomain.cloud -e BM_SHIELD_SYNC_ID="*****" -e BM_SHIELD_TAG=dsb-shield-app1 -e BS_SSL=true -e BS_SSL_KEYSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_KEYSTORE_PASSWORD=keystore -e BS_SSL_TRUSTSTORE_FILE=/opt/sslconfig/baffleshield-keystore.jks -e BS_SSL_TRUSTSTORE_PASSWORD=keystore -e BS_SSL_TLS_VERSION=TLSv1.2 -e KMS_CONFIG_PROPERTIES="{'baffle_secret':'*****','kmsType': 'local'}" --device=/dev/sgx_enclave -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

I even tried to run the gsc container command without passing the environment variables.
docker run --device=/dev/sgx_enclave \ -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket \ gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

[dimakuv]

Have you tried to enter this GSC-generated Docker image and take a look around? With a command like:

docker run -it --entrypoint /bin/bash gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

When you enter the Bash session inside this GSC-generated image, you really don't see the /opt/ directory?

[NandiniKJ]

I tried to look inside the GSC Docker image, opt folder is there but its empty.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker run -it --entrypoint /bin/bash gsc-icr.io/data-security-broker/dsb-shield-postgresql-v1:v1 root@16ac5039e29b:/# ls bin boot dev etc gramine home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var root@16ac5039e29b:/# cd opt/ root@16ac5039e29b:/opt# ls root@16ac5039e29b:/opt#

[dimakuv]

And it's not empty in the original app Docker image? Are you sure? How is this possible...

[NandiniKJ]

Its not empty in the original Docker image. It has the start-baffle-shield.sh script in the defined path /opt/baffle/Release-Baffle.1.7.0.161/shield.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker exec -it 17e6f3692053 sh sh-4.4$ ls bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var sh-4.4$ cd opt/ sh-4.4$ ls baffle sslconfig sh-4.4$ cd baffle/ sh-4.4$ sh-4.4$ ls Release-Baffle.1.7.0.161 sh-4.4$ cd Release-Baffle.1.7.0.161/ sh-4.4$ ls migration shield sh-4.4$ cd shield/ sh-4.4$ ls BaffleCommonConfig BafflePrivacySchema baffle-shield-postgresql-final.jar bss_public.pem key_transfer log4j-baffleshield.properties log4j-slf4j-impl-2.17.2.jar start-baffle-shield.sh BaffleEntitySchema KmsConfig.properties bm-connector.jar credstore log4j-api-2.17.2.jar log4j-core-2.17.2.jar logs sh-4.4$

[dimakuv]

But you performed docker exec -it 17e6f3692053, i.e., you connected to an already-existing Docker container. Could it be that in this container, you or some script created the /opt/ contents?

What if you just do:

docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql-v1:v1

[NandiniKJ]

I tried the above command which you gave, the existing image has the baffle folder inside opt.

root@baremetal01-innovation-poc-sgx:~/gramine-poc/gsc-v2# docker run -it --entrypoint /bin/bash icr.io/data-security-broker/dsb-shield-postgresql:v1 [baffle@b0c3229b9f8a /]$ ls bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var [baffle@b0c3229b9f8a /]$ cd opt/ [baffle@b0c3229b9f8a opt]$ ls baffle sslconfig [baffle@b0c3229b9f8a opt]$ cd baffle [baffle@b0c3229b9f8a baffle]$ ls Release-Baffle.1.7.0.161 [baffle@b0c3229b9f8a baffle]$ cd Release-Baffle.1.7.0.161/ [baffle@b0c3229b9f8a Release-Baffle.1.7.0.161]$ cd shield/ [baffle@b0c3229b9f8a shield]$ ls baffle-shield-postgresql-final.jar bss_public.pem log4j-api-2.17.2.jar log4j-core-2.17.2.jar start-baffle-shield.sh bm-connector.jar key_transfer log4j-baffleshield.properties log4j-slf4j-impl-2.17.2.jar [baffle@b0c3229b9f8a shield]$

[dimakuv]

This makes no sense to me...

Could you:

1. Remove all Docker images, containers, etc. -- basically prune all Docker artifacts
2. Remove the build/ directory in GSC -- prune all GSC-temporary data
3. Re-run gsc build and gsc sign-image commands and attach the produced logs

In other words, re-do the whole GSC process from an absolutely clean state, and show us the logs.

[NandiniKJ]

I deleted all the Docker images and pruned it. Git cloned - https://github.com/gramineproject/gsc.git
Then ran the following commands:

cd gsc cp config.yaml.template config.yaml openssl genrsa -3 -out enclave-key.pem 3072 ./gsc build --insecure-args icr.io/data-security-broker/dsb-shield-postgresql:v1 test/generic.manifest

This time the build failed in Step 9. Please find the build output below.

`Step 9/29 : RUN cd /gramine && meson setup build/ --prefix="/gramine/meson_build_output" --buildtype=release -Ddirect=enabled -Dsgx=enabled -Ddcap=enabled -Dsgx_driver=upstream -Dsgx_driver_include_path=/gramine/driver && ninja -C build && ninja -C build install

---> Running in 319df51f93e9
The Meson build system
Version: 1.2.0
Source dir: /gramine
Build dir: /gramine/build
Build type: native build
Project name: gramine
Project version: 1.5postUNRELEASED
C compiler for the host machine: cc (gcc 9.4.0 "cc (Ubuntu 9.4.0-1ubuntu120.04.1) 9.4.0")
C linker for the host machine: cc ld.bfd 2.34
C++ compiler for the host machine: c++ (gcc 9.4.0 "c++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0")
C++ linker for the host machine: c++ ld.bfd 2.34
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program check-no-reloc.sh found: YES (/gramine/scripts/check-no-reloc.sh)
Program gen-pal-map.py found: YES (/gramine/scripts/gen-pal-map.py)
Program get-python-platlib.py found: YES (/gramine/scripts/get-python-platlib.py)
Program meson-clang-format.sh found: YES (/gramine/scripts/meson-clang-format.sh)
Fetching value of define "GLIBC" : 2
Program objcopy found: YES (/usr/bin/objcopy)
Program python3 found: YES (/usr/bin/python3)
WARNING: You should add the boolean check kwarg to the run_command call.
.......
cc common.o onefile.o fuzz_dtlsserver.o ../../tests/src/helpers.o ../../tests/src/bignum_helpers.o ../../tests/src/asn1_helpers.o ../../tests/src/psa_crypto_helpers.o ../../tests/src/psa_exercise_key.o ../../tests/src/threading_helpers.o ../../tests/src/random.o ../../tests/src/fake_external_rng_for_test.o ../../tests/src/certs.o ../../tests/src/drivers/test_driver_aead.o ../../tests/src/drivers/test_driver_asymmetric_encryption.o ../../tests/src/drivers/test_driver_pake.o ../../tests/src/drivers/test_driver_key_agreement.o ../../tests/src/drivers/test_driver_signature.o ../../tests/src/drivers/test_driver_key_management.o ../../tests/src/drivers/test_driver_cipher.o ../../tests/src/drivers/hash.o ../../tests/src/drivers/test_driver_mac.o ../../tests/src/drivers/platform_builtin_keys.o -L../../library -lmbedtls -lmbedx509 -lmbedcrypto -o fuzz_dtlsserver
make[2]: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/programs/fuzz'
make[1]: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/programs'
make: Leaving directory '/gramine/build/subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p'

• basename subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a
• cp -a subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/library/libmbedcrypto.a subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a
• basename subprojects/mbedtls-mbedtls-3.4.0/libmbedtls.a
• cp -a subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/library/libmbedtls.a subprojects/mbedtls-mbedtls-3.4.0/libmbedtls.a
• basename subprojects/mbedtls-mbedtls-3.4.0/libmbedx509.a
• cp -a subprojects/mbedtls-mbedtls-3.4.0/libmbedcrypto.a.p/library/libmbedx509.a subprojects/mbedtls-mbedtls-3.4.0/libmbedx509.a
  ninja: build stopped: subcommand failed.
  Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`

[aneessahib]

What you are now seeing is another new issue.

Please change this line to the below, and retry.

&& /usr/bin/python3 -B -m pip install 'tomli>=1.1.0' 'tomli-w>=0.4.0' 'meson==1.1.1'

[NandiniKJ]

I tried the above fix and the ninja issue is resolved, but got into another error.

`Step 12/29 : RUN apt-get update && env DEBIAN_FRONTEND=noninteractive apt-get install -y binutils expect libprotobuf-c-dev locales openssl python3 python3-cryptography python3-protobuf python3-pyelftools \python3-pip && /usr/bin/python3 -B -m pip install click jinja2 protobuf 'tomli>=1.1.0' 'tomli-w>=0.4.0' && apt-get remove -y python3-pip && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*

---> Running in 030aed51b7a5
/bin/sh: apt-get: command not found

Failed to build unsigned graminized Docker image gsc-icr.io/data-security-broker/dsb-shield-postgresql:v1-unsigned.`