WIP: Add binary shims #553

HarryMichal · 2020-09-07T13:47:00Z

This PR adds support for creating binary shims in toolbox containers for binaries available on the host. This was heavily inspired by code snippets in #145. There's also very basic support for running the shim binaries with sudo.

Closes #145

softwarefactory-project-zuul · 2020-09-07T13:51:45Z

Build failed.

shellcheck : FAILURE in 2m 08s
system-test-fedora-31 : SUCCESS in 3m 41s
system-test-fedora-32 : SUCCESS in 3m 13s
system-test-fedora-rawhide : FAILURE in 3m 49s

cjao · 2020-10-14T08:46:41Z

Using the technique of your sudo wrapper, I tried flatpak-spawn --host --watch-bus --forward-fd=1 --forward-fd=2 --env=TERM=xterm-256color sudo ls / but got

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper

Loosely based off containers/toolbox#145 Doesn't work with sudo and hard coded shims in the docker container. Pending better UIX with containers/toolbox#553 Signed-off-by: anthr76 <[email protected]>

1player · 2022-07-01T09:30:08Z

Is there any update to this PR?

softwarefactory-project-zuul · 2023-03-30T20:09:51Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,64a43547160cb0ea119cec2b62b4d9d820fafd5c

debarshiray · 2023-03-30T20:09:59Z

Let's see if I can rebase this pull request.

debarshiray

I wonder if we should turn host-wrapper and sudo-wrapper into private toolbox(1) commands? The flatpak-spawn --host forwarding can benefit from shared code.

debarshiray · 2023-03-31T12:14:54Z

data/libexec/sudo-wrapper

+
+# Shims are located under /usr/libexec/toolbox; the list should not include
+# "host-runner" and "sudo"
+shims=$(ls /usr/libexec/toolbox | grep -v "host-runner" | grep -v "sudo")


How about using a separate sub-directory for the shims? It would remove the need for this filtering. Moreover, if we do want users to place their own shims, then we can't trust them not to overwrite things like host-runner and sudo.

debarshiray · 2023-03-31T13:10:53Z

data/libexec/host-runner

+	executable="sudo $executable"
+fi
+
+exec flatpak-spawn --host --watch-bus --forward-fd=1 --forward-fd=2 --directory=$(pwd) --env=TERM=xterm-256color $executable "$@"


Do we really need to explicitly forward file descriptors 1 and 2? What about 0?

flatpak-spawn(1) has always forwarded file descriptors 0, 1 and 2. We don't do it explicitly elsewhere either.

debarshiray · 2023-03-31T13:11:57Z

data/libexec/host-runner

+	executable="sudo $executable"
+fi
+
+exec flatpak-spawn --host --watch-bus --forward-fd=1 --forward-fd=2 --directory=$(pwd) --env=TERM=xterm-256color $executable "$@"


We shouldn't hard code TERM. Instead: --env=TERM=$TERM :)

Also, shouldn't we forward all the other environment variables that we usually do?

softwarefactory-project-zuul · 2023-03-31T13:19:58Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,f534ed4737b8957f029d7df8466d45b5f465bcd8

softwarefactory-project-zuul · 2023-03-31T13:20:46Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,e640e56f85711ec3133d2bd9709b68aaf4bba691

softwarefactory-project-zuul · 2023-03-31T15:12:12Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,39d47748bd887ef1ace03c8f323cf81d7c1b0c85

softwarefactory-project-zuul · 2023-03-31T15:12:46Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,546c23b99d09d4846b1f9fbf2888110ede7414b5

softwarefactory-project-zuul · 2023-03-31T15:14:28Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,ed7f13751340bd7a91c88015fbe24c960a6cbcf9

softwarefactory-project-zuul · 2023-03-31T15:15:33Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,267184ddac2feead15ff7b043618959e55483037

softwarefactory-project-zuul · 2023-03-31T15:17:38Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,8df091444c505acad958cc52252f2b9548c39b75

softwarefactory-project-zuul · 2023-03-31T15:21:20Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,c531e0ec5810b6dc294b6c94748071eaf5db9946

debarshiray · 2023-03-31T15:24:19Z

profile.d/toolbox.sh

@@ -1,5 +1,9 @@
 # shellcheck shell=sh

+# Toolbox provides shim binaries for executing commands on the host from inside
+# of a toolbox container
+export PATH="/usr/libexec/toolbox:$PATH"


Should we really do this unconditionally for both hosts and containers? I think this should ideally happen only for Toolbx containers, not even generic Podman containers. Or did I misunderstand something?

softwarefactory-project-zuul · 2023-03-31T15:27:56Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/containers/toolbox for 553,49ac68e31d3ab07ed68c33e23daa9932b7275501

'host-runner' is a shell script meant to be symlinked against to execute commands on the host from inside of an OCI container. It makes use of 'flatpak-spawn' with it's option --host that takes care of the plumbing from a container to the host. The command itself can not be run as root because it relies on session bus, so instead the command, that is to be executed on the host, needs to be prepended with "sudo <sudo-options>" manually. 'sudo' is a shell script that is meant to be used before /bin/sudo to make it possible to run commands on the host from inside of a container with "sudo" prefixed. It checks first if the command, that is to be run with sudo, is to be run on the host by checking if it is a symlink to /usr/libexec/toolbox/host-runner. If it is not, it is executed normally with /bin/sudo. Both of these files should be put in /usr/libexec/toolbox which will make them available in /run/host/usr/libexec/toolbox in toolbox containers. Thanks to that there's no need to recreate already existing containers because those should have the host /usr already available.

…zation In previous commit the two files, 'host-runner' and 'sudo', were added. This commit adds all the necessary plumbing to make them available in already existing containers and in the new ones. Preparation is done by the 'toolbox init-container' command (the containers' entry-point) by creating symlinks in /usr/libexec/toolbox pointing to the two files under /run/host/usr/libexec/toolbox. If the /usr/libexec/toolbox directory does not exist, it gets created. The final touch is done by /etc/profile.d/toolbox.sh that was previously only used to printing welcome messages. It puts /usr/libexec/toolbox at the beginning of the PATH variable and exports it.

This command is meant to be used in toolbox containers and will be disabled on the host. The created binaries are put in /usr/libexec/toolbox/. The command utilizes the redirectPath function in cmd/initContainer.go.

softwarefactory-project-zuul · 2023-03-31T16:47:24Z

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/2f72edcc35f34582b1d4d85b5b9033b8

❌ unit-test FAILURE in 9m 42s
❌ unit-test-migration-path-for-coreos-toolbox FAILURE in 3m 28s
❌ unit-test-restricted FAILURE in 8m 51s
✔️ system-test-fedora-rawhide SUCCESS in 21m 39s
✔️ system-test-fedora-38 SUCCESS in 21m 09s
✔️ system-test-fedora-37 SUCCESS in 20m 59s
✔️ system-test-fedora-36 SUCCESS in 19m 39s

debarshiray · 2023-03-31T16:57:05Z

data/libexec/host-runner

+	executable="sudo $executable"
+fi
+
+exec flatpak-spawn --host --watch-bus --forward-fd=1 --forward-fd=2 --directory=$(pwd) --env=TERM=xterm-256color $executable "$@"


I don't think we need the --directory=$(pwd) because flatpak-spawn(1) has been retaining the current working directory since before the --directory option became a thing.

debarshiray · 2023-03-31T16:59:25Z

data/libexec/host-runner

+	executable="sudo $executable"
+fi
+
+exec flatpak-spawn --host --watch-bus --forward-fd=1 --forward-fd=2 --directory=$(pwd) --env=TERM=xterm-256color $executable "$@"


I am curious about the use of --watch-bus. It's not wrong, but given that flatpak-spawn(1) always forwards signals, I am curious if you came across a scenario where the container-side shim would cleanly exit without signals leaving the host-side running. I am trying to understand if we should use --watch-bus more widely.

HarryMichal added 3. Enhancement Improvement to an existing feature 5. Help Wanted Extra attention is needed labels Sep 7, 2020

HarryMichal added this to the Release 0.2.0 milestone Sep 7, 2020

HarryMichal mentioned this pull request Jan 8, 2021

Starting Flatpak applications fails from Toolbox #659

Closed

Base automatically changed from master to main March 25, 2021 22:25

debarshiray force-pushed the add-binary-shims branch from 7ca9a72 to 64a4354 Compare March 30, 2023 20:09

debarshiray reviewed Mar 31, 2023

View reviewed changes

debarshiray force-pushed the add-binary-shims branch from 64a4354 to f534ed4 Compare March 31, 2023 13:19

debarshiray force-pushed the add-binary-shims branch from f534ed4 to e640e56 Compare March 31, 2023 13:20

debarshiray force-pushed the add-binary-shims branch from e640e56 to 39d4774 Compare March 31, 2023 15:12

debarshiray force-pushed the add-binary-shims branch from 39d4774 to 546c23b Compare March 31, 2023 15:12

debarshiray force-pushed the add-binary-shims branch from 546c23b to ed7f137 Compare March 31, 2023 15:14

debarshiray force-pushed the add-binary-shims branch from ed7f137 to 267184d Compare March 31, 2023 15:15

debarshiray force-pushed the add-binary-shims branch from 267184d to 8df0914 Compare March 31, 2023 15:17

debarshiray force-pushed the add-binary-shims branch from 8df0914 to c531e0e Compare March 31, 2023 15:21

debarshiray reviewed Mar 31, 2023

View reviewed changes

debarshiray force-pushed the add-binary-shims branch from c531e0e to 49ac68e Compare March 31, 2023 15:27

debarshiray force-pushed the add-binary-shims branch 11 times, most recently from e9be361 to 86f76e2 Compare March 31, 2023 16:21

HarryMichal added 3 commits March 31, 2023 18:21

[WIP] cmd/createShim: Add command for creating shim binaries

9efa954

This command is meant to be used in toolbox containers and will be disabled on the host. The created binaries are put in /usr/libexec/toolbox/. The command utilizes the redirectPath function in cmd/initContainer.go.

debarshiray force-pushed the add-binary-shims branch from 86f76e2 to 9efa954 Compare March 31, 2023 16:21

debarshiray reviewed Mar 31, 2023

View reviewed changes

notfirefox mentioned this pull request Mar 23, 2024

Update F39 image so that xdg-open links to flatpak-xdg-open #1450

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add binary shims #553

WIP: Add binary shims #553

HarryMichal commented Sep 7, 2020

softwarefactory-project-zuul bot commented Sep 7, 2020

cjao commented Oct 14, 2020

1player commented Jul 1, 2022

softwarefactory-project-zuul bot commented Mar 30, 2023

debarshiray commented Mar 30, 2023

debarshiray left a comment

debarshiray Mar 31, 2023

debarshiray Mar 31, 2023

debarshiray Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

debarshiray Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

debarshiray Mar 31, 2023

debarshiray Mar 31, 2023

WIP: Add binary shims #553

Are you sure you want to change the base?

WIP: Add binary shims #553

Conversation

HarryMichal commented Sep 7, 2020

softwarefactory-project-zuul bot commented Sep 7, 2020

cjao commented Oct 14, 2020

1player commented Jul 1, 2022

softwarefactory-project-zuul bot commented Mar 30, 2023

debarshiray commented Mar 30, 2023

debarshiray left a comment

Choose a reason for hiding this comment

debarshiray Mar 31, 2023

Choose a reason for hiding this comment

debarshiray Mar 31, 2023

Choose a reason for hiding this comment

debarshiray Mar 31, 2023

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

debarshiray Mar 31, 2023

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Mar 31, 2023

softwarefactory-project-zuul bot commented Mar 31, 2023

debarshiray Mar 31, 2023

Choose a reason for hiding this comment

debarshiray Mar 31, 2023

Choose a reason for hiding this comment