Cannot use shell form for command specified in Compose file

[ijessen-mitll]

Description

A command override in a Compose service definition will never execute in shell form, only ever exec form. This is contrary to the documentation statement that "the value can also be a list, in a manner similar to Dockerfile". A Dockerfile CMD specified as a list triggers exec form, while a Dockerfile CMD specified as a plain string triggers shell form (see here).

Steps To Reproduce

Example compose.yaml:

services:

  exec-form:
    image: alpine:latest
    command: ["/bin/sh", "-c", "echo $$HOSTNAME"]

  shell-form-compose:
    image: alpine:latest
    command: echo $$HOSTNAME

  shell-form-dockerfile:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        CMD echo $$HOSTNAME
    image: alpine:latest

Results (notice the difference between the compose shell form output from the other two):

$ docker compose -f compose.yaml up
[+] Running 3/0
 ✔ Container config-exec-form-1              Created         0.0s 
 ✔ Container config-shell-form-compose-1     Created         0.0s 
 ✔ Container config-shell-form-dockerfile-1  Created         0.0s 


exec-form-1               | 03eae6e24cbf
shell-form-compose-1      | $HOSTNAME
shell-form-dockerfile-1   | 2d7438fcb6a1

Compose Version

Docker Compose version v2.31.0

Docker Environment

Client: Docker Engine - Community
 Version:    27.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Anything else?

If the behavior is as-intended, the Compose reference documentation should be updated to clarify that a string command will be executed in exec mode (contrary to the Dockerfile behavior).

[ndeloof]

This is the expected behavior
"in a manner similar to Dockerfile" indeed is weaving hands without a clear definition in docs, need to be clarified.

[ijessen-mitll]

For completeness - why is this intended behavior?

[ndeloof]

Basically: preserve backward compatibility

[polarathene]

@ijessen-mitll your reproduction has the Dockerfile service build an image with alpine:latest stored locally, that'd be used instead of the actual image when the other services run again 😅 image isn't needed for that service. Alternatively by default it would also pull the remote image or use the locally stored image from earlier pull, rather than run the intended build.

I don't think your comparison was correct either, so I've provided an extended version with heavy commentary for context (in case anyone else lands here).

Main issue with your comparison was the Dockerfile instructions CMD + ENTRYPOINT have some added complexity in behaviour, notably how the SHELL default works with the shell-form (it has no relevance when you override the instruction at runtime via compose or cli, thus you'd need an explicit shell unlike the Dockerfile).

---

Observations

shell-form-c and dockerfile-shell-form-b seem a bit off, other than that everything seems fine?

Here's a table of the different variants from the revised compose.yaml below:

service	expected output	failure
shell-form-a	❌	no interpolation
shell-form-b	✔️
shell-form-c	❌	blank line
exec-form-a	❌	no interpolation
exec-form-b	✔️
exec-form-c	✔️
exec-form-d	❌	no interpolation
dockerfile-shell-form-a	✔️
dockerfile-shell-form-b	❌	exits without any output
dockerfile-exec-form-a	❌	no interpolation
dockerfile-exec-form-b	❌	no interpolation
dockerfile-exec-form-c	❌	blank line

The blank line examples seem to be something like this:

docker run --rm alpine /bin/sh -c '/bin/sh -c echo "host is $HOSTNAME"'

However I kind of expected dockerfile-shell-form-b (exits without output) to behave like that (due to these docs), and dockerfile-exec-form-c should not have the implicit SHELL so no clue why that produces a blank line 🤔 (EDIT: Oh it's probably because it's not a single string input for sh -c, ha!)

shell-form-c must also only be passing echo? Even when quote wrapping the value it'll still behave that way unless using exec-form. But as the commented out example directly beneath it shows, if moving -c from entrypoint to command this works as expected. The docs do touch on this a little bit about command appearing as an executable + args or just args 🤷‍♂ (not Compose specific)

Reproduction

docker compose up --force-recreate --build

services:

  # NOTE: The shell-form variants below would fail YAML parsing with:
  # "mapping values are not allowed in this context"
  # - Workaround Variant A => single quote wrapping the full value
  # - Workaround Variants B + C => Replace `host:` with `host is`

  # No shell is used, so no interpolation of the variable HOSTNAME.
  # https://docs.docker.com/reference/dockerfile/#variable-substitution
  # NOTE: The Alpine image has no default entrypoint set, so the command runs without a shell.
  #   This differs from the Dockerfile CMD example below, which with shell-form uses SHELL:
  #   https://docs.docker.com/reference/dockerfile/#shell
  # NOTE: `echo` is a built-in command in the shell,
  #   /bin/echo is also available as a separate command which is called here,
  #   Docker locates the binary to execute via the PATH environment:
  # NOTE: The equivalent via docker CLI would instead output the Docker host hostname:
  #   `docker run --rm alpine echo "host: $HOSTNAME"`
  shell-form-a:
    image: alpine:latest
    command: 'echo "host: $$HOSTNAME"'

  # Explicit use of shell, interpolation is valid:
  shell-form-b:
    image: alpine:latest
    command: /bin/sh -c 'echo "host is $$HOSTNAME"'

  # Same as shell-form-b, except an entrypoint configures a shell explicitly.
  # NOTE: This actually fails and outputs a blank output for some reason,
  # expectation was for the command to be appended to the entrypoint as a string arg for `-c`.
  shell-form-c:
    image: alpine:latest
    # Blank output (exec + shell maybe incompatible?):
    # NOTE: Potentially because there is no equivalent CLI command for exec-form with `--entrypoint`?
    entrypoint: ["/bin/sh", "-c"]
    command: echo "host is $$HOSTNAME"
    # Correct output:
    # NOTE: Equivalent via CLI: `docker run --rm --entrypoint /bin/sh alpine -c 'echo "host is $HOSTNAME"'`
    #entrypoint: ["/bin/sh"]
    #command: -c 'echo "host is $$HOSTNAME"'


  # Equivalent of shell-form-a output (no shell environment, no interpolation):
  exec-form-a:
    image: alpine:latest
    command: ["echo", "host: $$HOSTNAME"]

  # Equivalent of shell-form-b or shell-form-c, uses a shell thus interpolation is valid:
  exec-form-b:
    image: alpine:latest
    command: ["/bin/sh", "-c", "echo host: $$HOSTNAME"]

  # Interoplation by shell works as intended with double quotes instead,
  # exec-form can also use either single or double quotes for it's YAML string values:
  # NOTE: Unlike shell-form-c, this is valid due to entrypoint + command using exec form.
  exec-form-c:
    image: alpine:latest
    entrypoint: ["/bin/sh", "-c"]
    command: ['echo "host: $$HOSTNAME"']

  # Despite using a shell, single quote wrapping around the variable has
  # the shell parse it as a string literal (skips interpolation):
  exec-form-d:
    image: alpine:latest
    entrypoint: ["/bin/sh", "-c"]
    command: ["echo 'host: $$HOSTNAME'"]


  # This example highlights the implicit `SHELL ["/bin/sh", "-c"]` for shell-form with CMD,
  # Docker CLI or Compose can both replace CMD at runtime which ignores SHELL:
  # NOTE: Uncomment the SHELL instruction to produce a failure at runtime.
  dockerfile-shell-form-a:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        #SHELL ["/bin/invalid"]
        CMD echo "host: $$HOSTNAME"

  # Exits, no output
  # NOTE: If ENTRYPOINT used shell-form, then CMD would be ignored (as would a runtime command override):
  # https://docs.docker.com/reference/dockerfile/#entrypoint
  dockerfile-shell-form-b:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        ENTRYPOINT ["/bin/sh", "-c"]
        CMD echo "host: $$HOSTNAME"


  # Same output as shell-form-a and exec-form-a: needs a shell environment to work.
  dockerfile-exec-form-a:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        CMD ["echo", "host: $$HOSTNAME"]

  # REFERENCE: The exec-form with CMD is not affected by SHELL:
  # NOTE: The same exemption applies shell vs exec forms for RUN and ENTRYPOINT too.
  # If either ENTRYPOINT or CMD use shell-form, those will still default to SHELL:
  # https://docs.docker.com/reference/dockerfile/#understand-how-cmd-and-entrypoint-interact
  dockerfile-exec-form-b:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        SHELL ["/bin/invalid"]
        CMD ["echo", "host: $$HOSTNAME"]

  # Same output as shell-form-c, blank.
  # EDIT: This has an error, it should have had a CMD like exec-form-c,
  # providing a single value for `-c`.
  dockerfile-exec-form-c:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        ENTRYPOINT ["/bin/sh", "-c"]
        CMD ["echo", "host: $$HOSTNAME"]

[polarathene]

Follow-up to previous comment addressing the blank / exit variants.

Expected output should be like x86_64 Linux for most of the containers configured below. Some variations are demonstrated with extra commentary, but the only failures covered are shell-form-d, dockerfile-shell-form-b, and dockerfile-shell-form-c; all which are explained beneath their "Bad" comments in the config below.

For Compose the mistake with shell-form syntax is that Compose always deconstructs the input into exec-form style. For shell strings that makes it important when the image or compose config has an entrypoint configured for a shell (or similar) to wrap the command in quotes so it's used as a single string arg. When no shell environment is relied upon, the direct command without an entrypoint can work well with the command split/converted into exec-form.

services:

  # Good
  shell-form-a:
    image: alpine:latest
    command: uname -m -o

  # Good
  # Unlike ENTRYPOINT, the shell-form is technically treated as the exec-form here,
  # there is no implicit use of SHELL instruction and the command is not ignored.
  shell-form-b:
    image: alpine:latest
    entrypoint: uname
    command: -m -o

  # Good
  # shell-form vs exec-form doesn't really exist in compose equivalents:
  shell-form-c:
    image: alpine:latest
    entrypoint: ["uname"]
    command: -m -o

  # Bad
  # Outputs only 'Linux', mimics `docker run --rm alpine /bin/sh -c uname`
  # Cause: The command is converted to exec-form, which is fine for CLI args
  # but in this case command is intended as a single value to `/bin/sh -c` arg,
  # where it only receives `uname` due to splitting the command string into separate components.
  shell-form-d:
    image: alpine:latest
    entrypoint: ["/bin/sh", "-c"]
    command: uname -m -o

  # Good
  # When the entrypoint sets a shell, shell-form syntax needs to ensure the command is provided to `-c` as a single string,
  # In compose and docker CLI, you can ensure this with quote wrapping, this varies for compose due to YAML inference of strings.
  shell-form-e:
    image: alpine:latest
    entrypoint: ["/bin/sh", "-c"]
    # Preserve the single quote:
    #command: "'uname -m -o'"
    # Multi-line string does not require nested quotes
    command: |
      'uname -m -o'


  # Good
  # No entrypoint/shell, so the exec-form should have the command properly deconstructed:
  exec-form-a:
    image: alpine:latest
    command: ["uname", "-m", "-o"]

  # Good
  # Since the entrypoint ends with the option `-c`, a single string value is expected,
  # Thus exec-form clearly treats it as such:
  exec-form-b:
    image: alpine:latest
    entrypoint: ["/bin/sh", "-c"]
    command: ["uname -m -o"]

  # Good
  # Alternative exec-form syntax (YAML list), with chaining of other commands in the shell:
  exec-form-c:
    image: alpine:latest
    #command: ["/bin/sh", "-c", "uname -m -o && echo hello && echo world"]
    command:
      - /bin/sh
      - -c
      - uname -m -o && echo hello && echo world

  # Good
  # This style of exec-form with YAML lists can use multi-line strings,
  # which for `sh -c` is a nicer way to chain multiple commands vs `&&`
  exec-form-d:
    image: alpine:latest
    command:
      - /bin/sh
      - -c
      - |
        uname -m -o
        echo hello
        echo world

  # Good
  # Unlike Compose or the Docker CLI, shell-form in Dockerfile provides a string to implicit SHELL:
  dockerfile-shell-form-a:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        CMD uname -m -o

  # Bad
  # Fails to run command, mimics: `docker run --rm alpine /bin/sh -c /bin/sh -c uname -m`
  # Cause: Effectively runs `/bin/sh -c /bin/sh`, ignoring the rest.
  # If using `tty: true` container doesn't immediately exit, the tty for the shell keeps it open instead.
  dockerfile-shell-form-b:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        ENTRYPOINT ["/bin/sh", "-c"]
        CMD uname -m -o

  # Bad
  # Outputs an arg parsing error from uname, mimics: `docker run --rm alpine uname -m /bin/sh -c -o -p`
  # Cause: Implicit SHELL prepended due to using CMD shell-form syntax
  dockerfile-shell-form-c:
    build:
      dockerfile_inline: |
        FROM alpine:latest
        ENTRYPOINT ["uname", "-m"]
        CMD -o -p

[ndeloof]

@polarathene your analysis is 100% correct, still this is a corner case and nobody complained about this until this issue has been opened, while compose is 10 years old :)

[polarathene]

Yeah no worries. I don't think it's a compose specific issue (other than being able to use entrypoint: ["/bin/sh", "-c"], which docker CLI cannot do with --entrypoint).

Both seem to process the value of command the same way (environment interpolation aside), it's just less obvious in compose.yaml when command is intended as a entire string since it needs extra guidance 👍