Invalid mount config for type "bind": destination can't be `/`

[claymcleod]

Describe the bug

I'm trying to run funnel locally (being driven by Sprocket), and I'm getting the following error from the Funnel engine.

docker: Error response from daemon: invalid volume specification: '/host_mnt/<path>/funnel-work-dir/<random-string>:/:rw': invalid mount config for type "bind": invalid specification: destination can't be '/'

To Reproduce

1. Download funnel and run with funnel server run (no arguments/options).
2. Download Sprocket (link).
3. Write a configuration for Sprocket to dispatch jobs to the Funnel server.

[backend]
type = "tes"
url = "http://localhost:8000/"
inputs = "file:///<path>/<to>/<your>/<folder>/inputs/"
outputs = "file:///<path>/<to>/<your>/<folder>/outputs/"
interval = 10
insecure = true

4. Write the example WDL file on the OpenWDL quickstart guide to example.wdl.
5. Try to run the WDL workflow with sprocket run example.wdl main.name="World" --overwrite --config config.toml

Expected behavior

The workflow to complete successfully.

Screenshots

Not applicable.

Environment (please complete the following information):

• Environment: Mac OSX 15.5
• Version: Funnel v0.11.0

Additional context

None.

[kellrott]

Do you have a copy of the TES packet that was sent?

[claymcleod]

Sure looks like this was POST-ed to http://localhost:8000/tasks.

{
  "name": "say_hello-0-89MQ7jos2wQm",
  "inputs": [
    {
      "path": "/mnt/command",
      "type": "FILE",
      "content": "echo \"Hello, Clay!\""
    }
  ],
  "outputs": [
    {
      "url": "file:///<path>/<to>/<your>/<folder>/outputs/say_hello-0-89MQ7jos2wQm-20250529-223055/work",
      "path": "/mnt/work",
      "type": "DIRECTORY"
    },
    {
      "url": "file:///<path>/<to>/<your>/<folder>/outputs/say_hello-0-89MQ7jos2wQm-20250529-223055/stdout",
      "path": "/stdout",
      "type": "FILE"
    },
    {
      "url": "file:///<path>/<to>/<your>/<folder>/outputs/say_hello-0-89MQ7jos2wQm-20250529-223055/stderr",
      "path": "/stderr",
      "type": "FILE"
    }
  ],
  "resources": {
    "cpu_cores": 1,
    "preemptible": false,
    "ram_gb": 2,
    "disk_gb": 1
  },
  "executors": [
    {
      "image": "ubuntu:latest",
      "command": [
        "bash",
        "-C",
        "/mnt/command"
      ],
      "workdir": "/mnt/work",
      "stdin": null,
      "stdout": "/stdout",
      "stderr": "/stderr",
      "env": null,
      "ignore_error": true
    }
  ]
}

[lbeckman314]

Overview

Was able to replicate this error with the following steps and will add additional logging to the file mapper used to set the inputs and outputs for the docker command in worker.go to see if we can find what may be the cause of this root mount error!

Testing Steps

Note

Using /tmp/testing as the working directory in the steps below...

example-task.json

{
  "name": "say_hello-example",
  "inputs": [
    {
      "path": "/mnt/command",
      "type": "FILE",
      "content": "echo 'Hello, World!'"
    }
  ],
  "outputs": [
    {
      "url": "file:///tmp/testing/work",
      "path": "/mnt/work",
      "type": "DIRECTORY"
    },
    {
      "url": "file:///tmp/testing/stdout",
      "path": "/stdout",
      "type": "FILE"
    },
    {
      "url": "file:///tmp/testing/stderr",
      "path": "/stderr",
      "type": "FILE"
    }
  ],
  "resources": {
    "cpu_cores": 1,
    "preemptible": false,
    "ram_gb": 2,
    "disk_gb": 1
  },
  "executors": [
    {
      "image": "ubuntu:latest",
      "command": [
        "bash",
        "-C",
        "/mnt/command"
      ],
      "workdir": "/mnt/work",
      "stdin": null,
      "stdout": "/stdout",
      "stderr": "/stderr",
      "env": null,
      "ignore_error": true
    }
  ]
}

1. Checking Versions

➜ which funnel
$HOME/.local/bin/funnel

➜ funnel version
version: 0.11.2

2. Starting Server

➜ funnel server run --LocalStorage.AllowedDirs /tmp/testing

3. Submitting Task

➜ funnel task create example-task.json
<TASK ID>

4. Root Mount Error ❌

➜ funnel task get <TASK ID> | jq -r '.logs[].logs[].stderr'
docker: Error response from daemon: invalid volume specification: '/host_mnt/$HOME/code/funnel/funnel-work-dir/<TASK ID>:/:rw': invalid mount config for type "bind": invalid specification: destination can't be '/'.
See 'docker run --help'.

Full Task (funnel task get <TASK ID>)

{
  "creationTime":  "2025-06-02T16:49:48.737726-07:00",
  "executors":  [
    {
      "command":  [
        "bash",
        "-C",
        "/mnt/command"
      ],
      "ignoreError":  true,
      "image":  "ubuntu:latest",
      "stderr":  "/stderr",
      "stdout":  "/stdout",
      "workdir":  "/mnt/work"
    }
  ],
  "id":  "d0v3h71urbu85gjo1nug",
  "inputs":  [
    {
      "content":  "echo 'Hello, World!'",
      "path":  "/mnt/command"
    }
  ],
  "logs":  [
    {
      "endTime":  "2025-06-02T16:49:51.177438-07:00",
      "logs":  [
        {
          "endTime":  "2025-06-02T16:49:51.141717-07:00",
          "exitCode":  125,
          "startTime":  "2025-06-02T16:49:48.856217-07:00",
          "stderr":  "docker: Error response from daemon: invalid volume specification: '/host_mnt/Users/beckmanl/code/funnel/funnel-work-dir/d0v3h71urbu85gjo1nug:/:rw': invalid mount config for type \"bind\": invalid specification: destination can't be '/'.\nSee 'docker run --help'.\n"
        }
      ],
      "metadata":  {
        "hostname":  "RNB11239"
      },
      "startTime":  "2025-06-02T16:49:48.791818-07:00",
      "systemLogs":  [
        "level='info' msg='Version' timestamp='2025-06-02T16:49:48.772216-07:00' task_attempt='0' executor_index='0' GitBranch='' GitUpstream='' BuildDate='2025-05-15T20:28:14Z' Version='0.11.2' GitCommit='75e41493'",
        "level='info' msg='Running command' timestamp='2025-06-02T16:49:50.959518-07:00' task_attempt='0' executor_index='0' cmd='/Users/beckmanl/.rd/bin/docker run -i --read-only --rm --name d0v3h71urbu85gjo1nug-0 --workdir /mnt/work --volume /Users/beckmanl/code/funnel/funnel-work-dir/d0v3h71urbu85gjo1nug:/:rw --volume /Users/beckmanl/code/funnel/funnel-work-dir/d0v3h71urbu85gjo1nug/mnt/command:/mnt/command:ro --volume /Users/beckmanl/code/funnel/funnel-work-dir/d0v3h71urbu85gjo1nug/mnt/work:/mnt/work:rw ubuntu:latest bash -C /mnt/command'",
        "level='error' msg='failed to run docker container' timestamp='2025-06-02T16:49:51.084975-07:00' task_attempt='0' executor_index='0' unknown='exit status 125'",
        "level='error' msg='Exec error' timestamp='2025-06-02T16:49:51.193015-07:00' task_attempt='0' executor_index='0' error='exit status 125'"
      ]
    }
  ],
  "name":  "say_hello-example",
  "outputs":  [
    {
      "path":  "/mnt/work",
      "type":  "DIRECTORY",
      "url":  "file:///tmp/testing/work"
    },
    {
      "path":  "/stdout",
      "url":  "file:///tmp/testing/stdout"
    },
    {
      "path":  "/stderr",
      "url":  "file:///tmp/testing/stderr"
    }
  ],
  "resources":  {
    "cpuCores":  1,
    "diskGb":  1,
    "ramGb":  2
  },
  "state":  "EXECUTOR_ERROR"
}

Questions + Next Steps

• Is Docker trying to mount the working directory at root (/) in the container? If so, why?
• Add debug logging to worker.go and file_mapper.go to see where root mount is being added...
• Try to replicate error with Sprocket and example.wdl using Clay's steps above, update thread with results...

[lbeckman314]

Steps to Reproduce 🐛

1. Install Sprocket

➜ git clone https://github.com/stjude-rust-labs/sprocket

➜ cd sprocket

➜ cargo build --release
   Compiling sprocket v0.13.0 (/Users/beckmanl/code/sprocket)
    Finished `release` profile [optimized] target(s) in 1m 45s

2. Run Funnel

➜ funnel server run --LocalStorage.AllowedDirs /tmp --Logger.Level debug

3. Run Sprocket

config.toml

[backend]
type = "tes"
url = "http://localhost:8000/"
inputs = "file:///tmp/inputs/"
outputs = "file:///tmp/outputs/"
interval = 10
insecure = true

example.wdl

version 1.2

task say_hello {
    input {
        String greeting
        String name
    }

    command <<<
        echo "~{greeting}, ~{name}!"
    >>>

    output {
        String message = read_string(stdout())
    }

    requirements {
        container: "ubuntu:latest"
    }
}

workflow main {
    input {
        String name
        Boolean is_pirate = false
    }

    Array[String] greetings = select_all([
        "Hello",
        "Hallo",
        "Hej",
        (
            if is_pirate
            then "Ahoy"
            else None
        ),
    ])

    scatter (greeting in greetings) {
        call say_hello {
            greeting,
            name,
        }
    }

    output {
        Array[String] messages = say_hello.message
    }
}

➜ ./target/release/sprocket run example.wdl main.name="World" --overwrite --config config.toml

error: task execution failed for task `say_hello` (id `say_hello-1`): process terminated with exit code 125: see `file:///tmp/outputs/say_hello-1-.../stdout` and `file:///tmp/outputs/say_hello-1-.../stderr` for task output and the related files in `$HOME/sprocket/output/calls/say_hello-1/attempts/0`
   ┌─ example.wdl:3:6
   │
 3 │ task say_hello {
   │      ^^^^^^^^^ this task failed to execute
   │
   ┌─ example.wdl:40:9
   │
40 │         call say_hello {
   │         ---- called from this location

error: aborting due to evaluation error

Notes ✍

/stderr and /stdout are likely being set here in the wdl-engine crate:

https://github.com/stjude-rust-labs/wdl/blob/wdl-v0.14.0/wdl-engine/src/backend/tes.rs#L83-L87

Funnel currently mounts host outputs/inputs using the parent directory of the host machine:

• Host Path: .../funnel-work-dir/<TASK ID>/stderr
• Host Directory (on host): .../funnel-work-dir/<TASK ID>
• Mount Path: /stderr
• Mount Directory (in container): /

Funnel then creates the following Executor (Docker) command:

docker run -i --read-only --rm --name <TASK ID>-0 \
    --workdir /mnt/work \
    --volume .../funnel-work-dir/<TASK ID>:/:rw \   <---- Root mount that's causing the error...
    --volume .../funnel-work-dir/<TASK ID>/mnt/command:/mnt/command:ro \
    --volume .../funnel-work-dir/<TASK ID>/mnt/work:/mnt/work:rw \
    ubuntu:latest bash -C /mnt/command

Docker sees that we're trying to mount the Host Directory to / and throws the destination can't be '/' error.

Alternatives 🤔

• Can Funnel simply mount the outputs as single files instead of trying to mount the Host Directory?
• If so, would we need to create "placeholder" files on the Host in order for Docker to mount them?

[kellrott]

Funnel can't mount placeholder files, because it is not required the user declare if a path is a file or a directory. So if the user has an output directory /output but doesn't say it's a directory, Funnel could read that as they was to create a file, think it's in the / directory and try to mount a stub file, rather than overwrite all of /. This is actually a larger issue, because if the user puts /var/stdout if we don't know it will be a file, we have be mount the entire /var directory and override any files that were there.

It's an effect of how the specification is written. I think any other TES implementation would come to the same conclusions. So it's not a Funnel thing. We may have to clarify this behavior in the specification.