GCP Batch: Possible to remove Bucket Prefix in Inputs + Outputs?

[lbeckman314]

Overview 🌀

Currently submitting GCP Storage job via Funnel requires that users add a "bucket prefix" to all inputs and outputs.

Expected Behavior ✔️

{
  "name": "Input/Output Test",
  "inputs": [{
    "url": "gs://tes-batch-integration/README.md.sha256",
    "path": "/README.md"          // <---- Arbitrary path (e.g. root) with no "bucket prefix" ✔️
  }],
  "outputs": [{
    "url": "gs://tes-batch-integration/README.md.sha256",
    "path": "/README.md.sha256"   // <---- Arbitrary path (e.g. root) with no "bucket prefix" ✔️
  }],
  "executors": [{
    "image": "alpine",
    "command": ["sha256sum", "/README.md | tee /README.md.sha256"]
  }]
}

Actual Behavior ❌

{
  "name": "Input/Output Test",
  "inputs": [{
    "url": "gs://tes-batch-integration/README.md.sha256",
    "path": "/mnt/disks/tes-batch-integration/README.md"          // <---- Path includes bucket prefix ❌
  }],
  "outputs": [{
    "url": "gs://tes-batch-integration/README.md.sha256",
    "path": "/mnt/disks/tes-batch-integration/README.md.sha256"   // <---- Path includes bucket prefix ❌
  }],
  "executors": [{
    "image": "alpine",
    "command": ["sha256sum", "/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"]
  }]
}

Next Steps ⚙️

• Check if it's possible for Google Storage inputs/outputs to be mounted at arbitrary (i.e. non-prefixed) paths in their volumes.
• Update gcp_batch/backend to be able to support converting user provided paths (e.g. /) to Google Storage paths (e.g. /mnt/disks/<BUCKET>/).

Additional Resources 📚

• GCP's API Sample demonstrating the expected "Mount Path" → MOUNT_PATH=/mnt/disks/gcs

Tip

Nextflow does have a toContainerMount function that handles converting user provided paths (e.g. /) to Google Storage paths (e.g. /mnt/disks/<BUCKET>/)

• Example Call

• Test Cases

[quinnwai]

Hoping this will be addressed in #1284.

Just a note that above that the above only tests multiple bucket access when hardcoded to. Code I think already existed to write to multiple buckets, I think the bigger problem is that we were forcing users to specify a /mnt/disk/ url when they shouldn't need to (#1283)

[lbeckman314]

Ah thanks for noting the issue with having to explicitly set the path with the /mnt/disk url.

Hmm... I can go ahead and update this issue to reflect that and keep #1283 as the issue addressing multiple buckets.

[lbeckman314]

Issue updated!