CWL 1.0 conformance test errors on HPC

[kinow]

On MN5, when I run the conformance tests for CWL 1.0 with 0.2.0rc2, I get 75 errors. WIth #1054 , this number is reduced to 26.

I used this command to get the tool + job of the failing tests.

$ awk '
/^tool:/ {
    getline
    sub(/^ *file:\/\//, "", $0)
    tool=$0
}

/^job:/ {
    getline
    sub(/^ *file:\/\//, "", $0)
    job=$0
}

/Returned non-zero but it should be zero/ {
    print "tool:", tool
    if (job != "") print "job:", job
    print ""
    tool=""; job=""
}
' v1.0/local/run.log

Then, for each failure I used this template to reproduce failures individually, replacing file/settings accordingly:

version: v1.0
workflows:
  workflow_name:
    type: cwl
    config:
      file: "/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat3-tool-mediumcut.cwl"
        # settings: ""

      docker:
        - step: /
          deployment:
            type: singularity
            config: {}

I may not have time to go through the 26 failures, but I will try to add a few here, and then continue in the comments when I have more time. I won't post the complete output of streamflow run --debug, but will include the step failure + traceback. The complete list of failures first:

Failures

(One failure appears to have escaped my script, but I can chase it later; I believe there are 25 entries below, out of 26)
tool:

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat3-tool-mediumcut.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/dir2.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat4-tool.cwl

tool:
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/metadata.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/tmap-tool.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/tmap-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat1-testcli.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-n-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/valueFrom-constant.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/initialworkdirrequirement-docker-out.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/search.cwl#main

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/bool-empty-inputbinding.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/revsort.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat5-tool.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/revsort-packed.cwl#main

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat3-tool-shortcut.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/binding-test.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/bwa-mem-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat1-testcli.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/bwa-mem-tool.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/stage-unprovided-file.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat3-tool-docker.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/empty-array-input.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/template-tool.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-job.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/inline-js.cwl
job: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/empty.json

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/docker-array-secondaryfiles.cwl

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/formattest3.cwl

Reproducing the first failures

tool: /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat3-tool-mediumcut.cwl

2026-06-07 16:40:33.263 DEBUG    Step /file1-token-transformer received inputs ['0']
2026-06-07 16:40:33.263 ERROR    Token file1 is not optional.
Traceback (most recent call last):
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/workflow/step.py", line 1942, in run
    await self.transform(inputs)
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/transformer.py", line 165, in transform
    self.get_output_name(): await self.processor.process(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        inputs, inputs[self.port_name]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 341, in process
    _check_token_type(
    ~~~~~~~~~~~~~~~~~^
        name=self.name,
        ^^^^^^^^^^^^^^^
    ...<4 lines>...
        check_file=True,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 79, in _check_token_type
    raise ProcessorTypeError(f"Token {name} is not optional.")
streamflow.core.exception.ProcessorTypeError: Token file1 is not optional.
2026-06-07 16:40:33.315 DEBUG    Step /__transfer__/file1 received termination token with Status FAILED
2026-06-07 16:40:33.315 DEBUG    FAILED Step /file1-token-transformer

/gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/dir2.cwl

2026-06-07 16:43:40.835 DEBUG    Step /indir-token-transformer received inputs ['0']
2026-06-07 16:43:40.835 ERROR    Token indir is not optional.
Traceback (most recent call last):
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/workflow/step.py", line 1942, in run
    await self.transform(inputs)
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/transformer.py", line 165, in transform
    self.get_output_name(): await self.processor.process(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        inputs, inputs[self.port_name]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 341, in process
    _check_token_type(
    ~~~~~~~~~~~~~~~~~^
        name=self.name,
        ^^^^^^^^^^^^^^^
    ...<4 lines>...
        check_file=True,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 79, in _check_token_type
    raise ProcessorTypeError(f"Token {name} is not optional.")
streamflow.core.exception.ProcessorTypeError: Token indir is not optional.
2026-06-07 16:43:40.837 DEBUG    Step /__transfer__/indir received termination token with Status FAILED
2026-06-07 16:43:40.837 DEBUG    FAILED Step /indir-token-transformer

/gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat4-tool.cwl

2026-06-07 16:46:19.788 ERROR    Token file1 is not optional.
Traceback (most recent call last):
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/workflow/step.py", line 1942, in run
    await self.transform(inputs)
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/transformer.py", line 165, in transform
    self.get_output_name(): await self.processor.process(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        inputs, inputs[self.port_name]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 341, in process
    _check_token_type(
    ~~~~~~~~~~~~~~~~~^
        name=self.name,
        ^^^^^^^^^^^^^^^
    ...<4 lines>...
        check_file=True,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/gpfs/projects/bsc/<USER>/cwl/streamflow/streamflow/cwl/processor.py", line 79, in _check_token_type
    raise ProcessorTypeError(f"Token {name} is not optional.")
streamflow.core.exception.ProcessorTypeError: Token file1 is not optional.
2026-06-07 16:46:19.789 DEBUG    Step /__transfer__/file1 received termination token with Status FAILED
2026-06-07 16:46:19.789 DEBUG    FAILED Step /file1-token-transformer

/gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/metadata.cwl:

(job /gpfs/projects/bsc//cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/cat-job.json)

2026-06-07 16:48:17.427 INFO     COPYING from /usr/share/zoneinfo/UCT/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/hello.txt to /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt on location 882b94fd-cbf8-4b48-a83a-741e02808569/8ec5b201-b0e0-448d-9e94-b60253b36f5d
2026-06-07 16:48:17.427 DEBUG    EXECUTING command ln -snf /usr/share/zoneinfo/UCT/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/hello.txt /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt 2>&1 on SubprocessShell with command `singularity exec instance://8ec5b201-b0e0-448d-9e94-b60253b36f5d sh`
2026-06-07 16:48:17.427 DEBUG    COMPLETED Step /__schedule__
2026-06-07 16:48:17.427 DEBUG    Step /__transfer__/numbering received termination token with Status COMPLETED
2026-06-07 16:48:17.427 DEBUG    COMPLETED Step /__transfer__/numbering
2026-06-07 16:48:17.428 INFO     COMPLETED copy from /usr/share/zoneinfo/UCT/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/hello.txt to /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt on location 882b94fd-cbf8-4b48-a83a-741e02808569/8ec5b201-b0e0-448d-9e94-b60253b36f5d
2026-06-07 16:48:17.428 DEBUG    EXECUTING command test -L /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt 2>&1 on SubprocessShell with command `singularity exec instance://8ec5b201-b0e0-448d-9e94-b60253b36f5d sh`
2026-06-07 16:48:17.429 DEBUG    Step / received inputs ['0', '0']
2026-06-07 16:48:17.429 DEBUG    Step /__transfer__/file1 received termination token with Status COMPLETED
2026-06-07 16:48:17.429 DEBUG    COMPLETED Step /__transfer__/file1
2026-06-07 16:48:17.429 DEBUG    Job /0 started
2026-06-07 16:48:17.429 DEBUG    Job /0 changed status to RUNNING
2026-06-07 16:48:17.429 DEBUG    Job /0 inputs: {
    "file1": {
        "basename": "hello.txt",
        "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
        "class": "File",
        "dirname": "/tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d",
        "location": "file:///tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt",
        "nameext": ".txt",
        "nameroot": "hello",
        "path": "/tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt",
        "size": 13
    },
    "numbering": null
}
2026-06-07 16:48:17.429 INFO     EXECUTING step / (job /0) on location 882b94fd-cbf8-4b48-a83a-741e02808569/8ec5b201-b0e0-448d-9e94-b60253b36f5d into directory /tmp/streamflow/7613b320-447f-4bf5-9bb8-f947be6f182c:
cat \
    /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt

[kinow]

Only a few tests executed, sorry -- a bit strapped for time as I need to submit some Slurm jobs & write more in the thesis.

I think these initial failures have a few interesting things that if fixed, may fix more tests out of the 26 total failures we have right now with 0.2.0rc2 for 1.0.

1. The first ones all seem to fail with streamflow.core.exception.ProcessorTypeError: Token file1 is not optional.. Is the Token a CWL input of a workflow?

2. The second one is very interesting. Look at the end, at the copy command in MN5:

2026-06-07 16:48:17.427 INFO     COPYING from /usr/share/zoneinfo/UCT/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/hello.txt to /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt on location 882b94fd-cbf8-4b48-a83a-741e02808569/8ec5b201-b0e0-448d-9e94-b60253b36f5d
2026-06-07 16:48:17.427 DEBUG    EXECUTING command ln -snf /usr/share/zoneinfo/UCT/gpfs/projects/bsc/<USER>/cwl/streamflow/common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/v1.0/v1.0/hello.txt /tmp/streamflow/012515d8-0320-4173-9e5b-890044e56741/00b8357a-5b8a-4332-8290-1f8cff42b48d/hello.txt 2>&1 on SubprocessShell with command `singularity exec instance://8ec5b201-b0e0-448d-9e94-b60253b36f5d sh`

Why is StreamFlow copying things from /usr/share/zoneinfo/UCT??? That's already odd, but it's also using that as prefix before the actual path, /gpfs/projects/.... I think somewhere we might be using a variable, or some call in Python or str(some_object_with_timezone) maybe?

Cheers

[LanderOtto]

Hi @kinow, thanks for further testing StreamFlow.

Regarding the errors reported: error 1 streamflow.core.exception.ProcessorTypeError: Token file1 is not optional, yes, the error is saying that the workflow input data is missing.

Are you also passing the input config?

• tool: cat3-tool-mediumcut.cwl job: cat-job.json
• tool: dir2.cwll job: dir-job.yml
• tool: cat4-tool.cwl l job: cat-job.json

Regarding error 2, it is probably a case erroneously handled by StreamFlow with Singularity binds.
Sorry, we do not have access to MN5 to test it. Can you share the full StreamFlow debug log?