Cannot launch the CWL Conformance Tests for CWL v1.1 on HPC/laptop

[kinow]

Hello! 👋

I tried launching the CWL Conformance Tests for v1.1 with StreamFlow on CESGA FT3, CSC LUMI, and BSC MN5. It failed on all environments with the same error

==================================== ERRORS ====================================
_ ERROR collecting cwl-v1.1-4feec74019b56dc5c51be905a208ff90797661de/conformance_tests.cwltest.yaml _
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/plugin.py:260: in collect
    name = utils.shortname(entry["id"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/utils.py:649: in shortname
    return [n for n in re.split("[/#]", name) if len(n)][-1]
                       ^^^^^^^^^^^^^^^^^^^^^^
/mnt/netapp2/Store_uni/home/ulc/cursos/[REDACTED]/micromamba/envs/python-3.13/lib/python3.13/re/__init__.py:267: in split
    return _compile(pattern, flags).split(string, maxsplit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   TypeError: expected string or bytes-like object, got 'int'
_ ERROR collecting cwl-v1.1-4feec74019b56dc5c51be905a208ff90797661de/conformance_tests.cwltest.yaml _
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/plugin.py:260: in collect
    name = utils.shortname(entry["id"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/utils.py:649: in shortname
    return [n for n in re.split("[/#]", name) if len(n)][-1]
                       ^^^^^^^^^^^^^^^^^^^^^^
/mnt/netapp2/Store_uni/home/ulc/cursos/[REDACTED]/micromamba/envs/python-3.13/lib/python3.13/re/__init__.py:267: in split
    return _compile(pattern, flags).split(string, maxsplit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   TypeError: expected string or bytes-like object, got 'int'
_ ERROR collecting cwl-v1.1-4feec74019b56dc5c51be905a208ff90797661de/conformance_tests.cwltest.yaml _
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/plugin.py:260: in collect
    name = utils.shortname(entry["id"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cwl-conformance-venv/lib/python3.13/site-packages/cwltest/utils.py:649: in shortname
    return [n for n in re.split("[/#]", name) if len(n)][-1]
                       ^^^^^^^^^^^^^^^^^^^^^^
/mnt/netapp2/Store_uni/home/ulc/cursos/[REDACTED]/micromamba/envs/python-3.13/lib/python3.13/re/__init__.py:267: in split
    return _compile(pattern, flags).split(string, maxsplit)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   TypeError: expected string or bytes-like object, got 'int'
- generated xml file: /mnt/lustre/scratch/nlsas/home/ulc/cursos/[REDACTED]/cwl-mpi/cwl-conformance-tests/runs/streamflow/junit.xml -
============================== 3 errors in 2.72s ===============================

You can find logs for the execution of v1.1 on CESGA FT3 (and the other HPCs) here: https://github.com/kinow/cwl-mpi/tree/0124cb20c4fb7d5d58e65d58bd16ed375425514b/cwl-conformance-tests/output/streamflow/ft3/v1.1/local

What I did to launch the tests was:

$ eval "$(micromamba shell hook -s bash)"
$ micromamba env create -y -n python-3.13 python=3.13
$ micromamba activate python-3.13
$ cd $LUSTRE
$ module load cesga/2020 singularity/4.2.2
$ singularity registry login --username <USER> docker://docker.io
$ git clone -o upstream https://github.com/alpha-unito/streamflow.git --branch=0.2.0rc2 --depth=1 --single-branch
$ cd streamflow
$ sed -i 's/-n auto/-n 4/' cwl-conformance-test.sh
$ cd $(realpath -P .)
$ export VERSION="v1.1"
$ export COMMIT="a062055fddcc7d7d9dbc53d28288e3ccb9a800d8"  # <------
$ export EXCLUDE="docker_entrypoint,stdin_shorcut,inplace_update_on_file_content"
$ export TMPDIR="$HPC_SCRATCH_DIR"  # tested without the temporary dir changes too!
$ export TEMPDIR="$HPC_SCRATCH_DIR"
$ export TMP="$HPC_SCRATCH_DIR"
$ rm -rf "$HPC_SCRATCH_DIR"
$ mkdir "$HPC_SCRATCH_DIR"
$ export MODE=local
$ mkdir -p "$VERSION/$MODE"
$ COMMIT="$COMMIT" EXCLUDE="$EXCLUDE" DOCKER="singularity" bash \
  cwl-conformance-test.sh > "$VERSION/$MODE/run.log" 2>&1 ; mv junit.xml \
  "$VERSION/$MODE/"

I did not install anything else at first. After it failed a couple of times, I tried installing nodejs, but it did not fix it. I reproduced the same failure on my laptop (Ubuntu 26.04 LTS, Python 3.13).

I tested the v1.1 commit used in the GH Actions CI YAML file, 6397014050177074c9ccd0d771577f7fa9f728a3,

streamflow/.github/workflows/ci-tests.yaml

Line 55 in dfc0fd5

- commit: "6397014050177074c9ccd0d771577f7fa9f728a3"

.

Using this commit, instead of the crash on CESGA FT3, it got stuck on this page 30+ minutes.

On my machine, it got stuck in another part:

Cheers

[kinow]

BTW, after I stopped the processes with CTRL + C, sigint, I got this on my laptop:

/home/kinow/Development/python/workspace/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/_pytest/main.py:353: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 126908293479600, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/home/kinow/Development/python/workspace/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/_pytest/main.py:353: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 129051114437808, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/home/kinow/Development/python/workspace/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/_pytest/main.py:353: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 130253358447792, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(
/home/kinow/Development/python/workspace/streamflow/cwl-conformance-venv/lib/python3.14/site-packages/_pytest/main.py:353: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: 125206802234544, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config.hook.pytest_sessionfinish(

And a simpler KeyboardInterrupt on CESGA FT3 (followed by some tracebacks).

In case it helps diagnosing the issue.

[GlassOfWhiskey]

The first error (with tag v1.1.0) is related to common-workflow-language/cwltest#302. That PR should fix the issue.
For the other one, we will investigate.