Component not working when combined with eqasim component

[balacmi]

I am creating a workflow with eqasim data loader, eqasim, and eqasim-matsim using Corsica data example. First two components finish successfully. However, eqasim-matsim fails with the following error:

CORSICA SCENARIO.
Cloning into '/odtp/odtp-workdir/scenario'...
Note: switching to '3c9b137b9e4b3c17e163cae2e170f18611adcf56'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3c9b137 Update corsica.md (#222)
Running Matsim PIPELINE
PREPARATION OF CONFIG FILE
Output file available: /odtp/odtp-workdir/scenario/config.yml.
INFO:numexpr.utils:Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO:numexpr.utils:NumExpr defaulting to 16 threads.
INFO:synpp:Found 64 stages
INFO:synpp:Found pipeline metadata in /odtp/odtp-workdir/cache/pipeline.json
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/synpp/__main__.py", line 14, in <module>
    synpp.run_from_yaml(config_path)
  File "/usr/local/lib/python3.10/dist-packages/synpp/pipeline.py", line 886, in run_from_yaml
    Synpp.build_from_yml(path).run_pipeline()
  File "/usr/local/lib/python3.10/dist-packages/synpp/pipeline.py", line 910, in run_pipeline
    return run(definitions, self.config, self.working_directory, flowchart_path=flowchart_path,
  File "/usr/local/lib/python3.10/dist-packages/synpp/pipeline.py", line 739, in run
    validation_token = stage["wrapper"].validate(context)
  File "/usr/local/lib/python3.10/dist-packages/synpp/pipeline.py", line 83, in validate
    return self.instance.validate(context)
  File "/odtp/odtp-workdir/scenario/data/gtfs/cleaned.py", line 61, in validate
    input_files = get_input_files("{}/{}".format(context.config("data_path"), context.config("gtfs_path")))
  File "/odtp/odtp-workdir/scenario/data/gtfs/cleaned.py", line 56, in get_input_files
    raise RuntimeError("Did not find any GTFS data (.zip) in {}".format(base_path))
RuntimeError: Did not find any GTFS data (.zip) in /odtp/odtp-workdir/data/gtfs_corsica

I suppose that the data folder from the previous component is not copied to the new component correctly or at all, somewhere here:
https://github.com/odtp-org/odtp-eqasim-matsim/blob/813ec4be27ca69229227aa411d2e64fa8e9e681a/app/app.sh#L16

My folder structure in the eqasim-matsim component looks liek this after unsuccessful execution:

6140	./odtp-input/eqasim-output
309084	./odtp-input/cache
315228	./odtp-input
8	./odtp-logs
6140	./odtp-output/eqasim-matsim-output
7676	./odtp-output
4	./repository
322920	.

[caviri]

Hello @balacmi, taking a look here. Yes. It seems like the input folder is not named correctly. Are you using the ODTP CLI? What's the dt repository you are using, this one? https://github.com/odtp-org/dt-corsica-mobility

It's easier to debug if you share the whole workflow in a repository following this structure as I would be able to take a look on the parameters and other details.

[balacmi]

Hi @caviri, I am using the ODTP Dashboard GUI, where I manually set up the flow. Looking at https://github.com/odtp-org/dt-corsica-mobility I can also see that 002.parameters file has TRANSFER_INPUT_TO_OUTPUT=TRUE. However, I do not see anywhere where this is actually used neither in dt-corsica-mobility nor in the eqasim component (I did not set it for my workflow). Am I missing something?

I did not use the parameters file, but I manually added parameters when setting up the execution. Is there a way to obtain the parameters that I used somewhere else besides taking a screenshot from the GUI?

[caviri]

Hi @balacmi,

Thanks. Yes, in this case it's necessary to set the the parameter TRANSFER_INPUT_TO_OUTPUT=TRUE. This is a workaround to allow the running of a DAG pipeline cycles as a sequence of steps. This connection below below. This is one of the internal reserved parameters used by the ODTP client, they are described in our component template but not in the documentation (https://github.com/odtp-org/odtp-component-template/blob/main/.env.dist). I'll open a ticket for this.

My recommendation in the GUI is to try to replicate the pipeline with all parameters and secrets there.

I did not use the parameters file, but I manually added parameters when setting up the execution. Is there a way to obtain the parameters that I used somewhere else besides taking a screenshot from the GUI?

@sabinem, regarding this is there a way to export the execution and steps from the GUI?

[balacmi]

Thank you @caviri ! I will try it like that and keep you updated.

[balacmi]

It works great with that parameter added. It would be great to add it to the documentation somewhere.