Custom Output Path Issue

[timdressler]

Hi all,

thanks for proving such a nice tool!
I noticed that when setting derivatives_output to a custom folder, the toolbox still adds data/derivatives/MEEGqc/ to the path which is a bit annoying. Have you considered to just use the path provided in derivatives_output without adding data/derivatives/MEEGqc/?
I am using version 1.1.2 (python API).

Thanks for considering!

Best,
Tim

[karellopez]

Hi @timdressler, thanks for the kind words, and this is a fair point. It is closely related to #135 (thanks @dnacombo), where we discussed the same behaviour, so let me explain the reasoning and then ask for your input, because I would be happy to add an optional mode for this.

Why the extra data/derivatives/MEEGqc/ appears. MEEGqc has two output modes:

1. Leave the output empty: results go to the derivatives folder inside your input dataset, <dataset>/derivatives/MEEGqc/.... This is the standard BIDS layout.
2. Set a custom derivatives_output: MEEGqc recreates the input BIDS root inside that folder and writes the results there. That is why you see data/ (your input dataset root) reappear, with derivatives/MEEGqc/ inside it.

There are two reasons for mode 2:

• BIDS compliance. In BIDS, a derivatives/ folder is expected to live inside a dataset root (<root>/derivatives/<pipeline>/). Recreating the dataset root in the output folder keeps the result a valid BIDS derivatives structure that other BIDS tools can read.
• Multiple datasets in one run. MEEGqc can process several BIDS datasets in a single run. Recreating each dataset's root inside the output folder keeps their outputs organized and clearly separated, instead of mixing them in one flat folder.

So it is a deliberate default rather than a bug. That said, I fully see the appeal of pointing the tool at a folder and getting the derivatives written straight there, and I would like to support it as an optional mode while keeping the BIDS-compliant one as the default.

Open question, for you and anyone following #135: what would the ideal optional structure be? A few candidates to react to:

• A) Use the path literally, <derivatives_output>/MEEGqc/.... Simplest, and closest to your request. For a run with several input datasets we would add a per-dataset subfolder to avoid mixing, for example <derivatives_output>/<dataset-name>/MEEGqc/....
• B) <derivatives_output>/derivatives/MEEGqc/.... Keeps a derivatives level but drops the recreated dataset root.
• C) The sibling layout @dnacombo sketched in Derivatives folder nesting a rawdata folder #135 (a derivatives/ next to the dataset). This one is not strictly BIDS-compliant, since derivatives would sit outside the dataset root, but we could offer it as a clearly labelled non-BIDS option.

Which fits your workflow best, and how would you expect multiple input datasets to be handled? Once we agree on a layout people find intuitive, I am happy to add it as an optional flag.

Thanks again for the thoughtful report!

Best,
Karel

[timdressler]

Hi Karel,

Thanks for your swift response! Sorry, I did not check closed issue regarding the topic, otherwise I would have commented there, sorry!
To make it short: I think option A is a very good idea!

I don't know how other people do it, but I try to structure my derivatives/ according to a broad "topic" (e.g., preprocessing/, analysis/, ...). With MEEGqc I wanted to add a quality_control/ subfolder (bids_root/derivatives/quality_control/MEEGqc/). Thus, option A would perfectly fit my needs. Also the per-dataset folders sound nice to me.

Thanks again for considering!

Best,
Tim

P.S.: Greetings to UOL! I miss my old uni ;)

[viranovskaya]

Hi Karel,

Option A looks clear to me. I’d be happy to implement it once my documentation PR is wrapped up, if nobody is already working on it.

I would keep the current BIDS-compliant behaviour as the default, add an opt-in literal mode, and use separate dataset folders when several datasets are processed. I would also add tests for all three cases.

Would that be useful? And would you prefer a named output_layout option or a boolean flag?

[karellopez]

Hi @viranovskaya,

Yes, that would be very useful, thanks! I’m currently busy with another project, so if you have time to implement this feature, that would be greatly appreciated.
I also agree with keeping the current BIDS-compliant behavior as the default and making the literal mode opt-in. I think a named output_layout option would be preferable to a boolean flag, as it is clearer and leaves room for additional layouts in the future if needed.

Thanks again for taking this on!

[viranovskaya]

Hi Karel, I implemented the opt-in output_layout="literal" mode discussed in #142. The existing BIDS-style layout remains the default. Literal mode writes a single dataset to <output>/MEEGqc and keeps multiple datasets separate under <output>/<dataset-name>/MEEGqc.

I also added unit coverage for the default, external BIDS, single-dataset literal and multi-dataset literal layouts, plus an end-to-end real-data test covering calculation and plotting. The full local test suite passes: 53 tests.

Draft PR: #144