WIP Rework mne bids path match

[waldie11]

PR Description

When _return_root_paths now uses ignore_nosub, it can benefit from from only exploring directories within root/sub-* instead of exploring the whole tree and subsequently applying the regex filter on visited files.

get_entity_vals is extended by an argument include_match to introduce a prefilter on regex path match. This is a whitelist approach, and extends both functionality and performance compared to a equivalent blacklist approach of existing option ignore_dirs.

Merge checklist

Maintainer, please confirm the following before merging.
If applicable:

• All comments are resolved
• This is not your own PR
• All CIs are happy
• PR title starts with [MRG]
• whats_new.rst is updated
• New contributors have been added to CITATION.cff
• PR description includes phrase "closes <#issue-number>"

[welcome]

Hello! 👋 Thanks for opening your first pull request here!
Please read the contributor guide, and please follow the steps outlined in the "Instructions for first-time contributors" section therein. ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.41%. Comparing base (b75e4ca) to head (df82998).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1355      +/-   ##
==========================================
+ Coverage   97.37%   97.41%   +0.03%     
==========================================
  Files          40       40              
  Lines        8966     9008      +42     
==========================================
+ Hits         8731     8775      +44     
+ Misses        235      233       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sappelhoff]

thanks @waldie11! we have a bit of a backlog right now so please don't worry if we take a bit longer to get back to you.

[waldie11]

@sappelhoff happy new year!
no worries, as i had seen some of the api changes in python-mne bringing down the test suite, i already expected priorities to be elsewhere.

[sappelhoff]

Hey @waldie11 could you please add an example for the feature that you introduce? Either by modifying an actual example that already exists, or by adding a numpy docstr example below the ones that already exist.

Could you please also resolve the conflicts AND follow the steps for first time contributors here? -->#1354

[waldie11]

Hi @sappelhoff ,

I think the true meat is rather a rework of _return_root_paths. This is not exactly a API function, so it feels a bit weird to write an example.

get_entity_vals seems to undergo some changes within upcoming 0.17 anyhow. I extended the docstring a bit on my feature contribution. Do you think this is sufficient? Neither the ignore_ parameters have an example. It benefits filtering a bids_path, which has deep directory trees for sourcedata, derivatives, subject or who-knows-what, where one wants to select prehand in which directories to look.

my_dataset/
  derivatives/
    downsampled/
      sub-01/
        micr/
          sub-01_sample-01_res-4x_TEM.png
          sub-01_sample-01_res-4x_TEM.json
  sub-01/
    micr/
      sub-01_sample-01_TEM.png
      sub-01_sample-01_TEM.json

(BIDS v1.8.0-1 p. 214)

[waldie11]

Idk how far you are interested in diving into this:

in test_path_benchmark I created a dummy bids compliant tree. If I browse this artificial tree with get_entity_vals, I extract a performance gain in about one order of magnitude by using include_match="sub-*/" in comparison to ignore_dirs=["derivatives", "sourcedata"]

This benchmark is eating up some CI runtime though. I tried keeping it lightweight.

[sappelhoff]

I will make time for this in the next days, latest in the weekend, thanks for your patience!

[sappelhoff]

Thanks for your work @waldie11! I left a few comments and suggestions to be addressed, but I think we can then go ahead and merge it!

[sappelhoff]

where are the 20 and 9 coming from? as with the 17 above, better define these as variables within the test, potentially with a quick inline comment, or even better, a descriptive var name

n_test_subs = 20
...

[sappelhoff]

please use f-string formatting whenever possible.

[sappelhoff]

is this not done automatically by pytest?

[waldie11]

Thanks for your kind improvements @sappelhoff !

[sappelhoff]

Thanks for the contribution!

[welcome]

🎉 Congrats on merging your first pull request! 🥳 Looking forward to seeing more from you in the future! 💪