BIDS-SQL: Persistent and Remote Metadata Querying

[paul188]

Title

BIDS-SQL: Persistent and Remote Metadata Querying

Short description and the goals for the OHBM BrainHack

This project addresses the scalability bottlenecks in current BIDS data handling. While pybids is the gold standard for single-study analysis, it lacks a native way to persist and query metadata across multiple large-scale datasets simultaneously. Currently, researchers must re-index data into memory for every session, which is slow and resource-heavy.

We are building a persistent BIDS metadata warehouse (using SQLite or PostgreSQL) designed to aggregate and index entire collections of datasets. Our goal is to enable complex, remote, SQL-based querying that allows you to search across massive cohorts (e.g., all T1w images across three different OpenNeuro studies) without needing to mirror the files locally or reload the index.

Brainhack Goals:

1. Translate the current pybids layout into a robust SQL schema.
2. Implement a persistent storage backend that eliminates costly startup re-indexing.
3. Develop functionality to allow metadata querying of remote BIDS data (stored on servers/cloud) without local file mirroring.
4. (Stretch goal) Optimize query paths for complex subsetting that are currently slow or difficult in existing tools.

Link to the Project

https://github.com/paul188/BIDS-SQL

Image/Logo for the OHBM brainhack website

https://drive.google.com/uc?export=download&id=1FJmXy1_H4yOVF7uqnoTTm729eRcAH2PC

Project lead

Paul Johannssen,
@github: paul188,
@discord: paul1887113
Affiliation: University Clinic Bonn, Germany

Main Hub

Bordeaux

Link to the Project pitch

No response

Other hubs covered by the leaders

• Bordeaux
• Hybrid (Asia / Pacific)
• Hybrid (Europe / Middle East / Africa)
• Hybrid (Americas)

Skills

• Python (Intermediate: essential for API and integration)
• SQL & Database Experience (Basic to Intermediate)
• BIDS Specification (Solid understanding of the BIDS structure)

(optional C++ for optimizing heavy queries)

Recommended tutorials for new contributors

• Introduction to the BIDS Specification
• The existing pybids tutorial (to understand current limitations)
• SQLite Tutorial (Basic SQL queries)
• FastAPI Tutorial

Good first issues

No response

Twitter summary

No response

Short name for the Discord chat channel (~15 chars)

bids-sql

Please read and follow the OHBM Code of Conduct

• I agree to follow the OHBM Code of Conduct during the hackathon

[gkiar]

Cool project! It seems pretty related to a project we've been working on for a few years. Would love to hear your thoughts and if there's some overlap/interest in joining forces? https://github.com/childmindresearch/bids2table/ . We've been able to do very fast remote indexing with b2t (eg all of open neuro in ~15 min), but haven't made any of the helper functions currently in pybids/a close mirror of the "layout" object.

cc @kaitj

[paul188]

Cool project! It seems pretty related to a project we've been working on for a few years. Would love to hear your thoughts and if there's some overlap/interest in joining forces? https://github.com/childmindresearch/bids2table/ . We've been able to do very fast remote indexing with b2t (eg all of open neuro in ~15 min), but haven't made any of the helper functions currently in pybids/a close mirror of the "layout" object.

cc @kaitj

Hi @gkiar thank you for reaching out !
Your repo looks super interesting will definitely look into it. Would love to chat more about that :)
Can you shoot me a quick mail to Paul.Johannssen@ukbonn.de and maybe we could find date+time for a Zoom ?

[gkiar]

Hey! Sorry for the lag on this. We just merged a PR with pybids style compatibility to bids2table and are about to release. Would love to see if this fits your needs/what you think? childmindresearch/bids2table#69

cc @kaitj @effigies