Summarization principle implementation

[ericearl]

@Lestropie I'm so glad someone could start work on this project. It's been on my backlog a loooooooooooong time.

Anyway, for this issue, I am wondering whether your issue #2 or issue #4 already cover the conceptual idea (as I understand it) of the "summarization principle" proposed on bids-standard/bids-2-devel#65.

I believe an implementation of this might look something like:

1. Given any BIDS dataset, find all unique (and minimal) combinations of JSON filename entities (the _key-value pairs) for each suffix. Broadly call this a "unique combination" for the sake of this GH issue.
2. For each unique combination, find all JSON metadata which is the same across all appearances of that unique combination and "summarize it" to the root level of the BIDS hierarchy.
3. Simultaneously remove these root-level JSON appearing fields from "leaf-level" (subfiles/subdirectories) JSONs, leaving behind only differing metadata.
4. This way, any differing metadata (metadata that is not common between the root-level JSON and all leaf-level JSONs) would not be found in the root-level summarized JSON.

Tagging @yarikoptic and @effigies to check if I missed something in this conceptual implementation. Give a thumbs up if it's fine, give a comment if I misunderstood the concept. And sorry in advance if I misunderstood the concept.

[Lestropie]

• While I'm a fan of forbidding key-value metadata overrides, I'm hesitant about the renaming of the principle. I'm not convinced that it's a drastic enough change to warrant a rename, even if it is the case that some people have a negative view of the IP and might be more amenable to "something different" if it has a different name. But it still involves a many-to-many mapping between data files and metadata files, and a precedence order still has to be established for non-JSON metadata files, so precluding overrides doesn't actually save that much complexity.

  (Indeed a principle rename might make naming of the relevant tooling difficult...)

• For what was achievable as a one-man job in three days, the focus has been strictly on reporting of datasets. No manipulation of datasets has even been attempted, and there's still some more things to iron out that I think need to take precedence over such.

  I will however note in that regard: The function currently named utils.metadata.load_metadata() on Outcomes from OHBM Hackathon 2025 #15 loads into a dictionary all metadata for all data files taking into account inheritance. These are the data that would form the input to any dataset manipulation algorithm. As a non-programmatic preview, the contents of this graph can be exported using the -g option.

• Given interest in auditing the prevalence of specifically key-value overloading through inheritance, over and above just inheriting data from multiple metadata files, the code in Outcomes from OHBM Hackathon 2025 #15 is able to detect this:

  • I've interpreted this as a "warning" irrespective of proposed modifications to specification to RECOMMEND against doing so (as opposed to prior specification that merely RECOMMENDED that such should be kept to a minimum). Personally I think that this should be discouraged for all retrospective specifications and forbidden in future specifications.

  • Running with the -w option will escalate this from a warning to an error, thereby facilitating auditing of its prevalence in existing datasets (and eventually perhaps inclusion as an external module for the validator).

  • Which data files and which key-value fields are affected can be exported using the -o option.

• The algorithmic implementations of how to optimise the metadata layout for each given rule set are likely to be ruleset-specific. As far as how to do it under that ruleset:

  • Any such algorithm should I think not operate on the JSON files in a dataset. It should instead operate on the full metadata graph where the key-value metadata are defined per data file. That ensures that any complexities around inheritance within the input datasets are resolved and the algorithm has a fresh slate to work with.

  • My initial instinct was that the outer algorithmic loop should be key-values, not filesystem path. But I'd need to flesh the idea out in full.

  • It's not adequate to just find a common filesystem ancestor for those metadata files that define a given metadata key-value pair; that common ancestor must also not define that metadata key-value pair for any other data file.

[ericearl]

Thank you for the thorough response. I will check out #15. And if you need any help using the tools on ALL OpenNeuro datasets, I have done this a number of times for phenotypic data checks and would be happy to do IP auditing checks. Fair warning: I don't pay attention to my GitHub notifications as much as my emails. Thanks again for all the time investment here!