Need more user friendly way of dealing with friend trees and dataframes

[acampove]

Feature description

Problem

We have multiple trees t1 t2 t3, etc. For each of them, we have a set of 100 files that contain them. I.e. we have 300 files. However the trees t1 contain most of the branches, we just add extra branches through the latter two sets of files. I.e. this is the classical friend tree issues.

Solution needed

We need something like:

from ROOT import RDataFrame

rdf = RDataFrame('treename')
rdf = rdf.AddFiles('/path/to/tree1/files/*.root')
rdf = rdf.AddFiles('/path/to/tree2/files/*.root')
rdf = rdf.AddFiles('/path/to/tree3/files/*.root')

Currently I am writting my own classes to do this, mostly because you haven't spent time building something that can do this. Could you please work on it?

Thanks.

Alternatives considered

No response

Additional context

No response

[pcanal]

With TTree you can do:

TChain c1('treename');
c1.AddFiles('/path/to/tree1/files/*.root')
TChain c2('treename');
c2.AddFiles('/path/to/tree2/files/*.root')
TChain c3('treename');
c3.AddFiles('/path/to/tree3/files/*.root')
c1.AddFriend(c2);
c1.AddFriend(c3);

rdf = RDataFrame(c1)

[vepadulano]

Dear @acampove ,

Thanks for reaching out to us and for showing us your use case! I really appreciate your suggestions for improvement.

In this particular case, I feel like there is an interface you might find quite useful. RDataFrame allows specifying the input dataset, including friends, via a JSON file (see the ROOT::RDF::Experimental::FromSpec API) (or programmatically via a class called RDatasetSpec).

For your case, something as simple as the following should give you a good starting point

# spec.json
{
    "samples": { "main": {"trees": ["mainTree"], "files": ["path/to/main/*.root"] } },
    "friends": {
        "friend1": {"trees":["nameOfFriend1"], "files": ["path/to/friend1/*.root"] }, 
        "friend2": {"trees":["nameOfFriend2"], "files": ["path/to/friend2/*.root"] }
    }
}

# main.py
df = ROOT.RDF.Experimental.FromSpec("spec.json")

[acampove]

Hi @pcanal

With TTree you can do:

TChain c1('treename');
c1.AddFiles('/path/to/tree1/files/*.root')
TChain c2('treename');
c2.AddFiles('/path/to/tree2/files/*.root')
TChain c3('treename');
c3.AddFiles('/path/to/tree3/files/*.root')
c1.AddFriend(c2);
c1.AddFriend(c3);

rdf = RDataFrame(c1)

Thanks for your reply. The thing is that we need a magical object that allows us to access the data. Something that we do not know how it works, it just works. Dataframes seem to allow this sort of thing, so I would rather use them directly without having to know that there is a TChain beneath it.

[acampove]

Dear @acampove ,

Thanks for reaching out to us and for showing us your use case! I really appreciate your suggestions for improvement.

In this particular case, I feel like there is an interface you might find quite useful. RDataFrame allows specifying the input dataset, including friends, via a JSON file (see the ROOT::RDF::Experimental::FromSpec API) (or programmatically via a class called RDatasetSpec).

For your case, something as simple as the following should give you a good starting point

# spec.json
{
    "samples": { "main": {"trees": ["mainTree"], "files": ["path/to/main/*.root"] } },
    "friends": {
        "friend1": {"trees":["nameOfFriend1"], "files": ["path/to/friend1/*.root"] }, 
        "friend2": {"trees":["nameOfFriend2"], "files": ["path/to/friend2/*.root"] }
    }
}

# main.py
df = ROOT.RDF.Experimental.FromSpec("spec.json")

Hi @vepadulano

Thanks for your reply. That goes in the right direction. However we do not use JSON files for configuration, we use YAML, at least within LHCb. YAML offers a superior way of doing this; for instance, we can use comments in YAML, not in JSON. YAML is also far more readable. However, that example should work also with YAML, i assume, because YAML is a superset of JSON.

How do I specify the index when the friends are added? Currently I am doing this. I.e. I join two pandas dataframes using an index I build myself from the event and run number. This should not be optional, given that otherwise the trees might be misaligned and that would be a disaster, if you want your MVA scores to be aligned with the actual data, for instance.

Another consideration is that events might have multiple candidates. Therefore the EventNumber and RunNumber would not be enough and you have to have a candidate number. Have you setup a mechanism to deal with that? Currently we drop all but the first candidate when we have multiple candidates.

Cheers.

[acampove]

Hi @vepadulano

OK, I just checked, this FromSpec only work with JSON, with YAML I see:

Traceback (most recent call last):
  File "/home/acampove/Packages/rx_data/test.py", line 3, in <module>
    df = RDF.Experimental.FromSpec('config.yaml')
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cppyy.gbl.nlohmann.detail.parse_error: ROOT::RDataFrame ROOT::RDF::Experimental::FromSpec(const string& jsonFile) =>
    parse_error: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: 's'

And this seems to suggest that the situation won't change any time soon. Which means that I will need to at least wrap your experimental feature within some function that does some YAML -> JSON conversion.

[acampove]

Hi,

So I am testing this feature and it seems to work. However, as mentioned before, I still will need some sort of wrapper. Nevertheless the wrapper is much thinner. Compare the original with the update version. I will keep playing with the new implementation and see if i find any problem.

[vepadulano]

Dear @acampove ,

Thank you for testing this feature and for the precious suggestions and feedback!

Let me be more concrete and try to boil down this issue to practical tasks. My understanding by reading your comments so far is that this issue is actually underpinning at least three desiderata from your side as a developer of an LHCb analysis workflow:

1. The dataset specification should be able to ingest YAML, on top of the already existing support for JSON. I understand the requirement, this sounds like an interesting improvement to the existing API and a good first issue for newcomers to the project, external contributors or collaborations perhaps through student projects!
2. You would like to join two or more datasets (i.e. in the TTree data format) before processing. This functionality is data-format-dependent, that is TTree might give you some way of doing it, RNTuple another, in-memory data formats like numpy arrays yet another one etc. In TTree, this functionality is currently available via TTreeIndex. An example is available in the RDataFrame docs, but notice that currently can only be done by creating the TTree instances first, then building the indices via BuildIndex.
3. You have a further requirement on the join use cases, where you don't only need two join fields, e.g. event number and run number, but you need a third join field for the candidate number. Note that TTreeIndex does not support more than two join fields, so you will not be able to get this functionality if the input data is in the TTree data format.

You will be glad to know that both points 2 and 3 are the subject of active development for both RDataFrame and RNTuple, the next-generation ROOT data format, and you will be able to perform complex joins like the ones you describe with RDataFrame using the user-friendly JSON dataset specification for RNTuple datasets.

[acampove]

Hi @vepadulano

but notice that currently can only be done by creating the TTree instances first, then building the indices via BuildIndex.

Then when you use the friend trees feature through FromSpec, the machine will assume that the entries have to be aligned in the way they are ordered? This means that if I have, somehow, introduced a list of files for the main and the friend trees with different orders, the entries will be paired incorrectly?

the user-friendly JSON dataset specification

I am pretty sure YAML is more user friendly

[vepadulano]

Then when you use the friend trees feature through FromSpec, the machine will assume that the entries have to be aligned in the way they are ordered? This means that if I have, somehow, introduced a list of files for the main and the friend trees with different orders, the entries will be paired incorrectly?

Dear @acampove ,

The lines I showed above with the JSON specification are equivalent to doing the following (also hinted by @pcanal in an earlier comment)

mainChain = ROOT.TChain("mainTree")
mainChain.Add("path/to/main/*.root")

friend1 = ROOT.TChain("nameOfFriend1")
friend1.Add("path/to/friend1/*.root")

friend2 = ROOT.TChain("nameOfFriend2")
friend2.Add("path/to/friend2/*.root")

mainChain.AddFriend(friend1)
mainChain.AddFriend(friend2)

This code is going to join the main TChain with the other two TChain datasets according to the entry index, sometimes also called a "positional join". There is no knowledge regarding the join condition(s). As such, if your dataset contains some specific intelligence regarding which events from the main TChain should be connected to which other events of the other TChains according to some specific columns (e.g. the event number or the run number), this is not considered. You can already infer this because in no place in the above code you are able to signal that there is this intelligence in the dataset.

In your question, you mention specifically the entries will be paired incorrectly?. This question can only be answered if you know how to pair the entries together, that is if you know which are the joining conditions, and of course if you offer this knowledge to the underlying data format or database system. The API offered by TTree to do so is via TTreeIndex as I was mentioning previously. Also note that as already mentioned joining two generic datasets together is extremely data-format-dependent, whether it is a ROOT data format or some other data format.