Scoring mesh size, and zero suppression.

[rokieffe]

It would bee nice if we could select THnSparse Class Reference for scoring 3D histogram. In the case we only have a few deposit pixel per event. In order to avoid storing zeros in all the unused bins.

That would help greatly to reduce files size.

[stewartboogert]

@rokieffe thanks for raising this issue.

@lnevay Robert raised this is the weekly meeting. He has a pixel detector which is being represented by a scoring mesh, so needs event by event and merged run histogram.

[lnevay]

At the time a lot of the analysis was written, THNSparse was badly documented and difficult to use so we avoided it. We can come back to it now as looks not too bad.

However, this would involve touching a lot of the analysis tools and accumulation. With root's usual lovely inheritance, it has all the same functions one would expect as any class inheriting TH1 but THnSparse does not inherit it. Annoyingly, this would mean I have to write specialised accumulators for it. Not a huge deal, but annoying the inheritance isn't there despite having the same interface we would use.

We're basically finished now with a feature of only storing the (now correct) per event average histogram for the whole run, so the per-event ones can be turned off, vastly reducing the output data size.

For your purpose @rokieffe, you also want say the energy or charge deposition from the scoring mesh? i.e. if you had a sampler in front of it, could you live with that information (kinematic variables only) and then only have the per-event average for the run? Or do you need the scored quantity per-event also?

[stewartboogert]

@lnevay so use a sampler combined with a scoring mesh. Seems like a good solution.

Are the scoring mesh improvements merged yet?

[rokieffe]

My way to use it is not really to do accumulation. I would like to be able to run macros on the output file on an event by event basis. To perform different digitization tasks on the energy deposited in silicon and convert it into “detector like” data. Then apply/test clustering algorithms on an event by event basis. That’s why I store the whole scoring mesh event by event. But if it is complicated, I could also make a kind of post processing after the simulation to run on the root file from BDSIM and store into a hand made Event structure containing std:vector holding the coordinates and content of the few cells touched (over millions since my scoring plane is 16666*122 pixels). To avoid storing all the zero pixels. For now a file with 100kevts is 26GB and I need to simulate 10Million events to get close to the data we would collect and fit for a measurement. I want to avoid storing/manipulating 2Tb of data (mostly full of zeros) . I hope this info helps to get a picture of what I’m trying to do.

[stewartboogert]

@rokieffe So Laurie already has a another fix to suppress the per-event storage of the scoring mesh. This was done before your requirement. @lnevay is this merged to develop?

As for needing per event histograms of the hits, Laurie suggests just placing a sampler directly in front to the mesh, but then it really depends on what quantity you wish to score. I am also not sure if this is the correct thing for your detector energy deposit. @rokieffe : @lnevay asked this question in his last comment

[rokieffe]

Using the sampler I would only know the position of the particle incoming in the silicon detector. I have been using the sampler a lot for my first studies. But what I want to simulate is the detector response on an event by event basis, that's why I decided to store the energy deposited in the silicon layer on an event per event basis. Then I can reproduce the sharing of charge between pixels and put a cut at 3 sigma from the expected silicon detector noise, apply clustering and so on... this can only be done on an event per event basis. I really think it might be simpler if I do as I say, like a post processing of the root file to recreate yet another lighter root file with the X and Y location of the energy deposit and the deposited energy event by event. Then I use this new file to perform conversion into electron/hole pairs in silicon, and apply reconstruction event per event. Like tracking between few layer of silicon sensors for example.