Combining samples post fit

[meschrei]

Hi,
I just wanted to ask if there is an easy way to get the summed yields and their uncertainties of a sub set of samples in cabinetry.
If you have a couple of samples for your signal or background that you want to be treated individually by the fit, but in the end you only care about the total amount of background or signal yields.
I saw that you can easily get the total, but that would contain signal as well as background in this example. Summing the yields is relatively easy, but getting the correct uncertainties is probably a bit more tricky.

So, is there a way to combine a subset of samples after the fit procedure in this way?

[alexander-held]

Hi, I'm afraid there is no easy way at the moment to get the yield uncertainty for a set of samples through cabinetry. To get the correct numbers for the uncertainties requires already grouping samples in the error propagation calculation which is accessed through cabinetry.model_utils.prediction (which uses cabinetry.model_utils.yield_stdev). This could be done by adding a new keyword argument that could e.g. take a dictionary of the form

{"new_sample_name": [sample_to_merge_1, sample_to_merge_2]}

and then internally merge samples. That also means some small modifications to the ModelPrediction object are needed, as this currently contains the pyhf model instance to read metadata from it (like sample names) but after manually merging samples, this metadata no longer would correspond to the rest of the info held by the ModelPrediction instance. We could replace the model in there by just the relevant metadata and modify that on the fly as needed. This probably requires some more changes across cabinetry but seems like a good idea in general.

Would something like what I describe above work for your use case? A similar question came up recently as well, so given the interest I think this is a feature that should be added, I'll create an issue for it.

[alexander-held]

see #502

[meschrei]

Thank you for the answer and for creating an issue for that.
Yes, I think something like that solution you described would work for my case. It does sound like a larger project that is going to take some time though. So, if there is some way I could help with this, let me know. Otherwise I'm going to see if I can find another way to deal with my specific case for the time being

[alexander-held]

If you would like to give the implementation a try, you are very welcome and I would be happy to help. The ModelPrediction change might need to touch a few places in the codebase but the other part should be fairly contained. For other possible solutions: if you can merge the samples at the config level (and build a new model that contains the merged sample), that could work. You could also merge at the workspace level. There are some setups where this will not work, for example if you want to merge samples where one has a normalization factor that is not shared by the other. Such cases could only be addressed by a proper implementation. I cannot promise that I would get to this in the next few days myself but I have some hope that it is quicker to implement than it might sound.