[RF][HF] ShapeFactor with initial shape creates invalid HistFactory XML

[kratsg]

Check duplicate issues.

• Checked for duplicates

Description

Expected: valid XML.

To summarize:

• ShapeFactor can have init values for its nuisance parameters
• The documentation for ShapeFactor indicates settings both for reading/writing the initial shape (see fhHigh in the source code)
• ShapeFactor inherits from HistogramUncertaintyBase which allows setting of some common attributes such as HistoName and HistoPath and InputFile.

I can write valid code (in python, but assumed to be about the same as in C++) to create a valid Measurement that ROOT is happy with, however it is not serializable.

Reproducer

Using the attached data/data.root which contains the histogram already with init values, you can run the following python code mainly inspired from the hf001_example.C example:

import ROOT
from ROOT.RooStats import HistFactory as HiFa
Measurement = HiFa.Measurement
Sample = HiFa.Sample
Channel = HiFa.Channel

InputFile = "data/data.root"

# Create the measurement
meas = Measurement("meas", "meas")
meas.SetOutputFilePrefix( "./results/example_UsingPython" )
meas.SetPOI( "poi" )
meas.AddConstantParam("alpha_syst1")
meas.AddConstantParam("Lumi")
meas.SetLumi( 1.0 )
meas.SetLumiRelErr( 0.10 )
meas.SetExportOnly( False )
meas.SetBinHigh( 2 )

chan0 = Channel("channel_0")
chan0.SetData("histchannel_0_data", InputFile)
chan0.SetStatErrorConfig( 0.05, "Poisson" )

sample_0 = Sample("sample_0", "histchannel_0_sample_0", InputFile)
sample_0.AddShapeFactor("shape_factor_0")
sample_0.GetShapeFactorList()[0].SetInputFile(InputFile)
sample_0.GetShapeFactorList()[0].SetHistoName("inits_shape_factor_0")
chan0.AddSample( sample_0 )

sample_1 = Sample("sample_1", "histchannel_0_sample_1", InputFile)
sample_1.AddNormFactor( "poi", 1, 0, 10 )
chan0.AddSample( sample_1 )

chan1 = Channel("channel_1")
chan1.SetData("histchannel_1_data", InputFile)
chan1.SetStatErrorConfig( 0.05, "Poisson" )

sample_2 = Sample("sample_2", "histchannel_1_sample_2", InputFile)
sample_2.AddShapeFactor("shape_factor_1")
chan1.AddSample( sample_2 )

# Add and build XML
meas.AddChannel( chan0 )
meas.AddChannel( chan1 )

meas.CollectHistograms()
meas.PrintTree()
meas.PrintXML()

One of the output XML files produced is meas_channel_0.xml below:

<!--
This xml file created automatically on:
2023-7-19
-->
<!DOCTYPE Channel  SYSTEM 'HistFactorySchema.dtd'>

  <Channel Name="channel_0" InputFile="" >

    <Data HistoName="histchannel_0_data" InputFile="data/data.root" HistoPath=""  />

    <StatErrorConfig RelErrorThreshold="0.05" ConstraintType="Poisson" />

    <Sample Name="sample_0"  HistoPath=""  HistoName="histchannel_0_sample_0"  InputFile="data/data.root"  NormalizeByTheory="True" >
      <ShapeFactor Name="shape_factor_0"  InputFile="data/data.root"  HistoName="inits_shape_factor_0"  HistoPath=""   />
    </Sample>


    <Sample Name="sample_1"  HistoPath=""  HistoName="histchannel_0_sample_1"  InputFile="data/data.root"  NormalizeByTheory="True" >
      <NormFactor Name="poi"  Val="1"  High="10"  Low="0"  Const="False"   />
    </Sample>



  </Channel>

and when I run this through hist2workspace, I get the following error:

$ hist2workspace meas.xml

RooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby
                Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
                All rights reserved, please read http://roofit.sourceforge.net/license.txt

[#2] INFO:HistFactory -- hist2workspace is less verbose now. Use -v and -vv for more details.

./meas_channel_0.xml:14: element ShapeFactor: validity error : No declaration for attribute InputFile of element ShapeFactor
"  InputFile="data/data.root"  HistoName="inits_shape_factor_0"  HistoPath=""
                                                                               ^
./meas_channel_0.xml:14: element ShapeFactor: validity error : No declaration for attribute HistoName of element ShapeFactor
"  InputFile="data/data.root"  HistoName="inits_shape_factor_0"  HistoPath=""
                                                                               ^
./meas_channel_0.xml:14: element ShapeFactor: validity error : No declaration for attribute HistoPath of element ShapeFactor
"  InputFile="data/data.root"  HistoName="inits_shape_factor_0"  HistoPath=""
                                                                               ^
[#2] ERROR:HistFactory -- Loading of xml document "./meas_channel_0.xml" failed
HistFactory - Exception
hist2workspace - Caught Exception: HistFactory - Exception

which is correctly erroring, but also somewhat confusing.

ROOT version

   ------------------------------------------------------------------
  | Welcome to ROOT 6.26/06                        https://root.cern |
  | (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for macosx64 on Jul 28 2022, 18:08:51                      |
  | From tags/v6-26-06@v6-26-06                                      |
  | With Apple clang version 14.0.0 (clang-1400.0.29.202)            |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

Installation method

homebrew

Operating system

MacOS (13.0.1 (22A400))

Additional context

This comes out of scikit-hep/pyhf#2247 (please read that for some context).

[guitargeek]

Hi @kratsg, thanks a lot for the detailed issue report!

I see the problem, but I don't understand exactly from your post and the linked pyhf discussion what the expected behavior is. Do you expect the XML schema to be updated to allow for this initial ShapeFactor nuisance parameter values, or do you want to prohibit that such a Measurement object can even be instantiated?

Both solutions have problems in my opinion. Restricting the measurement class would break usecases, and expanding the XML schema is promoting the use of a software component (HistFactory XML) that we consider as legacy. Both changes affect code used in production and are not bugfixes (but extending the HistFactory schema), so they can't be done for ROOT patch releases. So if I fix this one way or the other, the patch would only reach users by ROOT 6.30 which is who knows how many months away.

We are currently working on a new IO format to replace the HistFactory XML, which is based on JSON (in the context of the HEP statistics serialization standard), and we plan for it to be ready for general use in ROOT 6.30 too. In that case, we can make sure that the models of the user in the pyhf issue thread are supported, such that he can share his model with his ROOT-using colleague via JSON.

Therefore, what I suggest is to not start working on this issue now, and only if it turns out late in the 6.30 development cycle that we probably will not support this usecase in the HS3 JSON for some unforeseen reason, we can extend the XML schema. This way, we're sure that this problem is solved in ROOT 6.30.

Does this sound okay to you, or do you have other suggestions? Thanks!

[kratsg]

I see the problem, but I don't understand exactly from your post and the linked pyhf discussion what the expected behavior is. Do you expect the XML schema to be updated to allow for this initial ShapeFactor nuisance parameter values, or do you want to prohibit that such a Measurement object can even be instantiated?

I think the serialization of the workspace from the ROOT side needs to be fixed (something in between both solutions you have). It's very odd to me that ROOT generates XML via PrintXML that is not self-consistent with it's own schema, but certainly it's for a somewhat defunct thing that will be replaced by the HS3 spec in the future anyway....

I don't know for sure why this wouldn't be supported in the first place (I'm somewhat confused by some of the choices in the legacy codebase for allowing some NPs to be init-ed and some not allowed to be depending on the type of systematic).