Fixing wrong streamer info

[maxnoe]

Trying to read a ROOT file with custom objects inside, I get this error:

DeserializationError: while reading

    MHMcCollectionArea version 14 as uproot.dynamic.Model_MHMcCollectionArea_v14 (14798723 bytes)
        MH version 2 as uproot.dynamic.Model_MH_v2 (17 bytes)
            MParContainer version 0 as uproot.dynamic.Model_MParContainer_v0 (6 bytes)
                (base): <TObject None None at 0x7f74b59c1ac0>
Base classes for MParContainer: (TObject)

expected 6 bytes but cursor moved by 18 bytes (through MParContainer)
in file ./Output_flute.root
in object /MHMcCollectionAreaEtrue;1

Do I interpret this correctly that something is wrong with the streamer info for the MParContainer base class?

Any hints on what I can try to fix would be appreciated.

[jpivarski]

The DeserializationError print-out shows what has been deserialized up to the point where it encountered an error, which usually provides some clues. In this case, it thought that the MParContainer's version is 0, which is odd because most version numbers start with 1. (I don't know if that is a necessary rule.) The fact that both of the TObject members, fProcessID and fBits, are both None, is also very unusual. I would guess that the actual error happened before getting to MParContainer.

Is 14798723 a reasonable number of bytes for MHMcCollectionArea? That would be 14 MiB, something that carries a real dataset, maybe some arrays or lots of small objects.

The next steps for digging more deeply is to print out the streamers for these objects. If you've opened a file as f, that's the ReadOnlyDirectory; you can get streamers from the ReadOnlyFile within it as f.file, using the ReadOnlyFile.show_streamers method:

f.file.show_streamers()

or to narrow in on MParContainer and its dependencies,

f.file.show_streamers("MParContainer")

Each of these is a prescription for how-many-bytes-per-field in each object, and base classes ("BASE") are serialized as nested objects.

For TTree data (which this is not), there's a debug method to look at the raw bytes and diagnose what went wrong. (For TTrees, it is more difficult to get the correct interpretation of the data because the type information is spread around among TBranch metadata, TLeaf metadata, and streamers. For non-TTree data, it's always just streamers.) We don't have a method for looking at the raw bytes of an object, but here's a way to do it, if you want to give it a try:

key = f.key("path/to/object")

will give you a uproot.reading.ReadOnlyKey, and

chunk, cursor = key.get_uncompressed_chunk_cursor()

does all the reading and decompression but not any of the interpretation, which fails for your object. Then

cursor.debug(chunk, limit_bytes=80)

prints out raw bytes of the object, up to limit_bytes=80, with both decimal and character interpretations. The character interpretation is good for spotting words, like names, titles, and classnames, while the decimal interpretation is good for finding relevant (big-endian) numbers. This Cursor.debug method has a lot of options; it's sometimes also useful to set a dtype to interpret the data as floating point numbers. Each object begins with a byte count (e.g. the 14798723 bytes for MHMcCollectionArea), which are easy to spot because they show up as 64 decimals. ROOT's flag to indicate a byte count is a 64 in the high byte:

>>> np.array([uproot.const.kByteCountMask], ">i4").view("u1")
array([64,  0,  0,  0], dtype=uint8)

... Or you can post a small file and I'll look into it when I can find the time. But the above procedure is what I would do: (1) print out the streamers, (2) print out the raw bytes, and (3) follow the raw bytes through the streamers to see what assumption is wrong.

The code generated from the streamers is a class returned by

f.file.class_of("MHMcCollectionArea")   # also in uproot.dynamic after the first time you attempt to read it

which has a class_code attribute (string) that you can print out. That auto-generated code is what is failing. The wrong assumption would be in the translation of streamer data into that Python code.

[maxnoe]

Thanks for the quick and detailed response, I have a deeper look later.

Looking at the code, MParContainer indeed has a ClassDef(MParContainer, 0), and the size of 14MB uncompressed is also not unrealistic, the file is 11MB in total and most of the data should be in that object.

I'll try if I can figure something out, I can also look into the file using the software used to write it, if that helps narrowing it down (proprietary software build on root)

[maxnoe]

Looking at the f.file.show_streamers, I get this for f.file.show_streamers("MH"):

In [15]: f.file.show_streamers('MH')
TObject (v1)
    fUniqueID: unsigned int (TStreamerBasicType)
    fBits: unsigned int (TStreamerBasicType)

MParContainer (v0): TObject (v1)

MH (v2): MParContainer (v0)
    fSerialNumber: unsigned char (TStreamerBasicType)
    fNumExecutions: unsigned int (TStreamerBasicType)

But a KeyError(1) for f.file.show_streamers('MHMcCollectionArea'): (not the full TB):

~/.local/lib/python3.9/site-packages/uproot/streamers.py in _dependencies(self, streamers, out)
    452         out.append((self.name, self.class_version))
    453         for element in self.elements:
--> 454             element._dependencies(streamers, out)
    455 
    456 

~/.local/lib/python3.9/site-packages/uproot/streamers.py in _dependencies(self, streamers, out)
    770                 streamer = streamer_versions[max(streamer_versions)]
    771             else:
--> 772                 streamer = streamer_versions[base_version]
    773             if (
    774                 streamer is not None

However, in the full f.file.show_streamers, this appears for MHMcCollectionArea:

MHMcCollectionArea (v14): MH (v2)
    fHistAllorig: TH3D* (TStreamerObjectPointer)
    fHistAll: TH3D* (TStreamerObjectPointer)
    fHistWeights: TH3D* (TStreamerObjectPointer)
    fHistSel: TH3D* (TStreamerObjectPointer)
    fHistCol: TH3D* (TStreamerObjectPointer)
    fHistColCoarse: TH2D* (TStreamerObjectPointer)
    fHistEestOverEtrue: TH2D* (TStreamerObjectPointer)
    fHistRadOverEtrue: TH2D* (TStreamerObjectPointer)
    fIsFineBinOK: TH2C* (TStreamerObjectPointer)
    fHistSelInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin_orig: TClonesArray (TStreamerObject)
    fAreaMax: double (TStreamerBasicType)
    fMinEvents: int (TStreamerBasicType)
    fMaxRelError: float (TStreamerBasicType)
    fMinFractionOfGoodBins: float (TStreamerBasicType)
    fMaxRelEtrueDepartureFromEest: float (TStreamerBasicType)
    kWobblePatch: bool (TStreamerBasicType)
    fAzBins: int (TStreamerBasicType)
    fUseEnergyEst: bool (TStreamerBasicType)
    fMcEventsNtuple: TNtupleD* (TStreamerObjectPointer)
    fStoreMCEvents: bool (TStreamerBasicType)
    fMcRingAnalysis: bool (TStreamerBasicType)
    fOffsetRingMin: double (TStreamerBasicType)
    fOffsetRingMax: double (TStreamerBasicType)
    fOffsetRingMed: double (TStreamerBasicType)
    fBinsx: MBinning (TStreamerObject)
    fBinsy: MBinning (TStreamerObject)
    fBinsz: MBinning (TStreamerObject)

[jpivarski]

So when it tries to evaluate streamer_versions[base_version], it's passing base_version = 1 (which does not exist) instead of 2 (which does, and is what's clearly listed in the full print-out)?

That's definitely sounding like some Uproot bug. Could you get the 11 MB file to me somehow (here, publicly, or to jpivarski at GMail, privately) and I'll at least figure out why it thinks the dependency has the wrong base_version?

[maxnoe]

Here is the file (gzipped for github):
Output_flute.root.gz

[jpivarski]

I just got a chance to look into it, and it's looking more like a weird file than a bug in Uproot.

Usually, we're able to show all the streamers with their dependencies—I've been focusing first on this problem of hitting a KeyError when trying to walk up the dependency chain. To print out one streamer without dependencies, you can do:

>>> f.file.streamers["MHMcCollectionArea"][14].show()
MHMcCollectionArea (v14): MH (v2)
    fHistAllorig: TH3D* (TStreamerObjectPointer)
    fHistAll: TH3D* (TStreamerObjectPointer)
    fHistWeights: TH3D* (TStreamerObjectPointer)
    fHistSel: TH3D* (TStreamerObjectPointer)
    fHistCol: TH3D* (TStreamerObjectPointer)
    fHistColCoarse: TH2D* (TStreamerObjectPointer)
    fHistEestOverEtrue: TH2D* (TStreamerObjectPointer)
    fHistRadOverEtrue: TH2D* (TStreamerObjectPointer)
    fIsFineBinOK: TH2C* (TStreamerObjectPointer)
    fHistSelInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin_orig: TClonesArray (TStreamerObject)
    fAreaMax: double (TStreamerBasicType)
    fMinEvents: int (TStreamerBasicType)
    fMaxRelError: float (TStreamerBasicType)
    fMinFractionOfGoodBins: float (TStreamerBasicType)
    fMaxRelEtrueDepartureFromEest: float (TStreamerBasicType)
    kWobblePatch: bool (TStreamerBasicType)
    fAzBins: int (TStreamerBasicType)
    fUseEnergyEst: bool (TStreamerBasicType)
    fMcEventsNtuple: TNtupleD* (TStreamerObjectPointer)
    fStoreMCEvents: bool (TStreamerBasicType)
    fMcRingAnalysis: bool (TStreamerBasicType)
    fOffsetRingMin: double (TStreamerBasicType)
    fOffsetRingMax: double (TStreamerBasicType)
    fOffsetRingMed: double (TStreamerBasicType)
    fBinsx: MBinning (TStreamerObject)
    fBinsy: MBinning (TStreamerObject)
    fBinsz: MBinning (TStreamerObject)

You had to first look at the f.file.streamers["MHMcCollectionArea"] dict to see that the only defined version number was 14, or maybe use ReadOnlyFile.streamer_named, which gives you the maximum version in a file if you don't specify.

The KeyError happens because TNtupleD (above) depends on TTree version 19, which depends on TAttLine version 1, but your file only has version 2 (brought in by the histograms). Maybe ROOT doesn't write multiple versions of the same class's streamer to a file? At the very least, I should put in some contingency in the dependency-walking code to not crash if the version is not available.

That will be a PR soon, but for now it shows you what you're working with:

>>> f.file.show_streamers("MHMcCollectionArea")
MBinning (v1): MParContainer (v0)
    fEdges: TArrayD (TStreamerObjectAny)
    fType: unsigned char (TStreamerBasicType)

TRefTable (v3): TObject (v1)
    fSize: int (TStreamerBasicType)
    fParents: TObjArray* (TStreamerObjectPointer)
    fOwner: TObject* (TStreamerObjectPointer)
    fProcessGUIDs: vector<string> (TStreamerSTL)

TBranch (v12): TNamed (v1), TAttFill (v1)
    fCompress: int (TStreamerBasicType)
    fBasketSize: int (TStreamerBasicType)
    fEntryOffsetLen: int (TStreamerBasicType)
    fWriteBasket: int (TStreamerBasicType)
    fEntryNumber: long long (TStreamerBasicType)
    fOffset: int (TStreamerBasicType)
    fMaxBaskets: int (TStreamerBasicType)
    fSplitLevel: int (TStreamerBasicType)
    fEntries: long long (TStreamerBasicType)
    fFirstEntry: long long (TStreamerBasicType)
    fTotBytes: long long (TStreamerBasicType)
    fZipBytes: long long (TStreamerBasicType)
    fBranches: TObjArray (TStreamerObject)
    fLeaves: TObjArray (TStreamerObject)
    fBaskets: TObjArray (TStreamerObject)
    fBasketBytes: int* (TStreamerBasicPointer)
    fBasketEntry: long long* (TStreamerBasicPointer)
    fBasketSeek: long long* (TStreamerBasicPointer)
    fFileName: TString (TStreamerString)

TBranchRef (v1): TBranch (v12)
    fRefTable: TRefTable* (TStreamerObjectPointer)

TObjArray (v3): TSeqCollection (v0)
    fLowerBound: int (TStreamerBasicType)
    fLast: int (TStreamerBasicType)

TTree (v19): TNamed (v1), TAttLine (v1), TAttFill (v1), TAttMarker (v2)
    fEntries: long long (TStreamerBasicType)
    fTotBytes: long long (TStreamerBasicType)
    fZipBytes: long long (TStreamerBasicType)
    fSavedBytes: long long (TStreamerBasicType)
    fFlushedBytes: long long (TStreamerBasicType)
    fWeight: double (TStreamerBasicType)
    fTimerInterval: int (TStreamerBasicType)
    fScanField: int (TStreamerBasicType)
    fUpdate: int (TStreamerBasicType)
    fDefaultEntryOffsetLen: int (TStreamerBasicType)
    fNClusterRange: int (TStreamerBasicType)
    fMaxEntries: long long (TStreamerBasicType)
    fMaxEntryLoop: long long (TStreamerBasicType)
    fMaxVirtualSize: long long (TStreamerBasicType)
    fAutoSave: long long (TStreamerBasicType)
    fAutoFlush: long long (TStreamerBasicType)
    fEstimate: long long (TStreamerBasicType)
    fClusterRangeEnd: long long* (TStreamerBasicPointer)
    fClusterSize: long long* (TStreamerBasicPointer)
    fBranches: TObjArray (TStreamerObject)
    fLeaves: TObjArray (TStreamerObject)
    fAliases: TList* (TStreamerObjectPointer)
    fIndexValues: TArrayD (TStreamerObjectAny)
    fIndex: TArrayI (TStreamerObjectAny)
    fTreeIndex: TVirtualIndex* (TStreamerObjectPointer)
    fFriends: TList* (TStreamerObjectPointer)
    fUserInfo: TList* (TStreamerObjectPointer)
    fBranchRef: TBranchRef* (TStreamerObjectPointer)

TNtupleD (v1): TTree (v19)
    fNvar: int (TStreamerBasicType)

TH2C (v3): TH2 (v4), TArrayC (v1)

TH2 (v4): TH1 (v7)
    fScalefactor: double (TStreamerBasicType)
    fTsumwy: double (TStreamerBasicType)
    fTsumwy2: double (TStreamerBasicType)
    fTsumwxy: double (TStreamerBasicType)

TH2D (v3): TH2 (v4), TArrayD (v1)

TAtt3D (v1)

TCollection (v3): TObject (v1)
    fName: TString (TStreamerString)
    fSize: int (TStreamerBasicType)

TSeqCollection (v0): TCollection (v3)

TList (v5): TSeqCollection (v0)

THashList (v0): TList (v5)

TArray (v1)
    fN: int (TStreamerBasicType)

TArrayD (v1): TArray (v1)
    fArray: double* (TStreamerBasicPointer)

TAttAxis (v4)
    fNdivisions: int (TStreamerBasicType)
    fAxisColor: short (TStreamerBasicType)
    fLabelColor: short (TStreamerBasicType)
    fLabelFont: short (TStreamerBasicType)
    fLabelOffset: float (TStreamerBasicType)
    fLabelSize: float (TStreamerBasicType)
    fTickLength: float (TStreamerBasicType)
    fTitleOffset: float (TStreamerBasicType)
    fTitleSize: float (TStreamerBasicType)
    fTitleColor: short (TStreamerBasicType)
    fTitleFont: short (TStreamerBasicType)

TAxis (v9): TNamed (v1), TAttAxis (v4)
    fNbins: int (TStreamerBasicType)
    fXmin: double (TStreamerBasicType)
    fXmax: double (TStreamerBasicType)
    fXbins: TArrayD (TStreamerObjectAny)
    fFirst: int (TStreamerBasicType)
    fLast: int (TStreamerBasicType)
    fBits2: unsigned short (TStreamerBasicType)
    fTimeDisplay: bool (TStreamerBasicType)
    fTimeFormat: TString (TStreamerString)
    fLabels: THashList* (TStreamerObjectPointer)

TAttMarker (v2)
    fMarkerColor: short (TStreamerBasicType)
    fMarkerStyle: short (TStreamerBasicType)
    fMarkerSize: float (TStreamerBasicType)

TAttFill (v2)
    fFillColor: short (TStreamerBasicType)
    fFillStyle: short (TStreamerBasicType)

TAttLine (v2)
    fLineColor: short (TStreamerBasicType)
    fLineStyle: short (TStreamerBasicType)
    fLineWidth: short (TStreamerBasicType)

TString (v2)

TNamed (v1): TObject (v1)
    fName: TString (TStreamerString)
    fTitle: TString (TStreamerString)

TH1 (v7): TNamed (v1), TAttLine (v2), TAttFill (v2), TAttMarker (v2)
    fNcells: int (TStreamerBasicType)
    fXaxis: TAxis (TStreamerObject)
    fYaxis: TAxis (TStreamerObject)
    fZaxis: TAxis (TStreamerObject)
    fBarOffset: short (TStreamerBasicType)
    fBarWidth: short (TStreamerBasicType)
    fEntries: double (TStreamerBasicType)
    fTsumw: double (TStreamerBasicType)
    fTsumw2: double (TStreamerBasicType)
    fTsumwx: double (TStreamerBasicType)
    fTsumwx2: double (TStreamerBasicType)
    fMaximum: double (TStreamerBasicType)
    fMinimum: double (TStreamerBasicType)
    fNormFactor: double (TStreamerBasicType)
    fContour: TArrayD (TStreamerObjectAny)
    fSumw2: TArrayD (TStreamerObjectAny)
    fOption: TString (TStreamerString)
    fFunctions: TList* (TStreamerObjectPointer)
    fBufferSize: int (TStreamerBasicType)
    fBuffer: double* (TStreamerBasicPointer)
    fBinStatErrOpt: TH1::EBinErrorOpt (TStreamerBasicType)

TH3 (v5): TH1 (v7), TAtt3D (v1)
    fTsumwy: double (TStreamerBasicType)
    fTsumwy2: double (TStreamerBasicType)
    fTsumwxy: double (TStreamerBasicType)
    fTsumwz: double (TStreamerBasicType)
    fTsumwz2: double (TStreamerBasicType)
    fTsumwxz: double (TStreamerBasicType)
    fTsumwyz: double (TStreamerBasicType)

TH3D (v3): TH3 (v5), TArrayD (v1)

TObject (v1)
    fUniqueID: unsigned int (TStreamerBasicType)
    fBits: unsigned int (TStreamerBasicType)

MParContainer (v0): TObject (v1)

MH (v2): MParContainer (v0)
    fSerialNumber: unsigned char (TStreamerBasicType)
    fNumExecutions: unsigned int (TStreamerBasicType)

MHMcCollectionArea (v14): MH (v2)
    fHistAllorig: TH3D* (TStreamerObjectPointer)
    fHistAll: TH3D* (TStreamerObjectPointer)
    fHistWeights: TH3D* (TStreamerObjectPointer)
    fHistSel: TH3D* (TStreamerObjectPointer)
    fHistCol: TH3D* (TStreamerObjectPointer)
    fHistColCoarse: TH2D* (TStreamerObjectPointer)
    fHistEestOverEtrue: TH2D* (TStreamerObjectPointer)
    fHistRadOverEtrue: TH2D* (TStreamerObjectPointer)
    fIsFineBinOK: TH2C* (TStreamerObjectPointer)
    fHistSelInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin: TClonesArray (TStreamerObject)
    fHistColInEestBin_orig: TClonesArray (TStreamerObject)
    fAreaMax: double (TStreamerBasicType)
    fMinEvents: int (TStreamerBasicType)
    fMaxRelError: float (TStreamerBasicType)
    fMinFractionOfGoodBins: float (TStreamerBasicType)
    fMaxRelEtrueDepartureFromEest: float (TStreamerBasicType)
    kWobblePatch: bool (TStreamerBasicType)
    fAzBins: int (TStreamerBasicType)
    fUseEnergyEst: bool (TStreamerBasicType)
    fMcEventsNtuple: TNtupleD* (TStreamerObjectPointer)
    fStoreMCEvents: bool (TStreamerBasicType)
    fMcRingAnalysis: bool (TStreamerBasicType)
    fOffsetRingMin: double (TStreamerBasicType)
    fOffsetRingMax: double (TStreamerBasicType)
    fOffsetRingMed: double (TStreamerBasicType)
    fBinsx: MBinning (TStreamerObject)
    fBinsy: MBinning (TStreamerObject)
    fBinsz: MBinning (TStreamerObject)

[jpivarski]

PR #437.