Cannot process simple Caliper sampling file

[mkre]

I created a Caliper sampling profile with
CALI_CONFIG=hatchet-sample-profile,source.function=true,source.location=true,source.module=true,sample.frequency=1000 ./a.out
using Caliper's cxx-example.cpp (see here).

The sampling output file is attached here: sample_profile.cali.txt
(I had to add .txt extension to be able to upload it.)

Next, I tried printing a tree of the recorded data using the following script (using from_caliperreader based on #161):

#! /usr/bin/env python3

import hatchet as ht

if __name__ == '__main__':
    gf = ht.GraphFrame.from_caliperreader('sample_profile.cali')
    print(gf.tree())

This resulted in the following error:

Traceback (most recent call last):
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: Node({'name': 'main', 'type': 'function'})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./hatchet_test.py", line 9, in <module>
    print(gf.tree())
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/hatchet/util/deprecated.py", line 14, in wrapper
    return f(*args, **kwargs)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/hatchet/graphframe.py", line 1050, in tree
    return ConsoleRenderer(unicode=unicode, color=color).render(
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/hatchet/external/console.py", line 153, in render
    result += self.render_frame(root, dataframe)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/hatchet/external/console.py", line 264, in render_frame
    node_metric = dataframe.loc[df_index, self.primary_metric]
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexing.py", line 1097, in __getitem__
    return self._getitem_tuple(key)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexing.py", line 1280, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexing.py", line 1000, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexing.py", line 1343, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexing.py", line 1293, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/generic.py", line 4095, in xs
    loc = index.get_loc(key)
  File "/u/ydfb4q/.pyenv/versions/3.8.5/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: Node({'name': 'main', 'type': 'function'})

[michaelmckinsey1]

@mkre It appears the cxx-example.cpp program executes too quickly for the sampler to profile main and mainloop. The gf.dataframe in your profile contains performance information for only foo, and the calltree gf.graph contains regions main, mainloop, and foo. The gf.tree() function prints the performance information alongside the calltree, so it errors because of the missing performance information. See this profile where I added dummy loops to give the sampler time to profile the main and mainloop missing functions sample_profile.cali.txt. You can also increase the sampling frequency parameter, however there is a limit on precision.

To visualize the tree for your profile, you can try the GraphFrame.squash() function, to match the graph regions to the regions in the dataframe, e.g.:

#! /usr/bin/env python3

import hatchet as ht

if __name__ == '__main__':
    gf = ht.GraphFrame.from_caliperreader('sample_profile.cali')
    gf = gf.squash()
    print(gf.tree())

If sampling is not required for your analysis, you can instead use the hatchet-region-profile configuration to see a performance profile of the execution.

[mkre]

Thanks, that helps to circumvent the error, and wasn't quite obvious from the documentation. However, now I have issues seeing all the relevant data in the Hatchet output with my real-life application. Not sure if it is related to the issue reported above. Please let me know if I should create a separate ticket for that.

Running cali-query with

select
  source.function#cali.sampler.pc as Function,
  sum(count) as Samples,
  sum(scount) as Time,
  percent_total(count) as Percent
group by
  source.function#cali.sampler.pc
format
  table
order by
  sum#count DESC

to create a flat profile as described here gives me the following 5 hot spots:

Function                                                     Samples Time     Percent
AMGSystem::GaussSeidel::backw~~const&, AMGArgs const&) const    4912 4.912000 7.232144
AMGSystem::GaussSeidel::forwa~~const&, AMGArgs const&) const    4221 4.221000 6.214756
void DiffusionFlux::evalInter~~on const&, bool const&) const    4162 4.162000 6.127888
void MomentumEquation::Viscou~~lux<3>(FvRegion const&) const    3711 3.711000 5.463861

However, following the steps described here to create a flat profile with Hatchet gives different hot spots using the sample sample_profile.cali file:

                                                      nid   count   time
name
FaceTerm::MomentumEquation::ViscousFlux              70.0  3711.0  3.711
FaceTerm::PressureGradientEquation::AccumulateW...   64.0  2186.0  2.186
FaceTerm::DeltaVDissipationEquation::Accumulate...   72.0  2168.0  2.168
FaceTerm::TurbDiffusionTerm<TurbulentKineticEne...  141.0  2091.0  2.091
FaceTerm::TurbDiffusionTerm<SpecificDissipation...  150.0  2071.0  2.071

None of the AMGSystem::... functions are present. I also cannot see these functions in the tree visualization (after squashing to get around the original error).

What am I missing here?

[slabasan]

Hi @mkre! Would you be able to share your file so we can take a look?

[mkre]

@slabasan sure, here you go: sample_profile.cali.txt

(remove the .txt extension)

[michaelmckinsey1]

Hi @mkre, sorry for the delay. Thank you for sharing the profiles with us. I believe we were able to come up with a solution with the issues that you were seeing here, and that fix has now been merged in develop (#169). If this is still not working for you, please let us know.

[mkre]

Thanks @michaelmckinsey1. I'll give it a try as soon as a new release has been tagged.

[slabasan]

Hi @mkre: Are you just needing a release tagged? Or do you need the release to be up on pypi?

[mkre]

Having it on pypi would be most convenient.

[slabasan]

Hi @mkre: A new llnl-hatchet==2025.1.1 release is available on pypi.

[mkre]

Thanks. I finally had a chance to test the new version. Using the example data from above, there is still a mismatch between the flat profiles obtained from cali-query vs hatchet:

> cali-query -Q cali-query.txt sample_profile.cali | (head -n5; echo "..."; tail -n5)
Function                                                     Samples Time     Percent
AMGSystem::GaussSeidel::backw~~const&, AMGArgs const&) const    5028 5.028000 7.440842
AMGSystem::GaussSeidel::forwa~~const&, AMGArgs const&) const    4173 4.173000 6.175543
void DiffusionFlux::evalInter~~on const&, bool const&) const    4114 4.114000 6.088231
void MomentumEquation::Viscou~~lux<3>(FvRegion const&) const    3800 3.800000 5.623548
...
void StarMklField::FieldEngin~~bool, true> >&, float const&)       1 0.001000 0.001480
AbstractPartitionConfigManage~~rtitionConfigManager>() const       1 0.001000 0.001480
ConvectionFlux::evalPermeable~~dary(FvBoundary const&) const       1 0.001000 0.001480
__cxxabiv1::__si_class_type_i~~nfo::__dyncast_result&) const       1 0.001000 0.001480
Subject::notify(Properties&)                                       1 0.001000 0.001480

cali-query.txt

> python hatchet_flat_profile.py
/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/hatchet/graphframe.py:495: FutureWarning: The provided callable <function mean at 0x7f22d52e0510> is currently using SeriesGroupBy.mean. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "mean" instead.
  agg_df = self.dataframe.groupby("node").agg(agg_dict)
                                                      nid   count   time
name
void MomentumEquation::ViscousFlux::interiorFlu...   97.0  3800.0  3.800
AMGSystem::makeNextLevel(unsigned int const&, u...  145.0  3645.0  3.645
AMGSystem::residual(std::vector<float, std::all...  334.0  3386.0  3.386
void PressureGradientEquation::AccumulateWeight...   90.0  2167.0  2.167
void DeltaVDissipationEquation::AccumulateDissi...  104.0  2155.0  2.155
...                                                   ...     ...    ...
FvTerm::~FvTerm()                                   450.0     1.0  0.001
void StarMklField::FieldEngine<StarMklField::As...  369.0     1.0  0.001
void StarMklField::FieldEngine<StarMklField::As...  363.0     1.0  0.001
void StarMklField::FieldEngine<StarMklField::As...  424.0     1.0  0.001
void StarMklField::FieldEngine<StarMklField::As...  393.0     1.0  0.001

[321 rows x 3 columns]

hatchet_flat_profile.py

Also, when I try to create a flamegraph, hatchet is crashing:

> python hatchet_flamegraph.py
Traceback (most recent call last):
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    return self._engine.get_loc(casted_key)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "pandas/_libs/index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 191, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 228, in pandas._libs.index.IndexEngine._get_loc_duplicates
KeyError: Node({'name': 'SimulationIterator::startSimulation', 'type': 'function'})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/u/ydfb4q/hatchet_flamegraph.py", line 8, in <module>
    folded_stack.write(gf.to_flamegraph())
                       ~~~~~~~~~~~~~~~~^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/hatchet/graphframe.py", line 1149, in to_flamegraph
    folded_stack + str(self.dataframe.loc[df_index, "name"]) + " "
                       ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexing.py", line 1185, in __getitem__
    return self._getitem_tuple(key)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexing.py", line 1369, in _getitem_tuple
    return self._getitem_lowerdim(tup)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexing.py", line 1066, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexing.py", line 1432, in _getitem_axis
    return self._get_label(key, axis=axis)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexing.py", line 1382, in _get_label
    return self.obj.xs(label, axis=axis)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/generic.py", line 4323, in xs
    loc = index.get_loc(key)
  File "/u/ydfb4q/venv/caliper-hatchet/lib/python3.14/site-packages/pandas/core/indexes/base.py", line 3819, in get_loc
    raise KeyError(key) from err
KeyError: Node({'name': 'SimulationIterator::startSimulation', 'type': 'function'})

hatchet_flamegraph.py

Any idea what's going on here? I am happy to provide more information.

I don't have permissions to reopen this issue. Let me know if I should create a new one instead.