Trim event duration based on tree hierarchy in CallGraph lib#274
Closed
hjli-creator wants to merge 1 commit intofacebookresearch:mainfrom
Closed
Trim event duration based on tree hierarchy in CallGraph lib#274hjli-creator wants to merge 1 commit intofacebookresearch:mainfrom
hjli-creator wants to merge 1 commit intofacebookresearch:mainfrom
Conversation
meta-cla
bot
added
the
CLA Signed
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D78594843 |
hjli-creator
added a commit
to hjli-creator/HolisticTraceAnalysis
that referenced
this pull request
Jul 19, 2025
…kresearch#274) Summary: ### What? This commit improves CallStackGraph lib in HTA package to handle edge case from Kineto trace data: In traces with Python call stack info, when child Python function doesn't end before parent, it disturbs the assumed time order of events which is very important when constructing call stack graph. Because we know a child function cannot end after its parent, if it is found in the trace it is basically wrong information and we can preprocess the data by truncating the event duration. We can rely on Python event argument "Python id" and "Python parent id" to detect parent-child, then it is just a BFS tree traversal, and for each child, make sure its end time doesn't exceed its parent. ### Why? Incorrect call stack tree structure impacts the consumers of CallStackGraph, for example accuracy of Python overhead detection, because it depends on the leaf event interval. We have seen a few traces with unexpected high overhead that turned out incorrect. The CPA (Critical Path Analysis) also depends on CallStackGraph. ### How? 1. First we need to add "Python id" and "Python parent id" into the trace DataFrame. 1. Then we introduce a boolean flag "preprocess_trace_data", if true then we run the truncating algorithm before constructing CallStackGraph. 1. The algorithm is just BFS, first scan the DataFrame to create parent child edges, then BFS and truncate each child event's duration if needed. Note if a parent function invokes a child in different thread, child's duration doesn't need to be truncated so it is treated as a root event. Differential Revision: D78594843
hjli-creator
force-pushed
the
export-D78594843
branch
from
61af952 to
ffe3292
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D78594843 |
…kresearch#274) Summary: ### What? This commit improves CallStackGraph lib in HTA package to handle edge case from Kineto trace data: In traces with Python call stack info, when child Python function doesn't end before parent, it disturbs the assumed time order of events which is very important when constructing call stack graph. Because we know a child function cannot end after its parent, if it is found in the trace it is basically wrong information and we can preprocess the data by truncating the event duration. We can rely on Python event argument "Python id" and "Python parent id" to detect parent-child, then it is just a BFS tree traversal, and for each child, make sure its end time doesn't exceed its parent. ### Why? Incorrect call stack tree structure impacts the consumers of CallStackGraph, for example accuracy of Python overhead detection, because it depends on the leaf event interval. We have seen a few traces with unexpected high overhead that turned out incorrect. The CPA (Critical Path Analysis) also depends on CallStackGraph. ### How? 1. First we need to add "Python id" and "Python parent id" into the trace DataFrame. 1. Then we introduce a boolean flag "preprocess_trace_data", if true then we run the truncating algorithm before constructing CallStackGraph. 1. The algorithm is just BFS, first scan the DataFrame to create parent child edges, then BFS and truncate each child event's duration if needed. Note if a parent function invokes a child in different thread, child's duration doesn't need to be truncated so it is treated as a root event. Differential Revision: D78594843
hjli-creator
force-pushed
the
export-D78594843
branch
from
ffe3292 to
d5c5533
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D78594843 |
Contributor
|
This pull request has been merged in 3011e1b. |
DaylonSrinivasan
pushed a commit
to DaylonSrinivasan/HolisticTraceAnalysis
that referenced
this pull request
Jul 29, 2025
Summary: Reverts facebookresearch#274, which caused a missing key error: "index::python_id" Reviewed By: hjli-creator Differential Revision: D79185506
DaylonSrinivasan
pushed a commit
to DaylonSrinivasan/HolisticTraceAnalysis
that referenced
this pull request
Jul 29, 2025
Summary: Reverts facebookresearch#274, which caused a missing key error: "index::python_id" Reviewed By: hjli-creator Differential Revision: D79185506
facebook-github-bot
pushed a commit
that referenced
this pull request
Jul 29, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
What?
This commit improves CallStackGraph lib in HTA package to handle edge case from Kineto trace data: In traces with Python call stack info, when child Python function doesn't end before parent, it disturbs the assumed time order of events which is very important when constructing call stack graph. Because we know a child function cannot end after its parent, if it is found in the trace it is basically wrong information and we can preprocess the data by truncating the event duration. We can rely on Python event argument "Python id" and "Python parent id" to detect parent-child, then it is just a BFS tree traversal, and for each child, make sure its end time doesn't exceed its parent.
Why?
Incorrect call stack tree structure impacts the consumers of CallStackGraph, for example accuracy of Python overhead detection, because it depends on the leaf event interval. We have seen a few traces with unexpected high overhead that turned out incorrect. The CPA (Critical Path Analysis) also depends on CallStackGraph.
How?
Differential Revision: D78594843