Add pprof profile offline symbolization tool #32496

SangJunBak · 2025-05-15T01:34:36Z

See commit messages for details

Motivation

Followup of #32423 and https://github.com/MaterializeInc/database-issues/issues/8908 where the debug tool scrapes the profiles and us, internally, can run this python script internally to spin up a docker container with a UI similar to pprof.me.

To easily test:

# Start the emulator with internal http port exposed 
docker rm materialized && docker run -d --name materialized -p
 6878:6878 materialize/materialized:latest

# Download the heap profile
curl -o envd.pprof.gz http://localhost:6878/prof/heap
# login 
aws sso login --profile mz-cloud-production-engineering-on-call
export AWS_PROFILE=mz-cloud-production-engineering-on-call

# Run the script and wait for "Serving web UI on http://localhost:8080"
bin/pyactivate -m materialize.visualize_pprof_profile.visualize_pprof_profile envd.pprof.gz

Open to suggestions given it's my first time writing a script like this!

Note: We still need to implement CPU profiling in an unsymbolized format then backport it into LTS. We only have memory profiles.

We also have to figure out why the latest binary for the emulator is like over a gigabyte if they're supposed to be stripped 😅

Tips for reviewer

Checklist

This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

SangJunBak · 2025-05-15T01:47:54Z

ci/builder/requirements.txt

@@ -36,6 +36,7 @@ pdoc==15.0.3
 # We can revert back to standard pg8000 versions once https://github.com/tlocke/pg8000/pull/161 is released
 pg8000@git+https://github.com/tlocke/pg8000@46c00021ade1d19466b07ed30392386c5f0a6b8e
 prettytable==3.16.0
+protobuf==5.29.3


Specifically used this version since that's the version used in profile.proto

This will be auto-upgraded by dependabot. Is that a problem?

SangJunBak · 2025-05-15T01:49:01Z

misc/python/materialize/visualize_pprof_profile/README.md

+
+```bash
+# Set up AWS credentials
+export AWS_PROFILE=mz-cloud-production-engineering-on-call


So for me, this is the only AWS profile with access to that bucket. For some reason, mz-core-admin doesn't work. Let me know if I should change it but I assume this is the same for most and although it's kinda suspicious using an overprivileged profile, I'd rather have one that works than not

I guess some devs won't have access to this profile. We should probably make the S3 bucket more available.

SangJunBak · 2025-05-15T01:50:53Z

misc/python/materialize/visualize_pprof_profile/profile.proto

@@ -0,0 +1,233 @@
+// Copyright 2016 Google Inc. All Rights Reserved.


As said in the readme, this is received from a CURL. Not sure if we need to do something to account for this apache license.

A potential worry is the pprof.gz provided doesn't match this proto, but given protobufs are supposed to be backwards compatible and because this .proto file (one used for pprof files) is most likely stable, I thought it'd be okay

Noticing this is failing the check-copyright lint test. I can fix this via changing the header to

// Copyright Materialize, Inc. and contributors. All rights reserved. // // Use of this software is governed by the Business Source License // included in the LICENSE file. // // As of the Change Date specified in that file, in accordance with // the Business Source License, use of this software will be governed // by the Apache License, Version 2.0.

but this seems quite suspicious to just override a copyright header like that. Any advice?

I have added an ignore for the copyright check in the linters.

SangJunBak · 2025-05-15T01:54:07Z

Would appreciate a review from @def- since you know this code the best and @teskje for an optional stakeholder/code review!

- Introduced a new tool for symbolizing and visualizing pprof profiles using debug symbols from S3. - Updated `requirements.txt` to include `protobuf==5.29.3`.

…rom copyright check

def-

Maybe I'm a bit confused of what the purpose of this is. Is it just doing the same as the instructions in "Analyzing pprof for a release build" section of https://www.notion.so/materialize/analyzing-pprof-for-a-release-build-3fa5a68aef994d90b3c94bca6eea4da8 ? That always worked well for me, I'm not sure if we need another tool for it.

def- · 2025-05-15T03:47:17Z

ci/deploy_mz-debug/README.md


 # macOS
 bin/pyactivate -m ci.deploy_mz-debug.macos

 # Linux
 bin/pyactivate -m ci.deploy_mz-debug.linux
 ```
+where AWS_PROFILE is the profile with access to the materialize-binaries S3 bucket in the Materialize Core account.


I don't think devs have access to the Materialize Core account normally? At least I don't.

What I always used to use is https://debuginfo.dev.materialize.com/buildid/d6a8b86c62ce2b7a6fd146a048ebba77/executable, which is publicly available. I think that's easier instead of using S3.

def- · 2025-05-15T03:55:05Z

misc/python/materialize/visualize_pprof_profile/visualize_pprof_profile.py

+            )
+
+        # Install graphviz in the container
+        subprocess.run(


Shouldn't this just be a Dockerfile?

SangJunBak · 2025-05-15T04:50:06Z

Maybe I'm a bit confused of what the purpose of this is. Is it just doing the same as the instructions in "Analyzing pprof for a release build" section of notion.so/materialize/analyzing-pprof-for-a-release-build-3fa5a68aef994d90b3c94bca6eea4da8 ? That always worked well for me, I'm not sure if we need another tool for it.

@def- oh no way I didn't know this existed!

I think if the public URL didn't exist, I'd argue for a tool. But given we'll most likely need a doc that points to this tool after someone generates the mz-debug zip anyways, it makes sense to close this in favor of the doc.

Another thing that isn't clear is the binaries in the debuginfo s3 bucket don't seem stripped (i.e. materilaized:latest's binary is over a gigabyte) but an extra step that the guide doesn't account for is not only supplying the path to the $PPROF_BINARY_PATH/buildid/binary, but also the debuginfo $PPROF_BINARY_PATH/debuginfo.debug. Regardless, we can probably add this step as a code block in the notion doc.

SangJunBak · 2025-05-15T11:06:47Z

On second thought: I think a good meet in the middle solution would be to generate a bash script per profile in the debug tool and expect users (us when debugging internally) to have pprof, graphviz, and curl installed. That way users can just run the script and don't have to look up a notion doc.

I have an ongoing notion doc for using the debug tool and I'll be sure to document the script as well as the manual way (the notion doc you posted). I'll create a separate PR for this and close this one out!

def- · 2025-05-15T14:06:11Z

One thing that would be nice in the script is to be able to pass all kinds of parameters to pprof that it supports.

SangJunBak requested review from def- and teskje May 15, 2025 01:34

SangJunBak marked this pull request as ready for review May 15, 2025 01:44

SangJunBak requested a review from a team as a code owner May 15, 2025 01:44

SangJunBak force-pushed the jun/#8908/symbolize-offline branch from 0dc3c62 to 3d04676 Compare May 15, 2025 01:45

SangJunBak commented May 15, 2025

View reviewed changes

SangJunBak force-pushed the jun/#8908/symbolize-offline branch 2 times, most recently from 647f017 to 6bbea6f Compare May 15, 2025 01:53

Add pprof profile visualization tool

fa2e919

- Introduced a new tool for symbolizing and visualizing pprof profiles using debug symbols from S3. - Updated `requirements.txt` to include `protobuf==5.29.3`.

SangJunBak force-pushed the jun/#8908/symbolize-offline branch from 6bbea6f to fa2e919 Compare May 15, 2025 03:06

lint: Exclude misc/python/materialize/visualize_pprof_profile files f…

e352783

…rom copyright check

def- force-pushed the jun/#8908/symbolize-offline branch from 3ee9379 to e352783 Compare May 15, 2025 03:41

def- reviewed May 15, 2025

View reviewed changes

SangJunBak closed this May 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pprof profile offline symbolization tool #32496

Add pprof profile offline symbolization tool #32496

SangJunBak commented May 15, 2025 •

edited

Loading

SangJunBak May 15, 2025

def- May 15, 2025

SangJunBak May 15, 2025

def- May 15, 2025

SangJunBak May 15, 2025 •

edited

Loading

SangJunBak May 15, 2025

def- May 15, 2025

SangJunBak commented May 15, 2025 •

edited

Loading

def- left a comment

def- May 15, 2025

def- May 15, 2025

def- May 15, 2025

SangJunBak commented May 15, 2025 •

edited

Loading

SangJunBak commented May 15, 2025 •

edited

Loading

def- commented May 15, 2025

		@@ -0,0 +1,233 @@
		// Copyright 2016 Google Inc. All Rights Reserved.

Add pprof profile offline symbolization tool #32496

Add pprof profile offline symbolization tool #32496

Conversation

SangJunBak commented May 15, 2025 • edited Loading

Motivation

Tips for reviewer

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak May 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented May 15, 2025 • edited Loading

def- left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SangJunBak commented May 15, 2025 • edited Loading

SangJunBak commented May 15, 2025 • edited Loading

def- commented May 15, 2025

SangJunBak commented May 15, 2025 •

edited

Loading

SangJunBak May 15, 2025 •

edited

Loading

SangJunBak commented May 15, 2025 •

edited

Loading

SangJunBak commented May 15, 2025 •

edited

Loading

SangJunBak commented May 15, 2025 •

edited

Loading