How do you search for relevant Traces?

[Dhrumil-Sentry]

Traces can be quite helpful in debugging application problems in SSR frameworks like NextJS or problems that occur in one of the many distributed services powering your apps.

As such, finding the right trace makes debugging a lot easier. We’re working on enabling this workflow in Sentry.

Coming Soon: Trace Search!

We’ll soon ship a Trace search that lets you find traces with particular spans.

E.g. you can search for traces that have database spans that take over 2s

You’ll also be able to query traces with slow spans that impact a user who complained about slow experience:

Your feedback is much appreciated

As we work on enabling this Trace Search we’d love to get your take on the following questions:

1. How do you typically query for traces?
   1. Do you use specific filters or parameters?
   2. What key information do you look for in a trace? (e.g., duration, spans, error messages, user info, etc.)
2. What are the most common queries you run? Could you share any examples?
   1. Do you query for aggregates on top of your traces? For e.g avg duration of db spans ?
3. Can you describe a recent scenario where querying for traces helped you solve a problem?

[smeubank]

I have worked with some different tools, which were typically self hosted log bucket style tools at big enterprise companies I worked at in the past, or AWS cloudwatch free tier bucket for my private (read useless) projects. I was a daily user and somewhat of a super user in my respective "tribe" for ELK stacks.

These were my top workflows:

1. find root cause: by search for example traces and sessions

• this is the core use case - ID errors and/or performance issues via human report or alerts
  • i NEED to be able to get an alert then jump into a search
• tailor the search results via some query language, and the configure the columns in the table view
• search criteria: title of span, span type (op for example), stage, sessionId, traceId
• sort chronologically obviously - see cross project
  • this can be confusing visually in search result with session vs trace, but this was crucial to understand user flow and associated signals on the timeline of user behavior in frontend and each downstream actions from the trace, both can be useful to track down root cause

2. create saved searches

• everything above, i do not want to have to do twice
• i want to be able to save a URL or similar to get back to that view whenever I need, and just input a new title, session or traceID
• this should translate well into creating alerts which ultimately could with the link as part of the alert message take me back to workflow 1 above quickly
• and to worklfow 3 below

3. create widgets for dashbaords

• all the above is me looking for patterns on some level
• if i can easily translate that into visualizations for myself and for internal comms, it would be 👌 👨‍🍳 💋

[Dhrumil-Sentry]

Thanks a lot for this feedback !

i NEED to be able to get an alert then jump into a search

Do you think we should surface relevant Traces in Issue and Metric Alerts more prominently too?

[smeubank]

yes

with the focus on traces, I'd be interested in getting somewhere with a Trace Details/Insights Page ->

• link/summarize
• list of trace samples
• widgets with stats relevant to that metric

[smeubank]

but the simple ask is something mechanical, get alert, get a link to either a preconfigure link with saved query for the relevant trace(s)

• saved queries could be perisited somehow
• or just the URL with ? query params

[drazisil-codecov]

How do you typically query for traces?

By trace id. Which doesn't work because they are dynamically sampled out due the shear volume (we have ingest at 100%)

Do you use specific filters or parameters?

Trace id, and sometimes the variables in the URL or GraphQL call

What key information do you look for in a trace? (e.g., duration, spans, error messages, user info, etc.)

I'm looking for errors, log messages, or DB calls

What are the most common queries you run? Could you share any examples?

trace:33d2697502b043babfa71c33a357525b

title:/upload/{service}/{repo}/commits codecov

Do you query for aggregates on top of your traces? For e.g avg duration of db spans ?

I do not. I'm generally focused on trouble a specific customer issue at a time

Can you describe a recent scenario where querying for traces helped you solve a problem?

"Why did we return 500 to a customer?"

"What happened with this external API call?"

"How is a customer loading a page that dies before the backend finishes fetching the data?"

[Dhrumil-Sentry]

Why did we return 500 to a customer?

@drazisil-codecov would you search for traces with status code 500 for the same API/page in case you can't find the trace for that customer?

[drazisil-codecov]

I could, but given that 500 is a general error and I don't have any other details, there no guarantee that the cause would be the same.

[Dhrumil-Sentry]

The Trace Search is available in Beta for all Orgs! Go to Explore->Traces to start searching your traces directly.

Please let us know if you have any feedback 🙏

[FlorinAndrei]

Is there a way to search for traces via the Sentry API? https://docs.sentry.io/api/

I know the span attribute (key and value), I just want to use the API programmatically to get the related traces, and extract some timing info from them.

@Dhrumil-Sentry

[Dhrumil-Sentry]

Hi, @FlorinAndrei we don't have APIs published to support fetching trace-level information today. Could you please share more details about your use case?

[FlorinAndrei]

I use Playwright https://playwright.dev/ to simulate user interaction with the website: go to site, log in, enter a search string, click Search, wait for results, declare success if a certain result appears. From Playwright I get the overall success / failure status of the whole test.

But I also want to time various steps in the process (specifically: the "wait for results" step). For this, I force tracing for these requests in Sentry, so the spans are generated with a certain span k/v attribute, which I could use then to search for the spans. The spans contain all the events in the trace, including their timing info. I am hoping to get that info out of Sentry via the API.

The workflow is easy to do via the Sentry website, but I want to automate it.

@Dhrumil-Sentry

[Dhrumil-Sentry]

Thanks for this info Florian - We don't have any plans on publishing an API to get event-level details for queried spans for the next 1-2 quarters at least but we will update here once this API is published