-
Notifications
You must be signed in to change notification settings - Fork 29
Description
Is your feature request related to a problem? Please describe
When filtering traces in the Jaeger UI, we are currently unable to use several helpful search operators or search through span logs. The lack of these advanced trace querying capabilities makes it difficult to filter out noise, limits our ability to pinpoint specific errors, and slows down diagnostic workflows.
Describe the solution you'd like
I would like the VictoriaMetrics backend to parse and support the following search operations originating from the Jaeger UI. I have broken these down into four specific requests:
1. Log Search
Request: Enable querying against span logs in addition to span tags.
This allows searching through textual log data attached to spans, rather than just indexed tags, enabling us (especially, support engineers 🙂 ) to find raw payloads and unstructured errors. Since this capability exists in the Jaeger OpenSearch backend, I'm hoping it can be replicated here.
Example Queries:
exception.message="The first argument must be.*"
exception.type=ERR_INVALID_ARG_TYPE
Use Cases:
- Finding traces where an application logged a specific error message or payload (like a raw transaction ID) that wasn't captured as a formalized span tag.
- Identifying spans where a background job or worker task reported a specific lifecycle event in its logs (e.g., "Starting payment sync for user_123").
- Tracing intermittent issues where the only symptom is a specific warning string emitted in the application logs, rather than a formal exception or error tag.
2. Regex / Wildcard Matching
Request: Allow searching for spans using partial string matches.
This enables matching dynamic or highly variable data (like changing URLs or generated IDs) without needing exact string matches. This is supported by the Jaeger OpenSearch backend as well.
Example Queries:
exception.message="The first argument must be.*"
http.user_agent=".*Chrome.*"
http.route="example.com/api/v.*"
Use Cases:
- Finding all traces with a specific exception prefix, even if the exact trailing message or stack trace varies dynamically.
- Searching for dynamic tags like User-Agents or API versions where only a portion of the string is known or relevant.
- Filtering for dynamically generated HTTP endpoints, paths, or database table names that share a common pattern.
3. OR Operator
Request: Support the pipe character | to filter for multiple specific values within a single tag.
This allows a single query to match spans containing any of a specified list of values, eliminating the need to run multiple disjointed queries. This is supported by the Jaeger OpenSearch backend as well.
Example Queries:
http.status_code=500|501
tenant.id=tenant_A|tenant_B
service.version=v1.2.0|v1.2.1
Use Cases:
- Filtering for multiple specific HTTP error status codes simultaneously (e.g., checking for 500, 501, or 503 errors in a single view).
- Finding traces belonging to a specific subset of user IDs, tenant IDs, or geographic regions.
4. NOT / Exclusion Operator
Request: Implement an exclusion operator (!= or -) to filter out specific tags or exception types.
This explicitly excludes spans containing a certain tag or value, acting as a powerful noise-reduction tool to isolate genuine anomalies. This was NOT supported by the Jaeger OpenSearch backend, but it would be a highly useful addition.
Example Query:
error!=true
Use Cases:
- Filtering out known, noisy background tasks or health checks from search results to focus entirely on actual user traffic.
- Excluding specific, expected exceptions that are known to clutter the trace view during an active incident investigation (I cannot express enough how many times I have dealt with this).
Describe alternatives you've considered
No response
Additional information
No response