v2.5.0-rc.0

Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. #3552 (@teyyubismayil)
• [BUGFIX] Correctly parse traceql queries with > 1024 character attribute names or static values. #3571 (@joe-elliott)
• [BUGFIX] Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants. [#3612](#3...

v2.4.2

• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)

v2.4.1

• [BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)

v2.4.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
• [ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
• [BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)

v2.4.0-rc.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)

v2.3.1

Bugfixes

• [BUGFIX] Include statusMessage intrinsic attribute in tag search. #3084 (@rcrowe)
• [BUGFIX] Fix compactor ignore configured S3 headers #3149 (@Batkilin)
• [BUGFIX] Readd session token to s3 credentials. #3144 (@farodin91)

v2.3.0

Breaking Changes

• Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
• Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

• [CHANGE] Update Go to 1.21 #2486 (@zalegrala)
• [CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
• [CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
• [CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
• [CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
  v1: azure-storage-blob-go original (now deprecated) SDK
  v2: azure-sdk-for-go
• [CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
• [CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
• [CHANGE] Overrides module refactor #2688 (@mapno)
  Added new defaults block to the overrides' module. Overrides change to indented syntax.
  Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Features

• [FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
• [FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
• [FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
• [FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
• [FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
• [FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
• [ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
• [ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
• [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
• [ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
• [ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
• [ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
• [ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
• [ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
• [ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@MarcinGinszt)
• [ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
• [ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
• [ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
• [ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
• [ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
• [ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
• [ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
• [ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
• [ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
• [ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
  Breaking Change Removed deprecated config option: distributor.log_received_traces
• [ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
• [ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry settings. #3019 (@joe-elliott)

Bugfixes

• [BUGFIX] Unescape tag names #2894 (@fabrizio-grafana)
• [BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
• [BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
• [BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
• [BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
• [BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
• [BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
  This query will now correctly tell you %age of requests that are within SLO:

  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
  

  BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
• [BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
• [BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
• [BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
• [BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
• [BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
• [BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
• [BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

v2.2.4

Changes

• [CHANGE] Update alpine image version to 3.18 to patch CVE-2022-48174. #3046 (@joe-elliott)
• [CHANGE] Bump Jaeger query docker image to 1.50.0 #2998 (@pavolloffay)

v2.3.0-rc.0

Breaking Changes

• Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
• Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

• [CHANGE] Update Go to 1.21 #2486 (@zalegrala)
• [CHANGE] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
• [CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
• [CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
• [CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
• [CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
  v1: azure-storage-blob-go original (now deprecated) SDK
  v2: azure-sdk-for-go
• [CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
• [CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
• [CHANGE] Overrides module refactor #2688 (@mapno)
  Added new defaults block to the overrides' module. Overrides change to indented syntax.
  Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Features

• [FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
• [FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
• [FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
• [FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
• [FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
• [FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
• [ENHANCEMENT] Unescape tag names #2894 (@fabrizio-grafana)
• [ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
• [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
• [ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
• [ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
• [ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
• [ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
• [ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
• [ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@MarcinGinszt)
• [ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
• [ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
• [ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
• [ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
• [ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
• [ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
• [ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
• [ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
• [ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
• [ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
  Breaking Change Removed deprecated config option: distributor.log_received_traces
• [ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
• [ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry
  settings. #3019 (@joe-elliott)

Bugfixes

• [BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
• [BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
• [BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
• [BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
• [BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
• [BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
  This query will now correctly tell you %age of requests that are within SLO:

  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
  

  BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
• [BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
• [BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
• [BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
• [BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
• [BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
• [BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
• [BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

v2.2.3

Bugfixes

• Fix S3 credentials providers configuration #2889 (@mapno)