|
| 1 | +--- |
| 2 | +title: Version 2.4 release notes |
| 3 | +menuTitle: V2.4 |
| 4 | +description: Release notes for Grafana Tempo 2.4 |
| 5 | +weight: 40 |
| 6 | +--- |
| 7 | + |
| 8 | +# Version 2.4 release notes |
| 9 | + |
| 10 | +The Tempo team is pleased to announce the release of Tempo 2.4. |
| 11 | + |
| 12 | +This release gives you: |
| 13 | + |
| 14 | +* New features, including multi-tenant queries and experimental TraceQL metrics queries |
| 15 | +* Performance enhancements, thanks to the addition of new caching tiers |
| 16 | +* Cost savings, thanks to polling improvements that reduce calls to object storage |
| 17 | + |
| 18 | +As part of this release, vParquet3 has also been promoted to the new default storage format for traces. For more about why we’re so excited about vParquet3, refer to [Accelerate TraceQL queries at scale with dedicated attribute columns in Grafana Tempo](/blog/2024/01/22/accelerate-traceql-queries-at-scale-with-dedicated-attribute-columns-in-grafana-tempo/). |
| 19 | + |
| 20 | +Read the [Tempo 2.4 blog post](/blog/2023/11/01/grafana-tempo-2.3-release-faster-trace-queries-traceql-upgrades/) for more examples and details about these improvements. |
| 21 | + |
| 22 | +These release notes highlight the most important features and bugfixes. For a complete list, refer to the [Tempo changelog](https://github.com/grafana/tempo/releases). |
| 23 | + |
| 24 | +## Features and enhancements |
| 25 | + |
| 26 | +The most important features and enhancements in Tempo 2.4 are highlighted below. |
| 27 | + |
| 28 | +### Multi-tenant queries |
| 29 | + |
| 30 | +Tempo now allows you to query multiple tenants at once. We’ve made multi-tenant queries compatible with streaming ([first released in v2.2]({{< relref "./v2-2#get-traceql-results-faster" >}})) so you can get query results as fast as possible. To learn more, refer to [Cross-tenant federation]({{< relref "../operations/cross_tenant_query" >}}) and [Enable multi-tenancy]({{< relref "../operations/multitenancy" >}}). [PRs [3262](https://github.com/grafana/tempo/pull/3262), [3087](https://github.com/grafana/tempo/pull/3087)] |
| 31 | + |
| 32 | +### TraceQL metrics (experimental) |
| 33 | + |
| 34 | +We’re excited to announce the addition of metrics queries to the TraceQL language. Metric queries extend trace queries by applying a function to trace query results. |
| 35 | +This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs. |
| 36 | + |
| 37 | +In this case, we are calculating the rate of the erroring spans coming from the service `foo`. Rate is a `spans/sec` quantity. |
| 38 | + |
| 39 | +``` |
| 40 | +{ resource.service.name = "foo" && status = error } | rate() |
| 41 | +``` |
| 42 | + |
| 43 | +In addition, you can use Grafana Explore to [query and visualize the metrics]({{< relref "../operations/traceql-metrics" >}}) with the Tempo data soruce in Grafana or Grafana Cloud. |
| 44 | + |
| 45 | + |
| 46 | + |
| 47 | +For more information, refer to the [TraceQL metrics]({{< relref "../operations/traceql-metrics" >}}) documentation. [PRs [3227](https://github.com/grafana/tempo/pull/3227) [#3252](https://github.com/grafana/tempo/pull/3252), [3258](https://github.com/grafana/tempo/pull/3258)] |
| 48 | + |
| 49 | +To learn more about the TraceQL syntax, see the [TraceQL documentation]({{< relref "../traceql" >}}). For information on planned future extensions to the TraceQL language, refer to [future work]({{< relref "../traceql/architecture" >}}). |
| 50 | + |
| 51 | +#### TraceQL performance improvements |
| 52 | + |
| 53 | +We continue to make query performance improvements so you spend less time waiting on results to your TraceQL queries. Below are some notable PRs that made it into this release: |
| 54 | + |
| 55 | +* Improve TraceQL regex performance in certain queries. [PR [3139](https://github.com/grafana/tempo/pull/3139)] |
| 56 | +* Improve TraceQL performance in complex queries. [[PR 3113](https://github.com/grafana/tempo/pull/3113)] |
| 57 | +* TraceQL/Structural operators performance improvement. [[PR 3088](https://github.com/grafana/tempo/pull/3088)] |
| 58 | + |
| 59 | +### vParquet3 is now the default block format |
| 60 | + |
| 61 | +Tempo 2.4 makes [vParquet3]({{< relref "../configuration/parquet" >}}) the default storage format. |
| 62 | + |
| 63 | +We’re excited about [vParquet3]({{< relref "../configuration/parquet" >}}) relative to prior formats because of its support for [dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}), which help speed up queries on your largest and most queried attributes. We've seen excellent performance improvements when running it ourselves, and by promoting it to the default, we're signaling that it is ready for broad adoption. |
| 64 | + |
| 65 | +Dedicated attribute columns, available using vParquet3, improve query performance by storing the largest and most frequently used attributes in their own columns, rather than in the generic attribute key-value list. |
| 66 | +For more information, refer to |
| 67 | +[Accelerate TraceQL queries at scale with dedicated attribute columns in Grafana Tempo](/blog/2024/01/22/accelerate-traceql-queries-at-scale-with-dedicated-attribute-columns-in-grafana-tempo/). [[PR 2526](https://github.com/grafana/tempo/pull/2526)] |
| 68 | + |
| 69 | +If you had manually configured vParquet3, we recommend removing it to move forward with Tempo defaults. |
| 70 | + |
| 71 | +To read more about the design of vParquet3, refer to [the design proposal](https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-05%20vParquet3.md). For general information, refer to [the Apache Parquet schema]({{< relref "../operations/schema" >}}). |
| 72 | + |
| 73 | +### Additional caching layers |
| 74 | + |
| 75 | +Tempo has added two new caches to improve TraceQL query performance. The frontend-search cache handles job search caching. The parquet-page cache handles page level caching. Refer to the [Cache section]({{< relref "../configuration#cache" >}}) of the Configuration documentation for how to configure these new caching layers. As part of adding these new caching layers, we’ve refactored our caching interface. This includes breaking changes described in “Breaking Changes”. [PRs [3166](https://github.com/grafana/tempo/pull/3166), [3225](https://github.com/grafana/tempo/pull/3225), [3196](https://github.com/grafana/tempo/pull/3196)] |
| 76 | + |
| 77 | +### Polling improvements for cost reduction |
| 78 | + |
| 79 | +We’ve improved how Tempo polls object storage, ensuring that we reuse previous results. This has dramatically reduced the number of requests Tempo makes to the object store. Not only does this reduce the load on your object store, for many, it will save you money (since most hosted object storage solutions charge per request). |
| 80 | + |
| 81 | +We’ve also added the `list_blocks_concurrency` parameter to allow you to tune the number of list calls Tempo makes in parallel to object storage so you can select the value that works best for your environment. We’ve set the default value to `3`, which should work well for the average Tempo cluster. [[PR 2652](https://github.com/grafana/tempo/pull/2652)] |
| 82 | + |
| 83 | +### Other enhancements and improvements |
| 84 | + |
| 85 | +In addition, the following improvements have been made in Tempo 2.4: |
| 86 | + |
| 87 | +* Improved Tempo error handling on writes, so that one erroring trace doesn't result in an entire batch of traces being dropped. [PR 2571](https://github.com/grafana/tempo/pull/2571) |
| 88 | +* Added per-tenant compaction window. [PR 3129](https://github.com/grafana/tempo/pull/3129) |
| 89 | +* Added `--max-start-time` and `--min-start-time` flag to tempo-cli command `analyse blocks`. [PR 3250](https://github.com/grafana/tempo/pull/3250) |
| 90 | +* Added per-tenant configurable `remote_write` headers to metrics-generator. [#3175](https://github.com/grafana/tempo/pull/3175) |
| 91 | +* Added variable expansion support to overrides configuration. [PR 3175](https://github.com/grafana/tempo/pull/3175) |
| 92 | +* Added HTML pages `/status/overrides` and `/status/overrides/{tenant}`. [PR 3244](https://github.com/grafana/tempo/pull/3244) [#3332](https://github.com/grafana/tempo/pull/3332) |
| 93 | +* Precalculate and reuse the vParquet3 schema before opening blocks. [PR 3367](https://github.com/grafana/tempo/pull/3367) |
| 94 | +* Made the trace ID label name configurable for remote written exemplars. [PR 3074](https://github.com/grafana/tempo/pull/3074) |
| 95 | +* Performance improvements in span filtering. [PR 3025](https://github.com/grafana/tempo/pull/3025) |
| 96 | +* Introduced localblocks process configuration option to select only server spans. [PR 3303](https://github.com/grafana/tempo/pull/3303) |
| 97 | + |
| 98 | +## Upgrade considerations |
| 99 | + |
| 100 | +When [upgrading]({{< relref "../setup/upgrade" >}}) to Tempo 2.4, be aware of these considerations and breaking changes. |
| 101 | + |
| 102 | +### Transition to vParquet 3 |
| 103 | + |
| 104 | +vParquet3 format is now the default block format. It is production ready and we highly recommend switching to it for improved query performance and [dedicated attribute columns]({{< relref "../operations/dedicated_columns" >}}). |
| 105 | + |
| 106 | +Upgrading to Tempo 2.4 modifies the Parquet block format. Although you can use Tempo 2.3 with vParquet2 or vParquet3, you can only use Tempo 2.4 with vParquet3. |
| 107 | + |
| 108 | +With this release, the first version of our Parquet backend, vParquet, is being deprecated. Tempo 2.4 will still read vParquet1 blocks. However, Tempo will exit with error if they are manually configured. [[PR 3377](https://github.com/grafana/tempo/pull/3377/files#top)] |
| 109 | + |
| 110 | +For information on upgrading, refer to [Upgrade to Tempo 2.4]({{< relref "../setup/upgrade" >}}) and [Choose a different block format]({{< relref "../configuration/parquet#choose-a-different-block-format" >}}) . |
| 111 | + |
| 112 | +### Updated, removed, or renamed configuration parameters |
| 113 | + |
| 114 | +<table> |
| 115 | + <tr> |
| 116 | + <td>Parameter |
| 117 | + </td> |
| 118 | + <td>Comments |
| 119 | + </td> |
| 120 | + </tr> |
| 121 | + <tr> |
| 122 | + <td><code>autocomplete_filtering_enabled</code> |
| 123 | + </td> |
| 124 | + <td>Set to <code>true</code> by default [PR <a href="https://github.com/grafana/tempo/pull/3178">3178</a>] |
| 125 | + </td> |
| 126 | + </tr> |
| 127 | + <tr> |
| 128 | + <td><code>distributor.log_received_traces</code> |
| 129 | + </td> |
| 130 | + <td>Use the <code>distributor.log_received_spans</code> configuration block instead. [PR <a href="https://github.com/grafana/tempo/pull/3008">#3008</a>] |
| 131 | + </td> |
| 132 | + </tr> |
| 133 | + <tr> |
| 134 | + <td><code>tempo_query_frontend_queries_total{op="searchtags|metrics"}</code> |
| 135 | + </td> |
| 136 | + <td>Removed deprecated frontend metrics configuration option |
| 137 | + </td> |
| 138 | + </tr> |
| 139 | + <tr> |
| 140 | + <td> |
| 141 | +``` |
| 142 | +storage: |
| 143 | + trace: |
| 144 | + cache: |
| 145 | + search: |
| 146 | + cache_control: |
| 147 | + background_cache: |
| 148 | + memcached: |
| 149 | +``` |
| 150 | + |
| 151 | + </td> |
| 152 | + <td>These fields have been removed in favor of the new cache configuration. Refer to Cache configuration refactored. |
| 153 | + |
| 154 | + </td> |
| 155 | + </tr> |
| 156 | +</table> |
| 157 | + |
| 158 | +The distributor now returns 200 for any batch containing only `trace_too_large` and `max_live_traces` errors. The number of discarded spans are still reflected in the `tempo_discarded_spans_total metrics`. |
| 159 | + |
| 160 | +### Removed experimental websockets support for search streaming |
| 161 | + |
| 162 | +GPRC is now the supported method for streaming results. |
| 163 | +Websockets support for search streaming has been removed. |
| 164 | +Websocket support was initially added due to conflicts with GRPC, HTTP, and TLS. |
| 165 | +Those issues were corrected [in PR 3300](https://github.com/grafana/tempo/pull/3300). [PR 3307](https://github.com/grafana/tempo/pull/3307) |
| 166 | + |
| 167 | +### Cache configuration refactored |
| 168 | + |
| 169 | +The major cache refactor to allow multiple role-based caches to be configured. [[PR 3166](https://github.com/grafana/tempo/pull/3166)] |
| 170 | +This change resulted in the following fields being deprecated. |
| 171 | +These have all been migrated to a top level `cache:` field. |
| 172 | + |
| 173 | +For more information about the configuration, refer to the [Cache]({{< relref "../configuration#cache" >}}) section. |
| 174 | + |
| 175 | +The old configuration block looked like this: |
| 176 | + |
| 177 | +```yaml |
| 178 | +storage: |
| 179 | + trace: |
| 180 | + cache: |
| 181 | + search: |
| 182 | + cache_control: |
| 183 | + background_cache: |
| 184 | + memcached: |
| 185 | + redis: |
| 186 | +``` |
| 187 | +
|
| 188 | +With the new configuration, you create your list of caches,- with either `redis` or `memcached` cluster with your config, then define the types of data and roles. |
| 189 | + |
| 190 | +```yaml |
| 191 | +cache: |
| 192 | + caches: |
| 193 | + - memcached: |
| 194 | + host: <some memcached cluster> |
| 195 | + roles: |
| 196 | + - bloom |
| 197 | + - parquet-footer |
| 198 | + - memcached: |
| 199 | + host: <some memcached cluster> |
| 200 | + roles: |
| 201 | + - frontend-search |
| 202 | +``` |
| 203 | + |
| 204 | +## Security fixes |
| 205 | + |
| 206 | +The following vulnerabilities have been addressed: |
| 207 | + |
| 208 | +* Addressed [CVE-2023-5363](https://github.com/advisories/GHSA-xw78-pcr6-wrg8). |
| 209 | +* Updated the `memcached` default image in jsonnet for multiple CVEs [PR 3310](https://github.com/grafana/tempo/pull/3310) |
| 210 | + |
| 211 | +## Bugfixes |
| 212 | + |
| 213 | +For a complete list, refer to the [Tempo changelog](https://github.com/grafana/tempo/releases). |
| 214 | + |
| 215 | +* Prevent building parquet iterators that would loop forever. [PR 3159](https://github.com/grafana/tempo/pull/3159) |
| 216 | +* Sanitize name in mapped dimensions in span-metrics processor. [PR 3171](https://github.com/grafana/tempo/pull/3171) |
| 217 | +* Fixed an issue where cached footers were requested then ignored. [PR 3196](https://github.com/grafana/tempo/pull/3196) |
| 218 | +* Fixed a panic in autocomplete when the query condition had the wrong type. [PR 3277](https://github.com/grafana/tempo/pull/3277) |
| 219 | +* Fixed TLS when GRPC is enabled on HTTP. [PR 3300](https://github.com/grafana/tempo/pull/3300) |
| 220 | +* Correctly return 400 when max limit is requested on search. [PR 3340](https://github.com/grafana/tempo/pull/3340) |
| 221 | +* Fixed autocomplete filters sometimes returning erroneous results. [PR 3339](https://github.com/grafana/tempo/pull/3339) |
| 222 | +* Fixed trace context propagation between query-frontend and querier. [PR 3387](https://github.com/grafana/tempo/pull/3387) |
| 223 | +* Fixed parsing of span.resource.xyz attributes in TraceQL. [PR 3284](https://github.com/grafana/tempo/pull/3284) |
| 224 | +* Changed exit code if config is successfully verified. [PR 3174](https://github.com/grafana/tempo/pull/3174) |
| 225 | +* The tempo-cli analyze blocks command no longer fails on compacted blocks. [PR 3183](https://github.com/grafana/tempo/pull/3183) |
| 226 | +* Moved waitgroup handling for poller error condition. [PR 3224](https://github.com/grafana/tempo/pull/3224) |
| 227 | +* Fixed head block excessive locking in ingester search. [PR 3328](https://github.com/grafana/tempo/pull/3328) |
| 228 | +* Fixed an issue with ingester failed to write traces to disk after a crash or unclean restart. [PR 3346](https://github.com/grafana/tempo/issues/3346) |
0 commit comments