grafana · knylander-grafana · Aug 18, 2026
@@ -51,7 +51,7 @@ Refer to [Use polling to monitor the backend status](polling/) for Tempo.
 
 ## Dashboards
 
-The [Tempo mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) has eight Grafana dashboards in the `dashboards` folder that you can download and import into your Grafana UI.
+The [Tempo mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) has ten Grafana dashboards in the `dashboards` folder that you can download and import into your Grafana UI.
 These dashboards work well when you run Tempo in a Kubernetes (k8s) environment and metrics scraped have the
 `cluster` and `namespace` labels.
 
@@ -137,6 +137,23 @@ It shows a limits table alongside distributor bytes and spans per second, live t
 
 Use this dashboard in multitenant deployments to identify tenants with high ingestion rates, query volumes, or storage growth.
 
+### Tempo Service Graph dashboard
+
+> This is available as `tempo-service-graph.json`.
+
+The Service Graph dashboard visualizes service-to-service topology discovered from the `traces_service_graph_connection_info` metric emitted by the metrics-generator's service-graphs connection-info subprocessor.
+
+Use this dashboard to see how services in your system call each other and to spot unexpected or missing connections in the traced topology.
+
+### Tempo Livestore dashboard
+
+> This is available as `tempo-livestore.json`.
+
+The Livestore dashboard monitors the health and performance of the live-store component, which serves recent trace data directly from memory and the write-ahead log (WAL) before it's flushed to the backend.
+It's organized into a triage cockpit plus rows for read latency and query impact, ingest lag and throughput, backpressure and readiness, the WAL completion path, live trace and tenant pressure, resource usage, and autoscaling, with most rows broken down by live-store zone.
+
+Use this dashboard to diagnose read latency, ingest lag, and backpressure issues on the live-store path, and to monitor autoscaling behavior for live-store and its supporting Warpstream agents.
+
 ## Rules and alerts
 
 The Rules and Alerts are available as [YAML files in the compiled mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin-compiled) on the repository.

@@ -0,0 +1,77 @@
+---
+title: Verify signed container images
+menuTitle: Verify container images
+description: Verify the cosign signature and SLSA build provenance of the container images Tempo publishes, before you deploy them.
+weight: 650
+---
+
+# Verify signed container images
+
+Starting with Tempo 3.1, Grafana Labs cryptographically signs every published Tempo container image and attaches a SLSA build provenance attestation to it.
+Use these artifacts to confirm that an image actually came from the `grafana/tempo` repository and to trace it back to the exact source commit and CI run that built it, before you pull it into a cluster.
+
+## Which images are signed
+
+The following images published to [Docker Hub](https://hub.docker.com/u/grafana) are signed and attested:
+
+- `grafana/tempo`
+- `grafana/tempo-vulture`
+- `grafana/tempo-query`
+- `grafana/tempo-cli`
+
+Signing and attestation run as a best-effort step after each image is published. They don't gate the release or block a deployment, so a signing failure never prevents an image from being available. It only means that image doesn't verify until it's re-signed.
+
+## What you get
+
+Each image ships with two supply chain artifacts:
+
+- **A keyless cosign signature.** Proves the image was produced by the `sign-and-attest.yml` workflow in the `grafana/tempo` repository and hasn't been tampered with since. Grafana Labs doesn't hold or manage a private signing key: the workflow signs using a short-lived certificate issued by [Sigstore's](https://www.sigstore.dev/) Fulcio CA in exchange for a GitHub Actions OIDC token, and records the signing event in the public Rekor transparency log.
+- **A SLSA build provenance attestation.** Binds the image's digest to the source commit, workflow, and run that built it. This is what lets you answer "was this image actually built by the Tempo CI pipeline from a specific commit, or did it come from somewhere else?"
+
+Both are pushed as OCI referrers alongside the image, so they travel with it wherever the image is stored or mirrored.
+
+## Before you begin
+
+Install the verification tools:
+
+- [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v2 or later, to verify the signature
+- The [GitHub CLI](https://cli.github.com/) (`gh`), to verify the provenance attestation
+
+## Verify an image
+
+Pull the image by digest, or resolve the digest for a tag you already have, so you verify the exact artifact you're about to deploy rather than a tag that could later move to a different image:
+
+```bash
+docker buildx imagetools inspect grafana/tempo:<TEMPO_VERSION> --format '{{.Manifest.Digest}}'
+```
+
+### Verify the signature
+
+Run `cosign verify` against the digest-pinned image, checking the certificate identity against the `sign-and-attest.yml` workflow in `grafana/tempo` and the GitHub Actions OIDC issuer:
+
+```bash
+cosign verify \
+  --certificate-identity-regexp '^https://github\.com/grafana/tempo/\.github/workflows/sign-and-attest\.yml@' \
+  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
+  grafana/tempo@<DIGEST>
+```
+
+A successful verification prints the certificate details and a `Verified OK` message.
+
+### Verify the provenance attestation
+
+Run `gh attestation verify` against the same digest-pinned image, scoped to the `grafana/tempo` repository:
+
+```bash
+gh attestation verify oci://grafana/tempo@<DIGEST> --repo grafana/tempo
+```
+
+A successful verification confirms the attestation is signed by `grafana/tempo` and prints the predicate type and the workflow run that produced it.
+
+## Result
+
+If both commands succeed, you've confirmed that the image digest you're about to deploy was built and published by the Tempo CI pipeline from a known commit, and hasn't been altered since. If either command fails, don't deploy the image, and treat it as unverified until you can confirm why verification failed.
+
+{{< admonition type="note" >}}
+The verification identity (the workflow path plus the OIDC issuer) is a public contract. It stays the same across all four signed images, so you can reuse the same `--certificate-identity-regexp` and `--certificate-oidc-issuer` values regardless of which component you're verifying.
+{{< /admonition >}}
@@ -144,6 +144,23 @@ Tempo changes the default connection behavior of the memcached cache client to k
 
 These defaults raise the steady-state number of open connections per memcached server. To keep the previous behavior, set `max_idle_conns: 16` and `min_idle_conns_headroom_percentage: 0` in your cache configuration.
 
+### Trace by ID query sharding now scales with block count
+
+Trace by ID lookups are split into jobs across a block range. Tempo adds a new `query_frontend.trace_by_id.blocks_per_shard` setting that targets a number of blocks per job instead of a fixed job count, and it defaults to `30`, taking precedence over the older `query_shards` setting whenever it's non-zero. [[PR 7105](https://github.com/grafana/tempo/pull/7105)]
+
+Previously, `query_shards` (default `50`) split every trace by ID query into the same fixed number of jobs, regardless of how many blocks a tenant had. That value had to be tuned for the largest tenant in a cell, which left smaller tenants with too few blocks per job, and it had to be revisited as ingest volume changed. With `blocks_per_shard` set, Tempo instead computes the shard count dynamically from the tenant's current block count, so the number of jobs scales automatically as a tenant grows or shrinks, and different tenant sizes in the same cell are each sharded appropriately. The computed shard count never exceeds `query_frontend.max_outstanding_per_tenant`, so this change doesn't overload the query queue.
+
+This affects anyone who tuned `query_shards` for their workload, or who relies on trace by ID lookups producing a fixed, predictable number of sub-queries, for example in dashboards, alerting, or capacity planning.
+
+To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`:
+
+```yaml
+query_frontend:
+  trace_by_id:
+    blocks_per_shard: 0
+    query_shards: 50
+```
+
 ## Upgrade to Tempo 3.0
 
 Tempo 3.0 is a major release that replaces the ingester-based architecture with a new design that separates the read and write paths.

@@ -33,11 +33,16 @@ Consider the following resolutions:
 
 ## TempoDistributorUnhealthy
 
-This can happen when we have unhealthy distributors sticking around in the ring.
+This can happen when we have unhealthy distributors sticking around in the ring, for example after a distributor Pod is OOM-killed or otherwise
+terminated non-gracefully instead of shutting down cleanly.
 
-If this occurs access the [ring page](https://grafana.com/docs/tempo/latest/operations/consistent_hash_ring/) at `/distributor/ring`.
-Use the "Forget" button to forget and remove any unhealthy distributors from the ring. An unhealthy distributor or two has virtually no impact except to slightly
-increase the amount of memberlist traffic propagated by the cluster.
+Distributors automatically forget unhealthy ring entries after `2 * distributor.ring.heartbeat-timeout` (10 minutes with the default 5m timeout),
+so stale entries are normally cleared well within this alert's 15m `for` window and no manual action is needed.
+
+If you need an unhealthy distributor removed sooner than the auto-forget window, or if `distributor.ring.heartbeat-timeout` has been configured to
+a larger value in your deployment, access the [ring page](https://grafana.com/docs/tempo/latest/operations/consistent_hash_ring/) at
+`/distributor/ring` and use the "Forget" button to remove it. An unhealthy distributor or two has virtually no impact except to slightly increase
+the amount of memberlist traffic propagated by the cluster.
 
 
 ### Quick checks