|
| 1 | +--- |
| 2 | +title: User-configurable overrides |
| 3 | +menuTitle: Configure Tempo overrides through the user-configurable overrides API |
| 4 | +description: Configure Tempo overrides through the user-configurable overrides API |
| 5 | +weight: 90 |
| 6 | +--- |
| 7 | + |
| 8 | +# User-configurable overrides |
| 9 | + |
| 10 | +User-configurable overrides in Tempo let you change overrides for your tenant using an API. |
| 11 | +Instead of modifying a file or Kubernetes configmap, you (and other services relying on Tempo) can use this API to modify the overrides directly. |
| 12 | + |
| 13 | +### Architecture |
| 14 | + |
| 15 | + |
| 16 | + |
| 17 | +User-configurable overrides are stored in an object store bucket managed by Tempo. |
| 18 | + |
| 19 | +{{% admonition type="note" %}} |
| 20 | +We recommend using a different bucket for overrides and traces storage, but they can share a bucket if needed. |
| 21 | +When sharing a bucket, make sure any lifecycle rules are scoped correctly to not remove data of user-configurable overrides module. |
| 22 | +{{% /admonition %}} |
| 23 | + |
| 24 | +Overrides of every tenant are stored at `/{tenant name}/overrides.json`: |
| 25 | + |
| 26 | +``` |
| 27 | +overrides/ |
| 28 | +├── 1/ |
| 29 | +│ └── overrides.json |
| 30 | +└── 2/ |
| 31 | +└── overrides.json |
| 32 | +``` |
| 33 | + |
| 34 | +Tempo regularly polls this bucket and keeps a copy of the limits in-memory. When requesting the overrides for a tenant, the overrides module: |
| 35 | + |
| 36 | +1. Checks this override is set in the user-configurable overrides, if so return that value |
| 37 | +2. Checks if this override is set in the runtime config (configmap), if so return that value |
| 38 | +3. Returns the default value |
| 39 | + |
| 40 | +### Supported fields |
| 41 | + |
| 42 | +User-configurable overrides are designed to be a subset of the runtime overrides. Refer to [Overrides]{{< relref "../configuration/_index#overrides" >}} for information about all overrides. |
| 43 | +When a field is set in both the user-configurable overrides and the runtime overrides, the value from the user-configurable overrides will take priority. |
| 44 | + |
| 45 | +{{% admonition type="note" %}} |
| 46 | +`processors` is an exception: Tempo will merge values from both user-configurable overrides and runtime overrides into a single list. |
| 47 | +{{% /admonition %}} |
| 48 | + |
| 49 | +```yaml |
| 50 | +[forwarders: <list of strings>] |
| 51 | + |
| 52 | +metrics_generator: |
| 53 | + |
| 54 | + [processors: <list of strings>] |
| 55 | + [collection_interval: <duration>] |
| 56 | + [disable_collection: <bool> | default = false] |
| 57 | + |
| 58 | + processor: |
| 59 | + |
| 60 | + service_graphs: |
| 61 | + [histogram_buckets: <list of float>] |
| 62 | + [dimensions: <list of string>] |
| 63 | + [peer_attributes: <list of string>] |
| 64 | + [enable_client_server_prefix: <bool>] |
| 65 | + |
| 66 | + span_metrics: |
| 67 | + [histogram_buckets: <list of float>] |
| 68 | + [dimensions: <list of string>] |
| 69 | + [intrinsic_dimensions: <map string to bool>] |
| 70 | + [filter_policies: [ |
| 71 | + [ |
| 72 | + include/exclude: |
| 73 | + match_type: <string> # options: strict, regexp |
| 74 | + attributes: |
| 75 | + - key: <string> |
| 76 | + value: <any> |
| 77 | + ] |
| 78 | + ] |
| 79 | + [enable_target_info: <bool>] |
| 80 | + [target_info_excluded_dimensions: <list of string>] |
| 81 | +``` |
| 82 | +
|
| 83 | +### API |
| 84 | +
|
| 85 | +All API requests are handled on the `/api/overrides` endpoint. The module supports `GET`, `POST`, `PATCH`, and `DELETE` requests. |
| 86 | + |
| 87 | +This endpoint is tenant-specific. If Tempo is run in multitenant mode, all requests should have an appropriate `X-Scope-OrgID` header. |
| 88 | + |
| 89 | +If the tenant is run in distributed mode, only the query-frontend will accept API requests. |
| 90 | + |
| 91 | +#### Operations |
| 92 | + |
| 93 | +``` |
| 94 | +GET /api/overrides |
| 95 | +``` |
| 96 | +
|
| 97 | +Returns the current overrides and it's version. |
| 98 | +
|
| 99 | +Query-parameters: |
| 100 | +- `scope`: whether to return overrides from the API only `api` or merge it with the runtime overrides `merged`. Defaults to `api`. |
| 101 | +
|
| 102 | +Example: `curl -X GET -v -H "X-Scope-OrgID: 3" http://localhost:3100/tempo/api/overrides\?scope=merged` |
| 103 | +
|
| 104 | +``` |
| 105 | +POST /api/overrides |
| 106 | +``` |
| 107 | +
|
| 108 | +Update the overrides with the given payload. Note this will overwrite any existing overrides. |
| 109 | +
|
| 110 | +Example: `curl -X POST -v -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "{}" |
| 111 | +
|
| 112 | +``` |
| 113 | +PATCH /api/overrides |
| 114 | +``` |
| 115 | +
|
| 116 | +Update the existing overrides by patching it with the payload. It follows the JSON merge patch protocol ([RFC 7386](https://datatracker.ietf.org/doc/html/rfc7386)). |
| 117 | +
|
| 118 | +Example: `curl -X PUT -v -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "{\"forwarders\":null}" |
| 119 | +
|
| 120 | +``` |
| 121 | +DELETE /api/overrides |
| 122 | +``` |
| 123 | +
|
| 124 | +Delete the existing overrides. |
| 125 | +
|
| 126 | +Example: `curl -X DELETE -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides` |
| 127 | +
|
| 128 | +#### Versioning |
| 129 | +
|
| 130 | +To handle concurrent read and write operations, overrides are stored with a version in the backend. |
| 131 | +Whenever the overrides are returned, the response will have an Etag header with the current version. |
| 132 | +
|
| 133 | +``` |
| 134 | +$ curl -v http://localhost:3100/api/overrides |
| 135 | +... |
| 136 | +< HTTP/1.1 200 OK |
| 137 | +< Content-Type: application/json |
| 138 | +< Etag: 1697726795401423 |
| 139 | +< Date: Wed, 07 Feb 2024 17:49:04 GMT |
| 140 | +< Content-Length: 118 |
| 141 | +... |
| 142 | +``` |
| 143 | +
|
| 144 | +Requests that modify or delete overrides need to pass the current version using the `If-Match` header: |
| 145 | +
|
| 146 | +``` |
| 147 | +$ curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "..." |
| 148 | +``` |
| 149 | +This example uses uses overrides in the `overrides.json` file with the location in `pwd`: |
| 150 | +
|
| 151 | +`curl -X POST -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data @overrides.json` |
| 152 | +
|
| 153 | +If the version does not match the version in the backend, the request is rejected with HTTP error 412. |
| 154 | +
|
| 155 | +#### Conflicting runtime overrides check |
| 156 | +
|
| 157 | +Overrides set through the user-configurable overrides take priority over runtime overrides. |
| 158 | +This can lead to misleading scenarios because a value set in the runtime overrides is not actively being used. |
| 159 | +
|
| 160 | +To warn users about preexisting runtime overrides, there is an optional check for conflicting runtime overrides. |
| 161 | +If enabled requests are rejected if: |
| 162 | +
|
| 163 | +1. There are no user-configurable overrides yet for this tenant. |
| 164 | +2. There are runtime overrides set that contain overrides present in the user-configurable overrides. |
| 165 | +
|
| 166 | +The check can be enabled in the configuration: |
| 167 | +
|
| 168 | +```yaml |
| 169 | +overrides: |
| 170 | + user_configurable_overrides: |
| 171 | + api: |
| 172 | + check_for_conflicting_runtime_overrides: true |
| 173 | +``` |
| 174 | + |
| 175 | +You can bypass this check by setting the query parameter `skip-conflicting-overrides-check=true`: |
| 176 | + |
| 177 | +``` |
| 178 | +$ curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides?skip-conflicting-overrides-check=true --data "..." |
| 179 | +``` |
0 commit comments