Skip to content

Commit eab553f

Browse files
gizasbmorelli25
authored and
mergify[bot]
committed
[Prometheus] Remote - Write: Enforce Maxdecoding Length check (#48218)
* fix the snappy decompression Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co> * merging new config vars Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co> --------- Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co> Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co> (cherry picked from commit 38cc702)
1 parent f373665 commit eab553f

9 files changed

Lines changed: 363 additions & 22 deletions

File tree

‎changelog/fragments/1766493789-prometheus_remotew_maxdecodinglenght.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Kind can be one of:
2+
# - breaking-change: a change to previously-documented behavior
3+
# - deprecation: functionality that is being removed in a later release
4+
# - bug-fix: fixes a problem in a previous version
5+
# - enhancement: extends functionality but does not break or fix existing behavior
6+
# - feature: new functionality
7+
# - known-issue: problems that we are aware of in a given version
8+
# - security: impacts on the security of a product or a user’s deployment.
9+
# - upgrade: important information for someone upgrading from a prior version
10+
# - other: does not fit into any of the other categories
11+
kind: bug-fix
12+
13+
# Change summary; a 80ish characters long description of the change.
14+
summary: Enforces configurable size limits on incoming requests for remote_write metricset (max_compressed_body_bytes, max_decoded_body_bytes)
15+
16+
# Long description; in case the summary is not enough to describe the change
17+
# this field accommodate a description without length limits.
18+
# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
19+
#description:
20+
21+
# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
22+
component: metricbeat
23+
24+
# PR URL; optional; the PR number that added the changeset.
25+
# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
26+
# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
27+
# Please provide it if you are adding a fragment for a different PR.
28+
#pr: https://github.com/owner/repo/1234
29+
30+
# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
31+
# If not present is automatically filled by the tooling with the issue linked to the PR number.
32+
#issue: https://github.com/owner/repo/1234

‎docs/reference/metricbeat/metricbeat-metricset-prometheus-remote_write.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,25 @@ remote_write:
6868
#insecure_skip_verify: true
6969
```
7070

71+
## Request size limits [_request_size_limits]
72+
73+
```{applies_to}
74+
stack: ga 9.2.5, ga 9.3.1, ga 9.4
75+
```
76+
77+
To protect against resource exhaustion from malicious or oversized payloads, the remote_write metricset enforces configurable size limits on incoming requests:
78+
79+
* `max_compressed_body_bytes`: Maximum size of the compressed (snappy-encoded) request body in bytes. Requests exceeding this limit are rejected with HTTP 413 before being read into memory. Default: 2 MB (2097152 bytes).
80+
* `max_decoded_body_bytes`: Maximum size of the decompressed request body in bytes. The server checks the declared decoded size in the snappy header before allocating memory for decompression, preventing decompression bomb attacks. Default: 10 MB (10485760 bytes).
81+
82+
```yaml
83+
- module: prometheus
84+
metricsets: ["remote_write"]
85+
host: "localhost"
86+
port: "9201"
87+
max_compressed_body_bytes: 2097152 # 2 MB (default)
88+
max_decoded_body_bytes: 10485760 # 10 MB (default)
89+
```
7190

7291
## Histograms and types [_histograms_and_types_2]
7392

‎metricbeat/module/prometheus/remote_write/_meta/docs.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,25 @@ remote_write:
5757
#insecure_skip_verify: true
5858
```
5959

60+
## Request size limits [_request_size_limits]
61+
62+
```{applies_to}
63+
stack: ga 9.2.5, ga 9.3.1, ga 9.4
64+
```
65+
66+
To protect against resource exhaustion from malicious or oversized payloads, the remote_write metricset enforces configurable size limits on incoming requests:
67+
68+
* `max_compressed_body_bytes`: Maximum size of the compressed (snappy-encoded) request body in bytes. Requests exceeding this limit are rejected with HTTP 413 before being read into memory. Default: 2 MB (2097152 bytes).
69+
* `max_decoded_body_bytes`: Maximum size of the decompressed request body in bytes. The server checks the declared decoded size in the snappy header before allocating memory for decompression, preventing decompression bomb attacks. Default: 10 MB (10485760 bytes).
70+
71+
```yaml
72+
- module: prometheus
73+
metricsets: ["remote_write"]
74+
host: "localhost"
75+
port: "9201"
76+
max_compressed_body_bytes: 2097152 # 2 MB (default)
77+
max_decoded_body_bytes: 10485760 # 10 MB (default)
78+
```
6079

6180
## Histograms and types [_histograms_and_types_2]
6281

‎metricbeat/module/prometheus/remote_write/config.go‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,16 +19,27 @@ package remote_write
1919

2020
import "github.com/elastic/elastic-agent-libs/transport/tlscommon"
2121

22+
const (
23+
// DefaultMaxCompressedBodyBytes is the default maximum size of compressed request body (2MB)
24+
DefaultMaxCompressedBodyBytes int64 = 2 * 1024 * 1024
25+
// DefaultMaxDecodedBodyBytes is the default maximum size of decoded request body (10MB)
26+
DefaultMaxDecodedBodyBytes int64 = 10 * 1024 * 1024
27+
)
28+
2229
type Config struct {
23-
MetricsCount bool `config:"metrics_count"`
24-
Host string `config:"host"`
25-
Port int `config:"port"`
26-
TLS *tlscommon.ServerConfig `config:"ssl"`
30+
MetricsCount bool `config:"metrics_count"`
31+
Host string `config:"host"`
32+
Port int `config:"port"`
33+
TLS *tlscommon.ServerConfig `config:"ssl"`
34+
MaxCompressedBodyBytes int64 `config:"max_compressed_body_bytes"`
35+
MaxDecodedBodyBytes int64 `config:"max_decoded_body_bytes"`
2736
}
2837

2938
func defaultConfig() Config {
3039
return Config{
31-
Host: "localhost",
32-
Port: 9201,
40+
Host: "localhost",
41+
Port: 9201,
42+
MaxCompressedBodyBytes: DefaultMaxCompressedBodyBytes,
43+
MaxDecodedBodyBytes: DefaultMaxDecodedBodyBytes,
3344
}
3445
}

‎metricbeat/module/prometheus/remote_write/remote_write.go‎

Lines changed: 43 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818
package remote_write
1919

2020
import (
21+
"errors"
22+
"fmt"
2123
"io"
2224
"net/http"
2325

@@ -56,10 +58,12 @@ type RemoteWriteEventsGeneratorFactory func(ms mb.BaseMetricSet, opts ...RemoteW
5658

5759
type MetricSet struct {
5860
mb.BaseMetricSet
59-
server serverhelper.Server
60-
events chan mb.Event
61-
promEventsGen RemoteWriteEventsGenerator
62-
eventGenStarted bool
61+
server serverhelper.Server
62+
events chan mb.Event
63+
promEventsGen RemoteWriteEventsGenerator
64+
eventGenStarted bool
65+
maxCompressedBodyBytes int64
66+
maxDecodedBodyBytes int64
6367
}
6468

6569
func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
@@ -92,8 +96,14 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
9296
// MetricSetBuilder returns a builder function for a new Prometheus remote_write metricset using
9397
// the given namespace and event generator
9498
func MetricSetBuilder(genFactory RemoteWriteEventsGeneratorFactory) func(base mb.BaseMetricSet) (mb.MetricSet, error) {
99+
return MetricSetBuilderWithConfig(genFactory, defaultConfig())
100+
}
101+
102+
// MetricSetBuilderWithConfig returns a builder function for a new Prometheus remote_write metricset using
103+
// the given namespace, event generator, and a base config that will be merged with module config
104+
func MetricSetBuilderWithConfig(genFactory RemoteWriteEventsGeneratorFactory, baseConfig Config) func(base mb.BaseMetricSet) (mb.MetricSet, error) {
95105
return func(base mb.BaseMetricSet) (mb.MetricSet, error) {
96-
config := defaultConfig()
106+
config := baseConfig
97107
err := base.Module().UnpackConfig(&config)
98108
if err != nil {
99109
return nil, err
@@ -105,10 +115,12 @@ func MetricSetBuilder(genFactory RemoteWriteEventsGeneratorFactory) func(base mb
105115
}
106116

107117
m := &MetricSet{
108-
BaseMetricSet: base,
109-
events: make(chan mb.Event),
110-
promEventsGen: promEventsGen,
111-
eventGenStarted: false,
118+
BaseMetricSet: base,
119+
events: make(chan mb.Event),
120+
promEventsGen: promEventsGen,
121+
eventGenStarted: false,
122+
maxCompressedBodyBytes: config.MaxCompressedBodyBytes,
123+
maxDecodedBodyBytes: config.MaxDecodedBodyBytes,
112124
}
113125

114126
svc, err := httpserver.NewHttpServerWithHandler(base, m.handleFunc)
@@ -150,13 +162,35 @@ func (m *MetricSet) handleFunc(writer http.ResponseWriter, req *http.Request) {
150162
m.eventGenStarted = true
151163
}
152164

165+
// Limit the size of the compressed request body to prevent resource exhaustion
166+
req.Body = http.MaxBytesReader(writer, req.Body, m.maxCompressedBodyBytes)
167+
153168
compressed, err := io.ReadAll(req.Body)
154169
if err != nil {
170+
var maxBytesError *http.MaxBytesError
171+
if errors.As(err, &maxBytesError) {
172+
m.Logger().Warnf("Request body too large: exceeds %d bytes limit", m.maxCompressedBodyBytes)
173+
http.Error(writer, fmt.Sprintf("request body too large: exceeds %d bytes limit", m.maxCompressedBodyBytes), http.StatusRequestEntityTooLarge)
174+
return
175+
}
155176
m.Logger().Errorf("Read error %v", err)
156177
http.Error(writer, err.Error(), http.StatusInternalServerError)
157178
return
158179
}
159180

181+
// Check decoded length before allocating memory to prevent
182+
decodedLen, err := snappy.DecodedLen(compressed)
183+
if err != nil {
184+
m.Logger().Errorf("Decoded length error: %v", err)
185+
http.Error(writer, "Decoded length error", http.StatusBadRequest)
186+
return
187+
}
188+
if int64(decodedLen) > m.maxDecodedBodyBytes {
189+
m.Logger().Warnf("Decoded length too large: %d bytes exceeds %d max decoded bytes limit (maxDecodedBodyBytes)", decodedLen, m.maxDecodedBodyBytes)
190+
http.Error(writer, fmt.Sprintf("decoded length too large: %d bytes exceeds %d max decoded bytes limit (maxDecodedBodyBytes)", decodedLen, m.maxDecodedBodyBytes), http.StatusRequestEntityTooLarge)
191+
return
192+
}
193+
160194
reqBuf, err := snappy.Decode(nil, compressed)
161195
if err != nil {
162196
m.Logger().Errorf("Decode error %v", err)

0 commit comments

Comments
 (0)