Skip to content

Commit 851584f

Browse files
author
Jorge Niedbalski
committed
docs(processors): add conditional processing documentation
1 parent b7198cb commit 851584f

File tree

2 files changed

+222
-0
lines changed

2 files changed

+222
-0
lines changed

‎pipeline/processors/README.md

+15
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,17 @@
11
# Processors
22

3+
Processors are components that can modify, transform, or enhance data records as they flow through the Fluent Bit pipeline.
4+
5+
## Available processors
6+
7+
Fluent Bit offers the following processors:
8+
9+
- [Content Modifier](content-modifier.md): Manipulate message content, metadata/attributes for logs and traces
10+
- [Labels](labels.md): Add, update or delete labels in records
11+
- [Metrics Selector](metrics-selector.md): Select specific metrics
12+
- [OpenTelemetry Envelope](opentelemetry-envelope.md): Convert logs to OpenTelemetry format
13+
- [SQL](sql.md): Process records using SQL queries
14+
15+
## Features
16+
17+
- [Conditional Processing](conditional-processing.md): Apply processors only to records that meet specific conditions

‎pipeline/processors/conditional-processing.md

+207
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Conditional processing
2+
3+
Conditional processing allows you to selectively apply processors to log records based on field values. This feature enables you to create processing pipelines that apply processors only to records that match specific criteria.
4+
5+
## Configuration format
6+
7+
Conditional processing is available for processors in the YAML configuration format. To apply a processor conditionally, you add a `condition` block to the processor configuration:
8+
9+
```yaml
10+
- name: processor_name
11+
# Regular processor configuration...
12+
condition:
13+
op: and|or
14+
rules:
15+
- field: "$field_name"
16+
op: comparison_operator
17+
value: comparison_value
18+
# Additional rules...
19+
```
20+
21+
### Condition operators
22+
23+
The `op` field in the condition block specifies the logical operator to apply across all rules:
24+
25+
| Operator | Description |
26+
| --- | --- |
27+
| `and` | All rules must evaluate to true for the condition to be true |
28+
| `or` | At least one rule must evaluate to true for the condition to be true |
29+
30+
### Rules
31+
32+
Each rule consists of:
33+
34+
- `field`: The field to evaluate (must use [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md) with `$` prefix)
35+
- `op`: The comparison operator
36+
- `value`: The value to compare against
37+
38+
### Comparison operators
39+
40+
The following comparison operators are supported:
41+
42+
| Operator | Description |
43+
| --- | --- |
44+
| `eq` | Equal to |
45+
| `neq` | Not equal to |
46+
| `gt` | Greater than |
47+
| `lt` | Less than |
48+
| `gte` | Greater than or equal to |
49+
| `lte` | Less than or equal to |
50+
| `regex` | Matches regular expression |
51+
| `not_regex` | Does not match regular expression |
52+
| `in` | Value is in the specified array |
53+
| `not_in` | Value is not in the specified array |
54+
55+
### Field access
56+
57+
You can access record fields using [record accessor syntax](/administration/configuring-fluent-bit/classic-mode/record-accessor.md):
58+
59+
- Simple fields: `$field`
60+
- Nested fields: `$parent['child']['subchild']`
61+
62+
## Examples
63+
64+
### Simple condition
65+
66+
Process records only when the HTTP method is POST:
67+
68+
```yaml
69+
pipeline:
70+
inputs:
71+
- name: dummy
72+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
73+
tag: request.log
74+
processors:
75+
logs:
76+
- name: content_modifier
77+
action: insert
78+
key: modified_if_post
79+
value: true
80+
condition:
81+
op: and
82+
rules:
83+
- field: "$request['method']"
84+
op: eq
85+
value: "POST"
86+
```
87+
88+
### Multiple conditions with AND
89+
90+
Apply a processor only when both conditions are met:
91+
92+
```yaml
93+
pipeline:
94+
inputs:
95+
- name: dummy
96+
dummy: '{"request": {"method": "POST", "path": "/api/v1/sensitive-data"}}'
97+
tag: request.log
98+
processors:
99+
logs:
100+
- name: content_modifier
101+
action: insert
102+
key: requires_audit
103+
value: true
104+
condition:
105+
op: and
106+
rules:
107+
- field: "$request['method']"
108+
op: eq
109+
value: "POST"
110+
- field: "$request['path']"
111+
op: regex
112+
value: "\/sensitive-.*"
113+
```
114+
115+
### OR condition example
116+
117+
Flag records that meet any of multiple criteria:
118+
119+
```yaml
120+
pipeline:
121+
inputs:
122+
- name: dummy
123+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource", "status_code": 200, "response_time": 150}}'
124+
tag: request.log
125+
processors:
126+
logs:
127+
- name: content_modifier
128+
action: insert
129+
key: requires_performance_check
130+
value: true
131+
condition:
132+
op: or
133+
rules:
134+
- field: "$request['response_time']"
135+
op: gt
136+
value: 100
137+
- field: "$request['status_code']"
138+
op: gte
139+
value: 400
140+
```
141+
142+
### Using IN operator
143+
144+
Apply a processor when a value matches one of multiple options:
145+
146+
```yaml
147+
pipeline:
148+
inputs:
149+
- name: dummy
150+
dummy: '{"request": {"method": "GET", "path": "/api/v1/resource"}}'
151+
tag: request.log
152+
processors:
153+
logs:
154+
- name: content_modifier
155+
action: insert
156+
key: high_priority_method
157+
value: true
158+
condition:
159+
op: and
160+
rules:
161+
- field: "$request['method']"
162+
op: in
163+
value: ["POST", "PUT", "DELETE"]
164+
```
165+
166+
## Multiple processors with conditions
167+
168+
You can chain multiple conditional processors to create advanced processing pipelines:
169+
170+
```yaml
171+
pipeline:
172+
inputs:
173+
- name: dummy
174+
dummy: '{"log": "Error: Connection refused", "level": "error", "service": "api-gateway"}'
175+
tag: app.log
176+
processors:
177+
logs:
178+
- name: content_modifier
179+
action: insert
180+
key: alert
181+
value: true
182+
condition:
183+
op: and
184+
rules:
185+
- field: "$level"
186+
op: eq
187+
value: "error"
188+
- field: "$service"
189+
op: in
190+
value: ["api-gateway", "authentication", "database"]
191+
192+
- name: content_modifier
193+
action: insert
194+
key: paging_required
195+
value: true
196+
condition:
197+
op: and
198+
rules:
199+
- field: "$log"
200+
op: regex
201+
value: "(?i)(connection refused|timeout|crash)"
202+
- field: "$level"
203+
op: in
204+
value: ["error", "fatal"]
205+
```
206+
207+
This configuration would add the `alert` field to error logs from critical services, and add the `paging_required` field to errors containing specific critical patterns.

0 commit comments

Comments
 (0)