You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fluent Bit stream processor uses common SQL to perform record queries. The following section describe the features available and examples of it.
3
+
Stream processing in Fluent Bit uses SQL to perform record queries.
4
4
5
-
## Statements
5
+
For additional information, see the [stream processing README file](https://github.com/fluent/fluent-bit/tree/master/src/stream_processor).
6
6
7
-
You can find the detailed query language syntax in BNF form [here](https://github.com/fluent/fluent-bit/tree/master/src/stream_processor). The following section will be a brief introduction on how to write SQL queries for Fluent Bit stream processing.
7
+
## Statements
8
8
9
-
### SELECT Statement
9
+
Use the following SQL statements in Fluent Bit.
10
10
11
-
#### Synopsis
11
+
###SELECT
12
12
13
13
```sql
14
14
SELECT results_statement
@@ -18,201 +18,159 @@ SELECT results_statement
18
18
[GROUP BY groupby]
19
19
```
20
20
21
-
#### Description
21
+
Groups keys from records that originate from a specified stream, or from records that match a specific tag pattern.
22
22
23
-
Select keys from records coming from a stream or records matching a specific Tag pattern. Note that a simple `SELECT` statement **not** associated from a stream creation will send the results to the standard output interface \(stdout\), useful for debugging purposes.
23
+
{% hint style="info" %}
24
+
A `SELECT` statement not associated with stream creation will send the results to the standard output interface, which can be helpful for debugging purposes.
25
+
{% endhint %}
24
26
25
-
The query allows filtering the results by applying a condition using `WHERE` statement. We will explain `WINDOW` and `GROUP BY` statements later in aggregationfunctions section.
27
+
You can filter the results of this query by applying a condition through a `WHERE` statement. For information about the `WINDOW` and `GROUP BY` statements, see [Aggregation functions](#aggregation-functions).
26
28
27
29
#### Examples
28
30
29
-
Select all keys from records coming from a stream called _apache_:
31
+
Selects all keys from records that originate from a stream called `apache`:
30
32
31
33
```sql
32
34
SELECT*FROM STREAM:apache;
33
35
```
34
36
35
-
Select code key from records which Tag starts with _apache._:
37
+
Selects the `code` key from records with tags whose name begins with `apache`:
36
38
37
39
```sql
38
40
SELECT code AS http_status FROM TAG:'apache.*';
39
41
```
40
42
41
-
> Since the TAG selector allows the use of wildcards, we put the value between single quotes.
42
-
43
-
### CREATE STREAM Statement
44
-
45
-
#### Synopsis
43
+
### CREATE STREAM
46
44
47
45
```sql
48
46
CREATE STREAM stream_name
49
47
[WITH (property_name=value, [...])]
50
48
AS select_statement
51
49
```
52
50
53
-
#### Description
54
-
55
-
Create a new stream of data using the results from the `SELECT` statement. New stream created can be optionally re-ingested back into Fluent Bit pipeline if the property _Tag_ is set in the WITH statement.
51
+
Creates a new stream of data using the results from a `SELECT` statement. If the `Tag` property in the `WITH` statement is set, this new stream can optionally be re-ingested into the Fluent Bit pipeline.
56
52
57
53
#### Examples
58
54
59
-
Create a new stream called _hello_ from stream called _apache_:
55
+
Creates a new stream called `hello_` from a stream called `apache`:
60
56
61
57
```sql
62
58
CREATE STREAM hello ASSELECT*FROM STREAM:apache;
63
59
```
64
60
65
-
Create a new stream called hello for all records which original Tag starts with _apache_:
61
+
Creates a new stream called `hello` for all records whose original tag name begins with `apache`:
66
62
67
63
```sql
68
64
CREATE STREAM hello ASSELECT*FROM TAG:'apache.*';
69
65
```
70
66
71
67
## Aggregation Functions
72
68
73
-
Aggregation functions are used in `results_statement` on the keys, allowing to perform data calculation on groups of records. Group of records that aggregation functions apply on are determined by `WINDOW`keyword. When`WINDOW` is not specified, aggregation functions apply on the current buffer of records received, which may have non-deterministic number of elements. Aggregation functions can be applied on records in a window of a specific time interval\(see the syntax of `WINDOW` in select statement\).
69
+
You can use aggregation functions in the `results_statement` on keys, which lets you perform data calculation on groups of records. These groups are determined by the `WINDOW`key. If`WINDOW` is unspecified, aggregation functions are applied to the current buffer of records received, which might have a non-deterministic number of elements. You can also apply aggregation functions to records in a window of a specific time interval.
74
70
75
-
Fluent Bit streaming currently supports tumbling window, which is non-overlapping window type. That means, a window of size 5 seconds performs aggregation computations on records over a 5-second interval, and then starts new calculations for the next interval.
71
+
Fluent Bit uses a tumbling window, which is non-overlapping. For example, a window size of `5`performs aggregation computations on records during a five-second interval, then starts new calculations for the next interval.
76
72
77
-
In addition, the syntax support `GROUP BY` statement, which groups the results by the one or more keys, when they have the same values.
73
+
Additionally, you can use the `GROUP BY` statement to group results by one or more keys with matching values.
78
74
79
75
### AVG
80
76
81
-
#### Synopsis
82
-
83
77
```sql
84
78
SELECTAVG(size) FROM STREAM:apache WHERE method ='POST' ;
85
79
```
86
80
87
-
#### Description
88
-
89
-
Calculates the average of request sizes in POST requests.
81
+
Calculates the average size of POST requests.
90
82
91
83
### COUNT
92
84
93
-
#### Synopsis
94
-
95
85
```sql
96
-
SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (5 SECOND) GROUP BY host;
86
+
SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (X SECOND) GROUP BY host;
97
87
```
98
88
99
-
#### Description
100
-
101
-
Count the number of records in 5 second windows group by host IP addresses.
89
+
Counts the number of records in 5 second window, grouped by host IP addresses.
102
90
103
91
### MIN
104
92
105
-
#### Synopsis
106
-
107
93
```sql
108
94
SELECTMIN(key) FROM STREAM:apache;
109
95
```
110
96
111
-
#### Description
112
-
113
-
Gets the minimum value of a key in a set of records.
97
+
Returns the minimum value of a key in a set of records.
114
98
115
99
### MAX
116
100
117
-
#### Synopsis
118
-
119
101
```sql
120
-
SELECTMIN(key) FROM STREAM:apache;
102
+
SELECTMAX(key) FROM STREAM:apache;
121
103
```
122
-
123
-
#### Description
124
-
125
-
Gets the maximum value of a key in a set of records.
104
+
Returns the maximum value of a key in a set of records.
126
105
127
106
### SUM
128
107
129
-
#### Synopsis
130
-
131
108
```sql
132
109
SELECTSUM(key) FROM STREAM:apache;
133
110
```
134
111
135
-
#### Description
136
-
137
-
Calculates the sum of all values of key in a set of records.
112
+
Calculates the sum of all values of a key in a set of records.
138
113
139
114
## Time Functions
140
115
141
-
Time functions adds a new key into the record with timing data
116
+
Use time functions to add a new key with time data into a record.
142
117
143
118
### NOW
144
119
145
-
#### Synopsis
146
-
147
120
```sql
148
121
SELECT NOW() FROM STREAM:apache;
149
122
```
150
123
151
-
#### Description
152
-
153
-
Add system time using format: %Y-%m-%d %H:%M:%S. Output example: 2019-03-09 21:36:05.
124
+
Adds the current system time to a record using the format `%Y-%m-%d %H:%M:%S`. Output example: `2019-03-09 21:36:05`.
154
125
155
126
### UNIX\_TIMESTAMP
156
127
157
-
#### Synopsis
158
-
159
128
```sql
160
129
SELECT UNIX_TIMESTAMP() FROM STREAM:apache;
161
130
```
162
131
163
-
#### Description
164
-
165
-
Add current Unix timestamp to the record. Output example: 1552196165 .
132
+
Adds the current Unix time to a record. Output example: `1552196165`.
166
133
167
134
## Record Functions
168
135
169
-
Record functions append new keys to the record using values from the record context.
136
+
Use record functions to append new keys to a record using values from the record's context.
170
137
171
138
### RECORD\_TAG
172
139
173
-
#### Synopsis
174
-
175
140
```sql
176
141
SELECT RECORD_TAG() FROM STREAM:apache;
177
142
```
178
143
179
-
#### Description
180
-
181
144
Append Tag string associated to the record as a new key.
182
145
183
146
### RECORD\_TIME
184
147
185
-
#### Synopsis
186
-
187
148
```sql
188
149
SELECT RECORD_TIME() FROM STREAM:apache;
189
150
```
190
151
191
-
## WHERE Condition
152
+
## The WHERE condition
192
153
193
-
Similar to conventional SQL statements, `WHERE` condition is supported in Fluent Bit query language. The language supports conditions over keys and subkeys, for instance:
154
+
Similar to conventional SQL statements, Fluent Bit supports the `WHERE` condition. You can use this condition in both keys and subkeys. For example:
194
155
195
156
```sql
196
157
SELECTAVG(size) FROM STREAM:apache WHERE method ='POST'AND status =200;
197
158
```
198
159
199
-
It is possible to check the existence of a key in the record using record-specific function `@record.contains`:
160
+
You can confirm whether a key exists in a record by using the record-specific function `@record.contains`:
200
161
201
162
```sql
202
163
SELECTMAX(key) FROM STREAM:apache WHERE @record.contains(key);
203
164
```
204
165
205
-
And to check if the value of a key is/is not`NULL`:
166
+
And to check whether the value of a key is `NULL`:
206
167
207
168
```sql
208
169
SELECTMAX(key) FROM STREAM:apache WHERE key IS NULL;
209
170
```
210
171
172
+
Or similar:
173
+
211
174
```sql
212
175
SELECT*FROM STREAM:apache WHERE user IS NOT NULL;
213
176
```
214
-
215
-
#### Description
216
-
217
-
Append a new key with the record Timestamp in _double_ format: seconds.nanoseconds. Output example: 1552196165.705683 .
0 commit comments