Skip to content

Commit d6dc97e

Browse files
stream-processing: getting-started: fluent-bit-sql: general cleanup
Signed-off-by: Alexa Kreizinger <[email protected]>
1 parent 447e459 commit d6dc97e

File tree

1 file changed

+37
-79
lines changed

1 file changed

+37
-79
lines changed
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
1-
# Fluent Bit + SQL
1+
# Fluent Bit and SQL
22

3-
Fluent Bit stream processor uses common SQL to perform record queries. The following section describe the features available and examples of it.
3+
Stream processing in Fluent Bit uses SQL to perform record queries.
44

5-
## Statements
5+
For additional information, see the [stream processing README file](https://github.com/fluent/fluent-bit/tree/master/src/stream_processor).
66

7-
You can find the detailed query language syntax in BNF form [here](https://github.com/fluent/fluent-bit/tree/master/src/stream_processor). The following section will be a brief introduction on how to write SQL queries for Fluent Bit stream processing.
7+
## Statements
88

9-
### SELECT Statement
9+
Use the following SQL statements in Fluent Bit.
1010

11-
#### Synopsis
11+
### SELECT
1212

1313
```sql
1414
SELECT results_statement
@@ -18,201 +18,159 @@ SELECT results_statement
1818
[GROUP BY groupby]
1919
```
2020

21-
#### Description
21+
Groups keys from records that originate from a specified stream, or from records that match a specific tag pattern.
2222

23-
Select keys from records coming from a stream or records matching a specific Tag pattern. Note that a simple `SELECT` statement **not** associated from a stream creation will send the results to the standard output interface \(stdout\), useful for debugging purposes.
23+
{% hint style="info" %}
24+
A `SELECT` statement not associated with stream creation will send the results to the standard output interface, which can be helpful for debugging purposes.
25+
{% endhint %}
2426

25-
The query allows filtering the results by applying a condition using `WHERE` statement. We will explain `WINDOW` and `GROUP BY` statements later in aggregation functions section.
27+
You can filter the results of this query by applying a condition through a `WHERE` statement. For information about the `WINDOW` and `GROUP BY` statements, see [Aggregation functions](#aggregation-functions).
2628

2729
#### Examples
2830

29-
Select all keys from records coming from a stream called _apache_:
31+
Selects all keys from records that originate from a stream called `apache`:
3032

3133
```sql
3234
SELECT * FROM STREAM:apache;
3335
```
3436

35-
Select code key from records which Tag starts with _apache._:
37+
Selects the `code` key from records with tags whose name begins with `apache`:
3638

3739
```sql
3840
SELECT code AS http_status FROM TAG:'apache.*';
3941
```
4042

41-
> Since the TAG selector allows the use of wildcards, we put the value between single quotes.
42-
43-
### CREATE STREAM Statement
44-
45-
#### Synopsis
43+
### CREATE STREAM
4644

4745
```sql
4846
CREATE STREAM stream_name
4947
[WITH (property_name=value, [...])]
5048
AS select_statement
5149
```
5250

53-
#### Description
54-
55-
Create a new stream of data using the results from the `SELECT` statement. New stream created can be optionally re-ingested back into Fluent Bit pipeline if the property _Tag_ is set in the WITH statement.
51+
Creates a new stream of data using the results from a `SELECT` statement. If the `Tag` property in the `WITH` statement is set, this new stream can optionally be re-ingested into the Fluent Bit pipeline.
5652

5753
#### Examples
5854

59-
Create a new stream called _hello_ from stream called _apache_:
55+
Creates a new stream called `hello_` from a stream called `apache`:
6056

6157
```sql
6258
CREATE STREAM hello AS SELECT * FROM STREAM:apache;
6359
```
6460

65-
Create a new stream called hello for all records which original Tag starts with _apache_:
61+
Creates a new stream called `hello` for all records whose original tag name begins with `apache`:
6662

6763
```sql
6864
CREATE STREAM hello AS SELECT * FROM TAG:'apache.*';
6965
```
7066

7167
## Aggregation Functions
7268

73-
Aggregation functions are used in `results_statement` on the keys, allowing to perform data calculation on groups of records. Group of records that aggregation functions apply on are determined by `WINDOW` keyword. When `WINDOW` is not specified, aggregation functions apply on the current buffer of records received, which may have non-deterministic number of elements. Aggregation functions can be applied on records in a window of a specific time interval \(see the syntax of `WINDOW` in select statement\).
69+
You can use aggregation functions in the `results_statement` on keys, which lets you perform data calculation on groups of records. These groups are determined by the `WINDOW` key. If `WINDOW` is unspecified, aggregation functions are applied to the current buffer of records received, which might have a non-deterministic number of elements. You can also apply aggregation functions to records in a window of a specific time interval.
7470

75-
Fluent Bit streaming currently supports tumbling window, which is non-overlapping window type. That means, a window of size 5 seconds performs aggregation computations on records over a 5-second interval, and then starts new calculations for the next interval.
71+
Fluent Bit uses a tumbling window, which is non-overlapping. For example, a window size of `5` performs aggregation computations on records during a five-second interval, then starts new calculations for the next interval.
7672

77-
In addition, the syntax support `GROUP BY` statement, which groups the results by the one or more keys, when they have the same values.
73+
Additionally, you can use the `GROUP BY` statement to group results by one or more keys with matching values.
7874

7975
### AVG
8076

81-
#### Synopsis
82-
8377
```sql
8478
SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' ;
8579
```
8680

87-
#### Description
88-
89-
Calculates the average of request sizes in POST requests.
81+
Calculates the average size of POST requests.
9082

9183
### COUNT
9284

93-
#### Synopsis
94-
9585
```sql
96-
SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (5 SECOND) GROUP BY host;
86+
SELECT host, COUNT(*) FROM STREAM:apache WINDOW TUMBLING (X SECOND) GROUP BY host;
9787
```
9888

99-
#### Description
100-
101-
Count the number of records in 5 second windows group by host IP addresses.
89+
Counts the number of records in 5 second window, grouped by host IP addresses.
10290

10391
### MIN
10492

105-
#### Synopsis
106-
10793
```sql
10894
SELECT MIN(key) FROM STREAM:apache;
10995
```
11096

111-
#### Description
112-
113-
Gets the minimum value of a key in a set of records.
97+
Returns the minimum value of a key in a set of records.
11498

11599
### MAX
116100

117-
#### Synopsis
118-
119101
```sql
120-
SELECT MIN(key) FROM STREAM:apache;
102+
SELECT MAX(key) FROM STREAM:apache;
121103
```
122-
123-
#### Description
124-
125-
Gets the maximum value of a key in a set of records.
104+
Returns the maximum value of a key in a set of records.
126105

127106
### SUM
128107

129-
#### Synopsis
130-
131108
```sql
132109
SELECT SUM(key) FROM STREAM:apache;
133110
```
134111

135-
#### Description
136-
137-
Calculates the sum of all values of key in a set of records.
112+
Calculates the sum of all values of a key in a set of records.
138113

139114
## Time Functions
140115

141-
Time functions adds a new key into the record with timing data
116+
Use time functions to add a new key with time data into a record.
142117

143118
### NOW
144119

145-
#### Synopsis
146-
147120
```sql
148121
SELECT NOW() FROM STREAM:apache;
149122
```
150123

151-
#### Description
152-
153-
Add system time using format: %Y-%m-%d %H:%M:%S. Output example: 2019-03-09 21:36:05.
124+
Adds the current system time to a record using the format `%Y-%m-%d %H:%M:%S`. Output example: `2019-03-09 21:36:05`.
154125

155126
### UNIX\_TIMESTAMP
156127

157-
#### Synopsis
158-
159128
```sql
160129
SELECT UNIX_TIMESTAMP() FROM STREAM:apache;
161130
```
162131

163-
#### Description
164-
165-
Add current Unix timestamp to the record. Output example: 1552196165 .
132+
Adds the current Unix time to a record. Output example: `1552196165`.
166133

167134
## Record Functions
168135

169-
Record functions append new keys to the record using values from the record context.
136+
Use record functions to append new keys to a record using values from the record's context.
170137

171138
### RECORD\_TAG
172139

173-
#### Synopsis
174-
175140
```sql
176141
SELECT RECORD_TAG() FROM STREAM:apache;
177142
```
178143

179-
#### Description
180-
181144
Append Tag string associated to the record as a new key.
182145

183146
### RECORD\_TIME
184147

185-
#### Synopsis
186-
187148
```sql
188149
SELECT RECORD_TIME() FROM STREAM:apache;
189150
```
190151

191-
## WHERE Condition
152+
## The WHERE condition
192153

193-
Similar to conventional SQL statements, `WHERE` condition is supported in Fluent Bit query language. The language supports conditions over keys and subkeys, for instance:
154+
Similar to conventional SQL statements, Fluent Bit supports the `WHERE` condition. You can use this condition in both keys and subkeys. For example:
194155

195156
```sql
196157
SELECT AVG(size) FROM STREAM:apache WHERE method = 'POST' AND status = 200;
197158
```
198159

199-
It is possible to check the existence of a key in the record using record-specific function `@record.contains`:
160+
You can confirm whether a key exists in a record by using the record-specific function `@record.contains`:
200161

201162
```sql
202163
SELECT MAX(key) FROM STREAM:apache WHERE @record.contains(key);
203164
```
204165

205-
And to check if the value of a key is/is not `NULL`:
166+
And to check whether the value of a key is `NULL`:
206167

207168
```sql
208169
SELECT MAX(key) FROM STREAM:apache WHERE key IS NULL;
209170
```
210171

172+
Or similar:
173+
211174
```sql
212175
SELECT * FROM STREAM:apache WHERE user IS NOT NULL;
213176
```
214-
215-
#### Description
216-
217-
Append a new key with the record Timestamp in _double_ format: seconds.nanoseconds. Output example: 1552196165.705683 .
218-

0 commit comments

Comments
 (0)