Skip to content

Commit 402fe18

Browse files
mergify[bot]karenzone
mergify[bot]
and
karenzone
authored
Doc: Incorporate field ref deep dive content (#17484) (#17508)
Co-authored-by: Rob Bavey <[email protected]> (cherry picked from commit e13fcad) Co-authored-by: Karen Metts <[email protected]>
1 parent 12e6659 commit 402fe18

File tree

1 file changed

+163
-17
lines changed

1 file changed

+163
-17
lines changed

Diff for: ‎docs/reference/event-dependent-configuration.md

+163-17
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,28 @@
11
---
22
mapped_pages:
33
- https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html
4+
- https://www.elastic.co/guide/en/logstash/current/field-references-deepdive.html
45
---
56

67
# Accessing event data and fields [event-dependent-configuration]
78

89
A Logstash pipeline usually has three stages: inputs → filters → outputs. Inputs generate events, filters modify them, and outputs ship them elsewhere.
910

10-
All events have properties. For example, an Apache access log has properties like status code (200, 404), request path ("/", "index.html"), HTTP verb (GET, POST), client IP address, and so forth. Logstash calls these properties "fields".
11+
All events have properties.
12+
For example, an Apache access log has properties like status code (200, 404), request path ("/", "index.html"), HTTP verb (GET, POST), client IP address, and so forth.
13+
Logstash calls these properties "fields."
14+
15+
Event and data field types:
16+
- [Field references](#logstash-config-field-references)
17+
- [Sprintf format](#sprintf)
18+
- [Conditionals](#conditionals)
1119

12-
Some configuration options in Logstash require the existence of fields in order to function. Because inputs generate events, there are no fields to evaluate within the input block—​they do not exist yet!
1320

1421
::::{important}
15-
[Field references](#logstash-config-field-references), [sprintf format](#sprintf), and [conditionals](#conditionals) do not work in input blocks. These configuration options depend on events and fields, and therefore, work only within filter and output blocks.
22+
Some configuration options in Logstash require the existence of fields in order to function. Because inputs generate events, there are no fields to evaluate within the input block—​they do not exist yet!
23+
24+
[Field references](#logstash-config-field-references), [sprintf format](#sprintf), and [conditionals](#conditionals) do not work in input blocks.
25+
These configuration options depend on events and fields, and therefore, work only within filter and output blocks.
1626
::::
1727

1828

@@ -21,11 +31,12 @@ Some configuration options in Logstash require the existence of fields in order
2131

2232
When you need to refer to a field by name, you can use the Logstash [field reference syntax](https://www.elastic.co/guide/en/logstash/current/field-references-deepdive.html).
2333

24-
The basic syntax to access a field is `[fieldname]`. If you are referring to a **top-level field**, you can omit the `[]` and simply use `fieldname`. To refer to a **nested field**, specify the full path to that field: `[top-level field][nested field]`.
34+
The basic syntax to access a field is `[fieldname]`.
35+
If you are referring to a **top-level field**, you can omit the `[]` and use `fieldname`. To refer to a **nested field**, specify the full path to that field: `[top-level field][nested field]`.
2536

2637
For example, this event has five top-level fields (agent, ip, request, response, ua) and three nested fields (status, bytes, os).
2738

28-
```js
39+
```
2940
{
3041
"agent": "Mozilla/5.0 (compatible; MSIE 9.0)",
3142
"ip": "192.168.24.44",
@@ -40,9 +51,146 @@ For example, this event has five top-level fields (agent, ip, request, response,
4051
}
4152
```
4253

43-
To reference the `os` field, specify `[ua][os]`. To reference a top-level field such as `request`, you can simply specify the field name.
54+
To reference the `os` field, specify `[ua][os]`.
55+
To reference a top-level field such as `request`, specify the field name.
56+
57+
58+
### Why use field references? [field-reference-deep-dive]
59+
60+
You might find situations in which you need to refer to a field or collection of fields by name.
61+
You can accomplish this goal using the Logstash field reference syntax.
62+
63+
The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.
64+
When a field name contains square brackets, the brackets must be properly [ escaped](#formal-grammar-escape-sequences).
65+
66+
Field references can be expressed literally within [conditional statements](#conditionals) in your pipeline configurations,
67+
as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:
68+
69+
```
70+
filter {
71+
# +----literal----+ +----literal----+
72+
# | | | |
73+
if [@metadata][date] and [@metadata][time] {
74+
mutate {
75+
add_field {
76+
"[@metadata][timestamp]" => "%{[@metadata][date]} %{[@metadata][time]}"
77+
# | | | | | | | |
78+
# +----string-argument---+ | +--field-ref----+ +--field-ref----+ |
79+
# +-------- sprintf format string ----------+
80+
}
81+
}
82+
}
83+
}
84+
```
85+
86+
87+
### Formal grammar [formal-grammar]
88+
89+
Expand the section below if you would like to explore the formal grammar.
90+
91+
::::{dropdown} Deep Dive: Formal grammar for field references
92+
:name: formal grammar
93+
94+
#### Field Reference Literal [formal-grammar-field-reference-literal]
95+
96+
A _Field Reference Literal_ is a sequence of one or more _Path Fragments_ that can be used directly in Logstash pipeline [conditionals](#conditionals) without any additional quoting.
97+
Example: `[request]`, `[response][status]`).
98+
99+
```
100+
fieldReferenceLiteral
101+
: ( pathFragment )+
102+
;
103+
```
44104

45-
For more detailed information, see [*Field References Deep Dive*](https://www.elastic.co/guide/en/logstash/current/field-references-deepdive.html).
105+
#### Field Reference (Event APIs) [formal-grammar-field-reference]
106+
107+
The Event API's methods for manipulating the fields of an event or using the sprintf syntax are more flexible than the pipeline grammar in what they accept as a Field Reference.
108+
Top-level fields can be referenced directly by their _Field Name_ without the square brackets, and there is some support for _Composite Field References_, simplifying use of programmatically-generated Field References.
109+
110+
A _Field Reference_ for use with the Event API is therefore one of:
111+
112+
- a single _Field Reference Literal_; OR
113+
- a single _Field Name_ (referencing a top-level field); OR
114+
- a single _Composite Field Reference_.
115+
116+
```
117+
eventApiFieldReference
118+
: fieldReferenceLiteral
119+
| fieldName
120+
| compositeFieldReference
121+
;
122+
```
123+
124+
#### Path Fragment [formal-grammar-path-fragment]
125+
126+
A _Path Fragment_ is a _Field Name_ wrapped in square brackets, such as `[request]`).
127+
128+
```
129+
pathFragment
130+
: '[' fieldName ']'
131+
;
132+
```
133+
134+
#### Field Name [formal-grammar-field-name]
135+
136+
A _Field Name_ is a sequence of characters that are _not_ square brackets (`[` or `]`).
137+
138+
```
139+
fieldName
140+
: ( ~( '[' | ']' ) )+
141+
;
142+
```
143+
144+
#### Composite Field Reference [formal-grammar-event-api-composite-field-reference]
145+
146+
In some cases, you may need to programmatically _compose_ a Field Reference from one or more Field References,
147+
such as when manipulating fields in a plugin or when using the Ruby Filter plugin and the Event API.
148+
149+
```
150+
fieldReference = "[path][to][deep nested field]"
151+
compositeFieldReference = "[@metadata][#{fieldReference}][size]"
152+
# => "[@metadata][[path][to][deep nested field]][size]"
153+
```
154+
155+
##### Canonical Representations of Composite Field References [canonical-representations]
156+
157+
| Acceptable _composite field reference_ | Canonical _field reference_ representation |
158+
| ------------- | ------------- |
159+
| `[[deep][nesting]][field]` | `[deep][nesting][field]`
160+
| `[foo][[bar]][bingo]` | `[foo][bar][bingo]`
161+
| `[[ok]]` | `[ok]`
162+
163+
164+
A _Composite Field Reference_ is a sequence of one or more _Path Fragments_ or _Embedded Field References_.
165+
166+
```
167+
compositeFieldReference
168+
: ( pathFragment | embeddedFieldReference )+
169+
;
170+
```
171+
172+
_Composite Field References_ are supported by the Event API, but are _not_ supported as literals in the Pipeline Configuration.
173+
174+
175+
#### Embedded Field Reference [formal-grammar-event-api-embedded-field-reference]
176+
177+
```
178+
embeddedFieldReference
179+
: '[' fieldReference ']'
180+
;
181+
```
182+
183+
An _Embedded Field Reference_ is a _Field Reference_ that is itself wrapped in square brackets (`[` and `]`), and can be a component of a _Composite Field Reference_.
184+
::::
185+
186+
### Escape sequences [formal-grammar-escape-sequences]
187+
188+
For {{ls}} to reference a field whose name contains a character that has special meaning in the field reference grammar, the character must be escaped.
189+
Logstash can be globally configured to use one of two field reference escape modes:
190+
191+
- `none` (default): no escape sequence processing is done. Fields containing literal square brackets cannot be referenced by the Event API.
192+
- `percent`: URI-style percent encoding of UTF-8 bytes. The left square bracket (`[`) is expressed as `%5B`, and the right square bracket (`]`) is expressed as `%5D`.
193+
- `ampersand`: HTML-style ampersand encoding (`&#` + decimal unicode codepoint + `;`). The left square bracket (`[`) is expressed as `&#91;`, and the right square bracket (`]`) is expressed as `&#93;`.
46194

47195

48196
## sprintf format [sprintf]
@@ -63,7 +211,7 @@ Instead of specifying a field name inside the curly braces, use the `%{{FORMAT}}
63211

64212
For example, if you want to use the file output to write logs based on the event’s UTC date and hour and the `type` field:
65213

66-
```js
214+
```
67215
output {
68216
file {
69217
path => "/var/log/%{type}.%{{yyyy.MM.dd.HH}}"
@@ -72,15 +220,10 @@ output {
72220
```
73221

74222
::::{note}
75-
The sprintf format continues to support [deprecated joda time format](http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html) strings as well using the `%{+FORMAT}` syntax. These formats are not directly interchangeable, and we advise you to begin using the more modern Java Time format.
223+
* The sprintf format continues to support [deprecated joda time format](http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html) strings as well using the `%{+FORMAT}` syntax. These formats are not directly interchangeable, and we advise you to begin using the more modern Java Time format.
224+
* A Logstash timestamp represents an instant on the UTC-timeline, so using sprintf formatters will produce results that may not align with your machine-local timezone.
76225
::::
77226

78-
79-
::::{note}
80-
A Logstash timestamp represents an instant on the UTC-timeline, so using sprintf formatters will produce results that may not align with your machine-local timezone.
81-
::::
82-
83-
84227
You can generate a fresh timestamp by using `%{{TIME_NOW}}` syntax instead of relying on the value in `@timestamp`. This is particularly useful when you need to estimate the time span of each plugin.
85228

86229
```js
@@ -195,7 +338,8 @@ output {
195338
}
196339
```
197340

198-
You can check for the existence of a specific field, but there’s currently no way to differentiate between a field that doesn’t exist versus a field that’s simply false. The expression `if [foo]` returns `false` when:
341+
You can check for the existence of a specific field, but there’s currently no way to differentiate between a field that doesn’t exist versus a field that’s simply false.
342+
The expression `if [foo]` returns `false` when:
199343

200344
* `[foo]` doesn’t exist in the event,
201345
* `[foo]` exists in the event, but is false, or
@@ -204,7 +348,9 @@ You can check for the existence of a specific field, but there’s currently no
204348
For more complex examples, see [Using Conditionals](/reference/config-examples.md#using-conditionals).
205349

206350
::::{note}
207-
Sprintf date/time format in conditionals is not currently supported. A workaround using the `@metadata` field is available. See [sprintf date/time format in conditionals](#date-time) for more details and an example.
351+
Sprintf date/time format in conditionals is not currently supported.
352+
A workaround using the `@metadata` field is available.
353+
See [sprintf date/time format in conditionals](#date-time) for more details and an example.
208354
::::
209355

210356

0 commit comments

Comments
 (0)