Skip to content

Update Stream and Filter logs docs #1364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Update Stream and Filter logs docs #1364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9bfa635
update logs timestamps
mdbirnstiehl May 5, 2025
da162b2
update timestamps
mdbirnstiehl May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 27 additions & 27 deletions solutions/observability/logs/filter-aggregate-logs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,28 +91,28 @@ Add some logs with varying timestamps and log levels to your data stream:
```console
POST logs-example-default/_bulk
{ "create": {} }
{ "message": "2023-09-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "message": "2025-04-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "create": {} }
{ "message": "2023-09-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "message": "2025-04-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "create": {} }
{ "message": "2023-09-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "message": "2025-04-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "create": {} }
{ "message": "2023-09-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
{ "message": "2025-04-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
```

For this example, let’s look for logs with a `WARN` or `ERROR` log level that occurred on September 14th or 15th. From Discover:
For this example, let’s look for logs with a `WARN` or `ERROR` log level that occurred on April 14th or 15th. From Discover:

1. Make sure **All logs** is selected in the **Data views** menu.
1. Add the following KQL query in the search bar to filter for logs with log levels of `WARN` or `ERROR`:

```text
log.level: ("ERROR" or "WARN")
```
1. Click the current time range, select **Absolute**, and set the **Start date** to `Sep 14, 2023 @ 00:00:00.000`.
1. Click the current time range, select **Absolute**, and set the **Start date** to `Apr 14, 2025 @ 00:00:00.000`.

![Set the time range start date](../../images/serverless-logs-start-date.png "")

1. Click the end of the current time range, select **Absolute**, and set the **End date** to `Sep 15, 2023 @ 23:59:59.999`.
1. Click the end of the current time range, select **Absolute**, and set the **End date** to `Apr 15, 2025 @ 23:59:59.999`.

![Set the time range end date](/solutions/images/serverless-logs-end-date.png "")

Expand All @@ -138,16 +138,16 @@ First, from **Developer Tools**, add some logs with varying timestamps and log l
```console
POST logs-example-default/_bulk
{ "create": {} }
{ "message": "2023-09-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "message": "2025-04-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "create": {} }
{ "message": "2023-09-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "message": "2025-04-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "create": {} }
{ "message": "2023-09-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "message": "2025-04-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "create": {} }
{ "message": "2023-09-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
{ "message": "2025-04-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
```

Let’s say you want to look into an event that occurred between September 14th and 15th. The following boolean query filters for logs with timestamps during those days that also have a log level of `ERROR` or `WARN`.
Let’s say you want to look into an event that occurred between April 14th and 15th. The following boolean query filters for logs with timestamps during those days that also have a log level of `ERROR` or `WARN`.

```console
POST /logs-example-default/_search
Expand All @@ -158,8 +158,8 @@ POST /logs-example-default/_search
{
"range": {
"@timestamp": {
"gte": "2023-09-14T00:00:00",
"lte": "2023-09-15T23:59:59"
"gte": "2025-04-14T00:00:00",
"lte": "2025-04-15T23:59:59"
}
}
},
Expand All @@ -183,27 +183,27 @@ The filtered results should show `WARN` and `ERROR` logs that occurred within th
...
"hits": [
{
"_index": ".ds-logs-example-default-2023.09.25-000001",
"_index": ".ds-logs-example-default-2025.04.25-000001",
"_id": "JkwPzooBTddK4OtTQToP",
"_score": 0,
"_source": {
"message": "192.168.1.101 Disk usage exceeds 90%.",
"log": {
"level": "WARN"
},
"@timestamp": "2023-09-15T08:15:20.234Z"
"@timestamp": "2025-04-15T08:15:20.234Z"
}
},
{
"_index": ".ds-logs-example-default-2023.09.25-000001",
"_index": ".ds-logs-example-default-2025.04.25-000001",
"_id": "A5YSzooBMYFrNGNwH75O",
"_score": 0,
"_source": {
"message": "192.168.1.102 Critical system failure detected.",
"log": {
"level": "ERROR"
},
"@timestamp": "2023-09-14T10:30:45.789Z"
"@timestamp": "2025-04-14T10:30:45.789Z"
}
}
]
Expand All @@ -223,19 +223,19 @@ First, from **Developer Tools**, add some logs with varying log levels to your d
```console
POST logs-example-default/_bulk
{ "create": {} }
{ "message": "2023-09-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "message": "2025-04-15T08:15:20.234Z WARN 192.168.1.101 Disk usage exceeds 90%." }
{ "create": {} }
{ "message": "2023-09-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "message": "2025-04-14T10:30:45.789Z ERROR 192.168.1.102 Critical system failure detected." }
{ "create": {} }
{ "message": "2023-09-15T12:45:55.123Z INFO 192.168.1.103 Application successfully started." }
{ "message": "2025-04-15T12:45:55.123Z INFO 192.168.1.103 Application successfully started." }
{ "create": {} }
{ "message": "2023-09-14T15:20:10.789Z WARN 192.168.1.104 Network latency exceeding threshold." }
{ "message": "2025-04-14T15:20:10.789Z WARN 192.168.1.104 Network latency exceeding threshold." }
{ "create": {} }
{ "message": "2023-09-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "message": "2025-04-10T14:20:45.789Z ERROR 192.168.1.105 Database connection lost." }
{ "create": {} }
{ "message": "2023-09-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
{ "message": "2025-04-20T09:40:32.345Z INFO 192.168.1.106 User logout initiated." }
{ "create": {} }
{ "message": "2023-09-21T15:20:55.678Z DEBUG 192.168.1.102 Database connection established." }
{ "message": "2025-04-21T15:20:55.678Z DEBUG 192.168.1.102 Database connection established." }
```

Next, run this command to aggregate your log data using the `log.level` field:
Expand Down Expand Up @@ -297,8 +297,8 @@ GET /logs-example-default/_search
"query": {
"range": {
"@timestamp": {
"gte": "2023-09-14T00:00:00",
"lte": "2023-09-15T23:59:59"
"gte": "2025-04-14T00:00:00",
"lte": "2025-04-15T23:59:59"
}
}
},
Expand Down
Loading
Loading