Skip to content

Commit a2684ea

Browse files
colleenmcginniscolleenmcginnis
committed
add migrated files
1 parent dfa7846 commit a2684ea

File tree

6 files changed

+1532
-0
lines changed

6 files changed

+1532
-0
lines changed

‎docs/docset.yml

Lines changed: 489 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/reference/index.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
mapped_pages:
3+
- https://www.elastic.co/guide/en/ecs-logging/nodejs/current/intro.html
4+
- https://www.elastic.co/guide/en/ecs-logging/nodejs/current/index.html
5+
---
6+
7+
# ECS Logging Node.js [intro]
8+
9+
Node.js ECS loggers are formatter plugins for your favorite logging libraries. They make it easy to format your logs into ECS-compatible JSON. In combination with [filebeat](https://www.elastic.co/products/beats/filebeat) you can send your logs directly to Elasticsearch and leverage [Kibana’s Logs app](docs-content://solutions/observability/logs/explore-logs.md) to inspect all logs in one single place.
10+
11+
The Node.js ECS logging formatters log structured JSON and support serialization of Error objects and HTTP Request and Response objects from Node.js core and popular web frameworks. A minimal log record includes the following fields:
12+
13+
```json
14+
{
15+
"@timestamp": "2021-01-13T21:32:38.095Z",
16+
"log.level": "info",
17+
"message": "hi",
18+
"ecs.version": "8.10.0"
19+
}
20+
```
21+
22+
::::{tip}
23+
Want to learn more about ECS, ECS logging, and other available language plugins? See the [ECS logging guide](ecs-logging://docs/reference/intro.md).
24+
::::
25+
26+
27+
Ready to jump into Node.js ECS logging?
28+
29+
* [ECS Logging with Pino](/reference/pino.md)
30+
* [ECS Logging with Winston](/reference/winston.md)
31+
* [ECS Logging with Morgan](/reference/morgan.md)
32+
33+
If you’d like to try out a tutorial using Node.js ECS logging with winston, see [Ingest logs from a Node.js web application using Filebeat](docs-content://manage-data/ingest/ingesting-data-from-applications/ingest-logs-from-nodejs-web-application-using-filebeat.md).
34+

‎docs/reference/morgan.md

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
---
2+
mapped_pages:
3+
- https://www.elastic.co/guide/en/ecs-logging/nodejs/current/morgan.html
4+
---
5+
6+
# ECS Logging with Morgan [morgan]
7+
8+
This Node.js package provides a formatter for the [morgan](https://github.com/expressjs/morgan#readme) logging middleware — commonly used with Express — compatible with [Elastic Common Schema (ECS) logging](ecs-logging://docs/reference/intro.md). In combination with the [Filebeat](https://www.elastic.co/beats/filebeat) shipper, you can [monitor all your logs](https://www.elastic.co/log-monitoring) in one place in the Elastic Stack.
9+
10+
11+
## Setup [_setup_3]
12+
13+
### Step 1: Install [morgan-setup-step-1]
14+
15+
```cmd
16+
$ npm install @elastic/ecs-morgan-format
17+
```
18+
19+
20+
### Step 2: Configure [morgan-setup-step-2]
21+
22+
```js
23+
const app = require('express')();
24+
const morgan = require('morgan');
25+
const { ecsFormat } = require('@elastic/ecs-morgan-format');
26+
27+
app.use(morgan(ecsFormat(/* options */))); <1>
28+
29+
// ...
30+
app.get('/', function (req, res) {
31+
res.send('hello, world!');
32+
})
33+
app.listen(3000);
34+
```
35+
36+
1. Pass the ECS formatter to `morgan()`.
37+
38+
39+
40+
### Step 3: Configure Filebeat [morgan-setup-step-3]
41+
42+
The best way to collect the logs once they are ECS-formatted is with [Filebeat](beats://docs/reference/filebeat/filebeat-overview.md):
43+
44+
:::::::{tab-set}
45+
46+
::::::{tab-item} Log file
47+
1. Follow the [Filebeat quick start](beats://docs/reference/filebeat/filebeat-installation-configuration.md)
48+
2. Add the following configuration to your `filebeat.yaml` file.
49+
50+
For Filebeat 7.16+
51+
52+
```yaml
53+
filebeat.inputs:
54+
- type: filestream <1>
55+
paths: /path/to/logs.json
56+
parsers:
57+
- ndjson:
58+
overwrite_keys: true <2>
59+
add_error_key: true <3>
60+
expand_keys: true <4>
61+
62+
processors: <5>
63+
- add_host_metadata: ~
64+
- add_cloud_metadata: ~
65+
- add_docker_metadata: ~
66+
- add_kubernetes_metadata: ~
67+
```
68+
69+
1. Use the filestream input to read lines from active log files.
70+
2. Values from the decoded JSON object overwrite the fields that {{filebeat}} normally adds (type, source, offset, etc.) in case of conflicts.
71+
3. {{filebeat}} adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
72+
4. {{filebeat}} will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
73+
5. Processors enhance your data. See [processors](beats://docs/reference/filebeat/filtering-enhancing-data.md) to learn more.
74+
75+
76+
For Filebeat < 7.16
77+
78+
```yaml
79+
filebeat.inputs:
80+
- type: log
81+
paths: /path/to/logs.json
82+
json.keys_under_root: true
83+
json.overwrite_keys: true
84+
json.add_error_key: true
85+
json.expand_keys: true
86+
87+
processors:
88+
- add_host_metadata: ~
89+
- add_cloud_metadata: ~
90+
- add_docker_metadata: ~
91+
- add_kubernetes_metadata: ~
92+
```
93+
::::::
94+
95+
::::::{tab-item} Kubernetes
96+
1. Make sure your application logs to stdout/stderr.
97+
2. Follow the [Run Filebeat on Kubernetes](beats://docs/reference/filebeat/running-on-kubernetes.md) guide.
98+
3. Enable [hints-based autodiscover](beats://docs/reference/filebeat/configuration-autodiscover-hints.md) (uncomment the corresponding section in `filebeat-kubernetes.yaml`).
99+
4. Add these annotations to your pods that log using ECS loggers. This will make sure the logs are parsed appropriately.
100+
101+
```yaml
102+
annotations:
103+
co.elastic.logs/json.overwrite_keys: true <1>
104+
co.elastic.logs/json.add_error_key: true <2>
105+
co.elastic.logs/json.expand_keys: true <3>
106+
```
107+
108+
1. Values from the decoded JSON object overwrite the fields that {{filebeat}} normally adds (type, source, offset, etc.) in case of conflicts.
109+
2. {{filebeat}} adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
110+
3. {{filebeat}} will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
111+
::::::
112+
113+
::::::{tab-item} Docker
114+
1. Make sure your application logs to stdout/stderr.
115+
2. Follow the [Run Filebeat on Docker](beats://docs/reference/filebeat/running-on-docker.md) guide.
116+
3. Enable [hints-based autodiscover](beats://docs/reference/filebeat/configuration-autodiscover-hints.md).
117+
4. Add these labels to your containers that log using ECS loggers. This will make sure the logs are parsed appropriately.
118+
119+
```yaml
120+
labels:
121+
co.elastic.logs/json.overwrite_keys: true <1>
122+
co.elastic.logs/json.add_error_key: true <2>
123+
co.elastic.logs/json.expand_keys: true <3>
124+
```
125+
126+
1. Values from the decoded JSON object overwrite the fields that {{filebeat}} normally adds (type, source, offset, etc.) in case of conflicts.
127+
2. {{filebeat}} adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.
128+
3. {{filebeat}} will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.
129+
::::::
130+
131+
:::::::
132+
For more information, see the [Filebeat reference](beats://docs/reference/filebeat/configuring-howto-filebeat.md).
133+
134+
135+
## Usage [morgan-usage]
136+
137+
```js
138+
const app = require('express')();
139+
const morgan = require('morgan');
140+
const { ecsFormat } = require('@elastic/ecs-morgan-format');
141+
142+
app.use(morgan(ecsFormat(/* options */))); <1>
143+
144+
app.get('/', function (req, res) {
145+
res.send('hello, world!');
146+
})
147+
app.get('/error', function (req, res, next) {
148+
next(new Error('boom'));
149+
})
150+
151+
app.listen(3000)
152+
```
153+
154+
1. See available options [below](#morgan-ref).
155+
156+
157+
Running this script (the full example is [here](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express.js)) and making a request (via `curl -i localhost:3000/`) will produce log output similar to the following:
158+
159+
```cmd
160+
% node examples/express.js | jq . # piping to jq for pretty-printing
161+
{
162+
"@timestamp": "2021-01-16T00:03:23.279Z",
163+
"log.level": "info",
164+
"message": "::1 - - [16/Jan/2021:00:03:23 +0000] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.64.1\"",
165+
"ecs.version": "8.10.0",
166+
"http": {
167+
"version": "1.1",
168+
"request": {
169+
"method": "GET",
170+
"headers": {
171+
"host": "localhost:3000",
172+
"accept": "*/*"
173+
}
174+
},
175+
"response": {
176+
"status_code": 200,
177+
"headers": {
178+
"x-powered-by": "Express",
179+
"content-type": "text/html; charset=utf-8",
180+
"etag": "W/\"d-HwnTDHB9U/PRbFMN1z1wps51lqk\""
181+
},
182+
"body": {
183+
"bytes": 13
184+
}
185+
}
186+
},
187+
"url": {
188+
"path": "/",
189+
"domain": "localhost",
190+
"full": "http://localhost:3000/"
191+
},
192+
"user_agent": {
193+
"original": "curl/7.64.1"
194+
}
195+
}
196+
```
197+
198+
199+
## Format options [morgan-format-options]
200+
201+
You can pass any [`format` argument](https://github.com/expressjs/morgan#morganformat-options) you would normally pass to `morgan()`, and the log "message" field will use the specified format. The default is [`combined`](https://github.com/expressjs/morgan#combined).
202+
203+
```js
204+
const app = require('express')();
205+
const morgan = require('morgan');
206+
const { ecsFormat } = require('@elastic/ecs-morgan-format');
207+
208+
app.use(morgan(ecsFormat({ format: 'tiny' }))); <1>
209+
// ...
210+
```
211+
212+
1. If "format" is the only option you are using, you may pass it as `ecsFormat('tiny')`.
213+
214+
215+
216+
## log.level [morgan-log-level]
217+
218+
The `log.level` field will be "error" for response codes >= 500, otherwise "info". For example, running [examples/express.js](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express.js) again, a `curl -i localhost:3000/error` will yield:
219+
220+
```cmd
221+
% node examples/express.js | jq .
222+
{
223+
"@timestamp": "2021-01-18T17:52:12.810Z",
224+
"log.level": "error",
225+
"message": "::1 - - [18/Jan/2021:17:52:12 +0000] \"GET /error HTTP/1.1\" 500 1416 \"-\" \"curl/7.64.1\"",
226+
"http": {
227+
"response": {
228+
"status_code": 500,
229+
...
230+
```
231+
232+
233+
## Log correlation with APM [morgan-apm]
234+
235+
This ECS log formatter integrates with [Elastic APM](https://www.elastic.co/apm). If your Node app is using the [Node.js Elastic APM Agent](apm-agent-nodejs://docs/reference/index.md), then a number of fields are added to log records to correlate between APM services or traces and logging data:
236+
237+
* Log statements (e.g. `logger.info(...)`) called when there is a current tracing span will include [tracing fields](ecs://docs/reference/ecs-tracing.md) — `trace.id`, `transaction.id`.
238+
* A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana — `service.name`, `service.version`, `service.environment`, `service.node.name`.
239+
* `event.dataset` enables [log rate anomaly detection](docs-content://solutions/observability/logs/inspect-log-anomalies.md) in the Elastic Observability app.
240+
241+
For example, running [examples/express-with-apm.js](https://github.com/elastic/ecs-logging-nodejs/blob/main/packages/ecs-morgan-format/examples/express-with-apm.js) and `curl -i localhost:3000/` results in a log record with the following:
242+
243+
```cmd
244+
% node examples/express-with-apm.js | jq .
245+
{
246+
// The same fields as before, plus:
247+
"service.name": "express-with-elastic-apm",
248+
"service.version": "1.1.0",
249+
"service.environment": "development",
250+
"event.dataset": "express-with-elastic-apm",
251+
"trace.id": "116d46f667a7600deed9c41fa015f7de",
252+
"transaction.id": "b84fb72d7bf42866"
253+
}
254+
```
255+
256+
These IDs match trace data reported by the APM agent.
257+
258+
Integration with Elastic APM can be explicitly disabled via the `apmIntegration: false` option, for example:
259+
260+
```js
261+
app.use(morgan(ecsFormat({ apmIntegration: false })));
262+
```
263+
264+
265+
## Reference [morgan-ref]
266+
267+
268+
### `ecsFormat([options])` [morgan-ref-ecsFormat]
269+
270+
* `options` `{{type-object}}` The following options are supported:
271+
272+
* `format` `{{type-string}}` A format **name** (e.g. *combined*), format function (e.g. `morgan.combined`), or a format string (e.g. *:method :url :status*). This is used to format the "message" field. Defaults to `morgan.combined`.
273+
* `convertErr` `{{type-boolean}}` Whether to convert a logged `err` field to ECS error fields. **Default:** `true`.
274+
* `apmIntegration` `{{type-boolean}}` Whether to enable APM agent integration. **Default:** `true`.
275+
* `serviceName` `{{type-string}}` A "service.name" value. If specified this overrides any value from an active APM agent.
276+
* `serviceVersion` `{{type-string}}` A "service.version" value. If specified this overrides any value from an active APM agent.
277+
* `serviceEnvironment` `{{type-string}}` A "service.environment" value. If specified this overrides any value from an active APM agent.
278+
* `serviceNodeName` `{{type-string}}` A "service.node.name" value. If specified this overrides any value from an active APM agent.
279+
* `eventDataset` `{{type-string}}` A "event.dataset" value. If specified this overrides the default of using `${serviceVersion}`.
280+
281+
282+
Create a formatter for morgan that emits in ECS Logging format.
283+

0 commit comments

Comments
 (0)