forked from pivotal-cf/docs-ops-guide
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathlogging-config-opsman.html.md.erb
186 lines (120 loc) · 12.5 KB
/
logging-config-opsman.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
---
title: Configuring Logging in PAS
owner: APM
---
This topic describes the types of logs that <%= vars.first_product_name %> generates. It also explains how to forward system logs to an external aggregator service, how to scale [Loggregator](../loggregator/architecture.html) component VMs to keep up with app log volume, and how to manage app traffic logging.
## <a id='types'></a> System Logs, App Logs, App Traffic Logs
<%= vars.first_product_name %> generates two types of logs, _system logs_ from <%= vars.product_name %> components and _app logs_ from hosted apps, as differentiated in the table below:
<table id='log-types' border="1" class="nice" >
<tr>
<th>Log Type</th>
<th>Originate from</th>
<th>Follow format</th>
<th>Stream from</th>
<th>Can to stream out to (configurable)</th>
<th>Visible to</th>
</tr><tr>
<th>System Logs</th>
<td>Platform components</td>
<td>Syslog standard</td>
<td>rsyslog agent</td>
<td>Component syslog drain</td>
<td>Operators</td>
</tr><tr>
<th rowspan=3>App Logs</th>
<td rowspan=3>Hosted apps</td>
<td>Format is up to the developer</td>
<td>Firehose<sup>1</sup></td>
<td>External data platform, optionally via nozzles</td>
<td rowspan=3>Developers and Operators</td>
</tr><tr>
<th colspan=3>(Optional) With <a href="../loggregator/architecture.html#adapter">Syslog Adapter</a></th>
</tr><tr>
<td bgcolor="#FBFBFB">Converted to syslog standard</td>
<td bgcolor="#FBFBFB">Syslog Adapter</td>
<td bgcolor="#FBFBFB">External syslog drain</td>
</tr>
</table>
<sup>1</sup>The Loggregator Firehose also streams component metrics.
<strong>App traffic logs are system logs.</strong> When app containers communicate, or attempt to communicate, their host cells generate _app traffic logs_. App traffic logs are system logs, not app logs. These logs come from host cells, not apps, and they carry no information from within the app. App traffic logs only show app communication behavior, as detected from outside by the host cell.
## <a id="log-cache"></a>Log Cache
Log Cache is a Loggregator feature that lets you filter and query app logs through a CLI plugin or API endpoints. Cached app logs are available on demand; you do not need to stream them continuously.
### <a id="log-cache-cli-examples"></a> Example Uses of the Log Cache CLI Plugin
To access cached logs with the Log Cache CLI plugin, you must first download and install the plugin.
[Download the Log Cache CLI plugin](https://plugins.cloudfoundry.org/#log-cache).
One you have installed the plugin, the basic command to access cached app logs is:
<pre class="terminal">$ cf tail [OPTIONS] [SOURCE ID/APP]</pre>
Here are some ways you can use Log Cache to filter app logs:
* `--start-time`: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with `--end-time` to view logs within a time period.
* `--end-time`: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with `--start-time` to view logs within a time period.
* `--json`: Output logs in JSON.
* `--follow`: Append exported logs to `stdout`.
For more information on using the Log Cache CLI, see [Log Cache CLI: Usage](https://github.com/cloudfoundry/log-cache-cli/blob/develop/README.md).
### <a id="log-cache-api-examples"></a> Example Uses of the Log Cache API
The Log Cache API is hosted on <%= vars.product_name %>, and references your system domain to return responses. The root URL for API calls is <code>ht<span>tps:</span>//log-cache.[YOUR-SYSTEM-DOMAIN]</code>.
The basic call to access and filter cached app logs is:
<pre class="terminal">GET ht<span>tps:</span>//log-cache.[YOUR-SYSTEM-DOMAIN]/v1/read/[SOURCE-ID]</pre>
Append the following parameters to the `GET` to customize app logs:
* `start_time`: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with `end_time` to view logs within a time period.
* `end_time`: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with `start_time` to view logs within a time period.
* `envelope_types`: Filters by Envelope Type. The available filters are: `LOG`, `COUNTER`, `GAUGE`, `TIMER`, and `EVENT`. Set an envelope type filter to emit logs of only that type. Specify this parameter multiple times to include more types.
* `limit`: Sets a maximum number of envelopes to request. The max limit is 1000. This value defaults to 100.
More API parameters are available to customize retrieved app logs. For more information, see [Log Cache: RESTful API Gateway](https://github.com/cloudfoundry/log-cache#restful-api-via-gateway).
## <a id='syslog-forward'></a> Enable Syslog Forwarding
<%= partial '../customizing/system_logging' %>
## <a id='container-metrics'></a> Include Container Metrics in Syslog Drains
Developers can monitor container metrics over the syslog protocol using the [CF Drain CLI plugin](https://github.com/cloudfoundry/cf-drain-cli). With the CF Drain CLI plugin, you can use the Cloud Foundry Command Line Interface (cf CLI) tool to set the app container to deliver container metrics to a syslog drain. Developers can then monitor the app container based on those metrics.
In PAS, the **Include container metrics in Syslog Drains** checkbox is enabled by default in the **System Logging** pane. If you have security concerns about streaming container metrics from your app, you can disable this checkbox.
For more information, see [Including Container Metrics in Syslog Drains](../devguide/deploy-apps/streaming-logs.html#container-metrics) in _Application Logging in Cloud Foundry_.
## <a id='scaling'></a> Scale Loggregator
Apps constantly generate app logs and <%= vars.product_name %> platform components constantly generate component metrics. The Loggregator system combines these data streams and handles them as follows. See [Loggregator Architecture](../loggregator/architecture.html) for more information.
* The [Loggregator](../loggregator/architecture.html#metron) agent running on each component or application VM collects and sends this data out to Doppler components.
* [Doppler](../loggregator/architecture.html#doppler) components temporarily buffer the data before periodically forwarding it to the Traffic Controller. When the log and metrics data input to a Doppler exceeds its buffer size for a given interval, data can be lost.
* The [Traffic Controller](../loggregator/architecture.html#traffic-controller) serves the aggregated data stream through the Firehose WebSocket endpoint.
Follow the instructions below to scale the Loggregator system. For guidance on monitoring and capacity planning, see [Monitoring <%= vars.product_name_full %>](../monitoring/).
### <a id="instances"></a> Add Component VM Instances
1. From the PAS tile, select **Resource Config**.
1. Increase the number in the **Instances** column of the component you want to scale. You can add instances for the following Loggregator components:
* **Loggregator Traffic Controller**
<p class="note"><strong>Note</strong>: The <a href="../loggregator/architecture.html#rlp">Reverse Log Proxy (RLP)</a> BOSH job is colocated on the Traffic Controller VM. If you want to scale Loggregator to handle more logs for syslog drains, you can add instances of the Traffic Controller.</p>
<p class="note"><strong>Note</strong>: The <a href="../loggregator/architecture.html#bosh-metrics">BOSH System Metrics Forwarder</a> job is colocated on the Traffic Controller VM. If you want to scale Loggregator to handle more BOSH component metrics, you can add instances of the Traffic Controller.</p>
* **Syslog Adapter**
* **Doppler Server**
<%= image_tag 'loggregator-vms.png' %>
1. Click **Save**.
1. Click **Review Pending Changes**, then **Apply Changes**.
## <a id="app-traffic-log"></a> App Traffic Logging
App traffic logging generates logs when app containers communicate with each other directly, or attempt to communicate, as allowed by [container-to-container networking (C2C) policies](../concepts/understand-cf-networking.html#c2cvsasg) and [Application Security Groups (ASGs)](../concepts/asg.html).
App traffic logging lets network security teams audit C2C traffic, by seeing allowed and denied packets, without needing access to the Cloud Controller or the apps themselves.
### <a id="enable-app-traffic-log"></a> Enable App Traffic Logging
To enable app traffic logging:
1. From Ops Manager, navigate to the **Pivotal Application Service** tile > **Networking** pane.
1. Under **Log traffic for all accepted/denied application packets**, select **Enable (will increase log volume)** or **Disable** to enable or disable app traffic logging.
<%= image_tag 'images/log-app-traffic-enable.png' %>
### <a id="log-behavior"></a> App Logging Behavior
App traffic logging generates log messages as follows:
* **TCP traffic** - Logs the first packet of every new TCP connection.
* **UDP traffic** - Logs UDP packets sent and received, up to a maximum per-second rate for each container. Set this rate limit in the **UDP logging interval** field (default: 100).
* **Packets denied** - Logs packets blocked by either a container-specific [networking policy](../concepts/understand-cf-networking.html#policies) or by [Application Security Group](./app-sec-groups.html) (ASG) rules applied across the space, org, or deployment. Logs packet denials up to a maximum per-second rate for each container, set in the **Denied logging interval** field (default: 1).
### <a id="log-format"></a> App Traffic Log Format and Contents
App traffic logs are formatted as described in the [cf-networking-release Traffic logging](https://github.com/cloudfoundry-incubator/cf-networking-release/blob/v1.10.x/docs/traffic_logging.md) documentation, following the `iptables-logger` format but without line breaks. For example, the first part of an app traffic log line looks like: `{"timestamp": "1500924070.182554722", "source": "cfnetworking.iptables", "message": "cfnetworking.iptables.ingress-allowed", "log_level": 1, "data": { "destination": { "container_id": "d5978989-1401-49ff-46cd-33e5","app_guid": "bc6f229d-5e4a-4c41-a63f-e8795496c283",`.
Each log message includes the following:
* Timestamp
* The GUID for the source or destination app that sent or was designated to receive the packet
* The protocol of the communication, `TCP` or `UDP`
* GUIDs for the container, space, and org running the source or destination app
* IP addresses and ports for both source and destination apps
* A `message` field recording whether the packet was allowed or denied, with one of the following four possibilities:
- ASG allowed packet to exit source app container
- C2C policy allowed packet to enter destination app container
- ASG prevented packet from exiting source app container
- C2C policy prevented packet from entering destination app container
* Additional information described in the [cf-networking-release](https://github.com/cloudfoundry-incubator/cf-networking-release/blob/v1.10.x/docs/traffic_logging.md).
#### <a id="denied-packet"></a> Denied Packet Causes
You can determine whether a denied-packet log resulted from a container networking policy or an ASG rule as follows:
* **Container networking policy**: Log `message` string includes `ingress-denied` and `packet direction` is `ingress`.
* **ASG rule**: Log `message` string includes `egress-denied` and `packet direction` is `egress`.
### <a id="scope-app-traffic-log"></a> Global vs. ASG-Level App Traffic Logging
<%= vars.product_name %> supports two mechanisms for enabling app traffic logging. [Setting](#enable-app-traffic-logging) **Log traffic** to **Enable** in Ops Manager enables app traffic logging globally for all ASGs and container policies. Setting the [`log` property](../concepts/asg.html#asg-logging-prereq) of an ASG to `true` enables app traffic logging at the individual ASG level.
Because these two mechanisms operate independently, <%= vars.product_name %> generates duplicate logs when app traffic logging is enabled globally and an ASG's `log` property is set to `true`. To avoid duplicate logs, Pivotal recommends setting the `log` property to `false` for all ASGs, or leaving it out entirely, when app traffic logging is [enabled globally](#app-traffic-log).
To focus on specific ASGs for log analysis, Pivotal recommends enabling app traffic logs globally and using a logging platform to filter traffic logs by ASG, rather than setting `log` at the individual ASG level.