Skip to content

Commit f9e1653

Browse files
constanca-mconstanca-m
authored
[encoding/googlecloudlogentry] Add support for Cloud Audit Log (#42036)
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description This PR parses cloud audit logs into log record attributes. Please see the updated README file for details. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Fixes #42035. <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests added and updated. <!--Describe the documentation added.--> #### Documentation README updated. <!--Please delete paragraphs that you did not use before submitting.-->
1 parent c967e0c commit f9e1653

21 files changed

+1513
-204
lines changed

‎.chloggen/gcp-add-audit-logs.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: breaking
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
7+
component: googlecloudlogentry_encoding
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Parse cloud audit logs into log record attributes instead of placing it in the body as is.
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [42035]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# If your change doesn't affect end users or the exported elements of any package,
21+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
22+
# Optional: The change log or logs in which this entry should be included.
23+
# e.g. '[user]' or '[user, api]'
24+
# Include 'user' if the change is relevant to end users.
25+
# Include 'api' if there is a change to a library API.
26+
# Default: '[user]'
27+
change_logs: []

‎extension/encoding/googlecloudlogentryencodingextension/README.md‎

Lines changed: 56 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515

1616
This extension can be used to unmarshall a [Cloud Logging LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) message type. The extension expects each log to take up 1 line, and it will decode as many logs as log lines received.
1717

18-
The following configuration options are supported:
18+
Currently, this extension [can parse the following logs](#supported-log-types) into log record attributes:
19+
- [Cloud audit logs](https://cloud.google.com/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog) (extension [mapping](#cloud-audit-logs))
20+
21+
For all others logs, the payload will be placed in the log record attribute. In this case, the following configuration options are supported:
1922

2023
* `handle_json_payload_as` (Optional): This controls how the json payload of the log entry is parsed into the body.
2124
The default `json` parses it as standard JSON, while `text` will the put the payload as a single string.
@@ -64,9 +67,9 @@ The [log entry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEn
6467
| `sourceLocation.file` | Log record attribute: `code.file.path` |
6568
| `sourceLocation.line` | Log record attribute: `code.line.number` |
6669
| `sourceLocation.function` | Log record attribute: `code.function.name` |
67-
| `protoPayload` | Placed on the record body as is |
68-
| `textPayload` | Placed on the record body as is |
69-
| `jsonPayload` | Placed on the record body as is |
70+
| `protoPayload` | Placed on the record body as is, unless log type is supported |
71+
| `textPayload` | Placed on the record body as is, unless log type is supported |
72+
| `jsonPayload` | Placed on the record body as is, unless log type is supported |
7073
| `split.uid` | Log record attribute: `gcp.split.uid` | |
7174
| `split.index` | Log record attribute: `gcp.split.index` | |
7275
| `split.totalSplits` | Log record attribute: `gcp.split.total` | |
@@ -104,4 +107,52 @@ The severity is mapped from [Google Cloud Log Severity](https://cloud.google.com
104107
| `ERROR`(500) | `ERROR`(17) | Error events are likely to cause problems. |
105108
| `CRITICAL`(600) | `FATAL`(21) | Critical events cause more severe problems or outages. |
106109
| `ALERT`(700) | `FATAL2`(22) | A person must take an action immediately. |
107-
| `EMERGENCY`(800) | `FATAL4`(24) | One or more systems are unusable. |
110+
| `EMERGENCY`(800) | `FATAL4`(24) | One or more systems are unusable. |
111+
112+
## Supported log types
113+
114+
Currently, these are the log types that are specifically parsed into log record attributes.
115+
116+
### Cloud Audit Logs
117+
118+
See the struct of the Cloud Audit Log payload in [AuditLog](https://cloud.google.com/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog). The fields are mapped this way in the extension:
119+
120+
121+
| Original field | Log record attribute |
122+
|----------------------------------------------------------------------------|---------------------------------------------------------------------|
123+
| `serviceName` | `gcp.audit.service.name` |
124+
| `methodName` | `gcp.audit.method.name` |
125+
| `resourceName` | `gcp.audit.resource.name` |
126+
| `resourceLocation.currentLocations` | `gcp.audit.resource.location.current` |
127+
| `resourceLocation.originalLocations` | `gcp.audit.resource.location.original` |
128+
| `resourceOriginalState` | _Currently not supported_ |
129+
| `numResponseItems` | `gcp.audit.response.items` |
130+
| `status.code` | `rpc.jsonrpc.error_code` |
131+
| `status.message` | `rpc.jsonrpc.error_message` |
132+
| `status.details` | _Currently not supported_ |
133+
| `authenticationInfo.principalEmail` | `user.email` |
134+
| `authenticationInfo.authoritySelector` | `gcp.audit.authentication.authority_selector` |
135+
| `authenticationInfo.thirdPartyPrincipal` | _Currently not supported_ |
136+
| `authenticationInfo.serviceAccountKeyName` | `gcp.audit.authentication.service_account.key.name` |
137+
| `authenticationInfo.serviceAccountDelegationInfo` | _Currently not supported_ |
138+
| `authenticationInfo.principalSubject` | `user.id` |
139+
| `authorizationInfo[*].resource` | Item entry `resource` in map `gcp.audit.authorization` |
140+
| `authorizationInfo[*].permission` | Item entry `permission` in map `gcp.audit.authorization` |
141+
| `authorizationInfo[*].granted` | Item entry `granted` in map `gcp.audit.authorization`` |
142+
| `authorizationInfo.resourceAttributes` | _Currently not supported_ |
143+
| `policyViolationInfo.orgPolicyViolationInfo.payload` | _Currently not supported_ |
144+
| `policyViolationInfo.orgPolicyViolationInfo.resourceType` | `gcp.audit.policy_violation.resource.type` |
145+
| `policyViolationInfo.orgPolicyViolationInfo.resourceTags` | `gcp.audit.policy_violation.resource.tags` |
146+
| `policyViolationInfo.orgPolicyViolationInfo.violationInfo[*].constraint` | Item entry `constraint` in map `gcp.audit.policy_violation.info` |
147+
| `policyViolationInfo.orgPolicyViolationInfo.violationInfo[*].errorMessage` | Item entry `error_message` in map `gcp.audit.policy_violation.info` |
148+
| `policyViolationInfo.orgPolicyViolationInfo.violationInfo[*].checkedValue` | Item entry `checked_value` in map `gcp.audit.policy_violation.info` |
149+
| `policyViolationInfo.orgPolicyViolationInfo.violationInfo[*].policyType` | Item entry `policy_type` in map `gcp.audit.policy_violation.info` |
150+
| `requestMetadata.callerIp` | `client.address` |
151+
| `requestMetadata.callerSuppliedUserAgent` | `user_agent.original` |
152+
| `requestMetadata.callerNetwork` | `gcp.audit.request.caller.network` |
153+
| `requestMetadata.requestAttributes` | _Currently not supported_ |
154+
| `requestMetadata.destinationAttributes` | _Currently not supported_ |
155+
| `request` | _Currently not supported_ |
156+
| `response` | _Currently not supported_ |
157+
| `metadata` | _Currently not supported_ |
158+
| `serviceData` | [GCP Deprecated field]<br>_Currently not supported_ |

‎extension/encoding/googlecloudlogentryencodingextension/extension_test.go‎

Lines changed: 61 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,11 @@ func TestHandleLogLine(t *testing.T) {
4545
expectedErr: `failed to handle log entry`,
4646
},
4747
{
48-
name: "valid",
49-
logLine: []byte(`{"logName": "projects/open-telemetry/logs/log-test"}`),
48+
name: "valid",
49+
logLine: []byte(`{
50+
"logName": "projects/open-telemetry/logs/log-test",
51+
"timestamp": "2024-05-05T10:31:19.45570687Z"
52+
}`),
5053
},
5154
}
5255

@@ -123,3 +126,59 @@ func TestUnmarshalLogs(t *testing.T) {
123126
})
124127
}
125128
}
129+
130+
func TestPayloads(t *testing.T) {
131+
// TODO Keep adding tests when adding new log types support
132+
133+
tests := []struct {
134+
name string
135+
logFilename string
136+
expectedFilename string
137+
expectedErr string
138+
}{
139+
{
140+
name: "audit log - activity",
141+
logFilename: "testdata/auditlog/activity.json",
142+
expectedFilename: "testdata/auditlog/activity_expected.yaml",
143+
},
144+
{
145+
name: "audit log - data access",
146+
logFilename: "testdata/auditlog/data_access.json",
147+
expectedFilename: "testdata/auditlog/data_access_expected.yaml",
148+
},
149+
{
150+
name: "audit log - policy",
151+
logFilename: "testdata/auditlog/policy.json",
152+
expectedFilename: "testdata/auditlog/policy_expected.yaml",
153+
},
154+
{
155+
name: "audit log - system event",
156+
logFilename: "testdata/auditlog/system_event.json",
157+
expectedFilename: "testdata/auditlog/system_event_expected.yaml",
158+
},
159+
}
160+
161+
extension := newTestExtension(t, Config{})
162+
for _, tt := range tests {
163+
t.Run(tt.name, func(t *testing.T) {
164+
t.Parallel()
165+
166+
data, err := os.ReadFile(tt.logFilename)
167+
require.NoError(t, err)
168+
169+
content := bytes.NewBuffer([]byte{})
170+
err = gojson.Compact(content, data)
171+
require.NoError(t, err)
172+
173+
logs, err := extension.UnmarshalLogs(content.Bytes())
174+
require.NoError(t, err)
175+
176+
// write expected log with:
177+
// golden.WriteLogs(t, tt.expectedFilename, logs)
178+
179+
expectedLogs, err := golden.ReadLogs(tt.expectedFilename)
180+
require.NoError(t, err)
181+
require.NoError(t, plogtest.CompareLogs(expectedLogs, logs))
182+
})
183+
}
184+
}

0 commit comments

Comments
 (0)