Skip to content

Commit afda915

Browse files
Kavindu-DodanKavindu-Dodan
authored
[extension/awslogs_encoding] implement streaming contract for WAF logs (open-telemetry#46223)
#### Description Based on contract introduced at open-telemetry#46211, this PR implements streaming for WAF logs. WAF logs change is focused at commit titled `streaming for WAF logs` #### Testing Unit tests and dedicated streaming tests #### Documentation Updated documentation on streaming contract --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
1 parent 363ae6a commit afda915

10 files changed

Lines changed: 421 additions & 280 deletions

File tree

‎.chloggen/feat_aws-logs-waf-logs.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Use this changelog template to create an entry for release notes.
2+
3+
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
4+
change_type: enhancement
5+
6+
# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
7+
component: extension/awslogs_encoding
8+
9+
# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
10+
note: Adopt encoding extension streaming contract for WAF logs
11+
12+
# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
13+
issues: [46214]
14+
15+
# (Optional) One or more lines of additional information to render under the primary note.
16+
# These lines will be padded with 2 spaces and then inserted directly into the document.
17+
# Use pipe (|) for multiline entries.
18+
subtext:
19+
20+
# If your change doesn't affect end users or the exported elements of any package,
21+
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
22+
# Optional: The change log or logs in which this entry should be included.
23+
# e.g. '[user]' or '[user, api]'
24+
# Include 'user' if the change is relevant to end users.
25+
# Include 'api' if there is a change to a library API.
26+
# Default: '[user]'
27+
change_logs: []

‎extension/encoding/awslogsencodingextension/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,7 @@ The table below summarizes streaming support details for each log type, along wi
185185
|---------------------|---------------------|-----------------------------|----------------------------------------------------------------------------------------------|
186186
| Network Firewall | Alert/Flow/TLS | Bytes processed | |
187187
| Subscription filter | - | Number of records processed | Supports processing multi-line inputs and offset tracks number of records that get processed |
188+
| WAF Logs | - | Bytes processed | |
188189

189190
## Produced Records per Format
190191

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/waf/benchmark_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ func BenchmarkUnmarshalLogs(b *testing.B) {
4444
},
4545
}
4646

47-
u := wafLogUnmarshaler{
47+
u := WafLogUnmarshaler{
4848
buildInfo: component.BuildInfo{},
4949
}
5050

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/waf/testdata/missing_webaclid_log.json‎

Lines changed: 1 addition & 100 deletions
Original file line numberDiff line numberDiff line change
@@ -1,100 +1 @@
1-
{
2-
"timestamp":1748208718574,
3-
"formatVersion":1,
4-
"terminatingRuleId":"Default_Action",
5-
"terminatingRuleType":"REGULAR",
6-
"action":"ALLOW",
7-
"terminatingRuleMatchDetails":[ ],
8-
"httpSourceName":"CF",
9-
"httpSourceId":"E3DTJP8YLL6OBQ",
10-
"ruleGroupList":[ ],
11-
"rateBasedRuleList":[
12-
{
13-
"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12",
14-
"rateBasedRuleName":"rule-1",
15-
"limitKey":"IP",
16-
"maxRateAllowed":10000,
17-
"evaluationWindowSec":300,
18-
"limitValue":"178.84.204.171"
19-
}
20-
],
21-
"nonTerminatingMatchingRules":[ ],
22-
"requestHeadersInserted":null,
23-
"responseCodeSent":null,
24-
"httpRequest":{
25-
"clientIp":"178.84.204.171",
26-
"country":"NL",
27-
"headers":[
28-
{
29-
"name":"host",
30-
"value":"dsx1234tsajqz63.cloudfront.net"
31-
},
32-
{
33-
"name":"user-agent",
34-
"value":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"
35-
},
36-
{
37-
"name":"accept",
38-
"value":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"
39-
},
40-
{
41-
"name":"accept-language",
42-
"value":"en-US,en;q=0.5"
43-
},
44-
{
45-
"name":"accept-encoding",
46-
"value":"gzip, deflate, br, zstd"
47-
},
48-
{
49-
"name":"referer",
50-
"value":"https://dsx88tsajqz63.cloudfront.net/"
51-
},
52-
{
53-
"name":"sec-fetch-dest",
54-
"value":"image"
55-
},
56-
{
57-
"name":"sec-fetch-mode",
58-
"value":"no-cors"
59-
},
60-
{
61-
"name":"sec-fetch-site",
62-
"value":"same-origin"
63-
},
64-
{
65-
"name":"dnt",
66-
"value":"1"
67-
},
68-
{
69-
"name":"sec-gpc",
70-
"value":"1"
71-
},
72-
{
73-
"name":"priority",
74-
"value":"u=6"
75-
},
76-
{
77-
"name":"te",
78-
"value":"trailers"
79-
}
80-
],
81-
"uri":"/favicon.ico",
82-
"args":"",
83-
"httpVersion":"HTTP/2.0",
84-
"httpMethod":"GET",
85-
"requestId":"n6LHLPqblIh_4qRsVj0940K9LxKyrkiUUE7lyMol1eTptabtlhHiXQ==",
86-
"fragment":"",
87-
"scheme":"https",
88-
"host":"dsx88tsajqz63.cloudfront.net"
89-
},
90-
"labels":[
91-
{
92-
"name":"awswaf:clientip:geo:country:NL"
93-
},
94-
{
95-
"name":"awswaf:clientip:geo:region:NL-NH"
96-
}
97-
],
98-
"ja3Fingerprint":"6f7889b9fb1a62a9577e685c1fcfa919",
99-
"ja4Fingerprint":"t13d1717h2_5b57614c22b0_3cbfd9057e0d"
100-
}
1+
{"timestamp":1748208718574,"formatVersion":1,"terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"CF","httpSourceId":"E3DTJP8YLL6OBQ","ruleGroupList":[],"rateBasedRuleList":[{"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12","rateBasedRuleName":"rule-1","limitKey":"IP","maxRateAllowed":10000,"evaluationWindowSec":300,"limitValue":"178.84.204.171"}],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"178.84.204.171","country":"NL","headers":[{"name":"host","value":"dsx1234tsajqz63.cloudfront.net"},{"name":"user-agent","value":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"},{"name":"accept","value":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"},{"name":"accept-language","value":"en-US,en;q=0.5"},{"name":"accept-encoding","value":"gzip, deflate, br, zstd"},{"name":"referer","value":"https://dsx88tsajqz63.cloudfront.net/"},{"name":"sec-fetch-dest","value":"image"},{"name":"sec-fetch-mode","value":"no-cors"},{"name":"sec-fetch-site","value":"same-origin"},{"name":"dnt","value":"1"},{"name":"sec-gpc","value":"1"},{"name":"priority","value":"u=6"},{"name":"te","value":"trailers"}],"uri":"/favicon.ico","args":"","httpVersion":"HTTP/2.0","httpMethod":"GET","requestId":"n6LHLPqblIh_4qRsVj0940K9LxKyrkiUUE7lyMol1eTptabtlhHiXQ==","fragment":"","scheme":"https","host":"dsx88tsajqz63.cloudfront.net"},"labels":[{"name":"awswaf:clientip:geo:country:NL"},{"name":"awswaf:clientip:geo:region:NL-NH"}],"ja3Fingerprint":"6f7889b9fb1a62a9577e685c1fcfa919","ja4Fingerprint":"t13d1717h2_5b57614c22b0_3cbfd9057e0d"}

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/waf/testdata/valid_log.json‎

Lines changed: 1 addition & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -1,101 +1 @@
1-
{
2-
"timestamp":1748208718574,
3-
"formatVersion":1,
4-
"webaclId":"arn:aws:wafv2:us-east-1:123456789101:global/webacl/open-telemetry-waf/e3132a63-134d-4da9-a0c4-b166ddd6de6c",
5-
"terminatingRuleId":"Default_Action",
6-
"terminatingRuleType":"REGULAR",
7-
"action":"ALLOW",
8-
"terminatingRuleMatchDetails":[ ],
9-
"httpSourceName":"CF",
10-
"httpSourceId":"E3DTJP8YLL6OBQ",
11-
"ruleGroupList":[ ],
12-
"rateBasedRuleList":[
13-
{
14-
"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12",
15-
"rateBasedRuleName":"rule-1",
16-
"limitKey":"IP",
17-
"maxRateAllowed":10000,
18-
"evaluationWindowSec":300,
19-
"limitValue":"178.84.204.171"
20-
}
21-
],
22-
"nonTerminatingMatchingRules":[ ],
23-
"requestHeadersInserted":null,
24-
"responseCodeSent":null,
25-
"httpRequest":{
26-
"clientIp":"178.84.204.171",
27-
"country":"NL",
28-
"headers":[
29-
{
30-
"name":"host",
31-
"value":"dsx1234tsajqz63.cloudfront.net"
32-
},
33-
{
34-
"name":"user-agent",
35-
"value":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"
36-
},
37-
{
38-
"name":"accept",
39-
"value":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"
40-
},
41-
{
42-
"name":"accept-language",
43-
"value":"en-US,en;q=0.5"
44-
},
45-
{
46-
"name":"accept-encoding",
47-
"value":"gzip, deflate, br, zstd"
48-
},
49-
{
50-
"name":"referer",
51-
"value":"https://dsx88tsajqz63.cloudfront.net/"
52-
},
53-
{
54-
"name":"sec-fetch-dest",
55-
"value":"image"
56-
},
57-
{
58-
"name":"sec-fetch-mode",
59-
"value":"no-cors"
60-
},
61-
{
62-
"name":"sec-fetch-site",
63-
"value":"same-origin"
64-
},
65-
{
66-
"name":"dnt",
67-
"value":"1"
68-
},
69-
{
70-
"name":"sec-gpc",
71-
"value":"1"
72-
},
73-
{
74-
"name":"priority",
75-
"value":"u=6"
76-
},
77-
{
78-
"name":"te",
79-
"value":"trailers"
80-
}
81-
],
82-
"uri":"/favicon.ico",
83-
"args":"",
84-
"httpVersion":"HTTP/2.0",
85-
"httpMethod":"GET",
86-
"requestId":"n6LHLPqblIh_4qRsVj0940K9LxKyrkiUUE7lyMol1eTptabtlhHiXQ==",
87-
"fragment":"",
88-
"scheme":"https",
89-
"host":"dsx88tsajqz63.cloudfront.net"
90-
},
91-
"labels":[
92-
{
93-
"name":"awswaf:clientip:geo:country:NL"
94-
},
95-
{
96-
"name":"awswaf:clientip:geo:region:NL-NH"
97-
}
98-
],
99-
"ja3Fingerprint":"6f7889b9fb1a62a9577e685c1fcfa919",
100-
"ja4Fingerprint":"t13d1717h2_5b57614c22b0_3cbfd9057e0d"
101-
}
1+
{"timestamp":1748208718574,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:123456789101:global/webacl/open-telemetry-waf/e3132a63-134d-4da9-a0c4-b166ddd6de6c","terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"CF","httpSourceId":"E3DTJP8YLL6OBQ","ruleGroupList":[],"rateBasedRuleList":[{"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12","rateBasedRuleName":"rule-1","limitKey":"IP","maxRateAllowed":10000,"evaluationWindowSec":300,"limitValue":"178.84.204.171"}],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"178.84.204.171","country":"NL","headers":[{"name":"host","value":"dsx1234tsajqz63.cloudfront.net"},{"name":"user-agent","value":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"},{"name":"accept","value":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"},{"name":"accept-language","value":"en-US,en;q=0.5"},{"name":"accept-encoding","value":"gzip, deflate, br, zstd"},{"name":"referer","value":"https://dsx88tsajqz63.cloudfront.net/"},{"name":"sec-fetch-dest","value":"image"},{"name":"sec-fetch-mode","value":"no-cors"},{"name":"sec-fetch-site","value":"same-origin"},{"name":"dnt","value":"1"},{"name":"sec-gpc","value":"1"},{"name":"priority","value":"u=6"},{"name":"te","value":"trailers"}],"uri":"/favicon.ico","args":"","httpVersion":"HTTP/2.0","httpMethod":"GET","requestId":"n6LHLPqblIh_4qRsVj0940K9LxKyrkiUUE7lyMol1eTptabtlhHiXQ==","fragment":"","scheme":"https","host":"dsx88tsajqz63.cloudfront.net"},"labels":[{"name":"awswaf:clientip:geo:country:NL"},{"name":"awswaf:clientip:geo:region:NL-NH"}],"ja3Fingerprint":"6f7889b9fb1a62a9577e685c1fcfa919","ja4Fingerprint":"t13d1717h2_5b57614c22b0_3cbfd9057e0d"}

‎extension/encoding/awslogsencodingextension/internal/unmarshaler/waf/testdata/valid_log_multi.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
{"timestamp":1748208718574,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:123456789101:global/webacl/open-telemetry-waf/e3132a63-134d-4da9-a0c4-b166ddd6de6c","terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"CF","httpSourceId":"E3DTJP8YLL6OBQ","ruleGroupList":[],"rateBasedRuleList":[{"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12","rateBasedRuleName":"rule-1","limitKey":"IP","maxRateAllowed":10000,"evaluationWindowSec":300,"limitValue":"178.84.204.171"}],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"178.84.204.171","country":"NL","headers":[{"name":"host","value":"dsx1234tsajqz63.cloudfront.net"},{"name":"user-agent","value":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:138.0) Gecko/20100101 Firefox/138.0"},{"name":"accept","value":"image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"},{"name":"accept-language","value":"en-US,en;q=0.5"},{"name":"accept-encoding","value":"gzip, deflate, br, zstd"},{"name":"referer","value":"https://dsx88tsajqz63.cloudfront.net/"},{"name":"sec-fetch-dest","value":"image"},{"name":"sec-fetch-mode","value":"no-cors"},{"name":"sec-fetch-site","value":"same-origin"},{"name":"dnt","value":"1"},{"name":"sec-gpc","value":"1"},{"name":"priority","value":"u=6"},{"name":"te","value":"trailers"}],"uri":"/favicon.ico","args":"","httpVersion":"HTTP/2.0","httpMethod":"GET","requestId":"n6LHLPqblIh_4qRsVj0940K9LxKyrkiUUE7lyMol1eTptabtlhHiXQ==","fragment":"","scheme":"https","host":"dsx88tsajqz63.cloudfront.net"},"labels":[{"name":"awswaf:clientip:geo:country:NL"},{"name":"awswaf:clientip:geo:region:NL-NH"}],"ja3Fingerprint":"6f7889b9fb1a62a9577e685c1fcfa919","ja4Fingerprint":"t13d1717h2_5b57614c22b0_3cbfd9057e0d"}
2+
{"timestamp":1683355579981,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-east-1:123456789101:global/webacl/open-telemetry-waf/e3132a63-134d-4da9-a0c4-b166ddd6de6c","terminatingRuleId":"RateBasedRule","terminatingRuleType":"RATE_BASED","action":"BLOCK","terminatingRuleMatchDetails":[],"httpSourceName":"APIGW","httpSourceId":"EXAMPLE11:rjvegx5guh:CanaryTest","ruleGroupList":[],"rateBasedRuleList":[{"rateBasedRuleId":"arn:aws:wafv2:us-east-1:123456789101_MANAGED:global/ipset/e3132a63-134d-4da9-a0c4-b166ddd6de6c_77ce5c35-14fa-4731-9710-86216d568f12_IPV4/77ce5c35-14fa-4731-9710-86216d568f12","rateBasedRuleName":"RateBasedRule","limitKey":"CUSTOMKEYS","maxRateAllowed":100,"evaluationWindowSec":"120","customValues":[{"key":"HEADER","name":"dogname","value":"ella"}]}],"nonTerminatingMatchingRules":[],"requestHeadersInserted":null,"responseCodeSent":null,"httpRequest":{"clientIp":"52.46.82.45","country":"FR","headers":[{"name":"X-Forwarded-For","value":"52.46.82.45"},{"name":"X-Forwarded-Proto","value":"https"},{"name":"X-Forwarded-Port","value":"443"},{"name":"Host","value":"rjvegx5guh.execute-api.eu-west-3.amazonaws.com"},{"name":"X-Amzn-Trace-Id","value":"Root=1-645566cf-7cb058b04d9bb3ee01dc4036"},{"name":"dogname","value":"ella"},{"name":"User-Agent","value":"RateBasedRuleTestKoipOneKeyModulePV2"},{"name":"Accept-Encoding","value":"gzip,deflate"}],"uri":"/CanaryTest","args":"","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":"Ed0AiHF_CGYF-DA="}}

0 commit comments

Comments
 (0)