Skip to content

Commit fdbe8ef

Browse files
agithomasagithomas
authored
[Amazon MQ] Add support for ActiveMQ general logs (elastic#13136)
* Add activemq general log support
1 parent acd934d commit fdbe8ef

File tree

15 files changed

+659
-6
lines changed

15 files changed

+659
-6
lines changed

‎packages/aws_mq/_dev/build/docs/README.md

+28-1
Original file line numberDiff line numberDiff line change
@@ -9,16 +9,35 @@ The Amazon MQ integration allows you to efficiently collect and monitor broker p
99

1010
**IMPORTANT: Extra AWS charges on API requests will be generated by this integration. Check [API Requests](https://www.elastic.co/docs/current/integrations/aws#aws-api-requests) for more details.**
1111

12+
## Setup
13+
14+
### ActiveMQ
15+
16+
To enable the `activemq_general_logs` integration, you must configure your ActiveMQ broker to publish general logs to Amazon CloudWatch Logs. Follow these steps:
17+
18+
1. **Assign Necessary Permissions**: Ensure the IAM user creating or managing the broker has the `logs:CreateLogGroup` permission. This allows Amazon MQ to create the required log groups in CloudWatch.
19+
20+
2. **Set Up a Resource-Based Policy**: Configure a policy that permits Amazon MQ to publish logs to your CloudWatch log groups. This involves granting `logs:CreateLogStream` and `logs:PutLogEvents` permissions.
21+
22+
3. **Enable Logging on the Broker**:
23+
24+
- Navigate to the [Amazon MQ console](https://console.aws.amazon.com/amazon-mq/).
25+
- During broker creation or by editing an existing broker, expand the **Additional settings** section.
26+
- In the **Logs** section, select the option to publish **General logs** to Amazon CloudWatch Logs.
27+
28+
For detailed instructions, refer to the [Amazon MQ Developer Guide](https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/configure-logging-monitoring-activemq.html#security-logging-monitoring-configure-cloudwatch-structure).
29+
1230
## Compatibility
1331

1432
This integration presently supports Amazon MQ for [Apache ActiveMQ](http://activemq.apache.org/) and [RabbitMQ](https://www.rabbitmq.com/) metrics.
1533

1634
## Data streams
1735

18-
The Amazon MQ integration collects Apache ActiveMQ and RabbitMQ metrics.
36+
The Amazon MQ integration collects metrics and logs from Apache ActiveMQ and metrics from RabbitMQ.
1937

2038

2139
Data streams:
40+
- `activemq_general_logs`: Collects ActiveMQ general logs, including system events, warnings, and errors, which are published to a designated Amazon CloudWatch log group.
2241
- `activemq_metrics`: Collects broker metrics and destination (queue and topic) metrics.
2342
- `rabbitmq_metrics`: Collects broker, queue and node metrics.
2443

@@ -46,6 +65,14 @@ documentation](https://docs.elastic.co/integrations/aws#requirements).
4665
data to Elastic, where the events will then be processed through the
4766
integration's ingest pipelines.
4867

68+
## Logs
69+
70+
### Collecting Amazon MQ ActiveMQ General Logs from CloudWatch
71+
72+
When general logging is enabled for your Amazon MQ ActiveMQ broker, it publishes the `activemq.log` file at the default `INFO` logging level to a designated log group. Please note that `DEBUG` logging is not supported.
73+
74+
{{event "activemq_general_logs"}}
75+
{{fields "activemq_general_logs"}}
4976

5077
## Metrics
5178

‎packages/aws_mq/changelog.yml

+5
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
- version: "0.5.0"
2+
changes:
3+
- description: Add activemq_general_logs dataset for ActiveMQ general logs.
4+
type: enhancement
5+
link: https://github.com/elastic/integrations/pull/13136
16
- version: "0.4.0"
27
changes:
38
- description: Add RabbitMQ overview dashboard.

‎packages/aws_mq/data_stream/activemq_general_logs/_dev/test/pipeline/test-amazonmq-activemq-general.log

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
2025-03-15 13:47:13,382 | WARN | Exception occurred for client ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7 (tcp://127.0.0.1:63868) processing: STOMP -> java.lang.SecurityException: User name [elastic] or password is invalid. | org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ Transport: ssl:///127.0.0.1:63868
2+
2025-03-15 13:47:13,381 | WARN | Failed to add Connection id=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientId=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientIP=tcp://127.0.0.1:63868 due to User name [elastic] or password is invalid. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: ssl:///127.0.0.1:63868

‎packages/aws_mq/data_stream/activemq_general_logs/_dev/test/pipeline/test-amazonmq-activemq-general.log-expected.json

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
{
2+
"expected": [
3+
{
4+
"@timestamp": "2025-03-15T13:47:13.382Z",
5+
"activemq": {
6+
"log": {
7+
"caller": "org.apache.activemq.transport.stomp.ProtocolConverter",
8+
"thread": "ActiveMQ Transport: ssl:///127.0.0.1:63868"
9+
}
10+
},
11+
"cloud": {
12+
"provider": "aws",
13+
"service": {
14+
"name": "amazonmq_activemq"
15+
}
16+
},
17+
"ecs": {
18+
"version": "8.11.0"
19+
},
20+
"event": {
21+
"original": "2025-03-15 13:47:13,382 | WARN | Exception occurred for client ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7 (tcp://127.0.0.1:63868) processing: STOMP -> java.lang.SecurityException: User name [elastic] or password is invalid. | org.apache.activemq.transport.stomp.ProtocolConverter | ActiveMQ Transport: ssl:///127.0.0.1:63868",
22+
"type": [
23+
"error"
24+
]
25+
},
26+
"log": {
27+
"level": "WARN"
28+
},
29+
"message": "Exception occurred for client ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7 (tcp://127.0.0.1:63868) processing: STOMP -> java.lang.SecurityException: User name [elastic] or password is invalid.",
30+
"tags": [
31+
"preserve_original_event",
32+
"preserve_duplicate_custom_fields"
33+
]
34+
},
35+
{
36+
"@timestamp": "2025-03-15T13:47:13.381Z",
37+
"activemq": {
38+
"log": {
39+
"caller": "org.apache.activemq.broker.TransportConnection",
40+
"thread": "ActiveMQ Transport: ssl:///127.0.0.1:63868"
41+
}
42+
},
43+
"cloud": {
44+
"provider": "aws",
45+
"service": {
46+
"name": "amazonmq_activemq"
47+
}
48+
},
49+
"ecs": {
50+
"version": "8.11.0"
51+
},
52+
"event": {
53+
"original": "2025-03-15 13:47:13,381 | WARN | Failed to add Connection id=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientId=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientIP=tcp://127.0.0.1:63868 due to User name [elastic] or password is invalid. | org.apache.activemq.broker.TransportConnection | ActiveMQ Transport: ssl:///127.0.0.1:63868",
54+
"type": [
55+
"error"
56+
]
57+
},
58+
"log": {
59+
"level": "WARN"
60+
},
61+
"message": "Failed to add Connection id=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientId=ID:b-cfab2617-b6fb-4a44-bd7a-052aa4cd96f4-1-45943-1741903831778-5:7, clientIP=tcp://127.0.0.1:63868 due to User name [elastic] or password is invalid.",
62+
"tags": [
63+
"preserve_original_event",
64+
"preserve_duplicate_custom_fields"
65+
]
66+
}
67+
]
68+
}

‎packages/aws_mq/data_stream/activemq_general_logs/_dev/test/pipeline/test-common-config.yml

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
fields:
3+
tags:
4+
- preserve_original_event
5+
- preserve_duplicate_custom_fields

‎packages/aws_mq/data_stream/activemq_general_logs/agent/stream/aws-cloudwatch.yml.hbs

+103
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
{{#unless log_group_name}}
2+
{{#unless log_group_name_prefix}}
3+
{{#if log_group_arn }}
4+
log_group_arn: {{ log_group_arn }}
5+
{{/if}}
6+
{{/unless}}
7+
{{/unless}}
8+
9+
{{#unless log_group_arn}}
10+
{{#unless log_group_name}}
11+
{{#if log_group_name_prefix }}
12+
log_group_name_prefix: {{ log_group_name_prefix }}
13+
{{/if}}
14+
{{#if include_linked_accounts_with_prefix }}
15+
include_linked_accounts_for_prefix_mode: {{ include_linked_accounts_with_prefix }}
16+
{{/if}}
17+
{{/unless}}
18+
{{/unless}}
19+
20+
{{#unless log_group_arn}}
21+
{{#unless log_group_name_prefix}}
22+
{{#if log_group_name }}
23+
log_group_name: {{ log_group_name }}
24+
{{/if}}
25+
{{/unless}}
26+
{{/unless}}
27+
28+
{{#unless log_group_arn}}
29+
region_name: {{ region_name }}
30+
{{/unless}}
31+
32+
{{#unless log_stream_prefix}}
33+
{{#if log_streams }}
34+
log_streams: {{ log_streams }}
35+
{{/if}}
36+
{{/unless}}
37+
38+
{{#unless log_streams}}
39+
{{#if log_stream_prefix }}
40+
log_stream_prefix: {{ log_stream_prefix }}
41+
{{/if}}
42+
{{/unless}}
43+
44+
{{#if start_position }}
45+
start_position: {{ start_position }}
46+
{{/if}}
47+
48+
{{#if scan_frequency }}
49+
scan_frequency: {{ scan_frequency }}
50+
{{/if}}
51+
52+
{{#if api_sleep }}
53+
api_sleep: {{ api_sleep }}
54+
{{/if}}
55+
56+
{{#if latency }}
57+
latency: {{ latency }}
58+
{{/if}}
59+
{{#if number_of_workers }}
60+
number_of_workers: {{ number_of_workers }}
61+
{{/if}}
62+
63+
{{#if credential_profile_name}}
64+
credential_profile_name: {{credential_profile_name}}
65+
{{/if}}
66+
{{#if shared_credential_file}}
67+
shared_credential_file: {{shared_credential_file}}
68+
{{/if}}
69+
{{#if api_timeout}}
70+
api_timeout: {{api_timeout}}
71+
{{/if}}
72+
{{#if default_region}}
73+
default_region: {{default_region}}
74+
{{/if}}
75+
{{#if access_key_id}}
76+
access_key_id: {{access_key_id}}
77+
{{/if}}
78+
{{#if secret_access_key}}
79+
secret_access_key: {{secret_access_key}}
80+
{{/if}}
81+
{{#if session_token}}
82+
session_token: {{session_token}}
83+
{{/if}}
84+
{{#if role_arn}}
85+
role_arn: {{role_arn}}
86+
{{/if}}
87+
{{#if proxy_url }}
88+
proxy_url: {{proxy_url}}
89+
{{/if}}
90+
tags:
91+
{{#if preserve_original_event}}
92+
- preserve_original_event
93+
{{/if}}
94+
{{#each tags as |tag i|}}
95+
- {{tag}}
96+
{{/each}}
97+
{{#contains "forwarded" tags}}
98+
publisher_pipeline.disable_host: true
99+
{{/contains}}
100+
{{#if processors}}
101+
processors:
102+
{{processors}}
103+
{{/if}}

‎packages/aws_mq/data_stream/activemq_general_logs/elasticsearch/ingest_pipeline/default.yml

+84
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
description: Pipeline for ActiveMQ general logs in AmazonMQ.
3+
processors:
4+
- rename:
5+
field: message
6+
target_field: event.original
7+
if: 'ctx.event?.original == null'
8+
description: 'Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.'
9+
- remove:
10+
field: message
11+
ignore_missing: true
12+
if: ctx.event?.original != null
13+
description: 'The `message` field is no longer required if the document has an `event.original` field.'
14+
- set:
15+
field: ecs.version
16+
value: 8.11.0
17+
- set:
18+
field: cloud.service.name
19+
value: amazonmq_activemq
20+
- set:
21+
field: cloud.provider
22+
value: aws
23+
- grok:
24+
field: event.original
25+
pattern_definitions:
26+
GREEDYMULTILINE: "(.|\\n|\\t)*"
27+
NOPIPEGREEDYDATA: "(\\n|(?! \\|).)*"
28+
THREAD_NAME: "((?! \n).)*"
29+
TIMESTAMP_DATA: "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{NUMBER}"
30+
patterns:
31+
- "%{TIMESTAMP_DATA:timestamp}%{SPACE}\\|%{SPACE}%{LOGLEVEL:log.level}%{SPACE}\\|%{SPACE}%{NOPIPEGREEDYDATA:message}%{SPACE}\\|%{SPACE}%{NOPIPEGREEDYDATA:activemq.log.caller}%{SPACE}\\|%{SPACE}%{THREAD_NAME:activemq.log.thread}%{SPACE}%{GREEDYMULTILINE:error.stack_trace}"
32+
ignore_missing: true
33+
- date:
34+
if: "ctx.event.timezone == null"
35+
field: timestamp
36+
target_field: "@timestamp"
37+
formats: ["yyyy-MM-dd HH:mm:ss,SSS"]
38+
- date:
39+
if: "ctx.event.timezone != null"
40+
field: timestamp
41+
target_field: "@timestamp"
42+
timezone: "{{{ event.timezone }}}"
43+
formats: ["yyyy-MM-dd HH:mm:ss,SSS"]
44+
- remove:
45+
field:
46+
- timestamp
47+
- script:
48+
if: "ctx.log?.level != null"
49+
lang: painless
50+
source: >-
51+
def err_levels = ["FATAL", "ERROR", "WARN"];
52+
if (err_levels.contains(ctx.log.level)) {
53+
ctx.event.type = ["error"];
54+
} else {
55+
ctx.event.type = ["info"];
56+
}
57+
- script:
58+
description: Drops null/empty values recursively
59+
lang: painless
60+
ignore_failure: true
61+
source: |
62+
boolean drop(Object o) {
63+
if (o == null || o == "") {
64+
return true;
65+
} else if (o instanceof Map) {
66+
((Map) o).values().removeIf(v -> drop(v));
67+
return (((Map) o).size() == 0);
68+
} else if (o instanceof List) {
69+
((List) o).removeIf(v -> drop(v));
70+
return (((List) o).length == 0);
71+
}
72+
return false;
73+
}
74+
drop(ctx);
75+
on_failure:
76+
- set:
77+
field: event.kind
78+
value: pipeline_error
79+
- set:
80+
field: error.message
81+
value: >-
82+
Processor '{{{ _ingest.on_failure_processor_type }}}'
83+
{{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}'
84+
{{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}'

‎packages/aws_mq/data_stream/activemq_general_logs/fields/agent.yml

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
- name: cloud
2+
type: group
3+
fields:
4+
- name: image.id
5+
type: keyword
6+
description: Image ID for the cloud instance.
7+
- name: host
8+
type: group
9+
fields:
10+
- name: containerized
11+
type: boolean
12+
description: >
13+
If the host is a container.
14+
15+
- name: os.build
16+
type: keyword
17+
example: "18D109"
18+
description: >
19+
OS build information.
20+
21+
- name: os.codename
22+
type: keyword
23+
example: "stretch"
24+
description: >
25+
OS codename, if any.
26+
27+
- name: input.type
28+
type: keyword
29+
description: Type of Filebeat input.

‎packages/aws_mq/data_stream/activemq_general_logs/fields/base-fields.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
- name: data_stream.type
2+
external: ecs
3+
- name: data_stream.dataset
4+
external: ecs
5+
- name: data_stream.namespace
6+
external: ecs
7+
- name: "@timestamp"
8+
external: ecs
9+
- name: event.module
10+
type: constant_keyword
11+
external: ecs
12+
value: aws
13+
- name: event.dataset
14+
type: constant_keyword
15+
description: Event dataset
16+
value: aws_mq.activemq_general_logs

‎packages/aws_mq/data_stream/activemq_general_logs/fields/fields.yml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
- name: activemq.log
2+
type: group
3+
fields:
4+
- name: caller
5+
type: keyword
6+
description: Name of the caller issuing the logging request (class or resource).
7+
- name: thread
8+
type: keyword
9+
description: Thread that generated the logging event.

‎packages/aws_mq/data_stream/activemq_general_logs/fields/input.yml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- name: aws.cloudwatch
2+
type: group
3+
fields:
4+
- name: message
5+
type: text
6+
description: |
7+
CloudWatch log message.

0 commit comments

Comments
 (0)