Open
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
Vector's internal metrics emits metrics with the "aggregated_histogram" type. As part of my pipeline I convert these metrics to logs to be able to ship them to a remote http_server vector source.
This might not be the optimal setup when one could use a native vector source instead, but it has uncovered that log_to_metric emits an ERROR when it encounters logs with the aggregated_histogram property, as discussed here.
Relevant log
2024-05-14T10:45:01.277458Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector::internal_events::log_to_metric: Internal log [Missing required metric details. Required one of gauge, distribution, histogram, summary, counter] has been suppressed 279 times.
2024-05-14T10:45:01.277528Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector::internal_events::log_to_metric: Missing required metric details. Required one of gauge, distribution, histogram, summary, counter error_code="missing_metric_details" error_type="parser_failed" stage="processing" internal_log_rate_limit=true
2024-05-14T10:45:01.277585Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 279 times.
2024-05-14T10:45:01.278201Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Missing required metric details. Required one of gauge, distribution, histogram, summary, counter" internal_log_rate_limit=true
2024-05-14T10:45:01.278310Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector::internal_events::log_to_metric: Internal log [Missing required metric details. Required one of gauge, distribution, histogram, summary, counter] is being suppressed to avoid flooding.
2024-05-14T10:45:01.278348Z ERROR transform{component_kind="transform" component_id=logs_to_metrics component_type=log_to_metric}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
Currently I work around this problem simply by using a route transform that matchese these events, and then using log_to_metric on that route's _unmatched route.
Configuration
sources:
logs_only_receiver:
type: http_server
address: 0.0.0.0:8081
decoding:
codec: json
vector_metrics:
type: internal_metrics
transforms:
metrics_to_logs:
type: metric_to_log
inputs: [ vector_metrics ]
# Drop aggregated histogram events because log_to_metric does not support it as of 0.38.1
drop_route:
type: route
route:
aggregated_histogram: .aggregated_histogram != null
inputs:
- logs_only_receiver
logs_to_metrics:
type: log_to_metric
metrics: []
all_metrics: true
inputs:
- drop_route._unmatched
sinks:
send_to_http_server:
type: http
inputs:
- metrics_to_logs
uri: http://localhost:8081
encoding:
codec: json
export_received_metrics:
type: prometheus_exporter
address: 0.0.0.0:9199
inputs: [ logs_to_metrics ]
Version
0.38.1
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response