[exporter/doris] Report rows filtered out by Doris during a successful stream load#49012
Open
vedatkilic wants to merge 1 commit into
Open
[exporter/doris] Report rows filtered out by Doris during a successful stream load#49012vedatkilic wants to merge 1 commit into
vedatkilic wants to merge 1 commit into
Conversation
…l stream load With max_filter_ratio > 0, Doris may drop rows from a batch and still return Status: Success; the count is only visible in the response's NumberFilteredRows field, which the exporter parsed but never read. Emit a WARN with the row counts and error URL, and increment a new otelcol_exporter_doris_filtered_rows counter, so partial data loss is alertable instead of invisible. Fixes open-telemetry#49011
|
|
Contributor
|
Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib. Important reminders:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
With
max_filter_ratio > 0, Doris may drop up to that fraction of rows from a batch and still returnStatus: Success— the loss is only visible in the stream load response'sNumberFilteredRowsfield, which the exporter parsed but never inspected (success()checksStatusonly). Today the only trace islog_response: true, which logs every response at INFO and relies on an operator reading them.This PR surfaces the loss at the moment it happens:
reportFilteredRowshook called from the logs/traces/metrics push paths: whenNumberFilteredRows > 0on an otherwise successful load it emits a WARN withfiltered_rows,loaded_rows, the stream loadlabel, and the Doriserror_url(which points at the per-row rejection detail).otelcol_exporter_doris_filtered_rowsmonotonic counter (mdatagentelemetry:section; this is the component's first internal metric, so the generatedTelemetryBuilderis wired into the common exporter — the per-signal constructors and factory now propagate the builder's error).Link to tracking issue
Fixes #49011
Testing
TestPushLogDataFilteredRows: fake stream load endpoint returnsStatus: SuccesswithNumberFilteredRows: 3; asserts the WARN fields (counts + error URL) via a zap observer and the counter value viametadatatest.AssertEqualExporterDorisFilteredRows.TestPushLogDataNoFilteredRowsStaysSilent: clean response produces no WARN.go test ./...for the module passes;gofmt/go vetclean.strict_modeon the load): a malformed row yieldsStatus: Success, NumberLoadedRows: 2, NumberFilteredRows: 1with no signal anywhere in the collector before this change.Documentation
metadata.yamltelemetry entry + regenerateddocumentation.mddescribe the new metric..chloggen.