Skip to content

[exporter/doris] Report rows filtered out by Doris during a successful stream load#49012

Open
vedatkilic wants to merge 1 commit into
open-telemetry:mainfrom
vedatkilic:dorisexporter-filtered-rows
Open

[exporter/doris] Report rows filtered out by Doris during a successful stream load#49012
vedatkilic wants to merge 1 commit into
open-telemetry:mainfrom
vedatkilic:dorisexporter-filtered-rows

Conversation

@vedatkilic

Copy link
Copy Markdown

Description

With max_filter_ratio > 0, Doris may drop up to that fraction of rows from a batch and still return Status: Success — the loss is only visible in the stream load response's NumberFilteredRows field, which the exporter parsed but never inspected (success() checks Status only). Today the only trace is log_response: true, which logs every response at INFO and relies on an operator reading them.

This PR surfaces the loss at the moment it happens:

  • A shared reportFilteredRows hook called from the logs/traces/metrics push paths: when NumberFilteredRows > 0 on an otherwise successful load it emits a WARN with filtered_rows, loaded_rows, the stream load label, and the Doris error_url (which points at the per-row rejection detail).
  • A new otelcol_exporter_doris_filtered_rows monotonic counter (mdatagen telemetry: section; this is the component's first internal metric, so the generated TelemetryBuilder is wired into the common exporter — the per-signal constructors and factory now propagate the builder's error).
  • Loads with zero filtered rows stay completely silent: no WARN, no counter points.

Link to tracking issue

Fixes #49011

Testing

  • TestPushLogDataFilteredRows: fake stream load endpoint returns Status: Success with NumberFilteredRows: 3; asserts the WARN fields (counts + error URL) via a zap observer and the counter value via metadatatest.AssertEqualExporterDorisFilteredRows.
  • TestPushLogDataNoFilteredRowsStaysSilent: clean response produces no WARN.
  • Full go test ./... for the module passes; gofmt/go vet clean.
  • The failure mode itself was reproduced against a live Doris cluster (3 BE, strict_mode on the load): a malformed row yields Status: Success, NumberLoadedRows: 2, NumberFilteredRows: 1 with no signal anywhere in the collector before this change.

Documentation

  • metadata.yaml telemetry entry + regenerated documentation.md describe the new metric.
  • Changelog entry added via .chloggen.

…l stream load

With max_filter_ratio > 0, Doris may drop rows from a batch and still
return Status: Success; the count is only visible in the response's
NumberFilteredRows field, which the exporter parsed but never read.
Emit a WARN with the row counts and error URL, and increment a new
otelcol_exporter_doris_filtered_rows counter, so partial data loss is
alertable instead of invisible.

Fixes open-telemetry#49011
@vedatkilic vedatkilic requested review from a team and atoulme as code owners June 11, 2026 07:41
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jun 11, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: vedatkilic / name: Vedat Kılıç (6fb0ddc)

@github-actions github-actions Bot added the first-time contributor PRs made by new contributors label Jun 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

  • Read our Contributing Guidelines.
  • Sign the CLA if you haven't already.
  • First-time contributors should have at most one PR not marked as draft until their first PR is merged.
  • If your change isn't one of our priority components, reviews may take more time.
  • Give reviewers at least a few days before pinging them for feedback.
  • If you need help or struggle to move your PR forward:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exporter/doris first-time contributor PRs made by new contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[exporter/doris] Rows filtered by Doris during a successful stream load are invisible

2 participants