Skip to content

[Geneva Exporter] Encode logs using LogsDataView#2810

Merged
jmacd merged 9 commits intoopen-telemetry:mainfrom
lalitb:geneva-zero-copy-log-records
May 5, 2026
Merged

[Geneva Exporter] Encode logs using LogsDataView#2810
jmacd merged 9 commits intoopen-telemetry:mainfrom
lalitb:geneva-zero-copy-log-records

Conversation

@lalitb
Copy link
Copy Markdown
Member

@lalitb lalitb commented May 1, 2026

Description

Updates the otel-arrow Geneva exporter to encode log payloads through the shared LogsDataView abstraction instead of converting logs back through OTLP proto objects.

Changes

  • Encode OTAP Arrow logs through OtapLogsView, the OTAP implementation of LogsDataView.
  • Encode OTLP byte logs through RawLogsData, the OTLP bytes implementation of LogsDataView.
  • Call the existing geneva-uploader encode_and_compress_logs(&view) API for both log paths.
  • Patch otap-df-pdata / otap-df-pdata-views so geneva-uploader uses the workspace view traits.
  • Add a TODO for compressed byte accounting until EncodedBatch::data_len() is exposed upstream.

How are these changes tested?

  • cargo fmt --all
  • CARGO_NET_GIT_FETCH_WITH_CLI=true cargo check -p otap-df-contrib-nodes --features "geneva-exporter otap-df-otap/crypto-ring"

Are there any user-facing changes?

None.

@lalitb lalitb marked this pull request as ready for review May 1, 2026 23:33
@lalitb lalitb requested a review from a team as a code owner May 1, 2026 23:33
@github-actions github-actions Bot added the rust Pull requests that update Rust code label May 1, 2026
@lalitb lalitb requested a review from utpilla May 1, 2026 23:35
@codecov
Copy link
Copy Markdown

codecov Bot commented May 1, 2026

Codecov Report

❌ Patch coverage is 73.57143% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.08%. Comparing base (11fdb97) to head (314b598).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2810      +/-   ##
==========================================
+ Coverage   86.06%   86.08%   +0.01%     
==========================================
  Files         705      705              
  Lines      265493   265593     +100     
==========================================
+ Hits       228498   228624     +126     
+ Misses      36471    36445      -26     
  Partials      524      524              
Components Coverage Δ
otap-dataflow 87.02% <73.57%> (+0.01%) ⬆️
query_abstraction 80.61% <ø> (ø)
query_engine 90.76% <ø> (ø)
otel-arrow-go 52.45% <ø> (ø)
quiver 92.25% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread rust/otap-dataflow/crates/contrib-nodes/src/exporters/geneva_exporter/mod.rs Outdated
Comment thread rust/otap-dataflow/crates/contrib-nodes/src/exporters/geneva_exporter/mod.rs Outdated
Comment thread rust/otap-dataflow/crates/pdata/src/views/otap/logs.rs Outdated
@lalitb
Copy link
Copy Markdown
Member Author

lalitb commented May 4, 2026

Thanks @AaronRM @utpilla, these are fair comments.
Lesson learned: I trusted Friday-evening-me to review this properly before raising it. That was clearly too much trust. I’ll clean this up properly and push an update.

@lalitb lalitb force-pushed the geneva-zero-copy-log-records branch from 9654a83 to 6699544 Compare May 4, 2026 20:01
Comment thread rust/otap-dataflow/crates/contrib-nodes/src/exporters/geneva_exporter/mod.rs Outdated
Comment thread rust/otap-dataflow/crates/contrib-nodes/src/exporters/geneva_exporter/mod.rs Outdated
Comment thread rust/otap-dataflow/crates/contrib-nodes/src/exporters/geneva_exporter/mod.rs Outdated
Comment thread rust/otap-dataflow/crates/pdata/src/views/otlp/bytes/logs.rs Outdated
Copy link
Copy Markdown
Contributor

@utpilla utpilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few minor non-blocking suggestions! Thanks @lalitb!

@lalitb
Copy link
Copy Markdown
Member Author

lalitb commented May 4, 2026

@AaronRM - Can you have a look when possible - as you have requested the changes, and it is now resolved.

// .map_err(|e| format!("Failed to encode logs from view: {}", e))?;

// Fallback path: Convert OTAP Arrow → OTLP bytes
mut otap_records @ OtapArrowRecords::Logs(_) => {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today is a good day to learn new Rust syntax.

@jmacd jmacd enabled auto-merge May 5, 2026 02:30
@jmacd jmacd added this pull request to the merge queue May 5, 2026
Merged via the queue into open-telemetry:main with commit 8b1e57d May 5, 2026
78 of 79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rust Pull requests that update Rust code

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants