feat(tracing): add tracing spans to all I/O sections #3795

fvaleye · 2025-09-30T12:50:21Z

Description

To better understand performance in the delta-rs crate, I added additional tracing to capture more detailed debug-level performance information.

Python now uses OpenTelemetry to collect tracing data emitted from Rust.
With this change, we gain true end-to-end visibility: Python spans can serve as parents of Rust spans (and vice versa), ensuring a continuous trace across both runtimes.

Related Issue(s)

close Introduce tracing spans to all I/O sections #3641

Documentation

https://docs.rs/tracing/latest/tracing/

github-actions · 2025-09-30T12:50:43Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

codecov · 2025-09-30T12:54:46Z

Codecov Report

❌ Patch coverage is 73.60775% with 109 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.98%. Comparing base (acd75d6) to head (610b4a6).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/tracing_otlp.rs	0.00%	37 Missing ⚠️
crates/core/src/operations/write/mod.rs	87.22%	6 Missing and 23 partials ⚠️
crates/core/src/kernel/transaction/mod.rs	76.85%	26 Missing and 2 partials ⚠️
python/src/lib.rs	0.00%	10 Missing ⚠️
crates/core/src/operations/filesystem_check.rs	71.42%	2 Missing ⚠️
crates/core/src/operations/vacuum.rs	71.42%	2 Missing ⚠️
crates/core/src/logstore/mod.rs	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3795      +/-   ##
==========================================
- Coverage   74.32%   73.98%   -0.35%     
==========================================
  Files         147      148       +1     
  Lines       39722    38882     -840     
  Branches    39722    38882     -840     
==========================================
- Hits        29523    28765     -758     
- Misses       8808     8852      +44     
+ Partials     1391     1265     -126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

rtyler

Generally very excited to have more tracing in place. Once tests pass I'm comfortable merging this

rtyler · 2025-09-30T13:05:27Z

crates/core/src/kernel/transaction/mod.rs

            let commit_or_bytes = this.commit_or_bytes;

            if this.table_data.is_none() {
+                tracing::debug!("committing initial table version 0");


👋 in a lot of places we actually just import use tracing::*; which means this module prefix isn't necessary.

IMHO it's reasonable for us to pull tracing::* into most modules 😄

Okay! It was a way to differentiate log from tracing.

Just imported with use tracing::*;, if you found it's ok for readability.
I didn't want to mix log and tracing crates, but it seems reasonable for now! We can tighten the imports or add explicit bridging later if the project grows.

Can we do it now to be more explicit or is tracing a very small library?

crates/core/src/delta_datafusion/mod.rs

crates/core/src/kernel/transaction/mod.rs

fvaleye · 2025-09-30T13:30:14Z

Generally very excited to have more tracing in place. Once tests pass I'm comfortable merging this

Same!

I’m re-requesting a review based on your feedback. I’m still waiting for the code coverage results, and we can proceed with a merge!

ion-elgreco · 2025-09-30T14:11:32Z

This is stdout/err tracing only, right?

fvaleye · 2025-09-30T14:28:08Z

This is stdout/err tracing only, right?

I used warning, info, and debug levels together with tracing, adding Spans with custom fields to track performance.

tracing isn’t limited to stdout or stderr. It’s a subscriber-agnostic instrumentation framework that emits structured events with typed fields, not just plain text logs.

Where these traces go is entirely up to the user: with tracing-subscriber, traces can be sent to distributed tracing backends like Jaeger, Zipkin, Tempo, Honeycomb, or Datadog, to OpenTelemetry collectors, to custom backends, or nowhere at all.

Example:

// Option 1: Console output
let subscriber = tracing_subscriber::fmt()
    .with_env_filter("deltalake=debug")
    .finish();
tracing::subscriber::set_global_default(subscriber)?;

Side note: Importantly, if no subscriber is configured, the overhead is nearly zero.

ion-elgreco · 2025-10-01T07:59:40Z

I get that, my point is rather, how would one configure it to use push to an OTEL endpoint from python ^^

fvaleye · 2025-10-01T08:12:10Z

I get that, my point is rather, how would one configure it to use push to an OTEL endpoint from python ^^

Ah, sorry! 😀
Indeed, we might need to wire tracing-opentelemetry + opentelemetry-otlp so spans are pushed to an OTLP endpoint (e.g. http://localhost:4317). On the Python side, we just point OpenTelemetry SDK at the same collector. Both Rust and Python traces then land in the same backend.

In Python, we might need to add a small API like deltalake.init_tracing("otel_endpoint") that sets up the exporter inside the Rust runtime, WDYT?

I can take time to add it, since we have many cases where the performances is evaluated from Python!

rtyler · 2025-10-01T12:55:37Z

I'm marking this as a draft, please change that when you're ready for this to be merged @fvaleye

…tracing Signed-off-by: Florian Valeye <[email protected]>

Resolved conflicts: - crates/core/src/delta_datafusion/mod.rs: Kept find_files implementation with tracing - crates/core/src/operations/optimize.rs: Merged logging with object_store initialization - crates/core/src/operations/write/mod.rs: Used DeltaPlanner::new() from main, maintained tracing - crates/core/src/protocol/checkpoints.rs: Used specific tracing imports from main

ion-elgreco · 2025-10-08T08:38:25Z

@fvaleye is this in a state it can be reviewed again?

fvaleye · 2025-10-08T08:40:44Z

@fvaleye is this in a state it can be reviewed again?

Yes!

crates/core/src/kernel/transaction/mod.rs

ion-elgreco · 2025-10-08T16:00:51Z

crates/core/src/kernel/transaction/mod.rs

-                            latest_version - steps,
-                            (latest_version - steps) + 1,
-                        )
+            async move {


The additional async move doesn't seem necessary, why this change?

It's more for adding an instrument to this part. instrument (commit_span).await
It ensures the span is correctly attached to the future and doesn't cause issues with await points.
But, I can extract the retry logic into a separate function and use #[instrument] macro on it, which would be cleaner.

Ah ok, yeah the diff was so big I didn't see that

Normal, with all these whitespaces 😄

crates/core/src/logstore/mod.rs

ion-elgreco · 2025-10-08T16:40:20Z

crates/core/src/logstore/storage/runtime.rs


 #[async_trait::async_trait]
 impl<T: ObjectStore + Clone> ObjectStore for DeltaIOStorageBackend<T> {
+    #[instrument(skip(self, bytes), fields(path = %location, size = bytes.content_length()))]


This wrapper is almost never hit at this point, maybe we need a simple wrapper during storage creation that wraps and instruments these methods.

I also wonder if the object store doesn't have some native tracing?

Good point!
I don't know, but we could add an issue to track this need.

@fvaleye yeah sound goods

ion-elgreco · 2025-10-08T16:42:22Z

crates/core/src/operations/write/mod.rs

Why are there changes being made here? Seems like a wrong resolution of merge conflict

Right, let me take a look!

Changed it, but there is a significant change due to the async traced block.

crates/core/src/operations/write/writer.rs

python/pyproject.toml

ion-elgreco · 2025-10-08T16:48:40Z

python/tests/test_tracing.py

+    with tracer.start_as_current_span("delta_write_operation") as span:
+        span.set_attribute("table.path", table_path)
+        span.set_attribute("operation.mode", "overwrite")
+
+        write_deltalake(table_path, sample_data, mode="overwrite")


I guess a follow up pr could be to do this for users directly?

Yes, good idea! 👍

roeap

Just flushing some first pass comments. awesome stuff!!

For everyone reviewing this, I recommend disabling whitespace changes.

roeap · 2025-10-08T11:18:36Z

crates/core/src/logstore/mod.rs

        .map_err(|err| -> TransactionError {
            match err {
                ObjectStoreError::AlreadyExists { .. } => {
+                    warn!("commit entry already exists");


does this need to be a warning? conflicting commits might be considered something we expect in the "normal" flow of things.

Good question!
I wanted to inform the user about the AlreadyExists error encountered, which falls between info and warn.

crates/core/src/operations/write/mod.rs

roeap · 2025-10-08T16:46:45Z

python/README.md

 objection store communication. Please file Github issue to request for critical
 openssl upgrade.

+## Tracing and Observability


I do agree that it is high time to update our README, but is this critical enough for people coming in, or should this be in the docs (or both?)

Not critical, but I wanted to create a section to improve our issue resolution by adding instructions on how to enable tracing. This would provide additional context when investigating issues.
I can definitely move this somewhere else.

python/README.md

Signed-off-by: Florian Valeye <[email protected]>

crates/core/src/logstore/storage/runtime.rs

Signed-off-by: Florian Valeye <[email protected]>

ion-elgreco · 2025-10-09T10:38:45Z

Just flushing some first pass comments. awesome stuff!!

For everyone reviewing this, I recommend disabling whitespace changes.

Yeah that definetly helps, should have enabled that earlier :P

Resolved conflicts in write operation by: - Using session refactoring from main (instead of state) - Keeping MetricObserver tracing from feature branch - Maintaining instrumentation spans for performance analysis

fvaleye requested review from hntd187, roeap and rtyler as code owners September 30, 2025 12:50

fvaleye added enhancement New feature or request binding/rust Issues for the Rust crate labels Sep 30, 2025

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch 2 times, most recently from 3f02b2c to 2641ccd Compare September 30, 2025 12:52

fvaleye changed the title ~~feat(tracing): Add tracing spans to all I/O sections~~ feat(tracing): add tracing spans to all I/O sections Sep 30, 2025

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from 2641ccd to e5812b1 Compare September 30, 2025 13:05

rtyler previously approved these changes Sep 30, 2025

View reviewed changes

fvaleye dismissed rtyler’s stale review via 4164b72 September 30, 2025 13:18

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch 3 times, most recently from 37d883c to ebed3e9 Compare September 30, 2025 13:24

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from ebed3e9 to fdd6946 Compare September 30, 2025 13:50

fvaleye requested a review from rtyler September 30, 2025 13:50

fvaleye enabled auto-merge (rebase) September 30, 2025 13:55

rtyler marked this pull request as draft October 1, 2025 12:55

auto-merge was automatically disabled October 1, 2025 12:55
Pull request was converted to draft

github-actions bot added the binding/python Issues for the Python package label Oct 1, 2025

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch 2 times, most recently from d810d7d to a6738b0 Compare October 1, 2025 14:39

feat(tracing): add environment variables documentation for OTEL with …

44b478c

…tracing Signed-off-by: Florian Valeye <[email protected]>

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch 4 times, most recently from 69785aa to b6040eb Compare October 7, 2025 17:11

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from b6040eb to ada34d4 Compare October 7, 2025 17:13

ion-elgreco requested changes Oct 8, 2025

View reviewed changes

roeap reviewed Oct 8, 2025

View reviewed changes

fvaleye added 5 commits October 8, 2025 18:58

fix(commit): revert post commit properties for initial table creation

9d4bc9b

Signed-off-by: Florian Valeye <[email protected]>

feat(version): add version to commit entry read error log

ae18dab

Signed-off-by: Florian Valeye <[email protected]>

chore: remove unused tracing from DeltaIOStorageBackend

4d25f4f

Signed-off-by: Florian Valeye <[email protected]>

chore: add path to multipart upload debug logs

d122206

Signed-off-by: Florian Valeye <[email protected]>

refactor: use SOURCE_COUNT_ID constant instead of hardcoded string

27405f7

Signed-off-by: Florian Valeye <[email protected]>

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from 9402e59 to 78bd04f Compare October 8, 2025 17:34

ion-elgreco reviewed Oct 9, 2025

View reviewed changes

crates/core/src/logstore/storage/runtime.rs Outdated Show resolved Hide resolved

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from 78bd04f to 4e669e1 Compare October 9, 2025 09:29

fix(writer): revert writer due to bad merge conflict

e04ffd1

Signed-off-by: Florian Valeye <[email protected]>

fvaleye force-pushed the feature/add-tracing-for-performance-analysis branch from 4e669e1 to e04ffd1 Compare October 9, 2025 09:37

Merge branch 'main' into feature/add-tracing-for-performance-analysis

7de306d

fvaleye added 3 commits October 11, 2025 14:35

Merge main into feature/add-tracing-for-performance-analysis

16041b7

Resolved conflicts in write operation by: - Using session refactoring from main (instead of state) - Keeping MetricObserver tracing from feature branch - Maintaining instrumentation spans for performance analysis

Merge branch 'main' into feature/add-tracing-for-performance-analysis

7ec96e8

Merge branch 'main' into feature/add-tracing-for-performance-analysis

1520e97

ion-elgreco approved these changes Oct 13, 2025

View reviewed changes

Merge branch 'main' into feature/add-tracing-for-performance-analysis

610b4a6

ion-elgreco enabled auto-merge (squash) October 13, 2025 08:05

ion-elgreco merged commit 4acc60b into delta-io:main Oct 13, 2025
28 of 29 checks passed

Uh oh!

feat(tracing): add tracing spans to all I/O sections #3795

feat(tracing): add tracing spans to all I/O sections #3795

Uh oh!

Conversation

fvaleye commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Documentation

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

codecov bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rtyler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ion-elgreco Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fvaleye commented Sep 30, 2025

Uh oh!

ion-elgreco commented Sep 30, 2025

Uh oh!

fvaleye commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ion-elgreco commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fvaleye commented Oct 1, 2025

Uh oh!

rtyler commented Oct 1, 2025

Uh oh!

ion-elgreco commented Oct 8, 2025

Uh oh!

fvaleye commented Oct 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fvaleye Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fvaleye commented Sep 30, 2025 •

edited

Loading

codecov bot commented Sep 30, 2025 •

edited

Loading

ion-elgreco Oct 8, 2025 •

edited

Loading

fvaleye commented Sep 30, 2025 •

edited

Loading

ion-elgreco commented Oct 1, 2025 •

edited

Loading

fvaleye Oct 8, 2025 •

edited

Loading