Skip to content

Conversation

@vsmanish1772
Copy link

@vsmanish1772 vsmanish1772 commented Oct 14, 2025

Description

Track the number of commit retry attempts in operationMetrics for write operations.

Changes

I have added num_retries to WriteMetrics struct which will be available inside operationMetrics inside delta log. I have used #[serde(default)] for backward compatibility with old delta logs.

The reason for the current approach was because operationMetrics is written to the delta log before we know how many retries were needed, making it difficult to include in the persisted log. Hence I do reserialise the commit log again.

Result

Delta log now includes num_retries in operationMetrics: {
"operationMetrics": {
"num_added_files": 5,
"num_retries": 2,
...
}
}

Related Issue(s)

Documentation

Manish Sogiyawar added 2 commits October 14, 2025 12:58
Track the number of commit retry attempts in operationMetrics for write operations.

## Changes

1. Add num_retries field to WriteMetrics struct
   - Initialized to 0 when metrics are created
   - Uses #[serde(default)] for backward compatibility with old delta logs

2. Add update_retry_count_in_bytes() helper method
   - Efficiently updates only the commitInfo action's operationMetrics
   - Avoids re-serializing all Add/Remove actions (important for high-concurrency scenarios)
   - Updates num_retries field right before each commit attempt

3. Update retry loop in PreparedCommit
   - Calls helper method before each write_commit_entry attempt
   - Ensures accurate retry count is persisted to delta log

## Performance

- Minimal overhead: only parses/serializes commitInfo (~1-2KB), not all actions
- Critical for high-concurrency workloads where retries are common

## Result

Delta log now includes num_retries in operationMetrics:
{
  "operationMetrics": {
    "num_added_files": 5,
    "num_retries": 2,
    ...
  }
}
@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Oct 14, 2025
@vsmanish1772
Copy link
Author

I'm not really happy with the approach or is it a right approach but I could think of any other solution as it was tricky. let me know if we can do something better.

Also I would like to add this change for all different operations i.e delete, merge, update etc. To keep the pr small I have started with write operation and will bring in subsequent pr's once we agree with approach here
@rtyler @ion-elgreco

@ion-elgreco
Copy link
Collaborator

Hey @vsmanish1772, as you are saying as well, I don't believe this should be the approach. We need to somehow defer the bytes creation at a later stage but I also wonder this will give overhead to re-update the commit bytes each time you retry, so maybe this should be an opt-in functionality only (@roeap @rtyler, thoughts?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/rust Issues for the Rust crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants