Vote: Atomic table_metadata updates alongside Operation::Update

[fecet]

TL;DR

Let a Lance Operation::Update commit atomically carry a small
Map<String, String> written into the manifest's table_metadata alongside
the row-level changes. (Mechanically the field lives on Transaction so
internal write paths don't need to know about it — details in the PR.)

Today the same UpdateMap exists inside Operation::UpdateConfig but cannot
ride along with a data commit, forcing streaming writers into a two-commit
race. Draft implementation: #6255. This thread is about
whether the format should grow this, not how.

The problem

A streaming writer / sink wants one piece of durable state per commit —
a cursor, an offset, a consumed-range — and it needs that state to land
atomically with the rows produced by Operation::Update. Lose the cursor
after landing rows → duplicates on restart. Land the cursor without the rows
→ gaps.

Operation::Update already is Lance's data-plane superset (remove + rewrite

• add fragments; used by update, merge_insert, and by a broker-style
  fold coordinator). What's missing is "... and record a cursor slot in the
  same commit".

Lance today offers nothing that checks all three boxes of "atomic with the
Update / readable in O(1) / safe under concurrent writers":

	atomic w/ Update	O(1) readback	concurrent writers
transaction_properties	yes	no (O(versions) scan of _transactions/*)	yes
UpdateConfig.table_metadata_updates	no (separate commit)	yes	yes
tags	no (not in the Update's manifest txn)	yes	no (one writer)

So every streaming workload on Lance either pays an O(N) startup scan or
races two commits. We've been carrying an out-of-tree patch and would rather
have the conversation upstream.

Value — what lands if this ships

All use cases are commits that pair an Operation::Update with a small
metadata write:

• Consumer progress slots — a consumer's merge_insert (which produces
  Operation::Update) atomically records its
  consumer={id}: {range}:{unacks} slot. Restart re-reads dataset.metadata
  in zero extra I/O.
• Broker / coordinator fold — a sole-writer broker folds many clients'
  fragment changes into a single Operation::Update, carrying the union of
  their cursor slots in one manifest write instead of two.
• Exactly-once sinks — any sink whose commit is already an
  Operation::Update (merge_insert by pk, or update-by-predicate) gets a
  clean home for its offset/watermark.

Concurrent disjoint keys commute and fuse; overlapping keys or replace:true
conflict — same semantics as the existing UpdateConfig path.

Relationship to #5960

#5960 proposes a richer, more comprehensive metadata story across the
transaction. Strictly more capable than this; if it lands soon we don't need
this field at all.

Two honest framings to pick between:

a) Ship this now as a bridge. Streaming workloads on Operation::Update
unblock today; when #5960 lands, this field either stays as the simple path
or gets aliased during a deprecation window.

b) Skip this, wait for #5960. If the comprehensive design is close
(weeks not months), the format commitment of an extra field isn't worth
the churn.

We'd carry the fork either way — the question for upstream is whether the
minimal version is worth merging in the meantime.

Prior art

Peer formats all have a same-shape slot — a transaction/snapshot-level string
map written atomically with the commit:

• Iceberg — Snapshot.summary (Map<String,String>), part of the
  atomic snapshot commit.
• Delta Lake — CommitInfo.tags per commit; SetTransaction action
  specifically for exactly-once sink offsets (appId → version).
• Hudi — HoodieCommitMetadata.extraMetadata on each timeline instant.

This proposal is the minimal equivalent for Lance, scoped to the
Operation::Update commit path.

Discussion questions

Regardless of how you vote, input on any of these would help us refine the
proposal (or decide to withdraw it):

1. Is UserOperation: Action-based Transactions for real diffs #5960 close enough that we should withdraw this and contribute
   there instead?
2. If this ships, should it eventually replace
   Operation::UpdateConfig.table_metadata_updates, or coexist long-term?
3. Are there use cases beyond Operation::Update that would want the same
   atomic-metadata slot, strong enough to broaden the scope?

Vote

Per lance.org/community/voting, this
is a format spec change (touches protos/transaction.proto):

• Passes with 3 binding +1 votes from PMC (excluding the proposer)
• Minimum voting period: 1 week from this post
• A single binding -1 is a veto and stops the proposal until resolved
• -1 votes without justification are invalid
• Please cast each vote as an independent comment on this Discussion
  (not as a reply inside another comment), so concerns get their own thread
• Use the scale: +1 / 0 / -1, annotated (binding) or (non-binding)

[wjones127]

This thread is about
whether the format should grow this, not how.

Generally, for format changes we vote on the specific spec changes. The protobuf changes are changes in the spec. The library changes can be considered separate an an implementation detail that doesn't need a vote (although seeing the library implementation can help build the case for the format changes).

I'm -1 on adding the field to Transaction.

I'd be +1 on adding the field to Update operation message instead. I believe this field should be superseded by #5960, so I'd prefer to put this interim fix into a message that will be unused in the final state. Does that seem acceptable to you?

[fecet]

Thanks, that makes sense.

We agree with your suggestion. Since this is intended as an interim fix and would likely be
superseded by #5960, it is better to put the field on the Update message instead of
Transaction. We've already updated #6255 to follow that approach, and the PR now carries the
field on the message rather than on Transaction.

[wjones127]

Sorry, we've left this stale for a while. We are starting to look at Transaction V2 now. Do you still need this? Or can we wait a few weeks for Transaction V2 to be implemented?