fix(table): allocate partition IDs from history

[mattfaltyn]

Closes #1987.

Summary

• preserve the persisted last-partition-id during metadata parsing so optimistic concurrency requirements still match the catalog
• allocate new partition field IDs above the greatest of 999, the persisted counter, and every historical partition field ID
• share the allocation floor between UpdateSpec and MetadataBuilder.AddPartitionSpec
• preserve the unchanged-byte fast path for metadata with no missing field IDs

Why

Format v2+ partition field IDs are unique across every spec in a table. When persisted metadata contains field ID 1000 but last-partition-id is 999, trusting only the counter can allocate 1000 to a different transform.

Repairing the counter at read time would make AssertLastAssignedPartitionID send 1000 to a catalog that still stores 999, causing a permanent false conflict. The allocation path now scans partition history while the requirement continues to use the catalog-visible persisted value. Adding a distinct field therefore asserts 999, allocates 1001, and lets the committed metadata self-heal.

Testing

• added end-to-end coverage for a stale counter with the greatest field ID in a non-current historical spec
• added direct builder coverage for missing-ID allocation above partition history
• hardened fixture mutation checks
• make test
• targeted go test -race coverage for partition allocation and commit paths
• make test-assert
• go vet ./table/...
• golangci-lint run --timeout=10m using v2.12.2
• git diff --check

No external documentation changes are needed because this restores safe partition evolution without changing the public API.

[zeroshade]

The stale-counter normalization correctly fixes the #1987 ID-collision and preserves the byte-identical fast path, but it also rewrites sub-999 counters on tables with no assigned partition field IDs, which desyncs the client's AssertLastAssignedPartitionID from a catalog that persisted the original value.

Re-review verification: 0 of 1 prior findings confirmed fixed at abc35da (each verified by mutating the fix and observing the suite go red, not by taking the claim on trust).

Smaller observations

• table/update_spec_test.go:1061 — PR description implies UpdateSpec coverage that the diff does not add: Bullet three claims the PR will 'cover the all-field-IDs-present case that previously allowed UpdateSpec to reuse a historical ID', and the Testing section lists TestUpdateSpecReuseHistoricalFieldID. That test is pre-existing (added by #1641) and is not touched by this diff -- it is only re-run. The diff adds exactly one test. Reword so the description matches what is actually added.

Verification performed

go build ./... (PASS); go test ./table/... (PASS: table, compaction, dv, internal, substrait); go test ./cmd/... (PASS); go test ./catalog/... (PASS: catalog, glue, hadoop, hive, internal, rest, planfake, sql); go test ./table/ -run '^TestP[0-9]+' with throwaway probes pr1988_probe_test.go + pr1988_probe2_test.go (13 probes, all ran; P13 discharged its own hypothesis); regression check -- restored table/metadata.go from HEAD~1 and ran the new test: FAIL 'expected: 1000, actual: 999', confirming the added test is non-vacuous. Probe files deleted and worktree confirmed clean via git status --porcelain.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer. The findings below are observations, not blockers; an Apache Iceberg Go maintainer — a real person — will take the next look at the PR. If you think a finding is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Iceberg Go handles maintainer review: CONTRIBUTING.md.

[zeroshade]

major — 999 assignment floor is conflated with the persisted counter, rewriting collision-free sub-999 values

lastAssignedID is seeded to iceberg.PartitionDataIDStart-1 (999) at line 2018 and then max'd with the persisted counter and all field IDs. The same variable is written back to last-partition-id at line 2077. For metadata with NO assigned partition field IDs, there is no collision to repair, yet any persisted counter below 999 (0, 5, 998) is rewritten to 999 and the unchanged-byte fast path is lost. Because update_spec.go:190-197 derives AssertLastAssignedPartitionID from this normalized value and rest.go:1626-1636 sends it to the catalog, a spec-changing commit against a REST catalog that persisted 0 now asserts 999 and is rejected -- before this PR the client sent 0 and matched. Fix: compute the persist target as max(persistedCounter, maxAssignedFieldID) as a variable separate from the assignment floor, so tables with no assigned field IDs are left untouched. If the 999 floor is deliberate (the linked issue does request 'the greatest of 999 and every explicit partition field ID'), add explicit test coverage for sub-999 counters and state the intentional mutation in the PR description, since the description currently claims only that already-consistent metadata is preserved.

Evidence

Probe P8 (sweep, unpartitioned v2, partition-specs [{spec-id:0,fields:[]}]): 'persisted=0 -> parsed=999 rewritten=true', 'persisted=5 -> parsed=999 rewritten=true', 'persisted=998 -> parsed=999 rewritten=true', 'persisted=999 -> parsed=999 rewritten=false'. Probe P10: 'client asserts 999 against catalog holding 0 -> requirement failed: last assigned partition id has changed: expected 999, found 0'. Probe P11: 're-serialized last-partition-id = 999 (was 0 on disk)'. Shape present in repo fixtures: cmd/iceberg/snapshots_test.go:45, cmd/iceberg/branch_tag_test.go:49, cmd/iceberg/partition_stats_test.go:44.

[mattfaltyn]

Thanks for catching this. Fixed in ebf0a16 by separating the persisted counter target from the 999 allocation cursor. Empty specs with sub-999 counters now retain the original bytes and catalog-visible value, while stale counters with assigned fields still normalize correctly.

[zeroshade]

minor — New early-return condition adds three branches, only one is tested

The condition at metadata.go:2047 introduces distinct branches: counter below max field ID (tested), counter above max field ID (must stay untouched), counter below the 999 floor with no field IDs, and stale counter combined with a missing field-id. Only the first has a test. I verified the untested ones behave as follows -- add cases for them so the condition is pinned: counter-above-max stays untouched, and stale-counter-plus-missing-field-id assigns correctly.

[mattfaltyn]

Thanks for identifying the missing branches. Added coverage in ebf0a16 for sub-999 counters with no assigned fields, counters above the greatest assigned field ID, and stale counters combined with a missing field ID. The first two cases also verify the unchanged-byte fast path.

[zeroshade]

minor — Missing-field re-marshal path silently drops unknown partition-spec keys

The new 'if len(missingFields) > 0' guard correctly keeps the stale-counter-only path from round-tripping specs through rawPartitionSpec (which carries only spec-id and fields). But the missing-field path it now wraps still does, so any other key on a partition-spec object is dropped on rewrite. This is pre-existing rather than introduced -- flagging it because the PR restructured exactly this block and the asymmetry between the two paths is now visible in the diff.

[mattfaltyn]

Thanks for flagging this. I confirmed the unknown partition-spec key loss predates this change and left it unchanged to keep #1987 focused. The stale-counter-only path introduced here continues to avoid partition-spec re-marshalling.

[mattfaltyn]

Thanks for the thorough review and clear reproduction. The counter-floor regression and requested boundary coverage are addressed in ebf0a16. I also updated the PR description to distinguish the newly added regression tests from the pre-existing historical-ID test. All local unit, race, assert, and lint checks pass.

[zeroshade]

The prior major is genuinely fixed — the 999 allocation floor is now separate from the persisted counter and sub-999 metadata round-trips byte-identically — leaving only an unpinned-by-tests write site at metadata.go:2081.

Re-review verification: 2 of 3 prior findings confirmed fixed at ebf0a16 (each verified by mutating the fix and observing the suite go red, not by taking the claim on trust).

Verification performed

go build ./table/... ./catalog/... (OK); go vet ./table/ (OK); go test ./table/ -run 'Partition|Preflight|Metadata|Spec' -count=1 (ok); go test ./table/ -count=1 -race (ok, 19.7s); go test ./catalog/sql/ -count=1 (ok); mutation runs M1 (early-return guard reverted -> TestParseMetadataBytesNormalizesStaleLastPartitionID FAIL, correctly pinned) and M2 (persist-site variable reverted -> full table package still ok, gap); throwaway probes table/pr1988_probe_test.go, pr1988_div_test.go, pr1988_repro_test.go all deleted; `git status --porcelain` empty.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Iceberg Go handles maintainer review: CONTRIBUTING.md.

[zeroshade]

minor — Floor/persist separation at the write site survives mutation of the whole table package

metadata.go:2081 persists normalizedLastPartitionID rather than lastAssignedID — the headline guarantee of commit ebf0a16. Substituting lastAssignedID there leaves every test in ./table passing. The two values only diverge at this line when no field IDs are missing and the persisted counter plus all partition field IDs are below 999, which is reachable with legacy v1 metadata whose partition field IDs predate the 1000 floor. All three added test cases hit the early return at metadata.go:2050 and never reach line 2081, so the separation is asserted in only one of its two directions. Behavior at head is correct; this is a coverage gap, not a live bug. Adding one case with a sub-1000 partition field ID and a lower counter (expect the greatest field ID, not 999) would pin it.

[zeroshade]

nit — End-to-end allocation half of the issue's suggested regression coverage is not asserted

Issue #1987 asks for coverage that asserts both that the parsed counter becomes 1000 and that the next distinct partition field receives 1001. TestParseMetadataBytesNormalizesStaleLastPartitionID asserts only the former. I verified the latter holds today, so this is purely about locking in the user-visible symptom (the cross-spec ID collision) rather than only its parse-level cause.

[mattfaltyn]

Adding one case with a sub-1000 partition field ID and a lower counter (expect the greatest field ID, not 999) would pin it.

Thanks for the precise mutation-based recommendation. I added TestAssignMissingPartitionFieldIDsNormalizesLegacyStaleCounter and confirmed it fails with actual 999 when the write site uses the allocation cursor.

Issue #1987 asks for coverage that asserts both that the parsed counter becomes 1000 and that the next distinct partition field receives 1001.

Great call. I extended TestParseMetadataBytesNormalizesStaleLastPartitionID with a bucket16 assertion for field ID 1001.

Both updates are in e01dbb2. The full test suite and golangci-lint pass. Thank you for the thorough review!

[zeroshade]

Test-only delta closes the last open finding: mutating the floor/persist separation at metadata.go:2081 now turns TestAssignMissingPartitionFieldIDsNormalizesLegacyStaleCounter RED (expected 9, actual 999), and the end-to-end AddField assertion demonstrates the 1000-collision.

Re-review verification: 3 of 4 prior findings confirmed fixed at e01dbb2 (each verified by mutating the fix and observing the suite go red, not by taking the claim on trust).

Verification performed

go build ./... (clean); go vet ./table/... (clean); go test ./table/... -run 'Partition|LastPartitionID|StaleCounter|StaleLastPartition' -timeout=180s -count=1 (all ok); go test ./table/ -race -timeout=600s -count=1 (ok, 20.234s); 7 in-place mutations of table/metadata.go (write site, early-return guard, field-loop tracker, post-assignment bump, re-marshal guard, builder prev floor, BindToSchema arg) each restored via git checkout; throwaway probe table/pr1988_probe_test.go written, run, and deleted. Final 'git status --porcelain' empty.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer. The maintainer approving this PR has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Iceberg Go handles maintainer review: CONTRIBUTING.md.

[zeroshade]

nit — Unknown-spec-key fidelity guard is itself unpinned by tests

The 'if len(missingFields) > 0' guard prevents the stale-counter-only path from round-tripping specs through rawPartitionSpec, which would drop spec-level keys the struct cannot represent. Replacing the condition with 'true' leaves the entire table package green, so the preservation behaviour has no regression test. TestAssignMissingPartitionFieldIDsNormalizesLegacyStaleCounter passes through this path but only asserts last-partition-id, never the spec bytes. Non-blocking; the guard is an improvement over pre-PR behaviour either way.

[laskoviymishka]

The core mechanic here is in good shape, and the earlier low-counter concern is genuinely addressed: the 999 allocation floor is now cleanly separated from the persisted counter, and sub-999 metadata round-trips byte-for-byte. Nice, tight change.

The one thing I'd want resolved before merge is the direction this PR actually targets, where the persisted counter is genuinely below an assigned field id and we bump it (999 to 1000) at read time. That normalized value flows straight into AssertLastAssignedPartitionID via BuildUpdates (update_spec.go:190), so against a REST/Java catalog that still stores 999 we'd send an assertion of 1000 and take a false 409. Because every reload re-normalizes the same bytes, it doesn't clear on retry, so the exact stale-counter table we're trying to help ends up permanently un-committable on a REST catalog, just failing a new way.

I left an inline with the shape I'd suggest: keep the persisted counter untouched at parse time and compute the allocation floor on the write side (in AddPartitionSpec) as max(lastPartitionID, max field id across specs). That's what Java does, it still gives the new field 1001 with no collision, and it lets AssertLastAssignedPartitionID(999) match the catalog so the server self-heals on commit. The current in-process test passes because it validates the assertion against the already-normalized metadata rather than the original 999, so it won't catch this.

Rest is small: a dead max that reads like a guard, plus a couple of test-hardening bits. Once the assertion path is sorted I'm happy to take another pass.

[laskoviymishka]

I think doing the repair at read time is the wrong layer here. Once we rewrite last-partition-id from 999 to 1000, Metadata().LastPartitionSpecID() returns 1000, and UpdateSpec.BuildUpdates (update_spec.go:190) feeds that straight into AssertLastAssignedPartitionID. Against a REST/Java catalog that still persists 999, that assertion is a guaranteed false 409, and because every reload re-normalizes the same bytes, it never clears. So for the exact stale-counter table this targets, spec evolution goes from silently allocating a colliding id to permanently un-committable on REST.

Java only recomputes last-partition-id from specs when the field is absent; present-but-stale is used verbatim, and the allocation floor is applied at evolution time, not at parse. I'd move the max-field-id scan to the write side: compute the floor as max(lastPartitionID, max field id across b.specs) in MetadataBuilder.AddPartitionSpec before binding, and leave the persisted counter untouched until a new spec actually builds. That keeps AssertLastAssignedPartitionID(999) matching the catalog, still gives the new field 1001 with no collision, and lets the server self-heal to 1001 on commit. wdyt?

[mattfaltyn]

move the max-field-id scan to the write side

Thanks—implemented in c9bae41 with one shared history-aware allocation floor.

[laskoviymishka]

This is the piece that gives false confidence on the case above. BuildUpdates emits AssertLastAssignedPartitionID from parsed, and we then validate it against that same normalized metadata (1000 == 1000), so it passes locally, but it never compares the emitted requirement against the original persisted 999, which is what a REST catalog checks. If we assert that update's AssertLastAssignedPartitionID requirement equals 999 here, this test goes red and surfaces the desync.

[mattfaltyn]

assert that update requirement equals 999

Great catch—covered, and the new field still receives 1001.

[laskoviymishka]

Small thing: this max is always lastAssignedID. Both cursors start from the same value and after the lastAssignedID++ above it's strictly greater, so the max can never pick normalizedLastPartitionID. It reads like it's guarding a case that can't happen. I'd just write normalizedLastPartitionID = lastAssignedID. Same idea for the field-scan update up at line 2045: it's dead when lastPartitionIDSet is false, so it could be gated on that. Not blocking.

[mattfaltyn]

this max is always lastAssignedID

Thanks—removed along with the parser-side normalization path.

[laskoviymishka]

This only exercises the fix if the Replace actually lands. If the fixture spacing ever changes and the replace silently no-ops, last-partition-id stays 1000 and both assertions pass without touching the fix. I'd add a require.Contains right after, asserting the "last-partition-id": 999 substring is present in data, so a missed replace fails loudly.

[mattfaltyn]

add a require.Contains

Good call—added explicit fixture guards so silent replacement failures go red.

[laskoviymishka]

While we're here, every stale-counter case uses a single spec in list form. Could we add a two-spec case where the stale counter sits below a field id in the second spec, and one using the V1 partition-spec key? The scan crosses all specs and the re-marshal path differs for the non-list form, so those are the two branches currently unexercised. Low priority.

[mattfaltyn]

add a two-spec case and one using the V1 partition-spec key

Thanks—the two-spec history case is covered; the existing V1 missing-ID test continues to cover its re-marshal path.

[mattfaltyn]

Thanks for the sharp review. Since e01dbb2, commit c9bae41:

• removes read-time counter normalization so AssertLastAssignedPartitionID keeps the catalog value (999)
• adds one shared history-aware allocation floor for UpdateSpec and MetadataBuilder.AddPartitionSpec, so the next distinct field receives 1001
• replaces the parser-normalization test with a guarded two-spec regression that checks both the 999 requirement and 1001 allocation
• adds direct builder coverage and removes the obsolete parser-only normalization cases and dead max logic
• updates the PR title, description, and API comment to reflect write-time repair

make test, targeted race coverage, assert tests, vet, lint, and diff checks are clean. Each inline review thread is also acknowledged with a concise quoted reply.