[default values] Add NestedField column-default APIs and Expressions.lit() (~upstream #9502)

[shanthoosh]

What

A faithful re-baseline of upstream apache/iceberg:main's column default-value API onto openhouse-1.2.0 — the API (Types.NestedField defaults, Expressions.lit()) plus SchemaParser serialization, so defaults are format-compatible end to end. Adapted only where 1.2 genuinely lacks the feature (v3-only bits are dropped); everything else lands as a copy from main. Not a git cherry-pick.

Types.NestedField

• Store defaults as Literal<?> (per main / #12211).
• Builder: withInitialDefault(Literal) / withWriteDefault(Literal) (+ deprecated Object overloads), initialDefault() / writeDefault() value accessors, initialDefaultLiteral() / writeDefaultLiteral(), and main's builder() entry points.
• castDefault surfaces a clear Cannot cast default value error instead of NPE-ing on an invalid cast.
• Dropped the v3-only UnknownType check.

SchemaParser

• Backport main's toJson/fromJson (per #11832) so initial-default / write-default round-trip through the schema JSON.
• Inert until a producer sets a default — schemas with no defaults serialize exactly as before.

Scope

• In scope: the column-default API + SchemaParser serialization.
• Out of scope: UpdateSchema, Avro, and reader wiring (the v3-only bits don't exist in 1.2).
• v2 on-disk compatibility is preserved at the catalog server, which strips the extra schema attributes before persisting metadata to disk.

Testing

• core TestSchemaParser — main's canonical testDocStrings + testPrimitiveTypeDefaultValues serialize→parse round-trip (the assertion that proves format compatibility), minus the two v3-only TimestampNano rows. 14 tests, 0 failures.
• Full :iceberg-api:test and schema-related :iceberg-core tests pass; :iceberg-api:revapi and spotlessCheck pass.

revapi

Keeps the NestedField serialization-id accepted break (still required, adding fields to a Serializable class). No unrelated churn.

🤖 Generated with Claude Code

[cbb330]

thanks for the change!

• This tracks upstream's #9502 as it merged (Oct 2024), not current apache/iceberg:main. Since then main moved default storage Object → Literal<?> (#12211) and wired defaults through SchemaParser (#11832). Let's re-baseline what this feature needs onto main so it stays a faithful cherry-pick and is format-compatible end to end — e.g. NestedField should take main's Literal<?>.

• SchemaParser is untouched here. I've shifted the requirements a bit on the client compatibility layer (sorry!) — let's also backport SchemaParser from main so we get the toJson/fromJson that persist initial-default. This is safe: the plan is for the OH server to parse initial-default back OUT of the schema before persisting to disk (that's what preserves v2 compatibility). It simplifies your design and the client, at the cost of some scope on the server — which we're fine with, since server logic is easier to iterate on.

• Update the PR description to reflect the new v2-compat requirement. Drop the "By design: defaults are NOT serialized — SchemaParser is untouched" line — that's the requirement we're changing. Reframe as: a faithful re-baseline of upstream/main's column-default API + SchemaParser serialization, inert until a producer sets a default, with no UpdateSchema/Avro/reader scope. Later, the catalog server strips the extra schema attributes before persisting to disk.

• For castDefault, prefer main's shape — it surfaces a clear error on an invalid cast instead of NPE-ing. Minor, and it comes for free with the copy.

• Tests: lean on main's canonical tests and drop our bespoke ones. Make sure a serialize→parse round-trip is among them — that's the assertion that actually proves format compatibility. Details in [0].

• revapi: keep the serialization-id accepted-break (still needed), but please revert the unrelated BaseMetadataTable::table() reordering — auto-accept churn, not part of this change.

General idea (which you've mostly followed): adapt only where 1.2 genuinely lacks the feature (the v3-only bits don't exist here); everything else should land as a copy from main.

[0] tests:

TestSchemaParser.testPrimitiveTypeDefaultValues — core/src/test/java/org/apache/iceberg/TestSchemaParser.java:139-157,
with its @MethodSource provider primitiveTypesAndDefaults at :108-137.

Exactly the proof we want — builds a schema with withInitialDefault/withWriteDefault, runs
SchemaParser.fromJson(SchemaParser.toJson(schema)), and asserts both survive:

  Schema serialized = SchemaParser.fromJson(SchemaParser.toJson(schema));
  assertThat(serialized.findField("col_with_default").initialDefault()).isEqualTo(defaultValue.value());
  assertThat(serialized.findField("col_with_default").writeDefault()).isEqualTo(defaultValue.value());

Copy both methods. When backporting to 1.2, drop the two v3-only rows from primitiveTypesAndDefaults — they use
types/APIs that don't exist in 1.2 (same bits you're dropping in the impl):
  - :120-123  Types.TimestampNanoType.withZone()    + Expressions.nanos(...)
  - :127-129  Types.TimestampNanoType.withoutZone() + Expressions.nanos(...)

The remaining 12 rows (boolean, int, long, float, double, date, timestamp tz/no-tz, string, uuid, fixed, binary,
decimal) all exist in 1.2 and cover the type matrix that matters. Imports: org.apache.iceberg.expressions.Literal,
plus already-present DateTimeUtil, ByteBuffer, BigDecimal, UUID.

Also worth grabbing: testDocStrings (:92-106) exercises the new NestedField builder path through SchemaParser —
cheap extra coverage of the builder you're re-baselining, no v3 deps.

[cbb330]

great work, looks like the needed changes (and no more) are faithfully cherry picked from master.

rollout is safe because no exiting client utilizes initial-default in any capacity, and in a future PR in openhouse catalog (direct consumer of this) we will strip the initial-default for v2 persisted tables.