fix(planner): emit executable DDL for NOT NULL columns on non-empty tables (#241)

closes
[TML-2067](https://linear.app/prisma-company/issue/TML-2067/planner-emits-unexecutable-ddl-for-not-null-columns-on-non-empty)

## Intent

Fix Postgres migration planning and execution for `ADD COLUMN ... NOT
NULL` on existing tables during `db update`.

Before this branch, two things were wrong:

- the planner could only safely add a NOT NULL column without an
explicit default by requiring the table to be empty
- the runner could skip `db update` work when the marker already matched
the destination, even if the live schema had drifted

This PR fixes that bug, extracts the first reusable multi-step planner
recipe, and adds an extension-aware seam for temporary-default
resolution. It does **not** try to ship a public TypeScript migration
API yet.

## What Changed

### Planner

- `buildAddColumnOperation(...)` now decides **when** the
temporary-default strategy applies.
- The actual 2-step recipe is extracted into
`buildAddNotNullColumnWithTemporaryDefaultOperation(...)`.
- Temporary-default lookup now flows through
`resolveTemporaryDefaultLiteral(...)`:
  1. ask a codec hook
  2. fall back to built-in Postgres defaults
  3. if neither can provide a safe value, keep the empty-table precheck
- `CodecControlHooks` now exposes `resolveTemporaryDefaultLiteral(...)`
so extensions and parameterized codecs can participate in this decision.
- `pgvector` implements that hook and returns a dimension-aware zero
vector literal.
- The built-in Postgres fallback now covers more safe cases, including
arrays (`'{}'`), `tsvector` (`''::tsvector`), `bytea` (`''::bytea`),
alias forms like `varchar` / `bpchar` / `varbit`, and length-aware zero
literals for `bit(n)`.

### Runner

- `db update` plans (`origin: null`) now always apply operations.
- `migration apply` plans (`origin` set) still skip when the marker
already matches the destination.

That preserves migration replay idempotency while allowing `db update`
to recover from live drift.

### Tests and Evidence

- `planner.behavior.test.ts` now verifies:
  - the extracted 2-step recipe for built-in types
  - extension-aware temporary defaults via `pgvector`
- fallback-to-empty-table behavior when a codec explicitly declines a
temporary default
- expanded built-in fallback coverage for arrays, `tsvector`, `bytea`,
and `bit(n)`
- `drift-schema.e2e.test.ts` now exercises the built-in recovery path:
  - initialize the schema
  - insert a row
  - manually drop a NOT NULL column
  - run `db update`
- verify the row was backfilled and that the temporary default was
removed
- `psl.pgvector-dbinit.test.ts` now exercises the extension path against
a live database:
  - initialize a schema with a `vector(3)` NOT NULL column
  - insert a row
  - manually drop the vector column
  - run `db update`
- verify the row was backfilled to `[0,0,0]` and that no default remains
on the column
- `runner.idempotency.integration.test.ts` now explicitly models
`migration apply` semantics by setting `origin`, so it still verifies
skip-on-marker-match in the right path.

### Supporting Changes

- Migration subsystem docs now describe the temporary-default strategy
as:
  - codec hook first
  - built-in fallback second
  - empty-table precheck only when no safe default is known

## Behavior Change

For a NOT NULL column without an explicit contract default:

- if the planner can resolve a safe temporary default, it emits:
  1. `ADD COLUMN ... DEFAULT <temporary> NOT NULL`
  2. `ALTER COLUMN ... DROP DEFAULT`
- existing rows keep the backfilled value
- future inserts must still provide an explicit value
- if no safe temporary default is known, the operation still requires an
empty table

## Scope and Deferrals

This PR intentionally stops at the internal planner boundary.

Deferred:

- public composable migration utilities for user-authored TypeScript
migrations
- a strategy registry or broader cross-target abstraction layer
- broader work on user-authored TS contracts / TS migrations

Those are follow-up design and implementation steps. This PR is the bug
fix plus the smallest useful extensibility seam.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* `db update` live reconciliation for recovering schema drift on
non-empty tables.
* Planner now supports a safe temporary-default strategy to add NOT NULL
columns without requiring empty tables.
* Extensions (e.g., vector) can provide temporary-defaults for
reconciliation.

* **Bug Fixes**
* Improved idempotency: migration-apply plans are skipped when already
at destination.

* **Documentation**
* Added architecture docs for the `db update` reconciliation flow and
planner behavior.

* **Tests**
* Expanded tests for NOT NULL migrations, pgvector recovery,
idempotency, and CLI error reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: jkomyno <12381818+jkomyno@users.noreply.github.com>

Author: jkomyno

docs/architecture docs/subsystems/7. Migration System.md

@@ -298,6 +298,16 @@ For CLI usage, output formats, and common failure modes, see the canonical comma
 
 Greenfield, brownfield-conservative, and brownfield-incremental paths are supported. Baselines help bootstrap fresh environments; incremental adoption expands contract coverage safely over time. See [ADR 122 — Database Initialization & Adoption](../adrs/ADR%20122%20-%20Database%20Initialization%20&%20Adoption.md).
 
+### `db update` (live reconciliation)
+
+`prisma-next db update` is the **reconciliation** entrypoint. It introspects the live database schema, diffs it against the destination contract, and applies the resulting operations.
+
+Unlike `migration apply`, which replays serialized edges, `db update` always works from a live introspection. Plans produced by the planner carry `origin: null` (no prior-state assertion). The runner treats this as "trust the planner": operations are always applied regardless of marker state. This ensures `db update` can recover from schema drift — for example, when a DBA manually drops a column, the planner detects the missing column and re-adds it even though the contract marker already matches the destination hash.
+
+By contrast, `migration apply` constructs plans with `origin: { storageHash }` from the migration chain. When the marker matches the destination, the runner skips operations (the migration was already applied). This provides safe idempotent replays.
+
+**NOT NULL columns without defaults.** When a direct `ADD COLUMN ... NOT NULL` would fail on existing rows, the planner chooses a safe reconciliation strategy. It first asks codec hooks for a type-specific identity literal, then falls back to built-in Postgres handling, and otherwise requires an empty-table precheck. The exact SQL recipe is an implementation detail of the planner and may evolve as additional strategies are added.
+
 ## Multi-Service Namespacing
 
 Per-service markers allow independent contracts within a single physical database (e.g., Postgres schemas). Each service has its own `prisma_contract.marker`; permissions and verification remain isolated. For engines without schemas, use separate databases. See namespacing notes in [ADR 021 — Contract Marker Storage](../adrs/ADR%20021%20-%20Contract%20Marker%20Storage.md).
@@ -333,5 +343,3 @@ Provide:
 - edges/<edgeId>/migration.json — manifests with ops and checks; optional tasks modules
 - graph.index.json (optional) — committed graph index when enabled
 - digests — content hashes for integrity
-
-

packages/2-sql/3-tooling/family/src/core/migrations/types.ts

@@ -164,6 +164,19 @@ export interface ExpandNativeTypeInput {
   readonly typeParams?: Record<string, unknown>;
 }
 
+/**
+ * Input for resolving an identity-value SQL literal used to backfill existing rows when
+ * adding a NOT NULL column without an explicit default.
+ *
+ * "Identity value" in the algebraic (monoid) sense: the neutral element for the type
+ * (0 for numbers, '' for strings, false for booleans, etc.).
+ */
+export interface ResolveIdentityValueInput {
+  readonly nativeType: string;
+  readonly codecId?: string;
+  readonly typeParams?: Record<string, unknown>;
+}
+
 export interface CodecControlHooks<TTargetDetails = unknown> {
   planTypeOperations?: (options: {
     readonly typeName: string;
@@ -194,6 +207,16 @@ export interface CodecControlHooks<TTargetDetails = unknown> {
    * Returns the expanded type string, or the original nativeType if no expansion is needed.
    */
   expandNativeType?: (input: ExpandNativeTypeInput) => string;
+  /**
+   * Resolves the identity value (monoid neutral element) as a SQL literal for safely adding
+   * a NOT NULL column without an explicit default to a non-empty table.
+   *
+   * Return semantics:
+   * - string: use this literal
+   * - null: explicitly no safe identity value is known; fall back to another strategy
+   * - undefined: no opinion; planner may use built-in fallbacks
+   */
+  resolveIdentityValue?: (input: ResolveIdentityValueInput) => string | null | undefined;
 }
 
 export interface SqlControlExtensionDescriptor<TTargetId extends string>

packages/2-sql/3-tooling/family/src/exports/control.ts

@@ -47,6 +47,7 @@ export type {
   CreateSqlMigrationPlanOptions,
   ExpandNativeTypeInput,
   PslScalarTypeDescriptor,
+  ResolveIdentityValueInput,
   SqlControlAdapterDescriptor,
   SqlControlExtensionDescriptor,
   SqlControlStaticContributions,

packages/3-extensions/pgvector/src/exports/control.ts

@@ -7,6 +7,16 @@ import { pgvectorOperationSignature, pgvectorPackMeta } from '../core/descriptor
 
 const PGVECTOR_CODEC_ID = 'pg/vector@1' as const;
 
+function buildVectorIdentityValue(typeParams: Record<string, unknown> | undefined): string | null {
+  const length = typeParams?.['length'];
+  if (typeof length !== 'number' || !Number.isInteger(length) || length <= 0) {
+    return null;
+  }
+
+  const zeroVector = `[${new Array(length).fill('0').join(',')}]`;
+  return `'${zeroVector}'::vector`;
+}
+
 const vectorControlPlaneHooks: CodecControlHooks = {
   expandNativeType: ({ nativeType, typeParams }) => {
     const length = typeParams?.['length'];
@@ -15,6 +25,7 @@ const vectorControlPlaneHooks: CodecControlHooks = {
     }
     return nativeType;
   },
+  resolveIdentityValue: ({ typeParams }) => buildVectorIdentityValue(typeParams),
 };
 
 const pgvectorDatabaseDependencies: ComponentDatabaseDependencies<unknown> = {

packages/3-targets/3-targets/postgres/src/core/migrations/planner-identity-values.ts

@@ -0,0 +1,129 @@
+import type { CodecControlHooks } from '@prisma-next/family-sql/control';
+import type { StorageColumn } from '@prisma-next/sql-contract/types';
+import { ifDefined } from '@prisma-next/utils/defined';
+
+/**
+ * Resolves the identity value (monoid neutral element) as a SQL literal for a column's type.
+ * Checks codec hooks first (extensions can provide type-specific identity values),
+ * then falls back to the built-in map.
+ */
+export function resolveIdentityValue(
+  column: StorageColumn,
+  codecHooks: Map<string, CodecControlHooks>,
+): string | null {
+  if (column.codecId) {
+    const hookDefault = codecHooks.get(column.codecId)?.resolveIdentityValue?.({
+      nativeType: column.nativeType,
+      codecId: column.codecId,
+      ...ifDefined('typeParams', column.typeParams),
+    });
+    if (hookDefault !== undefined) {
+      return hookDefault;
+    }
+  }
+
+  return buildBuiltinIdentityValue(column.nativeType, column.typeParams);
+}
+
+/**
+ * Returns the built-in identity value (monoid neutral element) as a SQL literal for the given
+ * PostgreSQL native type — e.g. 0 for integers, '' for text, false for booleans.
+ *
+ * This is the planner's fallback when no codec hook provides a type-specific identity value.
+ *
+ * Returns null for unrecognized types (for example enums and extension-owned types without a
+ * hook), which causes the planner to fall back to the empty-table precheck.
+ *
+ * @internal Exported for testing only.
+ */
+export function buildBuiltinIdentityValue(
+  nativeType: string,
+  typeParams?: Record<string, unknown>,
+): string | null {
+  const normalizedNativeType = normalizeIdentityValueNativeType(nativeType);
+
+  if (normalizedNativeType.endsWith('[]')) {
+    return "'{}'";
+  }
+
+  switch (normalizedNativeType) {
+    case 'text':
+    case 'character':
+    case 'bpchar':
+    case 'character varying':
+    case 'varchar':
+      return "''";
+
+    case 'int2':
+    case 'int4':
+    case 'int8':
+    case 'integer':
+    case 'bigint':
+    case 'smallint':
+    case 'float4':
+    case 'float8':
+    case 'real':
+    case 'double precision':
+    case 'numeric':
+    case 'decimal':
+      return '0';
+
+    case 'bool':
+    case 'boolean':
+      return 'false';
+
+    case 'uuid':
+      return "'00000000-0000-0000-0000-000000000000'";
+
+    case 'json':
+      return "'{}'::json";
+    case 'jsonb':
+      return "'{}'::jsonb";
+
+    case 'date':
+    case 'timestamp':
+    case 'timestamptz':
+    case 'timestamp with time zone':
+    case 'timestamp without time zone':
+      return "'epoch'";
+
+    case 'time':
+    case 'time without time zone':
+      return "'00:00:00'";
+    case 'timetz':
+    case 'time with time zone':
+      return "'00:00:00+00'";
+
+    case 'interval':
+      return "'0'";
+
+    case 'bytea':
+      return "''::bytea";
+    case 'tsvector':
+      return "''::tsvector";
+
+    case 'bit':
+      return buildBitIdentityValue(typeParams);
+    case 'bit varying':
+    case 'varbit':
+      return "B''";
+
+    default:
+      return null;
+  }
+}
+
+function normalizeIdentityValueNativeType(nativeType: string): string {
+  return nativeType.trim().toLowerCase().replace(/\s+/g, ' ');
+}
+
+function buildBitIdentityValue(typeParams?: Record<string, unknown>): string | null {
+  const length = typeParams?.['length'];
+  if (length === undefined) {
+    return "B'0'";
+  }
+  if (typeof length !== 'number' || !Number.isInteger(length) || length <= 0) {
+    return null;
+  }
+  return `B'${'0'.repeat(length)}'`;
+}

packages/3-targets/3-targets/postgres/src/core/migrations/planner-recipes.ts

@@ -0,0 +1,83 @@
+import { quoteIdentifier } from '@prisma-next/adapter-postgres/control';
+import type { CodecControlHooks, SqlMigrationPlanOperation } from '@prisma-next/family-sql/control';
+import type { StorageColumn } from '@prisma-next/sql-contract/types';
+import type { PostgresPlanTargetDetails } from './planner';
+import {
+  buildAddColumnSql,
+  columnExistsCheck,
+  columnHasNoDefaultCheck,
+  columnNullabilityCheck,
+  qualifyTableName,
+} from './planner-sql';
+import { buildTargetDetails } from './planner-target-details';
+
+export function buildAddColumnOperationIdentity(
+  schema: string,
+  tableName: string,
+  columnName: string,
+): Pick<
+  SqlMigrationPlanOperation<PostgresPlanTargetDetails>,
+  'id' | 'label' | 'summary' | 'target'
+> {
+  return {
+    id: `column.${tableName}.${columnName}`,
+    label: `Add column ${columnName} to ${tableName}`,
+    summary: `Adds column ${columnName} to table ${tableName}`,
+    target: {
+      id: 'postgres',
+      details: buildTargetDetails('table', tableName, schema),
+    },
+  };
+}
+
+export function buildAddNotNullColumnWithTemporaryDefaultOperation(options: {
+  readonly schema: string;
+  readonly tableName: string;
+  readonly columnName: string;
+  readonly column: StorageColumn;
+  readonly codecHooks: Map<string, CodecControlHooks>;
+  readonly temporaryDefault: string;
+}): SqlMigrationPlanOperation<PostgresPlanTargetDetails> {
+  const { schema, tableName, columnName, column, codecHooks, temporaryDefault } = options;
+  const qualified = qualifyTableName(schema, tableName);
+
+  return {
+    ...buildAddColumnOperationIdentity(schema, tableName, columnName),
+    operationClass: 'additive',
+    precheck: [
+      {
+        description: `ensure column "${columnName}" is missing`,
+        sql: columnExistsCheck({ schema, table: tableName, column: columnName, exists: false }),
+      },
+    ],
+    execute: [
+      {
+        description: `add column "${columnName}"`,
+        sql: buildAddColumnSql(qualified, columnName, column, codecHooks, temporaryDefault),
+      },
+      {
+        description: `drop temporary default from column "${columnName}"`,
+        sql: `ALTER TABLE ${qualified} ALTER COLUMN ${quoteIdentifier(columnName)} DROP DEFAULT`,
+      },
+    ],
+    postcheck: [
+      {
+        description: `verify column "${columnName}" exists`,
+        sql: columnExistsCheck({ schema, table: tableName, column: columnName }),
+      },
+      {
+        description: `verify column "${columnName}" is NOT NULL`,
+        sql: columnNullabilityCheck({
+          schema,
+          table: tableName,
+          column: columnName,
+          nullable: false,
+        }),
+      },
+      {
+        description: `verify column "${columnName}" has no default after temporary default removal`,
+        sql: columnHasNoDefaultCheck({ schema, table: tableName, column: columnName }),
+      },
+    ],
+  };
+}

packages/3-targets/3-targets/postgres/src/core/migrations/planner-reconciliation.ts

@@ -11,13 +11,13 @@ import { ifDefined } from '@prisma-next/utils/defined';
 import type { PlanningMode, PostgresPlanTargetDetails } from './planner';
 import {
   buildColumnTypeSql,
-  buildTargetDetails,
   columnExistsCheck,
   columnNullabilityCheck,
   constraintExistsCheck,
   qualifyTableName,
   toRegclassLiteral,
-} from './planner';
+} from './planner-sql';
+import { buildTargetDetails } from './planner-target-details';
 
 // ============================================================================
 // Public API