schema-evolution.md

id	schema-evolution
title	Schema Evolution
sidebar_label	Schema Evolution

Schema evolution is a common challenge in distributed systems where data structures change over time. ZIO Blocks provides two type classes—Into and As—that enable type-safe, compile-time verified transformations between different versions of your data types.

:::tip[Looking for data migrations?] For runtime, serializable migrations that can be stored and applied dynamically, see the Schema Migration documentation. :::

Overview

When your application evolves, you often need to:

• Add new fields to existing types
• Remove deprecated fields
• Rename fields
• Change field types (e.g., Int → Long)
• Convert between different representations of the same concept

ZIO Blocks handles these transformations with:

Type Class	Direction	Use Case
Into[A, B]	One-way (A → B)	Migrations, API responses, data import
As[A, B]	Bidirectional (A ↔ B)	Round-trip serialization, data sync

Into[A, B] - One-Way Conversion

Into[A, B] represents a one-way conversion from type A to type B with validation:

trait Into[-A, +B] {
  def into(a: A): Either[SchemaError, B]
}

Basic Usage

import zio.blocks.schema.Into

// Version 1 of our API
case class PersonV1(name: String, age: Int)

// Version 2 adds email and changes age to Long
case class PersonV2(name: String, age: Long, email: Option[String])

// Derive the conversion automatically
val migrate: Into[PersonV1, PersonV2] = Into.derived[PersonV1, PersonV2]

// Use it
val v1 = PersonV1("Alice", 30)
val v2 = migrate.into(v1)
// Right(PersonV2("Alice", 30L, None))

Field Matching Rules

The macro matches fields using the following priority:

1. Exact match: Same name + same type
2. Name match with coercion: Same name + convertible type (e.g., Int → Long)
3. Unique type match: Type appears only once in both source and target
4. Position + type match: Positional correspondence with matching type

Handling Missing Fields

When the target has fields not present in the source:

case class Source(name: String)
case class Target(name: String, age: Int = 25, nickname: Option[String])

val convert = Into.derived[Source, Target]
convert.into(Source("Bob"))
// Right(Target("Bob", 25, None))
//                     ↑    ↑
//               default   Option defaults to None

• Default values: Used when target field has a default
• Option types: Default to None when not present in source

Numeric Conversions

Built-in support for numeric widening and narrowing:

// Widening (lossless) - always succeeds
Into[Byte, Short]   // Byte → Short
Into[Int, Long]     // Int → Long 
Into[Float, Double] // Float → Double

// Narrowing (with validation) - may fail at runtime
Into[Long, Int]     // Fails if value > Int.MaxValue or < Int.MinValue
Into[Double, Float] // Fails if value out of Float range

Example with validation failure:

case class BigNumbers(value: Long)
case class SmallNumbers(value: Int)

val narrow = Into.derived[BigNumbers, SmallNumbers]

narrow.into(BigNumbers(42L))
// Right(SmallNumbers(42))

narrow.into(BigNumbers(Long.MaxValue))
// Left(SchemaError: "Value 9223372036854775807 is out of range for Int")

Collection Conversions

Automatic conversion between collection types:

case class ListData(items: List[Int])
case class VectorData(items: Vector[Long])

Into.derived[ListData, VectorData]
// Converts List → Vector AND Int → Long

Supported conversions:

• List, Vector, Set, Seq (interchangeable)
• Array ↔ Iterable
• Map[K1, V1] → Map[K2, V2]
• Option[A] → Option[B]
• Either[L1, R1] → Either[L2, R2]

:::note Converting to Set may remove duplicates. Converting from Set does not preserve any particular ordering. :::

Sealed Trait / Enum Conversions

Convert between coproduct types (sealed traits, enums):

// Scala 2
sealed trait StatusV1
object StatusV1 {
  case object Active extends StatusV1
  case object Inactive extends StatusV1
}

sealed trait StatusV2
object StatusV2 {
  case object Active extends StatusV2
  case object Inactive extends StatusV2
  case object Pending extends StatusV2  // New case added
}

// Scala 3
enum StatusV1 { case Active, Inactive }
enum StatusV2 { case Active, Inactive, Pending }

// Works - all V1 cases exist in V2
Into.derived[StatusV1, StatusV2]

Cases are matched by:

1. Name: Case names must match
2. Signature: For case classes, field types must be convertible

Nested Type Conversions

For nested types, provide implicit Into instances:

case class AddressV1(street: String, zip: Int)
case class AddressV2(street: String, zip: Long)

case class PersonV1(name: String, address: AddressV1)
case class PersonV2(name: String, address: AddressV2)

// The macro automatically uses Into[AddressV1, AddressV2] for the nested field
val convert = Into.derived[PersonV1, PersonV2]

Error Accumulation

When multiple fields fail validation, all errors are accumulated:

case class Source(a: Long, b: Long, c: Long)
case class Target(a: Int, b: Int, c: Int)

val convert = Into.derived[Source, Target]
convert.into(Source(Long.MaxValue, Long.MinValue, 42L))
// Left(SchemaError containing errors for BOTH 'a' and 'b')

As[A, B] - Bidirectional Conversion

As[A, B] extends Into[A, B] with a reverse conversion:

trait As[A, B] extends Into[A, B] {
  def from(input: B): Either[SchemaError, A]
  def reverse: As[B, A]
}

Basic Usage

import zio.blocks.schema.As

case class Point2D(x: Int, y: Int)
case class Coordinate(x: Int, y: Int)

val convert: As[Point2D, Coordinate] = As.derived[Point2D, Coordinate]

// Both directions work
convert.into(Point2D(1, 2))      // Right(Coordinate(1, 2))
convert.from(Coordinate(3, 4))   // Right(Point2D(3, 4))

// Swap directions
val reversed: As[Coordinate, Point2D] = convert.reverse

Restrictions for As

As has stricter requirements than Into to guarantee round-trip safety:

❌ No Default Values on Non-Matching Fields

Default values break round-trip when a field with a default exists in one type but not in the other:

case class WithDefault(name: String, age: Int = 25)
case class NoDefault(name: String)

// This will NOT compile (age has default but doesn't exist in NoDefault):
As.derived[WithDefault, NoDefault]
// Error: "Cannot derive As[...]: Default values break round-trip guarantee"

// Use Into instead for one-way conversion:
Into.derived[NoDefault, WithDefault]  // ✓ Works

However, default values ARE allowed when the field exists in both types:

case class PersonA(name: String, age: Int = 25)
case class PersonB(name: String, age: Int)

// This WILL compile (age exists in both types):
As.derived[PersonA, PersonB]  // ✓ Works

✅ Option Fields Are Allowed

Option fields work because None round-trips correctly:

case class TypeA(name: String, nickname: Option[String])
case class TypeB(name: String)

As.derived[TypeA, TypeB]  // ✓ Works

✅ Numeric Coercions Must Be Invertible

Numeric types can be coerced if the conversion works in both directions:

case class IntVersion(value: Int)
case class LongVersion(value: Long)

As.derived[IntVersion, LongVersion]
// ✓ Works: Int → Long (widening) and Long → Int (narrowing with validation)

Using As Where Into Is Expected

Since As[A, B] extends Into[A, B], you can use it anywhere an Into is required:

def migrate[A, B](data: A)(implicit into: Into[A, B]): Either[SchemaError, B] =
  into.into(data)

implicit val as: As[Point2D, Coordinate] = As.derived

migrate(Point2D(1, 2))  // Uses As as an Into

ZIO Prelude Newtype Support

Both Into and As automatically detect and validate ZIO Prelude newtypes:

import zio.prelude._

object Age extends Subtype[Int] {
  override def assertion = assert(between(0, 150))
}
type Age = Age.Type

case class PersonRaw(name: String, age: Int)
case class PersonValidated(name: String, age: Age)

val validate = Into.derived[PersonRaw, PersonValidated]

validate.into(PersonRaw("Alice", 30))
// Right(PersonValidated("Alice", Age(30)))

validate.into(PersonRaw("Bob", -5))
// Left(SchemaError: "Validation failed for field 'age': ...")

The macro automatically:

1. Detects that Age is a ZIO Prelude newtype
2. Calls Age.make(value) for validation
3. Converts Validation result to Either[SchemaError, _]

Scala 3 Opaque Type Support

In Scala 3, opaque types with companion validation are supported:

opaque type Email = String
object Email {
  def apply(value: String): Either[String, Email] =
    if (value.contains("@")) Right(value)
    else Left(s"Invalid email: $value")
    
  def unsafe(value: String): Email = value
}

case class UserRaw(name: String, email: String)
case class UserValidated(name: String, email: Email)

Into.derived[UserRaw, UserValidated]
// Automatically uses Email.apply for validation

The macro looks for:

1. apply(value: Underlying): Either[_, OpaqueType] - validation method
2. unsafe(value: Underlying): OpaqueType - fallback without validation

Structural Type Support

ZIO Blocks supports conversions involving structural types on JVM only, as they require reflection.

Platform Compatibility Matrix

Conversion	JVM	JS	Native	Notes
Product → Structural	✅	❌	❌	JVM only (reflection)
Structural → Product	✅	❌	❌	JVM only (reflection)

Key insight: Structural types require runtime reflection to access their members, which is only available on JVM. On JS and Native platforms, structural type conversions will fail at compile time with a helpful error message.

Structural Types (JVM Only)

Structural types are types defined by their members rather than their name:

// JVM ONLY - will fail at compile time on JS/Native
case class Person(name: String, age: Int)

// Structural type to case class
val into = Into.derived[{ def name: String; def age: Int }, Person]

// Case class to structural type  
val toStructural = Into.derived[Person, { def name: String; def age: Int }]

Compile-time error on non-JVM platforms:

Cannot derive Into[..., Person]: Structural type conversions are not supported on JS.

Structural types require reflection APIs (getClass.getMethod) which are only available on JVM.

Consider:
  - Using a case class instead of a structural type
  - Using a tuple instead of a structural type
  - Only using structural type conversions in JVM-only code

Working with Structural Types

import scala.language.reflectiveCalls

// Create a structural type instance
def makePerson(n: String, a: Int): { def name: String; def age: Int } = new {
  def name: String = n
  def age: Int = a
}

case class Person(name: String, age: Int)

// Convert structural → case class
val into = Into.derived[{ def name: String; def age: Int }, Person]
val result = into.into(makePerson("Alice", 30))
// Right(Person("Alice", 30))

Scala 2 vs Scala 3 Differences

Feature	Scala 2	Scala 3
Derivation syntax	Into.derived[A, B]	Into.derived[A, B]
Enum support	Sealed traits only	Scala 3 enums + sealed traits
Opaque types	N/A	✅ Supported
Structural types	JVM only (reflection)	JVM only (reflection)
ZIO Prelude newtypes	✅ assert { ... } syntax	✅ override def assertion syntax
Error messages	Detailed macro errors	Detailed macro errors

ZIO Prelude Newtype Syntax

Scala 2:

object Age extends Subtype[Int] {
  override def assertion = assert {
    between(0, 150)
  }
}

Scala 3:

object Age extends Subtype[Int] {
  override def assertion: Assertion[Int] =
    zio.prelude.Assertion.between(0, 150)
}

Best Practices

1. Prefer As When Round-Trip Is Required

// For data sync, use As
val sync: As[LocalModel, RemoteModel] = As.derived

// For one-way migrations, use Into
val migrate: Into[OldFormat, NewFormat] = Into.derived

2. Use Option for Truly Optional Fields

// Good: Optional field with Option
case class V2(name: String, email: Option[String])

// Avoid: Default values break As derivation
case class V2(name: String, email: String = "")

3. Provide Explicit Instances for Complex Nested Types

// When nested types need custom logic
implicit val addressConvert: Into[AddressV1, AddressV2] = 
  Into.derived[AddressV1, AddressV2]

// Now this works automatically
val personConvert = Into.derived[PersonV1, PersonV2]

4. Structural Types Are JVM-Only

// JVM-only: Structural types require reflection
type PersonLike = { def name: String }

// For cross-platform code, use case classes instead
case class PersonLike(name: String)

Complete Example

Here's a complete schema evolution example:

import zio.blocks.schema._

// API v1
object V1 {
  case class Address(street: String, city: String)
  case class Person(name: String, age: Int, address: Address)
}

// API v2 - adds fields, changes types
object V2 {
  case class Address(street: String, city: String, country: String = "US")
  case class Person(
    name: String, 
    age: Long,  // Changed from Int
    address: Address,
    email: Option[String]  // New field
  )
}

// Define conversions
object Migrations {
  // Address: one-way (v2 has default for country)
  implicit val addressMigrate: Into[V1.Address, V2.Address] = 
    Into.derived[V1.Address, V2.Address]
  
  // Person: one-way (v2 has new optional field)
  implicit val personMigrate: Into[V1.Person, V2.Person] = 
    Into.derived[V1.Person, V2.Person]
}

// Usage
import Migrations._

val oldPerson = V1.Person("Alice", 30, V1.Address("123 Main St", "NYC"))
val newPerson = personMigrate.into(oldPerson)
// Right(V2.Person("Alice", 30L, V2.Address("123 Main St", "NYC", "US"), None))

Error Handling

All conversions return Either[SchemaError, B] for explicit error handling:

val result = migrate.into(oldData)

result match {
  case Right(newData) => 
    // Success - use newData
    
  case Left(error) => 
    // Handle validation/conversion failure
    println(s"Migration failed: ${error.message}")
}

SchemaError provides:

• Detailed error messages
• Field path information
• Error accumulation (multiple errors combined)