Skip to content

Commit d874d87

Browse files
committed
ENH: Add schema migration support for versioned resources
1 parent 07ebe14 commit d874d87

17 files changed

Lines changed: 1524 additions & 20 deletions

File tree

ARCHITECTURE.md

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,10 @@ classDiagram
6565
6666
class UserFMUDirectory
6767
68-
class CacheManager
68+
class CacheManager {
69+
+get_revision_content(..., migration_manager)
70+
+restore_revision(..., migration_manager)
71+
}
6972
class LockManager {
7073
+acquire()
7174
+ensure_can_write()
@@ -75,12 +78,20 @@ classDiagram
7578
}
7679
7780
class PydanticResourceManager~PydanticResource~ {
81+
+migration_manager
7882
+load(force, store_cache)
7983
+save(model)
8084
+get_resource_diff(incoming_resource)
8185
+get_structured_model_diff(current_model, incoming_model)
8286
}
8387
88+
class MigrationManager~MigratableResource~ {
89+
+model_class
90+
+current_version: int
91+
+migrate_resource(data)
92+
+requires_backup(data)
93+
}
94+
8495
class MutablePydanticResourceManager~MutablePydanticResource~ {
8596
+get(key, default)
8697
+set(key, value)
@@ -129,6 +140,8 @@ classDiagram
129140
130141
FMUDirectoryBase *-- LockManager
131142
FMUDirectoryBase *-- CacheManager
143+
PydanticResourceManager o-- MigrationManager
144+
CacheManager ..> MigrationManager
132145
ProjectFMUDirectory *-- ProjectConfigManager
133146
ProjectFMUDirectory *-- ChangelogManager
134147
ProjectFMUDirectory *-- MappingsManager
@@ -140,10 +153,44 @@ The main library split is:
140153
- `FMUDirectoryBase` is the filesystem-centered abstraction. It owns the `.fmu` path, lock manager, cache manager, and generic read/write helpers.
141154
- `ProjectFMUDirectory` and `UserFMUDirectory` specialize the base class for project-local `.fmu/` directories and `$HOME/.fmu/`.
142155
- `PydanticResourceManager` is the generic resource engine for loading, saving, diffing, and caching JSON-backed Pydantic models.
156+
- `MigrationManager` applies registered, forward-only schema migrations and validates the result as the current Pydantic model.
143157
- `MutablePydanticResourceManager` adds dot-notation `get`, `set`, `update`, `reset`, and merge behavior for editable resources.
144158
- `ProjectConfigManager`, `UserConfigManager`, `MappingsManager`, and `LogManager` bind specific Pydantic models to managed files inside `.fmu/`.
145159
- Directory objects compose the correct managers and delegate resource operations to them.
146160

161+
## Schema Migration
162+
163+
`ProjectConfig`, `UserConfig`, and `InternalMappings` are versioned resources. Each
164+
model declares its current schema version, and each resource manager has a
165+
`MigrationManager` with the migration registry for that model.
166+
167+
Migration is forward-only. Data without `schema_version` is treated as version 1.
168+
Each registered function advances the data by one version. The manager rejects a
169+
newer schema, a missing migration step, an invalid version, or data that does not
170+
validate as the current model.
171+
172+
The migration boundary depends on the resource operation:
173+
174+
- **Load:** `PydanticResourceManager` decodes the stored JSON and asks
175+
`MigrationManager` for a validated current model. Migration happens in memory.
176+
Loading does not write the resource, create a cache revision, or add a changelog
177+
entry.
178+
- **Save:** The resource manager checks the write lock first. If the stored data is
179+
unversioned or older, it stores the original JSON as a normal cache revision
180+
before writing the current model. Normal cache retention applies.
181+
- **Cache read:** `CacheManager` uses the migration manager to return old revisions
182+
as current validated models.
183+
- **Restore:** `CacheManager` migrates the selected revision before writing it, so
184+
the restored resource uses the current schema. Project restore operations use the
185+
existing restore changelog entry.
186+
187+
There is no backward migration. After a current-schema resource is saved, an older
188+
`fmu-settings` release can reject it as newer than supported.
189+
190+
See the
191+
[schema migration guide](src/fmu/settings/_migrations/README.md)
192+
for the implementation, test, and coordinated release process.
193+
147194
## Runtime Flow
148195

149196
The full runtime spans multiple repositories, but `fmu-settings` owns the `.fmu/` directory operations used by the API and CLI.

CONTRIBUTING.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,8 @@ ruff check
3232
ruff format --check
3333
mypy src tests
3434
```
35+
36+
If you need to change the schema of `ProjectConfig`, `UserConfig`, or
37+
`InternalMappings`, see the
38+
[schema migration guide](src/fmu/settings/_migrations/README.md) for implementation
39+
and testing details.

README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,11 @@ ruff format --check
8181
mypy src tests
8282
```
8383

84+
If you need to change the schema of `ProjectConfig`, `UserConfig`, or
85+
`InternalMappings`, see the
86+
[schema migration guide](src/fmu/settings/_migrations/README.md) for implementation
87+
and testing details.
88+
8489
See [CONTRIBUTING.md](CONTRIBUTING.md) for more.
8590

8691
> [!NOTE]

src/fmu/settings/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
init_fmu_directory,
2222
init_user_fmu_directory,
2323
)
24+
from ._migrations import MigrationError
2425
from .models._enums import CacheResource
2526
from .models.mappings import (
2627
InternalBaseMapping,
@@ -47,6 +48,7 @@
4748
"InternalWellboreIdentifierMapping",
4849
"InternalWellboreMappings",
4950
"InvalidFMUProjectPathError",
51+
"MigrationError",
5052
"ProjectFMUDirectory",
5153
"REQUIRED_FMU_PROJECT_SUBDIRS",
5254
"UserFMUDirectory",

src/fmu/settings/_fmu_dir.py

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@
2222
from .models.project_config import ProjectConfig
2323
from .models.user_config import UserConfig
2424

25+
if TYPE_CHECKING:
26+
from ._migrations import MigrationManager
27+
2528
logger: Final = null_logger(__name__)
2629

2730
FMUConfigManager: TypeAlias = ProjectConfigManager | UserConfigManager
@@ -470,13 +473,19 @@ def restore_from_cache(
470473
if manager is self.config:
471474
previous_max_revisions = self._cache_manager.max_revisions
472475
restored_config = self.cache.get_revision_content(
473-
relative_path, revision_id, model_class=ProjectConfig
476+
relative_path,
477+
revision_id,
478+
model_class=ProjectConfig,
479+
migration_manager=self.config.migration_manager,
474480
)
475481
self._cache_manager.max_revisions = restored_config.cache_max_revisions
476482

477483
try:
478484
self.cache.restore_revision(
479-
relative_path, revision_id, model_class=ProjectConfig
485+
relative_path,
486+
revision_id,
487+
model_class=ProjectConfig,
488+
migration_manager=self.config.migration_manager,
480489
)
481490
except Exception:
482491
# Restore the previous runtime retention if config restore fails
@@ -495,7 +504,13 @@ def restore_from_cache(
495504
return
496505

497506
self.cache.restore_revision(
498-
relative_path, revision_id, model_class=manager.model_class
507+
relative_path,
508+
revision_id,
509+
model_class=cast("type[BaseModel]", manager.model_class),
510+
migration_manager=cast(
511+
"MigrationManager[BaseModel] | None",
512+
manager.migration_manager,
513+
),
499514
)
500515

501516
# Refresh the resource manager's in-memory cache
@@ -531,7 +546,13 @@ def get_cache_content(
531546
)
532547

533548
return self.cache.get_revision_content(
534-
relative_path, revision_id, model_class=manager.model_class
549+
relative_path,
550+
revision_id,
551+
model_class=cast("type[BaseModel]", manager.model_class),
552+
migration_manager=cast(
553+
"MigrationManager[BaseModel] | None",
554+
manager.migration_manager,
555+
),
535556
)
536557

537558
def _cacheable_resource_managers(

0 commit comments

Comments
 (0)