ENG: Preserve pre-migration cache revisions

Author: GibranAlfa

ARCHITECTURE.md

@@ -178,18 +178,22 @@ The migration boundary depends on the resource operation:
 - **Save:** The resource manager checks the write lock first. If the stored data is
   older, it attempts to store the exact original JSON under
   `.fmu/migration-backups/` before writing the current model. This backup is
-  best-effort: a file-system failure is logged and does not stop the save. The newly
-  written current data is still added to the cache.
+  best-effort: a file-system failure is logged and does not stop the save. The
+  original JSON is also stored as a cache revision. The newly written current
+  data is added as another cache revision.
 - **Cache read:** `CacheManager` uses the migration manager to return old revisions
   as current validated models.
 - **Restore:** `CacheManager` migrates the selected revision before writing it, so
-  the restored resource uses the current schema. Project restore operations use the
-  existing restore changelog entry.
+  the restored resource uses the current schema. If you roll back to an older
+  release while the pre-migration revision is still retained, restore that revision with
+  the older release to return the resource file to the older schema. Project restore
+  operations use the existing restore changelog entry.
 
 There is no backward migration. After a current-schema resource is saved, an older
-`fmu-settings` release can reject it as newer than supported. Migration backups are
-not loaded or restored by the library. Users must copy one back manually when they
-need to use the data with an older release.
+`fmu-settings` release can reject it as newer than supported. If the pre-migration
+cache revision is still retained, restore it with the older release. Migration
+backups are not loaded or restored by the library. Users must copy one back manually
+when the cache revision is no longer available.
 
 See the
 [schema migration guide](src/fmu/settings/_migrations/README.md)

src/fmu/settings/_fmu_dir.py

@@ -47,7 +47,7 @@ class FMUDirectoryBase:
     def __init__(
         self: Self,
         base_path: str | Path,
-        cache_revisions: int = CacheManager.MIN_REVISIONS,
+        cache_revisions: int = 10,
         *,
         lock_timeout_seconds: int = DEFAULT_LOCK_TIMEOUT,
     ) -> None:
@@ -56,7 +56,8 @@ def __init__(
         Args:
             base_path: The directory containing the .fmu directory or one of its parent
                 dirs
-            cache_revisions: Number of revisions to retain in the cache. Minimum is 5.
+            cache_revisions: Number of revisions to retain in the cache. Default is 10.
+                Minimum is 5.
             lock_timeout_seconds: Lock expiration time in seconds. Default 20 minutes.
 
         Raises:
@@ -334,20 +335,18 @@ def __init__(
         self.config = ProjectConfigManager(self)
         super().__init__(
             base_path,
-            CacheManager.MIN_REVISIONS,
+            10,
             lock_timeout_seconds=lock_timeout_seconds,
         )
         self._changelog = ChangelogManager(self)
         self._mappings = MappingsManager(self)
         try:
-            max_revisions = self.config.get(
-                "cache_max_revisions", CacheManager.MIN_REVISIONS
-            )
+            max_revisions = self.config.get("cache_max_revisions", 10)
             self._cache_manager.max_revisions = max_revisions
         except (FileNotFoundError, ValueError) as e:
             logger.warning(
                 f"Failed to load 'cache_max_revisions' from project config. "
-                f"Using default value '{CacheManager.MIN_REVISIONS}'. Error: {e}"
+                f"Using default value '10'. Error: {e}"
             )
 
     def update_validation_metadata(

src/fmu/settings/_migrations/README.md

@@ -2,16 +2,16 @@
 
 Use this guide when a stored `.fmu` resource needs a new schema version.
 
-The supported resources and their migration registries are:
+The supported resources and their migration function registries are:
 
 - `ProjectConfig`: `project_config/`
 - `UserConfig`: `user_config/`
 - `InternalMappings`: `mappings/`
 
 ## Decide whether to change the schema version
 
-Change the schema version when existing stored data needs a conversion before it
-can be used by the current model.
+Change the schema version when existing stored data needs a migration before it can
+be used by the current model.
 
 ### Changes that need a migration
 
@@ -40,11 +40,12 @@ old field name.
 Other changes that need a migration include:
 
 - Removing a stored field when its value must be moved or preserved elsewhere.
-- Making an optional field required and calculating its value from old data.
+- Making an optional field required and calculating its value from stored data from
+  an older schema.
 - Changing a field from one type to another, such as a string to a list of strings.
 - Moving flat fields into a nested model.
-- Changing the meaning of a value when old data must be converted to keep its
-  original meaning.
+- Changing the meaning of a value when stored data from an older schema must be
+  converted to keep its original meaning.
 
 ### Changes that do not need a migration
 
@@ -67,10 +68,6 @@ Other changes that do not normally need a migration include:
 - Changing documentation or field descriptions.
 - Adding a computed property that is not stored.
 
-Old stored data must still produce the correct current model when no migration is
-added. If the old data needs conversion or its meaning would change, add a schema
-version and migration function.
-
 ## Schema version contract
 
 Each migratable model declares one positive integer schema version. The literal and
@@ -108,15 +105,15 @@ class ProjectConfig(ResettableBaseModel):
 
     schema_version: Literal[2] = 2
     # ... unchanged fields ...
-    max_cache_revisions: int = Field(default=5, ge=5)
+    max_cache_revisions: int = Field(default=10, ge=5)
     # ... unchanged fields ...
 
     @classmethod
     def reset(cls: type[Self]) -> Self:
         """Reset the configuration to its defaults."""
         return cls(
             # ... unchanged defaults ...
-            max_cache_revisions=5,
+            max_cache_revisions=10,
             # ... unchanged defaults ...
         )
 ```
@@ -165,7 +162,7 @@ PROJECT_CONFIG_MIGRATIONS: dict[int, MigrationFunction] = {
 }
 ```
 
-The registry key is the source version. Key `1` registers the migration from
+The registry key is the source schema version. Key `1` registers the migration from
 version 1 to version 2.
 
 Keep every migration when later versions are added:
@@ -184,7 +181,7 @@ When you add a migration:
 - Keep `test_migration_manager.py` and the generic resource tests unchanged unless
   the framework behavior changes.
 - In `test_resource_migration.py`, update the affected resource tests that assert
-  its version, migration registry, or previous version data.
+  its version, migration function registry, or previous version data.
 - Add resource-specific tests under `tests/test_migrations/`. For example, a
   `ProjectConfig` migration can use `test_project_config_migration.py`.
 
@@ -207,7 +204,7 @@ uv run mypy src tests
 
 Migration is automatic during normal use:
 
-1. The resource manager reads old data.
+1. The resource manager reads stored data from an older schema.
 2. The migration manager converts it in memory.
 3. The resource manager returns the current validated model.
 4. The stored file remains unchanged until a save occurs.
@@ -217,20 +214,21 @@ On the first save:
 1. The write lock is checked.
 2. If the stored data needs migration, the resource manager tries to save a copy of
    the original JSON under `.fmu/migration-backups/`. This backup is separate from
-   the normal cache, is not removed automatically, and a failure to write it does not
-   stop the save.
+   the cache, is not removed automatically, and a failure to write it does not
+   stop the save. The original JSON is also added to a cache revision before
+   the migrated data is written.
 3. The current model is written with the new schema version.
-4. When automatic caching is enabled, the newly written data is added to a cache
-   revision.
+4. The newly written data is also added to a cache revision.
 
 Loading a resource does not create a changelog entry. A later user update or
 restore uses the existing changelog behavior.
 
-When an old cache revision is restored, it is migrated before it is written. The
-resource file therefore uses the current schema after the restore. Migration backups
-are only saved for manual recovery, and the library does not read or restore them. To
-roll back, first replace the current resource file with the appropriate backup, then
-run the older release.
+When an old cache revision is restored by the current release, it is migrated before
+it is written. The resource file therefore uses the current schema after the restore.
+If you roll back to an older release while the pre-migration revision is retained,
+restore that revision with the older release to return the resource file to the older
+schema. Migration backups are not read or restored by the library, copy one back
+manually when the cache revision is no longer available.
 
 Migrations are forward-only. After current-schema data is saved, an older
 `fmu-settings` release can reject it as newer than its supported schema.

src/fmu/settings/_migrations/manager.py

@@ -1,4 +1,4 @@
-"""Forward-only migration support for versioned resource data."""
+"""Forward-only migration support for versioned data."""
 
 from __future__ import annotations
 
@@ -20,7 +20,7 @@ class MigrationError(ValueError):
 
 
 class MigrationManager(Generic[MigratableResource]):
-    """Migrate versioned resource data to its current schema."""
+    """Migrate stored data to its current schema."""
 
     def __init__(
         self,
@@ -31,7 +31,7 @@ def __init__(
 
         Args:
             model_class: Current Pydantic model for the resource.
-            migrations: Migration functions keyed by their source version.
+            migrations: Migration functions keyed by their source schema version.
 
         Raises:
             TypeError: If the model does not declare one positive integer
@@ -42,30 +42,30 @@ def __init__(
         self.current_version = self._get_current_version()
 
     def migrate_resource(self, data: Any) -> MigratableResource:
-        """Migrate decoded JSON data with predefined migration function and validate it.
+        """Migrate decoded data with registered migration functions and validate it.
 
         Each migration function must increment the schema version by exactly one.
 
         Args:
             data: Decoded JSON data to migrate and validate.
 
         Returns:
-            The resource data as a validated current model.
+            The migrated data as a validated current model.
 
         Raises:
             MigrationError: If the data is not a JSON object, a schema version is
-                invalid or newer than supported, a required migration is missing or
-                fails, a migration function does not increment the schema version by
-                one, or the final data does not match the current model.
+                invalid or newer than supported, a required migration function is
+                missing or fails, a migration function does not increment the schema
+                version by one, or the final data does not match the current model.
         """
         if not isinstance(data, dict):
             raise MigrationError(
                 f"{self.model_class.__name__} resource must be a JSON object"
             )
-        source_version = self._get_source_version(data)
-        migration_steps = self._get_migration_steps(source_version)
+        source_schema_version = self._get_source_schema_version(data)
+        migration_steps = self._get_migration_steps(source_schema_version)
         migrated_data = copy.deepcopy(data) if migration_steps else data.copy()
-        migrated_data.setdefault("schema_version", source_version)
+        migrated_data.setdefault("schema_version", source_schema_version)
         for version, migration_function in migration_steps:
             try:
                 migrated_data = migration_function(migrated_data)
@@ -95,43 +95,44 @@ def migrate_resource(self, data: Any) -> MigratableResource:
         return validated_model
 
     def requires_migration(self, data: dict[str, Any]) -> bool:
-        """Return whether the data requires a migration before a write.
+        """Return whether stored data needs migration before a write.
 
-        This method does not run migrations. It confirms that all required forward
-        migration steps exist, then checks whether the source schema is older than
-        the current schema.
+        This check verifies that all required forward migration functions exist and
+        that the stored schema version is older than the current schema version.
 
         Args:
-            data: Existing resource data that a write would replace.
+            data: Existing stored data that a write would replace.
 
         Returns:
-            Whether the resource data requires migration.
+            Whether the stored data requires migration.
 
         Raises:
             MigrationError: If the schema version is invalid or newer than supported,
-                or a required migration step is missing.
+                or a required migration function is missing.
         """
-        source_version = self._get_source_version(data)
-        self._get_migration_steps(source_version)
-        return source_version < self.current_version
+        source_schema_version = self._get_source_schema_version(data)
+        self._get_migration_steps(source_schema_version)
+        return source_schema_version < self.current_version
 
     def _get_migration_steps(
-        self, source_version: int
+        self, source_schema_version: int
     ) -> list[tuple[int, MigrationFunction]]:
-        """Return and validate all migration steps needed by a source version."""
-        if source_version > self.current_version:
+        """Return and validate migration steps needed by a source schema version."""
+        if source_schema_version > self.current_version:
             raise MigrationError(
-                f"{self.model_class.__name__} schema version {source_version} is newer "
-                f"than supported version {self.current_version}; downgrade migration "
+                f"{self.model_class.__name__} schema version {source_schema_version} "
+                "is newer than supported version "
+                f"{self.current_version}; downgrade migration "
                 "is not supported"
             )
 
         steps: list[tuple[int, MigrationFunction]] = []
-        for version in range(source_version, self.current_version):
+        for version in range(source_schema_version, self.current_version):
             migration_function = self.migrations.get(version)
             if migration_function is None:
                 raise MigrationError(
-                    f"Missing {self.model_class.__name__} migration from schema "
+                    f"Missing {self.model_class.__name__} migration function from "
+                    "schema "
                     f"version {version} to {version + 1}"
                 )
             steps.append((version, migration_function))
@@ -179,8 +180,8 @@ def _get_current_version(self) -> int:
             )
         return current_version
 
-    def _get_source_version(self, data: dict[str, Any]) -> int:
-        """Get the source version, using the legacy version when it is absent."""
+    def _get_source_schema_version(self, data: dict[str, Any]) -> int:
+        """Return the stored schema version, using the legacy version when absent."""
         if "schema_version" not in data:
             return LEGACY_SCHEMA_VERSION
         return self._validate_version(data.get("schema_version"))

src/fmu/settings/_resources/cache_manager.py

@@ -39,13 +39,13 @@ class CacheManager:
     def __init__(
         self: Self,
         fmu_dir: FMUDirectoryBase,
-        max_revisions: int = 5,
+        max_revisions: int = 10,
     ) -> None:
         """Initialize the cache manager.
 
         Args:
             fmu_dir: The FMUDirectory instance.
-            max_revisions: Maximum number of revisions to retain. Default is 5.
+            max_revisions: Maximum number of revisions to retain. Default is 10.
                 Values below 5 are set to 5.
         """
         self._fmu_dir = fmu_dir

src/fmu/settings/_resources/pydantic_resource_manager.py

@@ -216,6 +216,8 @@ def _store_migration_backup(self: Self) -> None:
 
         source_version = data.get("schema_version", 1)
         self._write_migration_backup(content, source_version)
+        if self.automatic_caching:
+            self.fmu_dir.cache.store_revision(self.relative_path, content)
 
     def _write_migration_backup(self: Self, content: str, source_version: int) -> None:
         """Write a best-effort migration backup of the original content.

src/fmu/settings/models/project_config.py

@@ -114,7 +114,7 @@ class ProjectConfig(ResettableBaseModel):
     masterdata: Masterdata | None = None
     model: Model | None = None
     access: Access | None = None
-    cache_max_revisions: int = Field(default=5, ge=5)
+    cache_max_revisions: int = Field(default=10, ge=5)
     rms: RmsProject | None = None
     validation: ProjectValidation = Field(default_factory=ProjectValidation)
 
@@ -134,7 +134,7 @@ def reset(cls: type[Self]) -> Self:
             masterdata=None,
             model=None,
             access=None,
-            cache_max_revisions=5,
+            cache_max_revisions=10,
             rms=None,
             validation=ProjectValidation(),
         )