Make the EditorFileSystem cache more reliable

[Rindbee]

Closes godotengine/godot-proposals#13196

Known limitations

Due to the limited precision of file modification timestamps obtained by Godot, file movements are primarily tracked using UIDs. However, not all files are assigned a UID.

Details

The file timestamps are not precise enough

File change detection relies on the file's modification timestamp. However, the timestamp precision of the current file's modification time is 1 second.

And when processing files in batches, the modification timestamps of many files may all fall within the same second. This is especially noticeable when updating files using version control software such as Git.

This is unlikely to be a problem in most cases. However, some unexpected issues may arise in cases of file overwriting. For example:

1. If similar directory structures exist, such as res://a/x/ and res://b/x/.
2. You have already added these changes using git and committed them.
3. You accidentally removed the directories res://a/ and res://b/.
4. Using git restore . to restore these directories might be a good idea.
5. After restoration, the timestamps of these restored files and directories are typically within the same second.
6. Back to the editor. The timestamps of these files will be recorded.
7. If you delete res://a/ outside the editor and rename the original res://b/ to res://a/.
8. Back to the editor again. The content under res://a/x/ is likely not updated correctly.

Solutions

1. Ensure that each resource file is assigned a unique UID;
2. Determine if a file has been updated by relying on its modification time and UID (if both of these remain unchanged, then the file is considered unchanged; otherwise, the file is considered to have changed);
3. Always force checks for changes in directories (i.e., the addition or deletion of sub files/directories) in projects managed with git (identify via res://.git/).

Some types of file move actions cannot be tracked because no UID was assigned

The order in which file extensions are processed is as follows:

1. Importable file types (Their importer is either skip or keep);
2. Resource file type (The file extension list is json,po,mo,crt,pub,key, Core: Do not generate *.uid files for JSON, certificates, and translations #99540);
3. Text file types (The file extension list is txt,md,cfg,ini,log,json,yml,yaml,toml,xml, defined in docks/filesystem/textfile_extensions);
4. Other file types (The file extension list is ico,icns, defined in docks/filesystem/other_file_extensions);

Point 2 and Point 3 are somewhat repetitive. If we process 3 and 4 first, point 2 might be able to be merged into point 3.

TODO

However, from the perspective of using UID to track file movements, it might be better to assign UIDs to files with the above extensions.

The directory move actions could not be tracked because no UID was assigned

This may affect the updating of directory colors, favorites.

TODO

Perhaps the UID assigned to the directory could be recorded in a .uid file.

The @icon path for the script

Using the uid:// path allows for effective tracking; otherwise, it might be difficult to analyze the movement of @icon files. Furthermore, there is no method to update the @icon path in the script files.

Main processing flow

Main flowchart

Flowchart

flowchart TD
    S0(["Launch editor (first scan)"]) --> P00["Update global script classes"]
    P00 --> PA00["Scan"]
    PA00 --> C00{Already build tree?}
    C00 --> |No| PB03[[Build filesystem tree]]
    C00 --> |Yes| PB02[[Update filesystem tree]]
    PB02 --> PE00["Scan dirs"]
    PB03 --> PM01[Update uid actions]

    S1(["Application focus in"]) --> PB01["Scan full changes"]
    S2([Extensions changed]) --> PB01
    PB01 --> C01{Is scanning?}
    C01 --> |No|PB12["Mark fs dirty"]
    C01 --> |Yes|PB11["Full scan pending"]
    PB11 --> |Delayed to the end of the scan|PB01
    PB12 --> PD00["Scan dirty dirs"]

    S3([Dir operations in editor]) --> PB13["Mark dir dirty"]
    PB13 --> |Delayed to the end of the frame| PD00

    PD00 -->PE00
    PE00 -->PM01

    S4([File operations in editor]) --> PC00["Update file(s)"]
    PC00 --> |Delayed to the end of the frame|PM01

    PM01 --> C02{"Is first scan?"}
    C02 --> |No| PM02[Update global script classes]
    PM02 --> PM03[[Update scan actions]]
    C02 --> |Yes| PM03
    PM03 --> Z([End])

Loading

Trigger a scan/update

Several cases that trigger a scan/update:

1. Launch editor (scan fs recursively);
2. Application focus in (scan fs recursively);
3. Extensions changed (scan fs recursively);
4. Dir operations (scan specific dirs);
5. File operations (update and/or scan specific files and/or dirs);
6. Some plugins actively call certain functions (update and/or scan).

Details

In a sense, Case 1 is a special case of Case 2. One difference is that the file information is stored in the file in case 1, while it is stored in memory in case 2. Another difference is that in case 1, the editor is being launched, so it may read the file information first, rather than reading the file information in the cache file to ensure that certain modules are available.

In case 3, the fact that the directory timestamp has not changed should not be an obstacle to detecting new files. The reasons for the changes in the list of supported file extensions are likely as follows:

1. Modified the editor settings docks/filesystem/textfile_extensions and/or docks/filesystem/other_file_extensions.
2. Modify certain scripts that inherit from ResourceFormatLoader¹;
3. Activate/deactivate certain importer plugins²;

Cases 4 and 5 are usually caused by user interaction in the editor. Some file caches (such as uids and global script classes) are project-wide, but some processing methods are implemented on a file-by-file basis (depending on update_file()). Additionally, the current file timestamp precision is 1 second, making frequent calls to some methods unsuitable (#110048). These are the reasons for delaying processing until the end of the frame. It should be noted that if these file/dir operations affect the scripts mentioned in case 3, a new full scan may be triggered when the processing is completed.

Case 6 is probably the most unpredictable. Here are just some suggestions:

1. File write operations are best performed in the main thread;
2. Avoid frequent fs scanning.

Update file info

During the scanning process, most of the information in FileInfo is updated immediately. This information is generally unrelated to file movement. However, information that might be affected by file movement is updated with a delay.

Details

The FileInfo.status records the file status at the corresponding path:

1. File changes;
2. Type changes;
3. File categories;
4. Has custom uid support or not;
5. Whether it is treated as a resource file;
6. Whether its type is a Script;
7. Whether its type is a PackedScene;
8. Whether it is a global class alternative;
9. Whether it is the active global class alternative.

The FileInfo.class_name.icon_path is used not only for scripts but also for other types of resources. FileInfo.resource_script_class is used for any resource that has script with a global class name. These two might become invalid when the script changes. However, the script will definitely be in FileInfo.deps. Modify Object::get_property_list() to ensure that the first dep found in _find_resources() of ResourceFormatSaverBinaryInstance/ResourceFormatSaverTextInstance is the script.

Update UID and track paths

File changes can be tracked by monitoring changes in the UID. Since we obtain a snapshot of the file system during the scan, we need to analyze the changes in UIDs in a specific order to infer the changes to the files.

1. Verify whether the UID(f) is cached in ResourceUID;
2. Create and add a new UID;
3. Remove UID;
4. Attempting to add UID(f);
5. If UID(f) already exists in ResourceUID, try adding UID(m); if it still exists, try creating and adding a new UID.

Details

The above order is derived by working backward from the file changes. We may store the UID in both the FileInfo and the file itself. Read the new UID from the file to update the UID in memory.

Compound operations can be viewed as deleting first, and the UID is unique. Adding a UID before removing the existing one ensures the uniqueness of the newly added UID.

flowchart LR

A001(["Check uid change"]) --> C001{"Can have uid now?"}
C001 --> |Yes|A003["Check file status"]
C001 --> |No|A002["Remove UID(m) if valid"]
A002 --> E003([3])
A003 --> C002{"File removed?"}
C002 --> |Yes|A002
C002 --> |No|C003{"File added?"}
C003 --> |No|C005{"UID(m) valid?"}
C005 --> |Yes|C006{"UID(m) == UID(f)?"}
C006 --> |No|A008["Remove UID(m)"]
C006 --> |Yes|A007["Try to add UID(f) during first scan"]
A007 --> E001([1])
A008 --> E003
A008 --> C007{"UID(f) valid?"}
C005 --> |No|C007
C007 --> |Yes|A009["Try to add UID(f), UID(m)"]
A009 --> E004([4])
C007 --> |No|A010["Try to add UID(m)"]
A010 --> E004
C003 --> |Yes|C004{"UID(f) valid?"}
C004 --> |No|A006["Create a new UID"]
A006 --> E002([2])
C004 --> |Yes|A005["Try to add UID(f)"]
A005 --> E004

Loading

Analyze file changes based on path changes in UID

During each scan, we record the basic File operations in FileInfo.status and use UID to track path changes.

Based on the above UID tracking records, the file operations are inferred according to rules.

Details

File operation types include (U for UID, P for path):

1. Basic operations:
   1. File added (UID added), PaUa;
   2. File removed (UID removed), PrUr;
   3. File updated, Pu;
      1. UID added, Ua;
      2. UID removed, Ur;
      3. UID unchanged.
2. Compound operations:
   1. File moved/renamed, P0U0r + P1U0a;
   2. File overwriten, P0U0r + P1U1r + P1U0a + ...;
   3. File duplicated, P0U0r + P0U0a + P1U0a + ....

Update the caches after confirming the file has been moved

1. Update resource paths of instances in ResourceCache;
2. Update global script classes;
3. Update deps, resource script class, and icons;
4. Update project settings.

Jobs

Multi-dot extensions

Improve support for multi-dot extensions.

Tasks

• Add a String::validate_extension() method to validate whether the path's extension is in the provided list of extensions.
• Improved support for multi-dot extensions in the file system.
• Improved support for multi-dot extensions in file dialogs.

File status

Use status to record information such as the file category, type, and status.

Tasks

• Classification.
  • Custom uid support, tagged in EditorFileSystem::_category_validate().
  • Categorize files by file extension, tagged in EditorFileSystem::_category_validate().
    • Importable files.
    • Text files.
    • Other files.
  • Classified by type, tagged in EditorFileSystem::_type_analysis().
    • As resources.
    • Not a resource.
      • OtherFile.
      • TextFile.
      • Empty type.
• The existence status of files on the same path changes.
  • File removed, tagged in EditorFileSystem::_file_info_remove().
  • File added, handled in EditorFileSystem::_file_info_add().
  • File updated, handled in EditorFileSystem::_file_info_update().
• Type changes, tagged in EditorFileSystem::_type_analysis().
  • Type removed.
    • Clear the path of the cached instance, handled in EditorFileSystem::_update_scan_actions().
  • Type added.
  • Type changed, equivalent to removing first and then adding.
    • Clear the path of the cached instance, handled in EditorFileSystem::_update_scan_actions().
  • Type unchanged, just a file update.
    • Reload the cached instance.
      • Scripts.
      • Scenes.
      • Other resources, sent via the resources_reload signal, handled in EditorNode::_resources_changed().

File movement

Track changes to UIDs to analyze complex operations such as file movement.

Tasks

• Sort the processing order based on changes in UID, handled in EditorFileSystem::_create_actions_from_uid_change().
• Update ResourceUID cache in order, handled in EditorFileSystem::_update_scan_uid_actions().
• Track file changes based on changes in UID, handled in EditorFileSystem::_update_scan_uid_actions().
  • Categorizing file changes.
    • File untracked, the UID is no longer in use.
    • File moved, the file path using the UID has changed.
    • File duplicated, a new file path attempting to use the existing UID has been detected.
    • File tracked, the UID was used for the first time.
    • File retracked, the UID is already included in the file.
  • Process caches based on UID changes.
    • Resource path in cached instances.
      • Untracked files.
        • Cleared.
      • Moved files.
        • Updated.
          • Use the .tmp~ temporary suffix to avoid path swapping cases.
    • File owners and dependencies.
      • Untracked files.
        • Keep.
      • Moved files.
        • Update the deps in the owner's file info.
        • Rename dependencies in the owner files.
        • When the affected file owner is an opened scene file, reload occurs.
    • Project settings.
      • Update affected file paths.
    • Folder tracking has not yet been implemented. Use a .uid file to track?
      • Folder colors.
        • In editor, call FileSystemDock::_update_folder_colors_after_move().
        • Outside editor.
      • Favorites.
    • Update ResourceCache::resource_path_cache.
    • Provide a dialog that displays the changes.

Script class info

Update class information in the script.

Tasks

• Manage class info, handled in EditorFileSystem::_script_class_info_update().
  • Docs.
  • Icons.
    • Scripts.
      • class_info.icon_path³.
    • Resource files.
      • class_info.icon_path³.
• Manage global script class.
  • Cache global script class alternatives.
  • Handles updates for global script alternatives.
    • Class info removed, handled in EditorFileSystem::_global_script_class_info_remove().
    • Class info added, handled in EditorFileSystem::_global_script_class_info_add().
    • Class info updated, equivalent to removing first and then adding, handled in EditorFileSystem::_script_class_info_update().
  • Maintaining active global script classes.
    • Activate, handled in EditorFileSystem::_update_global_script_class_activation().
      • Register in ScriptServer.
      • Register in EditorData.
        • Class name.
        • Icon.
      • Generate its documentation.
    • Deactivate, handled in EditorFileSystem::_global_script_class_info_remove().
      • Unregister in ScriptServer.
      • Unregister in EditorData.
        • Class name.
        • Icon.
      • Clear its documentation.
    • Special class derived scripts.
      • ResourceFormatLoader.
      • ResourceFormatSaver.

Scenes

Update global scene groups and built-in scripts.

Tasks

• Global scene groups.
• Built-in scripts.
  • Docs.

Importable files

Tasks

• Verify whether (re)importing is necessary, handled in EditorFileSystem::_import_validate().

Known issues

An error occurs when updating the tscn file that uses ViewportTexture

ERROR: Path to node is invalid: 'SubViewportContainer/SubViewport'.
   at: _setup_local_to_scene (scene/main/viewport.cpp:200)

EditorFileSystem::_update_script_documentation() will load and use the PackedScene. Resources that are not configured with a local scene in the editor will return the root of the currently edited scene when Resource::get_local_scene() is called.

Footnotes

1. These scripts must be global script classes (class_name) and must be tool (@tool) scripts. ↩

2. Custom importer scripts must inherit from EditorImportPlugin and be tool (@tool) scripts, and use add_import_plugin()/remove_import_plugin() to add/remove them. ↩

3. The icon displayed as the file icon conflicts with the icon specified in the script's icon field. ↩ ↩²