Skip to content

bug(meta): corrupted meta files on abrupt shutdown cause JSONDecodeError on startup #187

Description

@akremenetsky

Description

When a node is powered off abruptly (power loss, crash), the meta driver files used by JsonFileStorageSingleton can become empty or contain truncated JSON. On the next boot, load() calls json.load(f) on the corrupted file and fails with:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This prevents all meta-based drivers (render, etc.) from starting, leaving the universal agent in a crash loop.

Root cause

JsonFileStorageSingleton.persist() (storage/common.py:47) uses an atomic write pattern (tmp file + os.replace) which is safe against crashes during the replace step. However, the file can still be corrupted if:

  1. The crash occurs during the initial write to the .tmp file, and the tmp file is later cleaned up without restoring the original.
  2. The meta file was created empty by a previous failed operation.
  3. A power loss happens during a write that predates the atomic pattern or bypasses it.

load() (common.py:35) has no defense against this — it opens the file and calls json.load() unconditionally.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions