Skip to content

Deterministic checkpoint fingerprint (canonical-JSON + sha256) #285

Description

@FernandoCelmer

Labels: enhancement, discovery
Related: #281

Context

When implementing the idempotency policy from #281, the engine needs a stable, cross-process hash of initial_context to compare against the stored checkpoint fingerprint. Python's built-in hash() is
not suitable:

  • PYTHONHASHSEED randomizes string hashes per process, so two
    runs on the same input produce different hash values.
  • dict and set iteration ordering historically influenced the
    hash of structures containing them.
  • hash() truncates to platform-dependent integer width.

Using hash() would silently invalidate every checkpoint on each process restart.

Concept

Compute the fingerprint as sha256(canonical_json(value)), where canonical_json produces deterministic output regardless of:

  • Insertion order of dict keys
  • Whitespace
  • Python version
  • Host architecture

Proposed implementation

# dotflow/core/fingerprint.py
import hashlib
import json
from typing import Any

def canonical_json(value: Any) -> str:
    """JSON output that is byte-identical for equal logical inputs."""
    return json.dumps(
        value,
        sort_keys=True,
        ensure_ascii=False,
        separators=(",", ":"),
        default=_default,
    )

def _default(obj):
    # Deterministic fallback for non-JSON types
    if hasattr(obj, "__dict__"):
        return {"__class__": type(obj).__name__, **obj.__dict__}
    return repr(obj)

def fingerprint(value: Any) -> str:
    """Stable sha256 of a canonical JSON encoding."""
    return hashlib.sha256(canonical_json(value).encode()).hexdigest()

Usage (preview, depends on #281)

from dotflow.core.fingerprint import fingerprint

stored = checkpoint.input_fingerprint
current = fingerprint(task.initial_context)
if stored != current:
    # apply on_input_change policy (#281)
    ...

Acceptance criteria

  • dotflow/core/fingerprint.py with fingerprint() and
    canonical_json()
  • Round-trip test: same input → same hash across two
    sub-processes spawned via multiprocessing.spawn
  • Property test: fingerprint({"a": 1, "b": 2}) == fingerprint({"b": 2, "a": 1})
  • Edge cases covered: nested dicts, lists, sets (sorted),
    dataclasses, datetime
  • Documented as the canonical hash function for the library

Why this matters

Issue #281 cannot ship without a deterministic fingerprint. Doing it right once, in a dedicated module, means every future feature (checkpoint cache keys, run replay IDs, task memoization) reuses the same primitive.


Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions