Content-hash-based change tracking for data imports

[jonathangreen]

Description

Replaces the timestamp-only change-detection logic in the data import pipeline
with a content-hash-based system. Previously, BibliographicData and
CirculationData used only the data source's "last updated" timestamp to decide
whether to re-apply incoming data to an Edition or LicensePool. This caused
two problems:

• Re-publishing the same content with a newer timestamp triggered unnecessary
  writes and work creation.
• A real change arriving with the same (or missing) timestamp was silently skipped.
  This branch reverts the original timestamp-throttling PR (Fix BibliographicData.has_changed to throttle updates when data sourc… #3198) and replaces it
  with a proper content-hash approach. A SHA-256 hash of the canonical, serialized
  form of the incoming data is stored on the database record after each import.
  Subsequent imports compare both the timestamp and the hash before deciding whether
  to apply an update.
  Key changes:
• New json_hash() / json_canonical() utilities (util/json.py) produce a
  stable, order-independent SHA-256 fingerprint of any JSON-serializable structure.
• BaseMutableData gains updated_at, created_at, as_of_timestamp,
  calculate_hash(), and should_apply_to(). The should_apply_to() method
  is now the single decision point for both bibliographic and circulation data.
• BibliographicData.has_changed() and CirculationData.has_changed() are
  removed and replaced by the shared should_apply_to() logic.
• Edition and LicensePool each gain an updated_at_data_hash column.
  LicensePool also gains created_at and updated_at columns to track when
  its CirculationData was first and most recently imported.
• Individual-license pools (e.g. ODL) always re-apply availability even when the
  hash matches, because license availability can change as licenses expire
  independently of feed content.
• Database migration f98e4049c87d adds all four new columns.

Motivation and Context

The original has_changed() implementation only compared timestamps, which is
insufficient: a data source can re-publish identical content with a newer timestamp,
or publish changed content with the same timestamp. Content hashing is the correct
primitive for detecting genuine data changes and avoiding redundant imports.

How Has This Been Tested?

• Updated unit tests for BibliographicData and CirculationData cover the
  new should_apply_to() logic, including the null-hash bootstrap case, the
  timestamp-is-older short-circuit, and the hash-match skip.
• New unit tests for json_canonical() and json_hash() verify ordering
  stability across dict keys, list items, and float precision.
• All existing integration tests for Boundless, OPDS, ODL, and Overdrive importers
  pass with the updated field names (updated_at in place of
  data_source_last_updated).
• Full test suite run via tox -e py312-docker -- --no-cov.

Checklist

• I have updated the documentation accordingly.
• All new and existing tests passed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.32%. Comparing base (45176e0) to head (d17bf96).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3199   +/-   ##
=======================================
  Coverage   93.31%   93.32%           
=======================================
  Files         502      502           
  Lines       46178    46227   +49     
  Branches     6315     6319    +4     
=======================================
+ Hits        43093    43143   +50     
+ Misses       2001     2000    -1     
  Partials     1084     1084           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[claude]

Claude finished @dbernstein's task in 12m 36s —— View job

---

PR Review: Content-hash-based change tracking for data imports

• Gather context and understand the request
• Read core implementation files
• Read test files and migration
• Post inline comments for bugs/issues
• Write summary review

---

The overall design is solid. The two-stage guard (timestamp short-circuit → hash comparison) is the right primitive, the canonicalization in json_canonical handles all the tricky edge cases (heterogeneous type ordering, multiple None values, bool/int disambiguation), and the hash cache invalidation via __setattr__ is clean.

A few cross-cutting observations:

Timestamp/hash coupling after force-apply (even_if_not_apparently_updated=True): The Edition and LicensePool paths handle this differently — _update_edition_timestamp only advances updated_at while circulation.apply always overwrites it. As noted in the inline comments, this creates an edge case on the Edition path where a force-applied stale hash combined with a pinned-at-high-water updated_at can cause subsequent imports with intermediate timestamps to be rejected by the < check even though their hash would differ. In practice this is harmless (data sources advance timestamps monotonically), but a comment in _update_edition_timestamp documenting the intentional decoupling between updated_at and updated_at_data_hash would help future readers.

Migration / bootstrap: The migration comment is thorough and accurate — all existing rows start with updated_at_data_hash = NULL, which should_apply_to correctly treats as "never imported", so the first full import after deployment re-establishes all hashes as expected.

ODL handling: The self.licenses is not None bypass is correctly placed in both CirculationData.apply (inner guard) and needs_apply (outer guard at import time), so ODL pools are always re-applied without the hash check.

No bugs found. The four inline comments are all Minor or Nit level.

[claude]

Minor: When even_if_not_apparently_updated=True is used with data whose as_of_timestamp is older than db_object.updated_at (i.e. force-applying stale bibliographic data), _update_edition_timestamp will not advance updated_at but will still overwrite updated_at_data_hash with the stale hash. Subsequent imports with a timestamp in the range (as_of_timestamp, db_object.updated_at) will be rejected by the < check here even though their hash would differ. The LicensePool path in circulation.apply does not have this issue because it always overwrites pool.updated_at. The inconsistency is harmless in practice (data sources advance timestamps monotonically) but worth documenting.

[claude]

Minor: pool.updated_at is unconditionally overwritten with self.as_of_timestamp, which can regress the stored timestamp when even_if_not_apparently_updated=True is combined with stale data (verified by the test). This differs from _update_edition_timestamp which only advances updated_at. The regression is actually beneficial here — it ensures subsequent imports with intermediate timestamps pass the < check — but the asymmetry with the Edition path is worth documenting as intentional.

[claude]

Nit: int and float share the same precedence (1). When a list contains both integers and floats, their relative order is determined by Python's native int/float comparison, which is well-defined. This is fine for the current use case, but it means [1, 1.0] and [1.0, 1] canonicalise to the same sorted output [1, 1.0] (since 1 == 1.0 in Python, the sort is stable and preserves input order). If that distinction ever matters, a separate precedence for float would be needed.

[claude]

Minor: updated_at_data_hash is unconditionally overwritten here, even when edition.updated_at was not advanced (i.e. as_of_timestamp <= edition.updated_at). This is correct — the hash should always reflect the data most recently applied — but it means the (updated_at, updated_at_data_hash) pair does not represent a coherent snapshot from a single import when old data is force-applied. Consider a brief comment to make this intentional decoupling explicit.