Skip to content

Fix HistoricForeignKey reverse prefetch_related (rebases #1159 and addresses review)#1604

Open
tomhampshire wants to merge 3 commits into
django-commons:masterfrom
tomhampshire:fix-historic-fk-prefetch
Open

Fix HistoricForeignKey reverse prefetch_related (rebases #1159 and addresses review)#1604
tomhampshire wants to merge 3 commits into
django-commons:masterfrom
tomhampshire:fix-historic-fk-prefetch

Conversation

@tomhampshire

@tomhampshire tomhampshire commented Apr 29, 2026

Copy link
Copy Markdown

Description

Rebases @mr-mare's PR #1159 onto current master, resolves the merge conflict in simple_history/tests/tests/test_models.py (which arose from the HistoricOneToOneFieldTest class added on master), and addresses the two review points raised by @ddabble in September 2024.

The branch contains three commits:

  1. Fix HistoricForeignKey when used together with prefetch_related() (authored by @mr-mare, cherry-picked from Fix HistoricForeignKey when used together with prefetch_related() #1159) - removes the spurious _apply_rel_filters() call inside HistoricRelationModelManager.get_queryset() that produced a WHERE pk = X AND pk IN (...) clause for prefetch queries, masking all but the first parent's related objects. Includes the original three single-level prefetch tests.

  2. Remove redundant prefetched-objects cache lookup in reverse manager - addresses the first review point. The outer RelatedManager produced by create_reverse_many_to_one_manager already short-circuits via self.instance._prefetched_objects_cache before delegating to super().get_queryset(), so the same lookup inside HistoricRelationModelManager.get_queryset() was redundant. Removing it also keeps the prefetch-build path clean: RelatedManager.get_prefetch_querysets() calls super().get_queryset() which would otherwise consult the per-instance cache while building a prefetch for many instances at once.

  3. Add nested-prefetch regression test and credit contributors - addresses the second review point with a new test for nested prefetch_related() across two HistoricForeignKey levels (the SubSub case from the description of With HistoricForeignKey prefetch_related doesn't return all related objects in some situations #1152). Adds a TestHistoricSubParticipantToHistoricParticipant model with a HistoricForeignKey to TestHistoricParticipanToHistoricOrganization. The new test exercises both the bare prefetch_related(\"a__b\") form and the equivalent fully-explicit Prefetch() form. The bare form fails on master with [] != [\"p2-sub1\", \"p2-sub2\"] and passes with the fix in place. Also adds @mr-mare and @tomhampshire to AUTHORS.rst.

Related Issue

Fixes #1152. Rebases and supersedes #1159 (with the original commit and authorship preserved).

Motivation and Context

Without this fix, prefetch_related() across a HistoricForeignKey reverse relation silently drops every parent's related objects except the first. The buggy code path is triggered by the natural Django idiom (prefetch_related(\"a__b\")) and produces no exception, no warning, no log line, just empty RelatedManager.all() returns and quietly wrong results downstream. We hit this in production this week, three years after #1152 was filed.

How Has This Been Tested?

Locally on Python 3.11 / Django 5.2.13 / SQLite using python runtests.py:

  • All 342 tests in simple_history.tests pass (was 340 + 2 new prefetch suites; now 342 + 1 new nested test = 343).
  • All 21 tests in simple_history.registry_tests pass.
  • Targeted HistoricForeignKeyTest (7 tests including the new nested test) all pass.
  • Confirmed the new nested test fails on master without the fix, with the same AssertionError: Lists differ: [] != [\"p2-sub1\", \"p2-sub2\"] expected from the bug.

pre-commit run clean across all changed files (bandit, black, codespell, flake8, isort, pyupgrade, django-upgrade, ...).

Also verified end-to-end against a real downstream codebase that uses HistoricForeignKey across multiple levels: a 4-parent prefetch_related(\"a__b\") correctly returned the expected child counts for every parent with the patched simple_history installed; with vanilla 3.8.0, only the first parent's children came back.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • I have run the pre-commit run command to format and lint.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • I have added my name and/or github handle to AUTHORS.rst
  • I have added my change to CHANGES.rst
  • All new and existing tests passed.

mr-mare and others added 3 commits April 29, 2026 12:33
The outer RelatedManager (constructed by create_reverse_many_to_one_manager)
already returns from self.instance._prefetched_objects_cache when a prefetch
cache exists, so HistoricRelationModelManager.get_queryset() does not need
to check the cache itself.

Removing it also keeps the prefetch-build path clean: get_prefetch_queryset
calls super().get_queryset(), which would otherwise consult the per-instance
cache when building a prefetch for many instances at once.

Suggested by @ddabble in review of django-commons#1159.
Adds TestHistoricSubParticipantToHistoricParticipant - a third-level
historic model with a HistoricForeignKey to a model that itself has a
HistoricForeignKey - and a regression test exercising bare nested
prefetch_related ("a__b") across two HistoricForeignKey reverses, as
described in the SubSub example of django-commons#1152. The test fails on master
(the second parent's children come back as []) and passes with the
fix in place.

Also lists the original PR author and the rebase author in AUTHORS.rst.

Addresses django-commons#1152 review feedback from @ddabble.
@tomhampshire tomhampshire force-pushed the fix-historic-fk-prefetch branch from fdd9f37 to ca25ed1 Compare April 29, 2026 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

With HistoricForeignKey prefetch_related doesn't return all related objects in some situations

2 participants