Skip to content

fix(retention): page through all expired records, remove 10k cap (#102) - #165

Merged
rajfirke merged 1 commit into
mainfrom
fix/102-retention-pagination
Sep 4, 2026
Merged

fix(retention): page through all expired records, remove 10k cap (#102)#165
rajfirke merged 1 commit into
mainfrom
fix/102-retention-pagination

Conversation

@rajfirke

@rajfirke rajfirke commented Sep 3, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

RetentionEngine.find_expired() called trail.query(end=cutoff, limit=10000) — a single fixed-size query. Any trail with more than 10,000 expired records would silently drop the remainder: preview() under-counted, and execute() left those records undeleted while reporting success. For a compliance retention engine enforcing EU AI Act Art. 12 archival requirements, this is unacceptable.

Fix: Page through the full result set using offset:

batch = self._QUERY_BATCH_SIZE
expired: list[dict[str, Any]] = []
offset = 0
while True:
    page = self._trail.query(end=cutoff, limit=batch, offset=offset)
    expired.extend(page)
    if len(page) < batch:
        break
    offset += batch
return expired

_QUERY_BATCH_SIZE = 10000 is a class attribute instead of a hardcoded literal so tests can shrink it to 2 and exercise the multi-batch path without inserting 10k rows.

Two new regression tests verify that find_expired() and execute() correctly handle a trail whose expired record count exceeds one batch.

Checklist

  • Tests added/updated for the change
  • ruff check src/ tests/ passes
  • ruff format --check src/ tests/ passes
  • mypy src/provena/ passes
  • pytest passes with no failures
  • CHANGELOG.md updated (if user-facing change)

Related Issues

Closes #102


Note: PR #158 covered the same fix but was blocked waiting on offset support in ContextTrail.query() (which was already shipped in v1.1.0). Applying it directly here with a clean rebase on current main.

RetentionEngine.find_expired() called trail.query(end=cutoff, limit=10000),
silently dropping every expired record past the 10,000th. A retention run on a
large trail would report success while leaving thousands of records that should
have been purged — a compliance gap for EU AI Act Art. 12 archival requirements.

The fix pages with trail.query(end=cutoff, limit=batch, offset=offset) until a
short page signals the end of results. _QUERY_BATCH_SIZE is a class attribute
(not a hardcoded literal) so tests can set it to 2 and exercise the multi-batch
path without needing 10k rows.

Closes #102
@rajfirke
rajfirke force-pushed the fix/102-retention-pagination branch from faa3ea4 to d6d39cb Compare September 3, 2026 18:27
@rajfirke
rajfirke merged commit a1660bc into main Sep 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RetentionEngine silently caps at 10,000 expired records per run

1 participant