TriggerFlow 4.1.4.3 snapshots multiply completed interrupt/resume payloads and provide no bounded retention API

[Maplemx]

Summary

A durable TriggerFlow execution snapshot can grow by several multiples of the application payload because completed pause/resume data is retained in several internal structures at once. The public runtime-event compaction policy does not compact these snapshot sections, and I could not find a public retention/projector API for completed interrupts, resume requests, or signal attempts.

This is not only an application-side “large payload” issue: one request/response value is copied into multiple framework-owned history structures after the interrupt has already reached a terminal state.

Environment

• Agently: 4.1.4.3
• Python: 3.10
• TriggerFlow durable execution with a snapshot store

Production observation

One closed/cancelled execution with no pending interrupts produced a 6,605,964-byte JSON snapshot.

Snapshot section	Bytes	Share
signal_net	2,754,161	41.7%
interrupts	2,171,383	32.9%
runtime_data	942,980	14.3%
resume_ledger	520,036	7.9%
sub_flow_frames	213,276	3.2%

The snapshot contained 13 interrupts (12 resumed, 1 cancelled) and 61 completed signal attempts. A large resume value was present in several paths, including:

• interrupt.response
• interrupt.resume_value
• interrupt.resume_requests.*.value
• resume_ledger.*.*.value
• completed signal-attempt metadata embedding the interrupt/resume fields
• signal.meta.resume.value

The application does pass non-trivial evidence/probe data through pause/resume, which amplifies the symptom, but the repeated terminal-history copies are framework-owned.

Minimal reproduction

import asyncio
import json
from agently import TriggerFlow


class MemoryStore:
    def __init__(self):
        self.state = None

    async def put_snapshot(self, run_id, state, **kwargs):
        self.state = state
        return {"ref": "memory"}

    async def get_snapshot(self, run_id):
        return self.state


async def main():
    async def pause(data):
        return await data.async_pause_for(
            payload={"request": "x" * 100_000},
            resume_to="next",
        )

    async def done(data):
        return data.input

    flow = TriggerFlow(name="snapshot-bloat-repro")
    flow.to(pause).to(done)

    store = MemoryStore()
    execution = flow.create_execution(
        auto_close=False,
        record_store=False,
        runtime_resources={"snapshot_store": store},
    )

    await execution.async_start(None)
    interrupt_id = next(iter(execution.get_pending_interrupts()))

    await execution.async_continue_with(
        interrupt_id,
        {"response": "y" * 100_000},
        resume_request_id="req-1",
    )

    snapshot = execution.save(require_idle=True)

    def size(value):
        return len(json.dumps(
            value,
            ensure_ascii=False,
            separators=(",", ":"),
        ).encode())

    print("snapshot", size(snapshot))
    print("interrupts", size(snapshot["interrupts"]))
    print("resume_ledger", size(snapshot["resume_ledger"]))
    print("signal_net", size(snapshot["signal_net"]))


asyncio.run(main())

Observed output:

snapshot 1306777
interrupts 501548
resume_ledger 100247
signal_net 602860

The same flow with tiny payloads is about 6,725 bytes. With the reproduction above, 200 KB of application request/response text becomes about 1.31 MB of serialized snapshot data.

Additional isolation showed approximately:

• 100 KB request alone -> 406 KB snapshot
• 100 KB response alone -> 907 KB snapshot

Compaction does not cover this state

I also tested execution.set_compaction_policy(...) with a 10 KB request and 10 KB response.

Before compaction: 136,916 bytes
After compaction: 137,555 bytes (the increase is compaction metadata)

The relevant sections were unchanged:

interrupts    51541 -> 51541
resume_ledger 10244 -> 10244
signal_net    62855 -> 62855

This appears consistent with the current implementation: runtime-event compaction applies to runtime_event_store events, while the snapshot serializer still includes full completed signal attempts, interrupt records, request values, response values, and the resume ledger.

Expected behavior

Full payloads are necessary for pending interrupts and active recovery boundaries. Once an interrupt/resume/signal attempt is terminal and a durable successor checkpoint exists, the serialized snapshot should be able to retain a bounded summary instead of repeated full values.

For example:

• pending interrupts retain the complete payload;
• resumed/cancelled interrupts retain IDs, status, timestamps, related signal/request IDs, and optionally a digest/artifact reference;
• completed signal attempts retain bounded metadata rather than embedding full interrupt/resume values;
• the idempotency ledger stores a digest/result reference instead of another full response copy;
• applications can configure latest-N history, a byte budget, or a snapshot projector/retention policy;
• load/resume validates summaries and does not prune anything needed by the active recovery boundary.

Suggested API / compatibility direction

A public snapshot_retention_policy or snapshot projector could be separate from runtime-event compaction, with defaults preserving current behavior for compatibility. Useful controls could include:

• retain full values for pending/active records;
• retain latest N terminal records;
• terminal-history byte budget;
• replace terminal values with digest + external artifact reference;
• load-time migration/projection for older snapshots.

It would also help if the documentation explicitly stated that set_compaction_policy(...) compacts runtime events but not TriggerFlow snapshot internals.

At-rest gzip is an effective temporary storage mitigation (the 6.61 MB sample compressed to about 1.02 MB at gzip level 1), but it does not reduce save-time object duplication, serialization CPU/memory, or in-process workspace size, so it does not address the underlying retention behavior.

[Maplemx]

Confirmed and implementation status

The report is reproducible. We confirmed that terminal interrupt/resume values can be repeated across framework-owned snapshot sections and that runtime-event compaction does not own this state.

A fix is now prepared in the current 4.1.4.4 development candidate:

• snapshot projection is a separate policy from runtime-event compaction;
• execution.set_snapshot_projection_policy(terminal_value_mode="digest", min_value_bytes=4096) replaces eligible large terminal values with deterministic digest summaries;
• pending interrupts and active recovery boundaries retain complete values;
• save/load, restart, resume idempotency, and pending-payload recovery remain intact;
• the built-in local RecordStore keeps the latest three snapshot versions by default and exposes configurable retention and explicit prune controls;
• docs and regression coverage describe the ownership boundary and opt-in compatibility behavior.

Observed on the issue-shaped A/B reproduction:

Mode	Snapshot size
compatibility default	1,307,086 bytes
digest projection	106,782 bytes

That is a 91.83% reduction. The fixed development commit also passed Pyright with 0 errors, the complete local suite (2463 passed, 2 skipped), the durable-recovery example, wheel build, and a clean Python 3.10 wheel-install smoke test.

Known bounds remain explicit: projection is opt-in for compatibility, last_signal can still be large, and this release does not introduce a whole-snapshot byte budget or claim distributed-provider retention support.

The issue will remain open and will not be labeled solved until the development branch is synchronized and the release is actually published.

---

已确认与实现状态

该问题可以复现。我们确认终态 interrupt/resume 值会在多个框架持有的 snapshot 区段中重复保存，并且 runtime-event compaction 并不负责这部分状态。

修复现已准备在当前 4.1.4.4 开发候选中：

• snapshot projection 与 runtime-event compaction 是两套独立策略；
• execution.set_snapshot_projection_policy(terminal_value_mode="digest", min_value_bytes=4096) 会把符合条件的大型终态值替换成确定性的 digest 摘要；
• pending interrupt 与活动恢复边界仍保留完整值；
• save/load、重启、resume 幂等性和 pending payload 恢复保持完整；
• 内置本地 RecordStore 默认保留最近三个 snapshot 版本，并提供可配置 retention 与显式 prune；
• 文档和回归测试明确了 owner 边界及 opt-in 兼容行为。

在与 issue 同形的 A/B 复现中观测到：

模式	Snapshot 大小
兼容默认模式	1,307,086 bytes
digest projection	106,782 bytes

降幅为 91.83%。修复后的开发提交还通过了 Pyright 0 错误、完整本地测试（2463 passed, 2 skipped）、durable-recovery 示例、wheel 构建和全新 Python 3.10 环境的 wheel 安装 smoke test。

当前边界也保持明确：projection 为了兼容性仍是 opt-in；last_signal 仍可能较大；本次发布未引入整个 snapshot 的字节预算，也不宣称已支持分布式 provider 的 retention。

在开发分支完成同步且版本实际发布前，本 issue 会继续保持 open，不标记 solved。

[Maplemx]

Development branch synchronized

The #331 implementation is now on remote dev at 395ffa67 and is included in the draft 4.1.4.4 release PR: #332.

Release-candidate validation completed with:

• source Pyright: 0 errors;
• clean default suite: 2,438 passed and 27 skipped; the 25 maintainer-local spec-runner skips passed separately;
• the durable-recovery example passed load, retention, explicit prune, idempotent resume, and durable-event checks;
• wheel/sdist build and fresh Python 3.10 installed-package smoke passed;
• the issue-shaped A/B result remains 1,307,086 B in compatibility-default mode versus 106,782 B with digest projection, a 91.83% reduction.

The issue remains open and is not marked solved yet. After 4.1.4.4 is actually published and the installed package is verified, the release closeout will add the published version/tag and mark it solved.

---

开发分支已同步

#331 的实现现已推送到远端 dev（395ffa67），并进入 4.1.4.4 草稿发布 PR：#332。

Release candidate 已完成以下校验：

• 源码 Pyright：0 errors；
• 干净默认套件：2,438 passed、27 skipped；其中 25 项维护者本地 spec runner 检查已单独全部通过；
• durable-recovery 示例通过 load、retention、显式 prune、幂等 resume 和 durable event 检查；
• wheel/sdist 构建和全新 Python 3.10 安装包 smoke 通过；
• issue 同形 A/B 结果仍为兼容默认模式 1,307,086 B，对比 digest projection 106,782 B，减少 91.83%。

本 issue 目前继续保持 open，不标记 solved。待 4.1.4.4 实际发布并验证安装包后，release closeout 会补充正式版本/tag，并将其标记 solved。

[Maplemx]

Released in Agently 4.1.4.4

The fix is now publicly available in Agently v4.1.4.4 and on PyPI.

Release closeout verification used a fresh Python 3.10 virtual environment and installed agently==4.1.4.4 directly from PyPI with the package cache disabled. It confirmed:

• installed distribution version 4.1.4.4;
• packaged py.typed metadata;
• the public set_snapshot_projection_policy(...) API;
• the complete durable-recovery example, including save/load, latest-N retention, explicit prune, idempotent duplicate resume, and persisted completion events.

The issue-shaped A/B evidence remains 1,307,086 B in compatibility-default mode versus 106,782 B with digest projection, a 91.83% reduction. Projection remains opt-in for compatibility, and whole-snapshot byte ceilings plus distributed-provider retention remain explicitly deferred rather than claimed as solved here.

This issue is now marked solved and intentionally remains open under the repository's solved-issue visibility convention.

---

已在 Agently 4.1.4.4 发布

修复现已通过 Agently v4.1.4.4 和 PyPI 正式发布。

发布收尾验证在全新的 Python 3.10 虚拟环境中完成，并关闭包缓存、直接从 PyPI 安装 agently==4.1.4.4。验证确认：

• 安装包版本为 4.1.4.4；
• 包含 py.typed 类型元数据；
• 公开 set_snapshot_projection_policy(...) API 可用；
• 完整 durable-recovery 示例通过，包括 save/load、latest-N retention、显式 prune、重复 resume 幂等处理和完成事件持久化。

Issue 同形 A/B 证据仍为兼容默认模式 1,307,086 B，对比 digest projection 106,782 B，减少 91.83%。为了兼容性，projection 仍保持 opt-in；整个 snapshot 的字节上限和分布式 provider retention 继续明确延期，不在本 issue 中宣称已经解决。

本 issue 现标记为 solved，并按照仓库的 solved issue 可见性约定继续保持 open。