Add failure-isolated loop iterations - #377
Merged
Merged
Conversation
Distinguish body activity failures from child runtime and protocol faults so continue-on-failure loops only consume expected application failures. Reject conditional loops combined with failure continuation, document the new loop policy and lifetime, and pin deterministic child input serialization.
Todd J. Green (tjgreen42)
marked this pull request as ready for review
September 7, 2026 02:34
Pino de Candia (pinodeca)
assigned
Thom Chiovoloni (thomcc-work) and unassigned
Thom Chiovoloni (thomcc-work)
Pino de Candia (pinodeca)
requested a review
from Thom Chiovoloni (thomcc-work)
September 8, 2026 19:31
Thom Chiovoloni (thomcc-work)
left a comment
Thom Chiovoloni (thomcc-work)
left a comment
Contributor
There was a problem hiding this comment.
I'm still not at the point of pg_durable understanding where I can confidently comment as to whether the change makes sense to do it this way. Really, the only thing I can comment on is general style/approach in the abstract. Because of this, I'm just going to submit this review as "comment" rather than "approving" but it seems mostly viable (assuming you look at the genuine problems and either address them, or don't think we need to).
I am glad I took a look though, since I hadn't considered handling signature changes that way. It's still not practical for cases like mine, (df.http is GRANT city, and it's apparently hard to migrate those properly), but it's a really good technique to know.
I also asked a robot to take a look at it on my end, and it surfaced two things, both of which seem to make sense to me, but I couldn't say for sure.
Thom Chiovoloni (thomcc-work)
added a commit
to thomcc-work/pg_durable
that referenced
this pull request
Sep 10, 2026
Several of my tasks require passing additional options to `df.http` and `df.http_multipart`. Unfortunately, adding new parameters to these functions is Hard. The normal approach (for example taken in microsoft#377 with the `df.loop`) is to keep the old function around, renamed (but with the same wrapper name). This is fine (and even avoids issues when `ALTER EXTENSION UPDATE` is not run), but `df.http` and `df.http_multipart` are functions that will likely have a bunch of `GRANT`s (and those grants are semantically meaningful to `pg_durable` beyond our ability to call the functions). If we try to capture and re-issue[^1] those grants from an extension, PG will record the grants as coming from the extension update script, and assume it doesn't need to provide them in pg_dump, so then the pg_restore won't have them, meaning logical restore and/or PG upgrades will be broken. The AI suggested the right approach was some catalog feng shui but that it would take a while to engineer. I asked on the PostgreSQL discord, and one of the PG committers (rhass) told me that this (trying to copy grants from one function to another) was something that you should never do, and to just have users reissue the grants on update. So... instead of that, we just add a combinator function that manipulates the durofut JSON directly to add the options. For example, you'd use it like: ```sql df.with_http_options( df.http(...), '{"options": "here"}'::jsonb ); ``` This admittedly is less ergonomic than adding an `options =>` parameter for `df.http`, but... well, yeah. If we want, we could make this into an operator, e.g. allowing `df.http(...) <some-operator> '{"options": "here"}'` or something like that? I don't have strong feelings. [^1]: This is ignoring the fact that the permissions may be different now, for example even ignoring the next issue this won't work quite right if a delegator of a grant became a superuser.
Pino de Candia (pinodeca)
approved these changes
Sep 10, 2026
Pino de Candia (pinodeca)
approved these changes
Sep 10, 2026
Thom Chiovoloni (thomcc-work)
added a commit
to thomcc-work/pg_durable
that referenced
this pull request
Sep 10, 2026
Several of my tasks require passing additional options to `df.http` and `df.http_multipart`. Unfortunately, adding new parameters to these functions is Hard. The normal approach (for example taken in microsoft#377 with the `df.loop`) is to keep the old function around, renamed (but with the same wrapper name). This is fine (and even avoids issues when `ALTER EXTENSION UPDATE` is not run), but `df.http` and `df.http_multipart` are functions that will likely have a bunch of `GRANT`s (and those grants are semantically meaningful to `pg_durable` beyond our ability to call the functions). If we try to capture and re-issue[^1] those grants from an extension, PG will record the grants as coming from the extension update script, and assume it doesn't need to provide them in pg_dump, so then the pg_restore won't have them, meaning logical restore and/or PG upgrades will be broken. The AI suggested the right approach was some catalog feng shui but that it would take a while to engineer. I asked on the PostgreSQL discord, and one of the PG committers (rhass) told me that this (trying to copy grants from one function to another) was something that you should never do, and to just have users reissue the grants on update. So... instead of that, we just add a combinator function that manipulates the durofut JSON directly to add the options. For example, you'd use it like: ```sql df.with_http_options( df.http(...), '{"options": "here"}'::jsonb ); ``` This admittedly is less ergonomic than adding an `options =>` parameter for `df.http`, but... well, yeah. If we want, we could make this into an operator, e.g. allowing `df.http(...) <some-operator> '{"options": "here"}'` or something like that? I don't have strong feelings. [^1]: This is ignoring the fact that the permissions may be different now, for example even ignoring the next issue this won't work quite right if a delegator of a grant became a superuser.
Thom Chiovoloni (thomcc-work)
added a commit
to thomcc-work/pg_durable
that referenced
this pull request
Sep 10, 2026
Several of my tasks require passing additional options to `df.http` and `df.http_multipart`. Unfortunately, adding new parameters to these functions is Hard. The normal approach (for example taken in microsoft#377 with the `df.loop`) is to keep the old function around, renamed (but with the same wrapper name). This is fine (and even avoids issues when `ALTER EXTENSION UPDATE` is not run), but `df.http` and `df.http_multipart` are functions that will likely have a bunch of `GRANT`s (and those grants are semantically meaningful to `pg_durable` beyond our ability to call the functions). If we try to capture and re-issue[^1] those grants from an extension, PG will record the grants as coming from the extension update script, and assume it doesn't need to provide them in pg_dump, so then the pg_restore won't have them, meaning logical restore and/or PG upgrades will be broken. The AI suggested the right approach was some catalog feng shui but that it would take a while to engineer. I asked on the PostgreSQL discord, and one of the PG committers (rhass) told me that this (trying to copy grants from one function to another) was something that you should never do, and to just have users reissue the grants on update. So... instead of that, we just add a combinator function that manipulates the durofut JSON directly to add the options. For example, you'd use it like: ```sql df.with_http_options( df.http(...), '{"options": "here"}'::jsonb ); ``` This admittedly is less ergonomic than adding an `options =>` parameter for `df.http`, but... well, yeah. If we want, we could make this into an operator, e.g. allowing `df.http(...) <some-operator> '{"options": "here"}'` or something like that? I don't have strong feelings. [^1]: This is ignoring the fact that the permissions may be different now, for example even ignoring the next issue this won't work quite right if a delegator of a grant became a superuser.
Thom Chiovoloni (thomcc-work)
added a commit
to thomcc-work/pg_durable
that referenced
this pull request
Sep 10, 2026
Several of my tasks require passing additional options to `df.http` and `df.http_multipart`. Unfortunately, adding new parameters to these functions is Hard. The normal approach (for example taken in microsoft#377 with the `df.loop`) is to keep the old function around, renamed (but with the same wrapper name). This is fine (and even avoids issues when `ALTER EXTENSION UPDATE` is not run), but `df.http` and `df.http_multipart` are functions that will likely have a bunch of `GRANT`s (and those grants are semantically meaningful to `pg_durable` beyond our ability to call the functions). If we try to capture and re-issue[^1] those grants from an extension, PG will record the grants as coming from the extension update script, and assume it doesn't need to provide them in pg_dump, so then the pg_restore won't have them, meaning logical restore and/or PG upgrades will be broken. The AI suggested the right approach was some catalog feng shui but that it would take a while to engineer. I asked on the PostgreSQL discord, and one of the PG committers (rhass) told me that this (trying to copy grants from one function to another) was something that you should never do, and to just have users reissue the grants on update. So... instead of that, we just add a combinator function that manipulates the durofut JSON directly to add the options. For example, you'd use it like: ```sql df.with_http_options( df.http(...), '{"options": "here"}'::jsonb ); ``` This admittedly is less ergonomic than adding an `options =>` parameter for `df.http`, but... well, yeah. If we want, we could make this into an operator, e.g. allowing `df.http(...) <some-operator> '{"options": "here"}'` or something like that? I don't have strong feelings. [^1]: This is ignoring the fact that the permissions may be different now, for example even ignoring the next issue this won't work quite right if a delegator of a grant became a superuser.
Thom Chiovoloni (thomcc-work)
added a commit
to thomcc-work/pg_durable
that referenced
this pull request
Sep 10, 2026
Several of my tasks require passing additional options to `df.http` and `df.http_multipart`. Unfortunately, adding new parameters to these functions is Hard. The normal approach (for example taken in microsoft#377 with the `df.loop`) is to keep the old function around, renamed (but with the same wrapper name). This is fine (and even avoids issues when `ALTER EXTENSION UPDATE` is not run), but `df.http` and `df.http_multipart` are functions that will likely have a bunch of `GRANT`s (and those grants are semantically meaningful to `pg_durable` beyond our ability to call the functions). If we try to capture and re-issue[^1] those grants from an extension, PG will record the grants as coming from the extension update script, and assume it doesn't need to provide them in pg_dump, so then the pg_restore won't have them, meaning logical restore and/or PG upgrades will be broken. The AI suggested the right approach was some catalog feng shui but that it would take a while to engineer. I asked on the PostgreSQL discord, and one of the PG committers (rhass) told me that this (trying to copy grants from one function to another) was something that you should never do, and to just have users reissue the grants on update. So... instead of that, we just add a combinator function that manipulates the durofut JSON directly to add the options. For example, you'd use it like: ```sql df.with_http_options( df.http(...), '{"options": "here"}'::jsonb ); ``` This admittedly is less ergonomic than adding an `options =>` parameter for `df.http`, but... well, yeah. If we want, we could make this into an operator, e.g. allowing `df.http(...) <some-operator> '{"options": "here"}'` or something like that? I don't have strong feelings. [^1]: This is ignoring the fact that the permissions may be different now, for example even ignoring the next issue this won't work quite right if a delegator of a grant became a superuser.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Long-running scheduled workflows should survive an isolated iteration failure instead of terminating permanently. This is motivated by the pg_textsearch compaction backstop in timescale/pg_textsearch#476, where one failed compaction must not prevent later scheduled attempts.
Summary
df.loop(body, condition DEFAULT NULL, continue_on_failure DEFAULT false)API for infinite and conditional loopsJOINsibling failures overdf.break()and recoverable activity failures inside failure-isolated iterations, while preserving historical first-error behavior elsewhere2^23), approximately 80 years at five-minute intervalsUpgrade note
Histories below iteration 100,000 replay unchanged. A loop that already recorded the old terminal-failure path at that boundary cannot replay under the new binary; drain such long-running loops before upgrading when continuity is required.
Testing