Eager commit cache transaction

[rama-reddy]

Hi,

I'm working on a use case where I want to eagerly cache nested tasks in Prefect. I've found that setting the transaction commit mode to CommitMode.EAGER helps achieve this, and also noticed that when I set a parent_task to eager, all nested child tasks inherit this behavior.

I'm aware that this will cause issues if any of the nested tasks return non-serializable values or futures, which will fail during caching.

I'm using DaskTaskRunner for distributed execution, and I'm wondering:

Are there any other side effects or caveats I should be aware of when using eager commit mode in this setup?

Appreciate any insights. Thanks.

[cicdw]

Hey @rama-reddy thanks for the question - the only other caveat is that eager commit mode increases the chances of cache hits within your workflow. Here's what I mean:

with transaction(commit_mode=CommitMode.LAZY):
    sub_task(2) 
    sub_task(2)

In this example, sub_task will execute twice even though it has been provided the same inputs. This is because its result will only commit to result storage once the parent transaction commits.

with transaction(commit_mode=CommitMode.EAGER):
    sub_task(2) 
    sub_task(2)

In this example, sub_task will only execute once since it commits its result to disk immediately.

So as long as you're aware of that possibility (and it depends on your caching configuration too!), that's the only other change in behavior that I can think of.