Stale Zombie Flows & Cleanup question

[jbw-vtl]

Bug summary

Hi team,

While going through our checks when upcoming from 3.4.0 to latest (3.4.12), we noticed the concurrency limits on our workpools did not seem to work as expected.
After some investigating, we noticed a lot of very old entries in the flow_run table still in state RUNNING.
This was causing the workpool to completely lock, which ended up blocking our queue.

Version info

3.4.12, however issue not linked to the version and been around for longer

Additional context

We found the documentation on zombie flows and will be exploring this
https://docs.prefect.io/v3/advanced/detect-zombie-flows

Considering this caused a total lock of the scheduler, is this planned to be directly integrated at some point?

Beyond the zombie flow cancelling, do you have general guidance for database cleanup when running open source?
I.e. recommended patterns to cleanup any flow runs & related artifacts after ~3 months.

We are considering running a custom job (either within or outside of prefect) that goes and runs a DELETE query on db with cascade, wondering whether there's a better way here.

[bdalpe]

Beyond the zombie flow cancelling, do you have general guidance for database cleanup when running open source?
I.e. recommended patterns to cleanup any flow runs & related artifacts after ~3 months.

We have some guidance in our documentation on regarding database maintenance for general cleanup: https://docs.prefect.io/v3/advanced/database-maintenance

[zzstoatzz]

hi @jbw-vtl - thanks for the write-up, I think your concerns are well-described by two existing issues

• Prevent Orphaned Slots in Concurrency Limits #17415
• Consider adding DB cleanup scripts to save space in Postgres #5813

as well as the "zombie" flow automation docs you linked and the maintenance docs linked above. would you like me to convert this to a discussion?

[jbw-vtl]

Definitely fine to convert into discussion

We will review everything and play around a bit to feed back what worked for us.