Prefect crashes my services

[slavaGanzin]

Bug summary

Hello, I like prefect very much and it's a huge step up from airflow-like garbage.

But there is a problem with python "let it fall" approach, because in contrast with Erlang were this idea emerged, Python doesn't work on the actors and messages concept because it is tragically synchronous (hello GIL and "asynchronous" asyncio library).

And also there is a problem with the fact that prefect is observability platform, it observes if something is working or not and that's it. You may think that by wrapping it into flow and task, you became a task workflow orchestration mumbo jumbo, but it's not: my code run some tasks, my code do some job, prefect - observe.

So in general, I don't care that much if prefect "Couln't authorize", "You limit is reached" or any other problems with prefect, if my tasks continue working. But I found out, that prefect have such inflated ego, that it crashes my services, because it couldn't login. Are you serious?

And you know what happens? I cannot get alerts from my prefect because my service couldn't log in to indicate that it failed. So I need to periodically check manually if my jobs are working fine. Then I may ask, for what reason do I have an observability tool.

Maybe if one of my services couldn't login, you may show me that in prefect UI and not silently crash my service? Just a suggestion

You need to think about it. But here is a quick fix.

1. Stop throwing exceptions outside if prefect cannot authorize, synchronize, find closing part of a log entry, show warnings in log and that's it
2. If prefect, couldn't authorize, synchronize or dance inside itself in the way it likes, continue working as everything was fine. I still need my jobs to working properly

You are digging a hole under yourself, unreliable workflow orchestration is worse than having none. I recommended prefect numerous times as a go to substitution for airflow and others, and now I feel uneasy about it

Version info

I don't use prefect locally on my machine, but it is what it is

$ prefect version

Traceback (most recent call last):
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn
    return asyncio.run(async_fn(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/site-packages/prefect/cli/root.py", line 142, in version
    integrations = get_prefect_integrations()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/slava/.asdf/installs/python/3.12.4/lib/python3.12/site-packages/prefect/cli/root.py", line 155, in get_prefect_integrations
    if dist.metadata["Name"].startswith("prefect-"):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'
An exception occurred.

Additional context

No response

[zzstoatzz]

hi @slavaGanzin - thanks for sharing your thoughts!

I converted this to a discussion because there wasn't a specific request here. I understand your concerns with respect to "interrupting real work with bookkeeping" etc but its most constructive if issues remain well-scoped to concrete bugs or enhancements. Discussions are great for getting your thoughts out and potentially catalyzing more specific issues.

In general we do attempt to avoid needless disruption of user code, so if you have specific examples where this is not the case (preferably with MREs!), we'd be happy to see them!

additionally, the error you encountered with prefect version should be addressed here #18190

[slavaGanzin]

Hi, Nate.

I don't notice that you use discussions in this repo. Makes sense
Thanks for transferring, though

Easiest issue to spot on: I incidentally deleted my api key, and all my jobs went down, because obviously it couldn't authorize. Totally understandable, but damn it was frustrating

I do kinda get feeling that most of the people use prefect as runner for their non critical work, but I use it for the most important processes in entire project, so it literally "broke production".

And it happens as well when I reach my limits, that maybe short lived downtime and maybe not. Can't understand cause/impact. Feels like magic. And that's a bad magic

Prefect cloud limits is not accustomed to spike usage. I totally adore your idea of having capped plans with transparent prices, it's not that common in a world of Datadogs, where they almost push you to unforced errors, that cost you thousands; but sometimes pay-as-you-go is what you needed

I initially tried to use it as generic queue observability (because I like your approach/Ui a lot more than competition) it failed miserably, because of those limits. So now I have additional otel just for that, but their tricky approaches with doing simple(spans and attributes) things remind me of Airflow. And as I know, we share that common trauma of using Airflow ;)

[slavaGanzin]

I'm open to pay, but this limitations pushes me to self-hosting

[slavaGanzin]

Again, I don't care about prefect being unable to parse my logs, why should my logs contain this?

--- Logging error ---
Traceback (most recent call last):
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/logging/handlers.py", line 337, in emit
self.console.print(message, soft_wrap=True)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/rich/console.py", line 1693, in print
renderables = self._collect_renderables(
objects,
...<5 lines>...
highlight=highlight,
)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/rich/console.py", line 1553, in _collect_renderables
self.render_str(
~~~~~~~~~~~~~~~^
renderable,
^^^^^^^^^^^
...<3 lines>...
highlighter=_highlighter,
^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/rich/console.py", line 1443, in render_str
rich_text = render_markup(
text,
...<2 lines>...
emoji_variant=self._emoji_variant,
)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/rich/markup.py", line 167, in render
raise MarkupError(
f"closing tag '{tag.markup}' at position {position} doesn't match any open tag"
) from None
rich.errors.MarkupError: closing tag '[/bin/sh -c pytest && python3 -c "from helpmetest import Help_Me_Test; print('Library import successful')" && robot --variable "RUN_ID:test__test__2024-08-03T17:17:59.703Z" --console NONE --listener ./listener.py --outputdir /tmp --exclude skip_on_build robottests/*.robot]' at position 50 doesn't match any open tag
Call stack:
File "/Users/slava/work/helpmetest/cli-helpmetest", line 1647, in
cli()
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 1161, in call
return self.main(*args, **kwargs)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
File "/Users/slava/work/helpmetest/cli-helpmetest", line 1144, in dev
run(
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/tasks.py", line 1130, in call
return run_task(
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/task_engine.py", line 1714, in run_task
return run_task_sync(**kwargs)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/task_engine.py", line 1487, in run_task_sync
engine.call_task_fn(txn)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/task_engine.py", line 874, in call_task_fn
result = call_with_parameters(self.task.fn, parameters)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/utilities/callables.py", line 210, in call_with_parameters
return fn(*args, **kwargs)
File "/Users/slava/work/helpmetest/cli-helpmetest", line 239, in run
out(line)
File "/Users/slava/work/helpmetest/cli-helpmetest", line 213, in out
log.info(trim(message))
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1919, in info
self.log(INFO, msg, *args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1957, in log
self.logger.log(level, msg, args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1592, in log
self._log(level, msg, args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1667, in _log
self.handle(record)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1686, in handle
self.callHandlers(record)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1744, in callHandlers
hdlr.handle(record)
File "/opt/homebrew/Cellar/[email protected]/3.13.5/Frameworks/Python.framework/Versions/3.13/lib/python3.13/logging/init.py", line 1027, in handle
self.emit(record)
File "/Users/slava/work/helpmetest/robot/.venv/lib/python3.13/site-packages/prefect/logging/handlers.py", line 343, in emit
self.handleError(record)
Message: '\x1b[35mbuild:robot \x1b[0m\x1b[36mINFO\x1b[0m[0104] Running: [/bin/sh -c pytest && python3 -c "from helpmetest import Help_Me_Test; print('Library import successful')" && robot --variable "RUN_ID:test__test__2024-08-03T17:17:59.703Z" --console NONE --listener ./listener.py --outputdir /tmp --exclude skip_on_build robottests/.robot]'