Allow for pipelines to ignore error and continue. #679
Conversation
J08nY
force-pushed
the
feat/pipeline-errorok
branch
4 times, most recently
from
ef3e3d4 to
b941324
Compare
J08nY
force-pushed
the
feat/pipeline-errorok
branch
from
b941324 to
021e6fc
Compare
dramatiq/middleware/pipelines.py
Outdated
| if exception is not None or message.failed: | ||
| actor = broker.get_actor(message.actor_name) | ||
|
|
||
| errorok = message.options.get("pipe_ignore_exception") or actor.options.get("pipe_ignore_exception") |
There was a problem hiding this comment.
@J08nY I'd rename the left-side to ignore_exception
dramatiq/middleware/pipelines.py
Outdated
| actor = broker.get_actor(message.actor_name) | ||
|
|
||
| errorok = message.options.get("pipe_ignore_exception") or actor.options.get("pipe_ignore_exception") | ||
| if (exception is not None or message.failed) and not errorok: |
There was a problem hiding this comment.
@J08nY exception is not None and message.failed shouldn't be considered equal - they may have different meanings in the actor implementation. Only explicitly thrown exceptions should be caught.
I'd also go for explicit .. and ignore_exception is not True
There was a problem hiding this comment.
That is why I had the name "errorok", to consider both cases of message failures. This can of course be split, but that requires a new parameter.
| has_run = True | ||
|
|
||
| # When I pipe some messages intended for that actor together and run the pipeline | ||
| pipe = do_nothing.message_with_options(pipe_ignore_exception=True) | should_run.message_with_options(pipe_ignore=True) |
There was a problem hiding this comment.
@J08nY with the above, the changes would require a redo
|
Looking at the code, if we allow pipelines to continue when Which could then mean the pipeline would be continued multiple times? each time the message is re-tried (default 20 times). This is obviously not good. |
I see. Would you mind including it somewhere in the README? I mean like a recommendation to try to use middleware for custom features, or make an issue if it is not possible and that new features are unlikely to get accepted. Would save time.
Hmm, I guess my understanding of what the different states can be and how they are handled is lacking. Would you suggest subclassing the Pipeline middleware and replacing the built-in one? In my use-cases I do not have/want retries and thus it is much simpler. |
|
@LincolnPuzey @Bogdanp gave me the go-ahead. Let's have a private conversation. @J08nY, I apologize for the poor communication. |
J08nY
force-pushed
the
feat/pipeline-errorok
branch
from
021e6fc to
3a31744
Compare
|
@synweap15 My apologies, I didn't know. Do you agree that continuing a pipeline multiple times due to retries, is something we should avoid? |
|
Are there some semantics of this concept that you want to ignore exceptions in an actor in a pipeline, that work together with retries? The only thing I can think of is to make the actor eat the exception and thus effectively disable retries. |
It is useful for me to have a pipeline of messages that execute. However, I want to keep executing the pipeline even if the individual message fails. This pull-request implements that using a new option on the message in the Pipelines middleware.