-
Notifications
You must be signed in to change notification settings - Fork 46
Write the crash group guide #1074
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
d892c68
Write crash group guide WiP
FelonEkonom 963d330
Add dupa test to debug crash group behaviour
FelonEkonom fde7789
Merge branch 'fix-connector-bug' into crash-groups-guide
FelonEkonom 75d8a08
WiP
FelonEkonom af625bf
Improve crash groups guide
FelonEkonom 479c829
Improve crash groups guide
FelonEkonom a4df817
Improve it once again
FelonEkonom 0a85933
Merge remote-tracking branch 'origin/master' into crash-groups-guide
FelonEkonom 766a75b
Remove leftover ;)
FelonEkonom 84f94c2
Update guides/useful_concepts/crash_groups.md
FelonEkonom 54c5016
Apply CR suggestions
FelonEkonom 4cebf40
One more fix
FelonEkonom f63a67f
Apply reviewr sugestion
FelonEkonom ba31ac4
Fix typo
FelonEkonom 9d569e7
Fix typo in README
FelonEkonom File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| # Crash Groups | ||
|
|
||
| Crash groups provide a mechanism to manage the lifecycle of children (elements or bins) within a pipeline when one of them fails. By grouping children together, you can ensure that a crash in one part of the pipeline triggers a coordinated restart or termination of related elements and bins, maintaining system consistency. | ||
|
|
||
| ## Overview | ||
|
|
||
| In Membrane, elements and bins are Elixir processes. By default, if a child (element or bin) that is not inside a crash group crashes, it leads to the crash of the whole pipeline. | ||
|
|
||
| The fundamental purpose of crash groups is to isolate the crash of a specific child from the rest of the pipeline. If an child is likely to crash (e.g., it interacts with unstable external resources), it is usually assigned to a crash group containing all children inextricably linked to its operation. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. | ||
|
|
||
| ## Defining Crash Groups | ||
|
|
||
| Crash groups are defined in the `spec` within your pipeline or bin. They are built upon the concept of **Children Groups**, which allow aggregating spawned children into easily identifiable groups. | ||
|
|
||
| To create a crash group, you must assign children to a group using the `group` option and set the `crash_group_mode` to `:temporary`. This turns a regular group into a crash group, enabling the crash handling behavior. | ||
|
|
||
| ```elixir | ||
| defmodule MyPipeline do | ||
| use Membrane.Pipeline | ||
|
|
||
| @impl true | ||
| def handle_init(_ctx, _opts) do | ||
| spec = | ||
| child(:source, MySource) | ||
| |> child(:filter, MyFilter) | ||
| |> child(:sink, MySink) | ||
|
|
||
| {[spec: {spec, group: :my_group, crash_group_mode: :temporary}], %{}} | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| In the example above, `:source`, `:filter`, and `:sink` are all assigned to the same group `:my_group`. Because `crash_group_mode` is set to `:temporary`, this group functions as a crash group. | ||
|
|
||
| ## Behaviour | ||
|
|
||
| When an child belonging to a crash group crashes: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "When a child" |
||
| 1. All other children in the same crash group are terminated by the pipeline. | ||
| 2. The pipeline's `c:Membrane.Pipeline.handle_crash_group_down/3` callback is invoked. | ||
|
varsill marked this conversation as resolved.
|
||
| 3. You can decide whether to restart the group, ignore the failure, or handle the situation in another way. | ||
|
|
||
| ## Flow of callbacks triggered by a crash within a crash group | ||
|
|
||
| Let's assume a `:filter` element spawned in `MyPipeline` raises an error with the message `"internal error"`. | ||
|
|
||
| ### Handling termination of the crash initiator | ||
|
|
||
| The first callback executed in `MyPipeline` will be: | ||
|
|
||
| ```elixir | ||
| @impl true | ||
| def handle_child_terminated(:filter, context, state) do | ||
| # ... | ||
| end | ||
| ``` | ||
|
|
||
| The `context` passed to this callback will contain a few extra fields: | ||
| * `context.exit_reason` - in this case, it equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. | ||
| * `context.group_name` - because `:filter` was spawned inside the `:my_group` group, this equals `:my_group`. If a child is spawned outside any crash group and terminates gracefully, the value of this field is `nil`. | ||
| * `context.crash_initiator` - the same as the child's reference, which is `:filter`. | ||
|
|
||
| Since `:filter` is no longer present in `context.children`, you could potentially respawn it here. However, it is recommended to do so later in `c:Membrane.Pipeline.handle_crash_group_down/3`. | ||
|
|
||
| ### Terminating other children within the failing crash group | ||
|
|
||
| Because one child from the crash group `:my_group` crashed ungracefully, the remaining children in that group will also be terminated. | ||
|
|
||
| Therefore, Membrane will terminate `:source` and `:sink` (in random order). After each termination, `MyPipeline` will execute the following callback: | ||
|
|
||
| ```elixir | ||
| @impl true | ||
| def handle_child_terminated(child, context, state) do | ||
| # ... | ||
| end | ||
| ``` | ||
|
|
||
| Each time, the `context` will contain the following extra fields: | ||
| * `context.exit_reason` - equals `{:shutdown, :membrane_crash_group_kill}`. | ||
| * `context.group_name` - equals `:my_group`. | ||
| * `context.crash_initiator` - equals `:filter`. | ||
|
|
||
| Note that the `context.children` map always contains only the children that are still alive. For example, if `:source` is terminated first, `handle_child_terminated(:source, context, state)` will contain only `:sink` in the `context.children` map. Subsequently, for `handle_child_terminated(:sink, context, state)`, `context.children` will be empty. | ||
|
|
||
| `MyPipeline` could potentially spawn children other than `:source`, `:filter`, and `:sink` - either in different crash groups or outside any crash group. In such cases, `context.children` would contain all of them normally, and these children would not be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of a child's crash to only those children within the same group. | ||
|
|
||
| ### Recovering from a crash group failure | ||
|
|
||
| Finally, when all members of the crash group are terminated, `MyPipeline` will execute: | ||
|
|
||
| ```elixir | ||
| @impl true | ||
| def handle_crash_group_down(:my_group, context, state) do | ||
| # ... | ||
| end | ||
| ``` | ||
|
|
||
| The `context` passed as the third argument to the `handle_crash_group_down/3` callback contains three additional fields: | ||
| - `context.crash_initiator` - the name or reference of the child that crashed first and caused the group to fail. In this case, it equals `:filter`. | ||
| - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. In this case, it equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. | ||
| - `context.members` - names/references of all children that were in the crash group. In this case, it equals `[:source, :filter, :sink]`. | ||
|
|
||
| When `handle_crash_group_down/3` is executed, you can be sure that all group members have already been terminated. This is the suggested place to recover from a group failer, e.g. by respawning all crash group members. Doing so in `handle_child_terminated/3` might lead to issues because the termination order of group members can vary. Moreover, if a pipeline or bin terminates its children gracefully (using the `t:Membrane.Pipeline.Action.remove_children()` action), the `c:Membrane.Pipeline.handle_child_terminated/3` callback will also be executed, but with `context.exit_reason` set to `normal`. | ||
|
|
||
|
|
||
| ## Callback Contexts | ||
|
|
||
| For more information about callback contexts, refer to the documentation for `t:Membrane.Pipeline.CallbackContext.t()`, `t:Membrane.Bin.CallbackContext.t()`, and `t:Membrane.Element.CallbackContext.t()`. | ||
|
|
||
| ## Bins | ||
|
|
||
| Although the example above demonstrates using crash groups within a `Membrane.Pipeline`, they function in the same way within a `Membrane.Bin`. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.