From d892c687f36518b403b70e7216a0f8ec2dfbfa48 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Wed, 18 Feb 2026 15:47:23 +0100 Subject: [PATCH 01/13] Write crash group guide WiP --- guides/useful_concepts/crash_groups.md | 73 ++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 guides/useful_concepts/crash_groups.md diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md new file mode 100644 index 000000000..1f2e93b2a --- /dev/null +++ b/guides/useful_concepts/crash_groups.md @@ -0,0 +1,73 @@ +# Crash groups + +Crash groups provide a mechanism to manage the lifecycle of elements within a pipeline when one of them fails. By grouping elements together, you can ensure that a crash in one part of the pipeline triggers a coordinated restart or termination of related elements, maintaining system consistency. + +## Overview + +In Membrane, elements and bins are Elixir processes. By default, if an element that is not inside a crash group crashes, it leads to the crash of the whole pipeline. + +The most fundamental functionality of crash groups is to separate the crash of a specific element from the rest of the pipeline. Usually, if an element is likely to crash (e.g. it interacts with unstable external resources), it is placed in a crash group along with other elements whose functioning is inextricably connected to it. This prevents a localized failure from bringing down the entire system and allows for controlled recovery of specific logical units. By doing so, it also cleans up components that would have to be restarted or killed anyway. + +## Defining Crash Groups + +Crash groups are defined in the `spec` within your pipeline or bin. You assign a crash group ID to a set of children. + +```elixir +defmodule MyPipeline do + use Membrane.Pipeline + + @impl true + def handle_init(_ctx, _opts) do + spec = + child(:source, MySource) + |> child(:filter, MyFilter) + |> child(:sink, MySink) + + {[spec: {spec, crash_group: :audio_processing}], %{}} + end +end +``` + +In the case above, the crash group ID is `:audio_processing`. + +## Behavior + +When an element belonging to a crash group crashes: +1. All other elements in the same crash group are terminated by the pipeline. +2. The pipeline's `handle_crash_group_down/3` callback is invoked. +3. You can decide whether to restart the group, ignore the failure, or terminate the pipeline. + +## Handling Failures + +To react to a crash group failure, implement the `handle_crash_group_down/3` callback: + +```elixir +@impl true +def handle_crash_group_down(crash_group_id, context, state) do + # Logic to restart the group or handle the error + {[], state} +end +``` + +`context` passed to `handle_crash_group_down/3` callback contiains 3 additional fields, that usually don't occur in contexts of other callbacks: + - `context.crash_initiator` - name or reference of the child that crashed, what caused crash group to explode. + - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. + - `context.members` - names/references of all children that were in the crash group. + +Question marks: + - is crash group initiator in context.members? + - what is the order in of: + * `handle_crash_group_down` + * `handle_child_terminated` + * removing children from `context.children`, so that it becames possible to respawn new children with the same names? + - what is the relation between `handle_child_pad_removed` and `handle_crash_group_down`? + + + +## Use Cases +## tutaj poniej mamy AI BS, no chodzi o to ze jak mamy element co sie wydupca, to nie chcemy zeby wszystko poszlo w piach, wiec wrzucamy go (i byc moze cos co i tak bysmy chcieli razem z nim zrestartowac) do crash groupy + +- **Atomic logical units:** When a group of elements (like an encoder and its associated parser) cannot function independently. +- **Resource Cleanup:** Ensuring that if a consumer crashes, the producer is also stopped to prevent buffered data from leaking memory. +- **Error Recovery:** Grouping elements that require a specific initialization sequence that must be repeated upon failure. + From 963d3300665f7278e922795d420d5cf9fef868f6 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Wed, 18 Feb 2026 16:52:56 +0100 Subject: [PATCH 02/13] Add dupa test to debug crash group behaviour --- test/membrane/dupa_test.exs | 84 +++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 test/membrane/dupa_test.exs diff --git a/test/membrane/dupa_test.exs b/test/membrane/dupa_test.exs new file mode 100644 index 000000000..1ae1700fa --- /dev/null +++ b/test/membrane/dupa_test.exs @@ -0,0 +1,84 @@ +defmodule Membrane.DupaTest do + use ExUnit.Case, async: true + + import Membrane.ChildrenSpec + import Mock + + alias Membrane.Core.Element.DiamondDetectionController.DiamondLogger + alias Membrane.Core.Element.DiamondDetectionController.PathInGraph.Vertex + alias Membrane.Testing + + require Membrane.Pad, as: Pad + + defmodule MockSource do + use Membrane.Source + + def_output_pad :output, + accepted_format: _any, + flow_control: :push + end + + defmodule MyPipeline do + use Membrane.Pipeline + + alias Membrane.Testing + alias Membrane.DupaTest.MockSource + + @impl true + def handle_init(_ctx, _opts) do + crash_group_spec = { + for i <- 2..5 do + child({:connector, i}, Membrane.Connector) + end, + group: :my_group, crash_group_mode: :temporary + } + + children_beyond_crash_group = [ + child(:source, MockSource) + |> child({:connector, 1}, Membrane.Connector), + child({:connector, 6}, Membrane.Connector) + |> child(:sink, Testing.Sink) + ] + + connector_links = + for i <- 1..5 do + get_child({:connector, i}) + |> get_child({:connector, i + 1}) + end + + spec = [crash_group_spec, children_beyond_crash_group, connector_links] + + {[spec: spec], %{}} + end + + @impl true + def handle_child_terminated(child, context, state) do + {child, context} |> IO.inspect(label: "HANDLE CHILD TERMINATED", limit: :infinity) + {[], state} + end + + @impl true + def handle_crash_group_down(group_name, context, state) do + {group_name, context} |> IO.inspect(label: "HANDLE CRASH GROUP DOWN", limit: :infinity) + {[], state} + end + + @impl true + def handle_child_pad_removed(child, pad, context, state) do + {child, pad, context} |> IO.inspect(label: "HANDLE CHILD PAD REMOVED", limit: :infinity) + {[], state} + end + end + + test "dupa" do + pipeline = Testing.Pipeline.start_link_supervised!(module: MyPipeline) + + Process.sleep(1500) + + {:ok, connector_pid} = Testing.Pipeline.get_child_pid(pipeline, {:connector, 3}) + Process.exit(connector_pid, :kill) + + Process.sleep(1500) + Testing.Pipeline.terminate(pipeline) + end +end From 75d8a0828ecd0e3923e6d9f2fac93c0c437d3ac5 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Mon, 23 Feb 2026 18:03:09 +0100 Subject: [PATCH 03/13] WiP --- CHANGELOG.md | 2 +- guides/useful_concepts/crash_groups.md | 69 +++++++++++++++++++++++--- test/membrane/dupa_test.exs | 28 +++++++++-- 3 files changed, 86 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 53fbd91c6..aa38b3e5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # Changelog ## Latest -* Handle removed pads properly in `Membrane.Connector` [#1075](https://github.com/membraneframework/membrane_core/pull/1075)/ +* Handle removed pads properly in `Membrane.Connector` [#1075](https://github.com/membraneframework/membrane_core/pull/1075) * Improve remove_link action docs * Deprecate `:components` option for `:unsafely_name_processes_for_observer` * Deprecate `:links` option for `:unsafely_name_processes_for_observer` in favour of `:report_links_to_observer` configuration entry diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index 1f2e93b2a..3cd51e784 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -10,7 +10,7 @@ The most fundamental functionality of crash groups is to separate the crash of a ## Defining Crash Groups -Crash groups are defined in the `spec` within your pipeline or bin. You assign a crash group ID to a set of children. +Crash groups are defined in the `spec` within your pipeline or bin. To create a crash group, you have to define a group name and set `crash_group_mode` to `:temporary`, like in the example below: ```elixir defmodule MyPipeline do @@ -23,13 +23,11 @@ defmodule MyPipeline do |> child(:filter, MyFilter) |> child(:sink, MySink) - {[spec: {spec, crash_group: :audio_processing}], %{}} + {[spec: {spec, group: :my_group, crash_group_mode: :temporary}], %{}} end end ``` -In the case above, the crash group ID is `:audio_processing`. - ## Behavior When an element belonging to a crash group crashes: @@ -45,14 +43,71 @@ To react to a crash group failure, implement the `handle_crash_group_down/3` cal @impl true def handle_crash_group_down(crash_group_id, context, state) do # Logic to restart the group or handle the error - {[], state} end ``` -`context` passed to `handle_crash_group_down/3` callback contiains 3 additional fields, that usually don't occur in contexts of other callbacks: - - `context.crash_initiator` - name or reference of the child that crashed, what caused crash group to explode. +## Flow of all callbacks reated to a crash within a crash group. + +Let's assume, that in `:filter` spawned in `MyPipeline` raised with message `"internal error"`. + +### Handling termination of the crash itiator + +The first callback, that will be executed in `MyPipeline`, is + +```elixir +@impl true +def handle_child_terminated(:filter, context, state) do + # ... +end +``` + +`context` passed to this callback will contain few extra fileds: + * `context.exit_reason` - in this case equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. + * `context.group_name` - because `:filter` was spawned inside `:my_group` group, it equals `:my_group`. If a child is spawned beyond any crash group and is terminated gracefully, value of this field is `nil`. + * `context.crash_initiator` - the same as child's reference, that is `:filter`. + +`:filter` won't be present in `context.children`, so you could respawn it here, however it is suggested to do it in `handle_crash_group_down/3` later. + +### Terminating other children within the crash group that explodes + +Because one of children from crash group `:my_group` ungracefully crashed, the rest of children from this group will be terminated as well. + +Therefore, Membrane will terminate `:source` and `:sink` in random order. After each termination, `MyPipeline` will execute + +```elixir +@impl true +def handle_child_terminated(child, context, state) do + # ... +end +``` + +callback. Each time, `context` will contain following extra fields: + * `context.exit_reason` - for these terminations, equals `{:shutdown, :membrane_crash_group_kill}`. + * `context.group_name` - equals `:my_group`. + * `context.crash_initiator` - TODO: continue + + +Note, that `context.children` map always contains only children that are still alive. E.g. if `:source` is terminated first, `hanlde_child_terminated(:source, context, state)` will contain only `:sink` in `context.children` map and for `hanlde_child_terminated(:sink, context, state)` `context.children` will be empty. + +Of course, `MyPipeline` could possibly spawn another children beyond `:source`, `:filter` and `:sink`, inside different crash groups or beyond any crash group. In such a case, `context.children` would contain all of them normally and these children wouldn't be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of the child's crash only to children with the same crash group. + +### Handling the crash group down + +Then, when all members of a crash group are terminated, `MyPipeline` will execute + +```elixir +@impl true +def handle_crash_group_down(:my_group, context, state) do + # ... +end +``` + +`context` passed as a third argument to `handle_crash_group_down/3` callback contains 3 additional fields, that usually don't occur in contexts of other callbacks: + - `context.crash_initiator` - name or reference of the child that crashed first and caused the crash group to explode. - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. - `context.members` - names/references of all children that were in the crash group. +TODO: continue + Question marks: - is crash group initiator in context.members? diff --git a/test/membrane/dupa_test.exs b/test/membrane/dupa_test.exs index 1ae1700fa..2d61c58e8 100644 --- a/test/membrane/dupa_test.exs +++ b/test/membrane/dupa_test.exs @@ -18,17 +18,33 @@ defmodule Membrane.DupaTest do flow_control: :push end + defmodule MockFilter do + use Membrane.Filter + + def_input_pad :input, + accepted_format: _any + + def_output_pad :output, + accepted_format: _any + + @impl true + def handle_buffer(_pad, _buffer, _ctx, state), do: {[], state} + + @impl true + def handle_parent_notification(:dupa, _ctx, state), do: raise("internal error") + end + defmodule MyPipeline do use Membrane.Pipeline alias Membrane.Testing - alias Membrane.DupaTest.MockSource - + alias Membrane.DupaTest.{MockFilter, MockSource} @impl true def handle_init(_ctx, _opts) do crash_group_spec = { for i <- 2..5 do - child({:connector, i}, Membrane.Connector) + child_def = if i == 3, do: MockFilter, else: Membrane.Connector + child({:connector, i}, child_def) end, group: :my_group, crash_group_mode: :temporary } @@ -75,8 +91,10 @@ defmodule Membrane.DupaTest do Process.sleep(1500) - {:ok, connector_pid} = Testing.Pipeline.get_child_pid(pipeline, {:connector, 3}) - Process.exit(connector_pid, :kill) + # {:ok, connector_pid} = Testing.Pipeline.get_child_pid(pipeline, {:connector, 3}) + # Process.exit(connector_pid, :kill) + + Testing.Pipeline.notify_child(pipeline, {:connector, 3}, :dupa) Process.sleep(1500) Testing.Pipeline.terminate(pipeline) From af625bfecd73edc676010b8c082125ceae475022 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 12:32:37 +0100 Subject: [PATCH 04/13] Improve crash groups guide --- guides/useful_concepts/crash_groups.md | 48 +++++--------------------- 1 file changed, 9 insertions(+), 39 deletions(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index 3cd51e784..3e17141d3 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -33,18 +33,7 @@ end When an element belonging to a crash group crashes: 1. All other elements in the same crash group are terminated by the pipeline. 2. The pipeline's `handle_crash_group_down/3` callback is invoked. -3. You can decide whether to restart the group, ignore the failure, or terminate the pipeline. - -## Handling Failures - -To react to a crash group failure, implement the `handle_crash_group_down/3` callback: - -```elixir -@impl true -def handle_crash_group_down(crash_group_id, context, state) do - # Logic to restart the group or handle the error -end -``` +3. You can decide whether to restart the group, ignore the failure, or handle this situation any other way. ## Flow of all callbacks reated to a crash within a crash group. @@ -64,7 +53,7 @@ end `context` passed to this callback will contain few extra fileds: * `context.exit_reason` - in this case equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. * `context.group_name` - because `:filter` was spawned inside `:my_group` group, it equals `:my_group`. If a child is spawned beyond any crash group and is terminated gracefully, value of this field is `nil`. - * `context.crash_initiator` - the same as child's reference, that is `:filter`. + * `context.crash_initiator` - the same as child's reference, which is `:filter`. `:filter` won't be present in `context.children`, so you could respawn it here, however it is suggested to do it in `handle_crash_group_down/3` later. @@ -82,10 +71,9 @@ end ``` callback. Each time, `context` will contain following extra fields: - * `context.exit_reason` - for these terminations, equals `{:shutdown, :membrane_crash_group_kill}`. + * `context.exit_reason` - equals `{:shutdown, :membrane_crash_group_kill}`. * `context.group_name` - equals `:my_group`. - * `context.crash_initiator` - TODO: continue - + * `context.crash_initiator` - equals `:filter`. Note, that `context.children` map always contains only children that are still alive. E.g. if `:source` is terminated first, `hanlde_child_terminated(:source, context, state)` will contain only `:sink` in `context.children` map and for `hanlde_child_terminated(:sink, context, state)` `context.children` will be empty. @@ -102,27 +90,9 @@ def handle_crash_group_down(:my_group, context, state) do end ``` -`context` passed as a third argument to `handle_crash_group_down/3` callback contains 3 additional fields, that usually don't occur in contexts of other callbacks: - - `context.crash_initiator` - name or reference of the child that crashed first and caused the crash group to explode. - - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. - - `context.members` - names/references of all children that were in the crash group. -TODO: continue - - -Question marks: - - is crash group initiator in context.members? - - what is the order in of: - * `handle_crash_group_down` - * `handle_child_terminated` - * removing children from `context.children`, so that it becames possible to respawn new children with the same names? - - what is the relation between `handle_child_pad_removed` and `handle_crash_group_down`? - - - -## Use Cases -## tutaj poniej mamy AI BS, no chodzi o to ze jak mamy element co sie wydupca, to nie chcemy zeby wszystko poszlo w piach, wiec wrzucamy go (i byc moze cos co i tak bysmy chcieli razem z nim zrestartowac) do crash groupy - -- **Atomic logical units:** When a group of elements (like an encoder and its associated parser) cannot function independently. -- **Resource Cleanup:** Ensuring that if a consumer crashes, the producer is also stopped to prevent buffered data from leaking memory. -- **Error Recovery:** Grouping elements that require a specific initialization sequence that must be repeated upon failure. +`context` passed as a third argument to `handle_crash_group_down/3` callback contains 3 additional fields: + - `context.crash_initiator` - name or reference of the child that crashed first and caused the crash group to explode. In this case, equals `:filter` + - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. In this case, equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. + - `context.members` - names/references of all children that were in the crash group. In this case, equals `[:source, :filter, :sink]` +When `handle_crash_group_down/3` is executed, you can be sure that all group members are already terminated. It is a suggested place, to e.g. respawn crash group. Doing so in `handle_child_terminated/3` might lead to some problems, because the order of group members termination might vary. Moreover, if pipeline or bin terminates its children gracefully, using `t:Membrane.Pipline.Action.remove_children()` action, `handle_child_terminated/3` callback will be also executed with `context.exit_reason` set to `normal`. From 479c829b2b58c1cf623fc57294b0961ec410cf22 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 12:45:52 +0100 Subject: [PATCH 05/13] Improve crash groups guide --- guides/useful_concepts/crash_groups.md | 59 ++++++++++++++------------ 1 file changed, 33 insertions(+), 26 deletions(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index 3e17141d3..a9da8ef09 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -1,4 +1,4 @@ -# Crash groups +# Crash Groups Crash groups provide a mechanism to manage the lifecycle of elements within a pipeline when one of them fails. By grouping elements together, you can ensure that a crash in one part of the pipeline triggers a coordinated restart or termination of related elements, maintaining system consistency. @@ -6,11 +6,11 @@ Crash groups provide a mechanism to manage the lifecycle of elements within a pi In Membrane, elements and bins are Elixir processes. By default, if an element that is not inside a crash group crashes, it leads to the crash of the whole pipeline. -The most fundamental functionality of crash groups is to separate the crash of a specific element from the rest of the pipeline. Usually, if an element is likely to crash (e.g. it interacts with unstable external resources), it is placed in a crash group along with other elements whose functioning is inextricably connected to it. This prevents a localized failure from bringing down the entire system and allows for controlled recovery of specific logical units. By doing so, it also cleans up components that would have to be restarted or killed anyway. +The fundamental purpose of crash groups is to isolate the crash of a specific element from the rest of the pipeline. If an element is likely to crash (e.g., it interacts with unstable external resources), it is usually placed in a crash group along with other elements whose operation is inextricably connected to it. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. ## Defining Crash Groups -Crash groups are defined in the `spec` within your pipeline or bin. To create a crash group, you have to define a group name and set `crash_group_mode` to `:temporary`, like in the example below: +Crash groups are defined in the `spec` within your pipeline or bin. To create a crash group, you must define a group name and set the `crash_group_mode` to `:temporary`, as shown in the example below: ```elixir defmodule MyPipeline do @@ -32,16 +32,16 @@ end When an element belonging to a crash group crashes: 1. All other elements in the same crash group are terminated by the pipeline. -2. The pipeline's `handle_crash_group_down/3` callback is invoked. -3. You can decide whether to restart the group, ignore the failure, or handle this situation any other way. +2. The pipeline's `c:Membrane.Pipeline.handle_crash_group_down/3` callback is invoked. +3. You can decide whether to restart the group, ignore the failure, or handle the situation in another way. -## Flow of all callbacks reated to a crash within a crash group. +## Flow of callbacks triggered by a crash within a crash group -Let's assume, that in `:filter` spawned in `MyPipeline` raised with message `"internal error"`. +Let's assume a `:filter` element spawned in `MyPipeline` raises an error with the message `"internal error"`. -### Handling termination of the crash itiator +### Handling termination of the crash initiator -The first callback, that will be executed in `MyPipeline`, is +The first callback executed in `MyPipeline` will be: ```elixir @impl true @@ -50,18 +50,18 @@ def handle_child_terminated(:filter, context, state) do end ``` -`context` passed to this callback will contain few extra fileds: - * `context.exit_reason` - in this case equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. - * `context.group_name` - because `:filter` was spawned inside `:my_group` group, it equals `:my_group`. If a child is spawned beyond any crash group and is terminated gracefully, value of this field is `nil`. - * `context.crash_initiator` - the same as child's reference, which is `:filter`. +The `context` passed to this callback will contain a few extra fields: + * `context.exit_reason` - in this case, it equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. + * `context.group_name` - because `:filter` was spawned inside the `:my_group` group, this equals `:my_group`. If a child is spawned outside any crash group and terminates gracefully, the value of this field is `nil`. + * `context.crash_initiator` - the same as the child's reference, which is `:filter`. -`:filter` won't be present in `context.children`, so you could respawn it here, however it is suggested to do it in `handle_crash_group_down/3` later. +Since `:filter` won't be present in `context.children`, you could potentially respawn it here. However, it is recommended to do so later in `c:Membrane.Pipeline.handle_crash_group_down/3`. -### Terminating other children within the crash group that explodes +### Terminating other children within the failing crash group -Because one of children from crash group `:my_group` ungracefully crashed, the rest of children from this group will be terminated as well. +Because one child from the crash group `:my_group` crashed ungracefully, the remaining children in that group will also be terminated. -Therefore, Membrane will terminate `:source` and `:sink` in random order. After each termination, `MyPipeline` will execute +Therefore, Membrane will terminate `:source` and `:sink` (in random order). After each termination, `MyPipeline` will execute the following callback: ```elixir @impl true @@ -70,18 +70,18 @@ def handle_child_terminated(child, context, state) do end ``` -callback. Each time, `context` will contain following extra fields: +Each time, the `context` will contain the following extra fields: * `context.exit_reason` - equals `{:shutdown, :membrane_crash_group_kill}`. * `context.group_name` - equals `:my_group`. * `context.crash_initiator` - equals `:filter`. -Note, that `context.children` map always contains only children that are still alive. E.g. if `:source` is terminated first, `hanlde_child_terminated(:source, context, state)` will contain only `:sink` in `context.children` map and for `hanlde_child_terminated(:sink, context, state)` `context.children` will be empty. +Note that the `context.children` map always contains only the children that are still alive. For example, if `:source` is terminated first, `handle_child_terminated(:source, context, state)` will contain only `:sink` in the `context.children` map. Subsequently, for `handle_child_terminated(:sink, context, state)`, `context.children` will be empty. -Of course, `MyPipeline` could possibly spawn another children beyond `:source`, `:filter` and `:sink`, inside different crash groups or beyond any crash group. In such a case, `context.children` would contain all of them normally and these children wouldn't be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of the child's crash only to children with the same crash group. +`MyPipeline` could potentially spawn other children outside `:source`, `:filter`, and `:sink`—either in different crash groups or outside any crash group. In such cases, `context.children` would contain all of them normally, and these children would not be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of a child's crash to only those children within the same group. ### Handling the crash group down -Then, when all members of a crash group are terminated, `MyPipeline` will execute +Finally, when all members of the crash group are terminated, `MyPipeline` will execute: ```elixir @impl true @@ -90,9 +90,16 @@ def handle_crash_group_down(:my_group, context, state) do end ``` -`context` passed as a third argument to `handle_crash_group_down/3` callback contains 3 additional fields: - - `context.crash_initiator` - name or reference of the child that crashed first and caused the crash group to explode. In this case, equals `:filter` - - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. In this case, equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. - - `context.members` - names/references of all children that were in the crash group. In this case, equals `[:source, :filter, :sink]` +The `context` passed as the third argument to the `handle_crash_group_down/3` callback contains three additional fields: + - `context.crash_initiator` - the name or reference of the child that crashed first and caused the group to fail. In this case, it equals `:filter`. + - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. In this case, it equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. + - `context.members` - names/references of all children that were in the crash group. In this case, it equals `[:source, :filter, :sink]`. -When `handle_crash_group_down/3` is executed, you can be sure that all group members are already terminated. It is a suggested place, to e.g. respawn crash group. Doing so in `handle_child_terminated/3` might lead to some problems, because the order of group members termination might vary. Moreover, if pipeline or bin terminates its children gracefully, using `t:Membrane.Pipline.Action.remove_children()` action, `handle_child_terminated/3` callback will be also executed with `context.exit_reason` set to `normal`. +When `handle_crash_group_down/3` is executed, you can be sure that all group members have already been terminated. This is the suggested place to respawn the crash group. Doing so in `handle_child_terminated/3` might lead to issues because the termination order of group members can vary. Moreover, if a pipeline or bin terminates its children gracefully (using the `t:Membrane.Pipeline.Action.remove_children()` action), the `c:Membrane.Pipeline.handle_child_terminated/3` callback will also be executed, but with `context.exit_reason` set to `normal`. + + +## Callback Contexts +For more info about callback contexts, see `t:Membrane.Pipeline.CallbackContext.t()`, `t:Membrane.Bin.CallbackContext.t()` and `t:Membrane.Element.CallbackContext.t()` docs. + +## Bins +Although the example above mentioned crash groups used from the `Membrane.Pipeline` level, they work the same in way in `Membrane.Bin`. \ No newline at end of file From a4df817e67d7e1199e3495df98544bd2339d1ab8 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 12:55:55 +0100 Subject: [PATCH 06/13] Improve it once again --- guides/useful_concepts/crash_groups.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index a9da8ef09..d78ae55f4 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -28,7 +28,7 @@ defmodule MyPipeline do end ``` -## Behavior +## Behaviour When an element belonging to a crash group crashes: 1. All other elements in the same crash group are terminated by the pipeline. @@ -55,7 +55,7 @@ The `context` passed to this callback will contain a few extra fields: * `context.group_name` - because `:filter` was spawned inside the `:my_group` group, this equals `:my_group`. If a child is spawned outside any crash group and terminates gracefully, the value of this field is `nil`. * `context.crash_initiator` - the same as the child's reference, which is `:filter`. -Since `:filter` won't be present in `context.children`, you could potentially respawn it here. However, it is recommended to do so later in `c:Membrane.Pipeline.handle_crash_group_down/3`. +Since `:filter` is no longer present in `context.children`, you could potentially respawn it here. However, it is recommended to do so later in `c:Membrane.Pipeline.handle_crash_group_down/3`. ### Terminating other children within the failing crash group @@ -79,7 +79,7 @@ Note that the `context.children` map always contains only the children that are `MyPipeline` could potentially spawn other children outside `:source`, `:filter`, and `:sink`—either in different crash groups or outside any crash group. In such cases, `context.children` would contain all of them normally, and these children would not be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of a child's crash to only those children within the same group. -### Handling the crash group down +### Recovering from a crash group failure Finally, when all members of the crash group are terminated, `MyPipeline` will execute: @@ -95,11 +95,13 @@ The `context` passed as the third argument to the `handle_crash_group_down/3` ca - `context.crash_reason` - the reason with which `context.crash_initiator` crashed. In this case, it equals `{%RuntimeError{message: "internal error"}, _stacktrace}`. - `context.members` - names/references of all children that were in the crash group. In this case, it equals `[:source, :filter, :sink]`. -When `handle_crash_group_down/3` is executed, you can be sure that all group members have already been terminated. This is the suggested place to respawn the crash group. Doing so in `handle_child_terminated/3` might lead to issues because the termination order of group members can vary. Moreover, if a pipeline or bin terminates its children gracefully (using the `t:Membrane.Pipeline.Action.remove_children()` action), the `c:Membrane.Pipeline.handle_child_terminated/3` callback will also be executed, but with `context.exit_reason` set to `normal`. +When `handle_crash_group_down/3` is executed, you can be sure that all group members have already been terminated. This is the suggested place to recover from a group failer, e.g. by respawning all crash group members. Doing so in `handle_child_terminated/3` might lead to issues because the termination order of group members can vary. Moreover, if a pipeline or bin terminates its children gracefully (using the `t:Membrane.Pipeline.Action.remove_children()` action), the `c:Membrane.Pipeline.handle_child_terminated/3` callback will also be executed, but with `context.exit_reason` set to `normal`. ## Callback Contexts -For more info about callback contexts, see `t:Membrane.Pipeline.CallbackContext.t()`, `t:Membrane.Bin.CallbackContext.t()` and `t:Membrane.Element.CallbackContext.t()` docs. + +For more information about callback contexts, refer to the documentation for `t:Membrane.Pipeline.CallbackContext.t()`, `t:Membrane.Bin.CallbackContext.t()`, and `t:Membrane.Element.CallbackContext.t()`. ## Bins -Although the example above mentioned crash groups used from the `Membrane.Pipeline` level, they work the same in way in `Membrane.Bin`. \ No newline at end of file + +Although the example above demonstrates using crash groups within a `Membrane.Pipeline`, they function in the same way within a `Membrane.Bin`. \ No newline at end of file From 766a75bb743ed9fafd0c5ba59ba50a96eb2e70e6 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 12:58:45 +0100 Subject: [PATCH 07/13] Remove leftover ;) --- test/membrane/dupa_test.exs | 102 ------------------------------------ 1 file changed, 102 deletions(-) delete mode 100644 test/membrane/dupa_test.exs diff --git a/test/membrane/dupa_test.exs b/test/membrane/dupa_test.exs deleted file mode 100644 index 2d61c58e8..000000000 --- a/test/membrane/dupa_test.exs +++ /dev/null @@ -1,102 +0,0 @@ -defmodule Membrane.DupaTest do - use ExUnit.Case, async: true - - import Membrane.ChildrenSpec - import Mock - - alias Membrane.Core.Element.DiamondDetectionController.DiamondLogger - alias Membrane.Core.Element.DiamondDetectionController.PathInGraph.Vertex - alias Membrane.Testing - - require Membrane.Pad, as: Pad - - defmodule MockSource do - use Membrane.Source - - def_output_pad :output, - accepted_format: _any, - flow_control: :push - end - - defmodule MockFilter do - use Membrane.Filter - - def_input_pad :input, - accepted_format: _any - - def_output_pad :output, - accepted_format: _any - - @impl true - def handle_buffer(_pad, _buffer, _ctx, state), do: {[], state} - - @impl true - def handle_parent_notification(:dupa, _ctx, state), do: raise("internal error") - end - - defmodule MyPipeline do - use Membrane.Pipeline - - alias Membrane.Testing - alias Membrane.DupaTest.{MockFilter, MockSource} - @impl true - def handle_init(_ctx, _opts) do - crash_group_spec = { - for i <- 2..5 do - child_def = if i == 3, do: MockFilter, else: Membrane.Connector - child({:connector, i}, child_def) - end, - group: :my_group, crash_group_mode: :temporary - } - - children_beyond_crash_group = [ - child(:source, MockSource) - |> child({:connector, 1}, Membrane.Connector), - child({:connector, 6}, Membrane.Connector) - |> child(:sink, Testing.Sink) - ] - - connector_links = - for i <- 1..5 do - get_child({:connector, i}) - |> get_child({:connector, i + 1}) - end - - spec = [crash_group_spec, children_beyond_crash_group, connector_links] - - {[spec: spec], %{}} - end - - @impl true - def handle_child_terminated(child, context, state) do - {child, context} |> IO.inspect(label: "HANDLE CHILD TERMINATED", limit: :infinity) - {[], state} - end - - @impl true - def handle_crash_group_down(group_name, context, state) do - {group_name, context} |> IO.inspect(label: "HANDLE CRASH GROUP DOWN", limit: :infinity) - {[], state} - end - - @impl true - def handle_child_pad_removed(child, pad, context, state) do - {child, pad, context} |> IO.inspect(label: "HANDLE CHILD PAD REMOVED", limit: :infinity) - {[], state} - end - end - - test "dupa" do - pipeline = Testing.Pipeline.start_link_supervised!(module: MyPipeline) - - Process.sleep(1500) - - # {:ok, connector_pid} = Testing.Pipeline.get_child_pid(pipeline, {:connector, 3}) - # Process.exit(connector_pid, :kill) - - Testing.Pipeline.notify_child(pipeline, {:connector, 3}, :dupa) - - Process.sleep(1500) - Testing.Pipeline.terminate(pipeline) - end -end From 84f94c2f8174d97e4d4f22cb74d405103f1470e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Feliks=20Pobiedzi=C5=84ski?= <38541925+FelonEkonom@users.noreply.github.com> Date: Tue, 24 Feb 2026 15:32:32 +0100 Subject: [PATCH 08/13] Update guides/useful_concepts/crash_groups.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Łukasz Kita --- guides/useful_concepts/crash_groups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index d78ae55f4..21f1277dc 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -77,7 +77,7 @@ Each time, the `context` will contain the following extra fields: Note that the `context.children` map always contains only the children that are still alive. For example, if `:source` is terminated first, `handle_child_terminated(:source, context, state)` will contain only `:sink` in the `context.children` map. Subsequently, for `handle_child_terminated(:sink, context, state)`, `context.children` will be empty. -`MyPipeline` could potentially spawn other children outside `:source`, `:filter`, and `:sink`—either in different crash groups or outside any crash group. In such cases, `context.children` would contain all of them normally, and these children would not be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of a child's crash to only those children within the same group. +`MyPipeline` could potentially spawn children other than `:source`, `:filter`, and `:sink` - either in different crash groups or outside any crash group. In such cases, `context.children` would contain all of them normally, and these children would not be interrupted by the crash of `:my_group` members. The main goal of crash groups is to limit the consequences of a child's crash to only those children within the same group. ### Recovering from a crash group failure From 54c5016e39b2f71f5b887fd05977b6f68da3d1a1 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 15:52:56 +0100 Subject: [PATCH 09/13] Apply CR suggestions --- guides/useful_concepts/crash_groups.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index 21f1277dc..b3fe68215 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -6,11 +6,13 @@ Crash groups provide a mechanism to manage the lifecycle of elements within a pi In Membrane, elements and bins are Elixir processes. By default, if an element that is not inside a crash group crashes, it leads to the crash of the whole pipeline. -The fundamental purpose of crash groups is to isolate the crash of a specific element from the rest of the pipeline. If an element is likely to crash (e.g., it interacts with unstable external resources), it is usually placed in a crash group along with other elements whose operation is inextricably connected to it. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. +The fundamental purpose of crash groups is to isolate the crash of a specific element from the rest of the pipeline. If an element is likely to crash (e.g., it interacts with unstable external resources), it is usually assigned to a crash group containing all elements inextricably linked to its operation. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. ## Defining Crash Groups -Crash groups are defined in the `spec` within your pipeline or bin. To create a crash group, you must define a group name and set the `crash_group_mode` to `:temporary`, as shown in the example below: +Crash groups are defined in the `spec` within your pipeline or bin. They are built upon the concept of **Children Groups**, which allow aggregating spawned children into easily identifiable groups. + +To create a crash group, you must assign children to a group using the `group` option and set the `crash_group_mode` to `:temporary`. This turns a regular group into a crash group, enabling the crash handling behavior. ```elixir defmodule MyPipeline do @@ -28,6 +30,8 @@ defmodule MyPipeline do end ``` +In the example above, `:source`, `:filter`, and `:sink` are all assigned to the same group `:my_group`. Because `crash_group_mode` is set to `:temporary`, this group functions as a crash group. + ## Behaviour When an element belonging to a crash group crashes: From 4cebf40b5898551497dd73b7480362ec355c46ff Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 15:58:48 +0100 Subject: [PATCH 10/13] One more fix --- guides/useful_concepts/crash_groups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index b3fe68215..79c07ee91 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -4,7 +4,7 @@ Crash groups provide a mechanism to manage the lifecycle of elements within a pi ## Overview -In Membrane, elements and bins are Elixir processes. By default, if an element that is not inside a crash group crashes, it leads to the crash of the whole pipeline. +In Membrane, elements and bins are Elixir processes. By default, if a child (element or bin) that is not inside a crash group crashes, it leads to the crash of the whole pipeline. The fundamental purpose of crash groups is to isolate the crash of a specific element from the rest of the pipeline. If an element is likely to crash (e.g., it interacts with unstable external resources), it is usually assigned to a crash group containing all elements inextricably linked to its operation. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. From f63a67fef4cb31150b55c2e6dc5372e98ec8b57c Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 16:04:45 +0100 Subject: [PATCH 11/13] Apply reviewr sugestion --- guides/useful_concepts/crash_groups.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index 79c07ee91..c7f76fbbd 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -1,12 +1,12 @@ # Crash Groups -Crash groups provide a mechanism to manage the lifecycle of elements within a pipeline when one of them fails. By grouping elements together, you can ensure that a crash in one part of the pipeline triggers a coordinated restart or termination of related elements, maintaining system consistency. +Crash groups provide a mechanism to manage the lifecycle of children (elements or bins) within a pipeline when one of them fails. By grouping children together, you can ensure that a crash in one part of the pipeline triggers a coordinated restart or termination of related elements and bins, maintaining system consistency. ## Overview In Membrane, elements and bins are Elixir processes. By default, if a child (element or bin) that is not inside a crash group crashes, it leads to the crash of the whole pipeline. -The fundamental purpose of crash groups is to isolate the crash of a specific element from the rest of the pipeline. If an element is likely to crash (e.g., it interacts with unstable external resources), it is usually assigned to a crash group containing all elements inextricably linked to its operation. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. +The fundamental purpose of crash groups is to isolate the crash of a specific child from the rest of the pipeline. If an child is likely to crash (e.g., it interacts with unstable external resources), it is usually assigned to a crash group containing all children inextricably linked to its operation. This prevents a localized failure from bringing down the entire system and allows for the controlled recovery of specific logical units. This approach also ensures that components generally needing a restart or termination are cleaned up correctly. ## Defining Crash Groups @@ -34,8 +34,8 @@ In the example above, `:source`, `:filter`, and `:sink` are all assigned to the ## Behaviour -When an element belonging to a crash group crashes: -1. All other elements in the same crash group are terminated by the pipeline. +When an child belonging to a crash group crashes: +1. All other children in the same crash group are terminated by the pipeline. 2. The pipeline's `c:Membrane.Pipeline.handle_crash_group_down/3` callback is invoked. 3. You can decide whether to restart the group, ignore the failure, or handle the situation in another way. From ba31ac4625b79968f963bbb2d2b2d98150507d61 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 16:15:27 +0100 Subject: [PATCH 12/13] Fix typo --- guides/useful_concepts/crash_groups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guides/useful_concepts/crash_groups.md b/guides/useful_concepts/crash_groups.md index c7f76fbbd..1516a1e97 100644 --- a/guides/useful_concepts/crash_groups.md +++ b/guides/useful_concepts/crash_groups.md @@ -34,7 +34,7 @@ In the example above, `:source`, `:filter`, and `:sink` are all assigned to the ## Behaviour -When an child belonging to a crash group crashes: +When a child belonging to a crash group crashes: 1. All other children in the same crash group are terminated by the pipeline. 2. The pipeline's `c:Membrane.Pipeline.handle_crash_group_down/3` callback is invoked. 3. You can decide whether to restart the group, ignore the failure, or handle the situation in another way. From 9d569e755e4718c6a375cf9c6b6ee9d8fd821c37 Mon Sep 17 00:00:00 2001 From: "feliks.pobiedzinski@swmansion.com" Date: Tue, 24 Feb 2026 16:23:10 +0100 Subject: [PATCH 13/13] Fix typo in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6a0f9be08..d069ba070 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ Plugins provide elements that you can use in your pipeline. Each plugin lives in **Formats** -Apart from plugins, Membrane has stream formats, which live in `membrane_X_format` repositories, where X is usually a codec or container, for example [mebrane_opus_format](https://github.com/membraneframework/mebrane_opus_format). Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container. +Apart from plugins, Membrane has stream formats, which live in `membrane_X_format` repositories, where X is usually a codec or container, for example [membrane_opus_format](https://github.com/membraneframework/membrane_opus_format). Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container. **Core**