diff --git a/guides/useful_concepts/assets/pads1.png b/guides/useful_concepts/assets/pads1.png new file mode 100644 index 000000000..b28510fc6 Binary files /dev/null and b/guides/useful_concepts/assets/pads1.png differ diff --git a/guides/useful_concepts/assets/pads2.png b/guides/useful_concepts/assets/pads2.png new file mode 100644 index 000000000..10884e0d8 Binary files /dev/null and b/guides/useful_concepts/assets/pads2.png differ diff --git a/guides/useful_concepts/pads.md b/guides/useful_concepts/pads.md new file mode 100644 index 000000000..d6fd0a002 --- /dev/null +++ b/guides/useful_concepts/pads.md @@ -0,0 +1,342 @@ +# Everything about pads + +When developing intuition about the structure of pipelines, pads are +something that can't be ignored. If you think about elements and bins (from now +on collectively referred to as components) as some +sort of containers or boxes in which processing happens, then pads are the parts +that these containers are connected with. + +![image](assets/pads1.png) + +There are some constraints regarding pads: + +* A pad of one component can only connect to a single pad of another component. + Only once these two pads are linked, communication through them can happen. +* One pad needs to be an input pad, and the other an output pad. +* The accepted formats of the pads need to match - stream formats passing + between elements through these pads need to match the accepted formats of both. + +When looking at the insides of components, the pads are their main way to +communicate with other components in the pipeline. There are four types +of information that can be exchanged between components through pads - +[stream formats](`t:Membrane.StreamFormat.t/0`), +[events](`t:Membrane.Event.t/0`), [buffers](`t:Membrane.Buffer.t/0`) and +`:end_of_stream`s. + +When a component receives information of one of these types from another one, it +receives it on a pad. The reference to this pad is then also available as an argument +of the callback that handles the received information. For example, an invocation +of a [`handle_buffer/4`](`c:Membrane.Element.WithInputPads.handle_buffer/4`) +callback would mean that a buffer `buffer` has arrived on a pad `some_pad`: + +```elixir +@impl true +def handle_buffer(some_pad, buffer, context, state) do + ... ^^^^^^^^ +end +``` + +When a component wants to send information to another +component in the pipeline, it should send it on a pad that's linked to it. It +can do that by using the pad reference in actions that send these types of +information, for example returning a +[`:buffer`](`t:Membrane.Element.Action.buffer/0`) action +from a callback would mean that a buffer `buffer` will be sent on a pad `some_pad`: + +```elixir +@impl true +def handle_something(..., _context, state) do + ... + {[buffer: {some_pad, buffer}], state} +end ^^^^^^^^ +``` + +## Defining pads + +To define what pads a component will have and how they'll behave, we use +[`def_input_pad/2`](`Membrane.Element.WithInputPads.def_input_pad/2`) +and [`def_output_pad/2`](`Membrane.Element.WithOutputPads.def_output_pad/2`) macros. +Input pads can only be defined for Bins, Sinks, Filters and Endpoints, and output +pads can only be defined for Bins, Sources, Filters and Endpoints. +The first argument for these macros is a name, which will then be used to +identify the pads. The second argument is a [pad spec](`t:Membrane.Pad.element_spec/0`) +keyword list, which is used to define how this pad will work. An option we'll +now focus on is [`availability`](`t:Membrane.Pad.availability/0`), which +determines if the pad is _static_ or _dynamic_. + +### Static pads + +Static pads are pretty straightforward - when a static pad is defined there +will always be exactly one instance of this pad and it's referenced by its +name. + +#### File Source Example + +An example of an element with only static pads is a [File Source](https://hexdocs.pm/membrane_file_plugin/Membrane.File.Source.html). +This element reads the contents of a file and sends them in batches through a static +output pad. The content of the buffers sent by this element is unknown - the file +that's being read can contain anything - so this pad has `:accepted_format` set to +`%RemoteStream{type: :bytestream}`. That means that any stream format that +matches on this struct can be sent on the output pad and this fact has to be +accounted for when linking a component after the source. + +A pipeline spec with a file source passing buffers to an MP4 demuxer could look +like this: + +```elixir +@impl true +def handle_init(_context, state) do + spec = + child(:source, %Membrane.File.Source{location: "my_file.mp4"}) + |> via_out(:output) + |> via_in(:input) + |> child(:mp4_demuxer, Membrane.MP4.ISOM.Demuxer) + + {[spec: spec], state} +end +``` + +This spec will link a pad named `:output` of the source to a pad named +`:input` of the demuxer. However, this can be shortened - if an output pad is +called `:output` or an input pad is called `:input`, their respective +[`via_in/3`](`Membrane.ChildrenSpec.via_in/3`) and +[`via_out/3`](`Membrane.ChildrenSpec.via_out/3`) calls can be omitted and +Membrane will automatically recognize and link them: + +```elixir +@impl true +def handle_init(_context, state) do + spec = + child(:source, %Membrane.File.Source{location: "my_file.mp4"}) + |> child(:mp4_demuxer, Membrane.MP4.ISOM.Demuxer) + + {[spec: spec], state} +end +``` + +### Dynamic pads + +Dynamic pads are a bit more complex. They're used when the number of pads of a +given type is variable - dependent on the processed stream or external factors. +The creation of these pads is controlled by the parent of the component - if a +[`:spec`](`t:Membrane.Pipeline.Action.spec/0`) action linking the dynamic pad is +being executed, then the pad is created dynamically and the component needs to +handle this, in most cases with +[`handle_pad_added/3`](`c:Membrane.Element.Base.handle_pad_added/3`). This +callback is called only for dynamic pads. + +Another thing that's different is the way the pad are referenced. The pad's name +can't just be used as the pad's reference, because it wouldn't be unique. +Dynamic pads are identified by [`Pad.ref/2`](`Membrane.Pad.ref/2`), which takes +the pad's name and some unique reference as arguments. The result is a unique +pad reference that is also associated with a given pad's specification through +its name. When a new pad is linked, its reference is made known to the element +through the first argument of +[`handle_pad_added/3`](`c:Membrane.Element.Base.handle_pad_added/3`). + +#### MP4 Demuxer Example + +An example of an element using dynamic pads is an +[MP4 Demuxer](https://hexdocs.pm/membrane_mp4_plugin/Membrane.MP4.Demuxer.ISOM.html). +This element has an input pad, from which it receives the contents of an MP4 +container, and output pads, on which it'll send the different tracks +that were in the container. The input pad can be static, however MP4 containers can +have different numbers and kinds of tracks, so the output pad needs to be dynamic. + +![image](assets/pads3.png) + +We'll consider the case when we don't have any prior information about the +tracks in this MP4 container. Because of this, the parent pipeline or bin of this +demuxer won't initially know how many pads should be linked. To solve this +problem, the demuxer will identify the tracks in the incoming stream and send a +message to its parent in the form of +[`{:new_tracks, [{track_id :: integer(), content :: struct()}]}`](https://hexdocs.pm/membrane_mp4_plugin/Membrane.MP4.Demuxer.ISOM.html#t:new_tracks_t/0). +The list consists of tuples corresponding to tracks, where the first +element is a track id and will be used to identify the corresponding pad, +and the second is a stream format contained in the track. + +Initially, the parent will only create the elements before and including the +demuxer: + +```elixir +@impl true +def handle_init(_context, state) do + spec = + child(:source, %Membrane.File.Source{location: "my_file.mp4"}) + |> child(:mp4_demuxer, Membrane.MP4.ISOM.Demuxer) + + {[spec: spec], state} +end +``` + +The source will start providing the demuxer with the MP4 container content, +from which the demuxer will identify tracks and notify its parent +about them. The parent now has to link an output pad of the demuxer for each +track received, which can look like this: + +```elixir +@impl true +def handle_child_notification({:new_tracks, tracks}, :mp4_demuxer, _context, state) do + spec = + Enum.map(tracks, fn {id, format} -> + get_child(:mp4_demuxer) + |> via_out(Pad.ref(:output, id)) + |> ... + end) + + {[spec: spec], state} +end +``` + +The elements that the output pads will be linked to should be based on what stream +format is in the `format` variable - different formats require different approaches. + +After this spec is returned, the demuxer will now have the +[`handle_pad_added/3`](`c:Membrane.Element.Base.handle_pad_added/3`) callback +called for each new linked pad: + +```elixir +@impl true +def handle_pad_added(Pad.ref(:output, track_id), _context, state) do + ... +end +``` + +It will now know that these pads are linked and +ready to pass buffers forward. + +An operation that's less common than linking dynamic pads, but also important, +is unlinking them. If the parent removed a child with +[`t:remove_children/0`](`t:Membrane.Pipeline.Action.remove_children/0`) action: + +```elixir +@impl true +def handle_something(..., _context, state) do + ... + {[remove_children: :some_child], state} +end +``` + +Then the child with name `:some_child` would be stopped and removed from the +pipeline, unlinking all its pads. If an input pad of this child happened to be +connected to our demuxer, then the +[`handle_pad_removed/3`](`c:Membrane.Element.Base.handle_pad_removed/3`) +would be called in the demuxer with a reference to the pad that was unlinked: + +```elixir +@impl true +def handle_pad_removed(Pad.ref(:output, unlinked_track_id), _context, state) do + ... +end +``` + +The demuxer should react to this information accordingly, for example it should +now know that it can no longer send buffers on this pad, because it has been +unlinked and essentially no longer exists. + +If a link has dynamic pads on both sides, the parent could also return a +[`:remove_link`](`t:Membrane.Pipeline.Action.remove_link/0`) action, +which would only remove the link, resulting in +[`handle_pad_removed/3`](`c:Membrane.Element.Base.handle_pad_removed/3`) +being called in children on both sides of it. + +## Life cycle of a pad + +The life cycle of components is explored more broadly in this +[guide](components_lifecycle.md). Here, we'll take a closer look at the life +cycle of a pad, mostly focusing on elements. + +### Creation + +Static pads are essentially created and linked at the same time as the whole +component and exist alongside it for its entire lifespan - they have to be +linked at the same time the component is created. + +Dynamic pads can be linked and unlinked throughout their components' lifespan. +There can also be multiple instances of a dynamic pad. +Because of this, each creation can be handled separately in +[`handle_pad_added/3`](`c:Membrane.Element.Base.handle_pad_added/3`) callback, +which is called every time a new dynamic pad is linked, and therefore +created. + +### Playback + +When an element is in `:stopped` +[playback](`t:Membrane.Playback.t/0`), nothing can be sent on its pads - the +pipeline is not ready. Only when an element enters `:playing` playback and +[`handle_playing/2`](`c:Membrane.Element.Base.handle_playing/2`) callback is +called can it assume that the pipeline is ready for communication and can +send on and receive information from its pads. + +### Removal + +Static pads are removed and unlinked only once their component is terminated. + +Dynamic pads can be removed during the lifespan of their component. For each removal +a [`handle_pad_removed/3`](`c:Membrane.Element.Base.handle_pad_removed/3`) +callback is called. + +## Pads in bins + +We know that when it comes to linking pads between components, bins behave just like +elements. However, when we take a look inside them, they are pretty different. +While elements can interact with information moving through their pads +directly with callbacks, bins are only containers for other components. They +don't interact with information moving through their pads directly, they forward +it to components inside them. + +The insides of a bin can be thought of as a pipeline of sorts. The components +are created with the [`:spec`](`t:Membrane.Bin.Action.spec/0`) action and the +bin is their parent. The way that these components can send and receive +information on the bin's pads is by linking with +[`bin_input`](`Membrane.ChildrenSpec.bin_input/1`) and +[`bin_output`](`Membrane.ChildrenSpec.bin_output/2`). These functions can be +thought of as the "interior" part of the bin's pad. + +![image](assets/pads2.png) + +For example, if a bin has +a dynamic input pad called `:input`, and a single static output pad called `:output`, +it can behave like this: + +```elixir +@impl true +def handle_setup(_ctx, state) do + spec = + child(:comp, SomeComponent) + |> bin_output(:output) + + {[spec: spec], state} +end +``` + +When the bin is being set up, it creates a `SomeComponent` child named `:comp`, +which connects its output pad to the `bin_output` corresponding to the bin's +`:output` pad. Now whenever this child sends or receives something on its output +pad, it's received by or sent to whatever component is connected to the bin's +`:output` pad. + +```elixir +@impl true +def handle_pad_added(Pad.ref(:input, id) = pad, _ctx, state) do + spec = + bin_input(pad) + |> child(SomeOtherComponent) + |> get_child(:comp) + + {[spec: spec], state} +end +``` + +Every time a new input pad of the bin is created, the +[`handle_pad_added/3`](`c:Membrane.Bin.handle_pad_added/3`) callback is called. +The bin has to connect this pad to something within a 5-second timeout, +otherwise a `LinkError` will be +raised. A new `SomeOtherComponent` child is created and its input pad is +connected to a `bin_input` corresponding to the newly created pad. Now whenever +this child sends or receives something on its input pad, it's received by or +sent to whatever component is connected to the bin's newly created pad. + +It's worth noting that pads of bins are only an abstraction. When a component +links with a bin, it actually links directly to one of the components inside of +it to avoid unnecessary forwarding of messages. diff --git a/mix.exs b/mix.exs index 45ee7456f..b8b7aecd1 100644 --- a/mix.exs +++ b/mix.exs @@ -87,7 +87,8 @@ defmodule Membrane.Mixfile do "guides/membrane_tutorials/digital_video_introduction/assets" => "assets", "guides/membrane_tutorials/h264/assets" => "assets", "guides/membrane_tutorials/broadcasting/assets" => "assets", - "guides/membrane_tutorials/glossary/assets" => "assets" + "guides/membrane_tutorials/glossary/assets" => "assets", + "guides/useful_concepts/assets" => "assets" }, nest_modules_by_prefix: [ Membrane.Bin, diff --git a/mix.lock b/mix.lock index aa6f8b6a0..eab11fc4f 100644 --- a/mix.lock +++ b/mix.lock @@ -8,7 +8,7 @@ "erlex": {:hex, :erlex, "0.2.7", "810e8725f96ab74d17aac676e748627a07bc87eb950d2b83acd29dc047a30595", [:mix], [], "hexpm", "3ed95f79d1a844c3f6bf0cea61e0d5612a42ce56da9c03f01df538685365efb0"}, "ex_doc": {:hex, :ex_doc, "0.39.3", "519c6bc7e84a2918b737aec7ef48b96aa4698342927d080437f61395d361dcee", [:mix], [{:earmark_parser, "~> 1.4.44", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "0590955cf7ad3b625780ee1c1ea627c28a78948c6c0a9b0322bd976a079996e1"}, "excoveralls": {:hex, :excoveralls, "0.18.3", "bca47a24d69a3179951f51f1db6d3ed63bca9017f476fe520eb78602d45f7756", [:mix], [{:castore, "~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "746f404fcd09d5029f1b211739afb8fb8575d775b21f6a3908e7ce3e640724c6"}, - "file_system": {:hex, :file_system, "1.0.1", "79e8ceaddb0416f8b8cd02a0127bdbababe7bf4a23d2a395b983c1f8b3f73edd", [:mix], [], "hexpm", "4414d1f38863ddf9120720cd976fce5bdde8e91d8283353f0e31850fa89feb9e"}, + "file_system": {:hex, :file_system, "1.1.1", "31864f4685b0148f25bd3fbef2b1228457c0c89024ad67f7a81a3ffbc0bbad3a", [:mix], [], "hexpm", "7a15ff97dfe526aeefb090a7a9d3d03aa907e100e262a0f8f7746b78f8f87a5d"}, "jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"}, "junit_formatter": {:hex, :junit_formatter, "3.4.0", "d0e8db6c34dab6d3c4154c3b46b21540db1109ae709d6cf99ba7e7a2ce4b1ac2", [:mix], [], "hexpm", "bb36e2ae83f1ced6ab931c4ce51dd3dbef1ef61bb4932412e173b0cfa259dacd"}, "makeup": {:hex, :makeup, "1.2.1", "e90ac1c65589ef354378def3ba19d401e739ee7ee06fb47f94c687016e3713d1", [:mix], [{:nimble_parsec, "~> 1.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "d36484867b0bae0fea568d10131197a4c2e47056a6fbe84922bf6ba71c8d17ce"},