Skip to content

Local inter-pipeline communication #1814

@lquerel

Description

@lquerel

To support more complex deployment modes, we need to support a local inter-pipeline communication mechanism, whether pipelines belong to the same group or not. Many scenarios can be supported with such a mechanism, for example:

  • A pipeline receiving traffic and splitting it across n specific pipelines based on a tenant ID.
  • A pipeline receiving traffic and load balancing it across n instances of the same pipeline, to support platforms where SO_REUSEPORT is not effective.

To support this kind of inter-pipeline communication, we want to introduce the concept of named channels and add a new pair of exporter and receiver that reference a named channel, thereby establishing inter-pipeline communication. In the longer term, we also want to allow defining a class for these inter-pipeline channels such as:

  • in-memory channel class
  • quiver channel class

Metadata

Metadata

Assignees

Labels

Type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions