Skip to content

A110-child-channel-plugins.md

Latest commit

 

History

History
437 lines (355 loc) · 18.7 KB

A110-child-channel-plugins.md

File metadata and controls

437 lines (355 loc) · 18.7 KB
Raw

A110: Child Channel Options

Abstract

There are several use cases where gRPC internally creates a "child channel". Because these channels are created internally rather than being created by the application, it is currently difficult to inject necessary configuration for these channels, which makes it hard to configure things like metrics or tracing from the application. This design proposes a mechanism to allow applications to pass configuration options to these child channels.

Background

Complex gRPC ecosystems often require the creation of auxiliary channels that are not directly instantiated by the user application. The primary examples are:

  1. xDS: When a user creates a channel with an xDS target, the gRPC library internally creates a separate channel to communicate with the xDS control plane. (Note: Child channel options are passed to the XdsClient when the XdsClient is created, and that XdsClient instance will use those same child channel options for any child channel it creates over its lifetime.)
  2. External Authorization (ext_authz): As described in gRFC A92, the gRPC server or client may create an internal channel to contact an external authorization service.
  3. External Processing (ext_proc): As described in gRFC A93, filters may create internal channels to call external processing servers.

The primary motivation for this feature is the need to configure observability on a per-child-channel basis.

  • StatsPlugins: Users use these plugins to configure metrics and tracing (as described in gRFC A66 and A72) so that telemetry from internal channels is correctly tagged and exported.
  • Interceptors: Users may need to apply specific interceptors (e.g., for logging, or tracing) to internal traffic.

Global configuration is not sufficient for these use cases for the reasons described in the 'Rationale' section below.

Related Proposals

Proposal

We introduce the concept of Child Channel Options. This is a configuration container attached to a parent channel/server that is strictly designated for use by its children.

The user API must allow "nesting" of channel options (specifying child channel options within parent channel options). A user creating a Parent Channel/Server P can provide a set of options O_child.

  • O_child is opaque to P. P does not apply these options to itself.
  • O_child is carried in P's state, available for extraction by internal components.
  • The configuration provided by O_child is strictly uniform across all child channels of a particular parent channel/server.

When an internal component (e.g., an xDS client factory) attached to P needs to create a Child Channel C:

  1. It retrieves O_child from P.
  2. It applies O_child to the configuration of C.
  3. It should also configure channel C to use O_child for its children.
  • Multi-level Propagation: The child options O_child apply recursively to all child channels, no matter how deeply nested. For example, if a parent channel P creates a child channel C (e.g., to an ext_authz server), and C itself is configured to use xds:/// (which requires creating an xDS client control plane channel), C will pass O_child to its own child channels.

The Child Channel C typically requires some internal configuration O_internal (e.g., target URIs, or internal interceptors).

  • Merge Rule: O_child and O_internal are merged. If the environment supports global channel options, O_child options override global channel options.
  • Conflict Resolution: Mandatory internal settings (O_internal) generally take precedence over user-provided child options (O_child) to ensure correctness.

In some cases, child channels may be shared across multiple parent channels/servers. For example, the xDS control plane channel is shared across multiple channels or servers as described in gRFC A27. However, it is possible for each parent channel or server to be created with different child options.

Consider an example where Parent Channels/Servers (P1, P2) point to the same target but provide different Child Channel Options (O_child1, O_child2):

  • Behavior: The shared client is created using the options from the first parent channel or server that triggers its creation (e.g., O_child1).
  • Subsequent Usage: When P2 requests the client, it receives the existing shared client. O_child2 is effectively ignored for that specific shared resource.

LB Policies and Resolvers

Some LB policies and resolvers may need to create child channels. We use grpclb as an example for how this plumbing will be handled in LB policies. Note that this proposal does not mandate any behavior changes for grpclb specifically.

To support this, the child channel options must be plumbed down into resolvers and LB policies. Each internal component that creates a child channel C is explicitly responsible for applying and propagating those options:

  • Java: When an LB policy creates an out-of-band (OOB) child channel via LoadBalancer.Helper (e.g., createResolvingOobChannelBuilder() or createOobChannel()), the Helper implementation (ManagedChannelImpl) is responsible for automatically calling channelConfigurator.configureChannelBuilder(builder) to apply the options to C and builder.childChannelConfigurator(channelConfigurator) to propagate O_child recursively to any further child channels. For resolvers and other internal components (e.g., XdsClient via GrpcXdsTransportFactory) that independently create a child channel C, channelConfigurator is plumbed via NameResolver.Args.getChildChannelConfigurator(). That internal component is responsible for explicitly calling both channelConfigurator.configureChannelBuilder(channelBuilder) and channelBuilder.childChannelConfigurator(channelConfigurator) when constructing C.
  • Go: A new field (ChildChannelOptions) will be added to resolver.BuildOptions and balancer.BuildOptions (passed when creating a resolver or LB policy) to contain the child channel options ([]grpc.DialOption). When a resolver (e.g., the xDS resolver creating a control plane XdsClient) or an LB policy (e.g., grpclb creating an out-of-band ClientConn) creates a child channel C (e.g., via grpc.NewClient), that resolver or LB policy is responsible for applying ChildChannelOptions to C and calling grpc.WithChildChannelOptions(ChildChannelOptions...) so that any further child channels created by C also inherit the options.
  • C-core: No special plumbing is needed to pass child channel options to LB policies and resolvers because they are already contained within the channel arguments (grpc_channel_args). When an LB policy or resolver (e.g., grpclb or an xDS resolver) creates a child channel C, that LB policy or resolver is responsible for propagating both the individual child channel args (O_child applied to C) and the GRPC_ARG_CHILD_CHANNEL_ARGS argument containing the child channel args (O_child propagated recursively) to C.

Language Implementations

Java

In Java, the configuration will be achieved by accepting functional interfaces. The API allows users to register a configurator on a ManagedChannelBuilder<?> or ServerBuilder<?>. When an internal library or component creates a child channel C:

  1. If created via LoadBalancer.Helper (createResolvingOobChannelBuilder() or createOobChannel()), ManagedChannelImpl automatically calls channelConfigurator.configureChannelBuilder(builder) and builder.childChannelConfigurator(channelConfigurator) on C's builder.
  2. If created independently by a resolver or internal transport factory (e.g., GrpcXdsTransportFactory), that component retrieves channelConfigurator via NameResolver.Args.getChildChannelConfigurator() and explicitly calls channelConfigurator.configureChannelBuilder(builder) and builder.childChannelConfigurator(channelConfigurator) on C's builder.
  • Configuration Interface

    Define a new public API interface, ChannelConfigurator, to encapsulate the configuration logic for channels.

    import io.grpc.ManagedChannelBuilder;

// Captures the intent of the plugin.
// Consumes a builder to modify it before further configuring the channel
public interface ChannelConfigurator {
   /**
   * Configures the given channel builder.
   *
   * @param builder the channel builder to configure
   */
  void configureChannelBuilder(ManagedChannelBuilder<?> builder);
}
  • API Changes
    • ManagedChannelBuilder: Add ManagedChannelBuilder#childChannelConfigurator(ChannelConfigurator channelConfigurator) to allow users to register this configurator.
    • XdsServerBuilder: Add XdsServerBuilder#childChannelConfigurator(ChannelConfigurator configurator) to allow users to provide configuration for any internal channels created by the server (e.g., connections to external authorization or processing services).
    • NameResolver.Args: Add NameResolver.Args#getChildChannelConfigurator() and NameResolver.Args.Builder#setChildChannelConfigurator( ChannelConfigurator channelConfigurator) to allow resolvers to access the child channel configurator when creating internal child channels.
  • Usage Example
    // Define the configurator for internal child channels
ChannelConfigurator myInternalConfig = new ChannelConfigurator() {
    @Override
    public void configureChannelBuilder(ManagedChannelBuilder<?> builder) {
        builder.maxInboundMessageSize(4 * 1024 * 1024);
    }
};

// Apply it to the parent channel
ManagedChannel channel = ManagedChannelBuilder.forTarget("xds:///my-service")
    .childChannelConfigurator(myInternalConfig) // <--- Configuration injected here
    .build();

Go

In Go, both the Client (grpc.NewClient) and the Server (NewGRPCServer) create internal child channels. We introduce mechanisms to pass DialOptions into these internal channels from both entry points.

  • New API for Child Channel Options

  • Client-Side: WithChildChannelOptions

    For standard clients, we introduce a DialOption wrapper.

    // WithChildChannelOptions returns a DialOption that specifies a list of 
// DialOptions to be applied to any internal child channels.
func WithChildChannelOptions(opts ...DialOption) DialOption {
    return newFuncDialOption(func(o *dialOptions) {
        o.childChannelOptions = opts
    })
}

  • Server-Side: ChildChannelOptions

    For xDS-enabled servers, we introduce a ServerOption wrapper. Since xds.NewGRPCServer creates an internal xDS client to fetch listener configurations, it requires a way to apply DialOptions (such as Socket Options or Stats Handlers) to that internal connection.

    // ChildChannelOptions returns a ServerOption that specifies a list of 
// DialOptions to be applied to the server's internal child channels 
// (e.g., the xDS control plane connection).
func ChildChannelOptions(opts ...DialOption) ServerOption {
    return newFuncServerOption(func(o *serverOptions) {
        o.childDialOptions = opts
    })
}

  • Resolvers and LB Policies: BuildOptions

    To allow resolvers and load balancers to access child channel options when creating child channels, ChildChannelOptions will be added to both resolver.BuildOptions and balancer.BuildOptions.

    type BuildOptions struct {
    // ... existing fields ...
    
    // ChildChannelOptions contains the options to be applied to any
    // internal child channels created by the resolver or load balancer.
    ChildChannelOptions []DialOption
}
  • Usage Example (User-Side Code)

    This design provides users with the flexibility to define independent configurations for parent and child channels within a single NewClient call. For example, a parent channel can be configured with transport security (mTLS) while the internal child channels (such as the xDS control plane connection) are configured with specific interceptors or a custom authority.

    func main() {
    // Define configuration specifically for the internal control plane
    internalOpts := []grpc.DialOption{
        // Inject the OTel handler here. It will only measure traffic on the
        // internal child channels (e.g., to the xDS server).
        grpc.WithStatsHandler(otelHandler)
    }

    // Create the Parent Channel
    conn, err := grpc.NewClient("xds:///my-service",
        // Parent channel configuration (Data Plane)
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        
        // Child channel configuration (Control Plane)
        // The OTel handler inside here applies ONLY to the child channels.
        grpc.WithChildChannelOptions(internalOpts...), 
    )
    
    if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }
    defer conn.Close()
  
    // ... use conn ...
}

Core (C/C++)

In gRPC Core, we utilize the existing ChannelArgs mechanism recursively to pass configuration to internal channels. We define a standard argument key whose value is a pointer to another grpc_channel_args structure. This "Nested Arguments" pattern allows the parent channel or server to carry a specific subset of arguments intended solely for its children.

  • Configuration Mechanism

    We define a new channel argument key. The value associated with this key is a pointer to a grpc_channel_args struct, managed via a pointer vtable to ensure correct ownership and copying.

    // A pointer argument key. The value is a pointer to a grpc_channel_args 
// struct containing the subset of options for child channels.
#define GRPC_ARG_CHILD_CHANNEL_ARGS "grpc.child_channel.args"
  • API Changes

    We add a helper method to the C++ ChannelArguments and ServerBuilder classes to simplify packing the nested arguments safely.

      // Sets the channel arguments to be used for child channels.
  void SetChildChannelArgs(const ChannelArguments& args);
  • Usage Example (User-Side Code)

    An example of how this will work on the channel side:

    // Create OTel StatsPlugin.
grpc::OpenTelemetryPluginBuilder ot_plugin_builder;
// ...set options on builder...
auto ot_plugin = ot_plugin_builder.Build();
assert(ot_plugin.ok());
// Add the StatsPlugin to both child args and parent args.
grpc::ChannelArguments child_args;
grpc::ChannelArguments parent_args;
ot_plugin->AddToChannelArguments(&child_args);
ot_plugin->AddToChannelArguments(&parent_args);
// Add child args to parent args.
parent_args.SetChildChannelArgs(child_args);
// Create channel with parent args.
auto channel = grpc::CreateCustomChannel(
    "xds:///my-service", credentials, parent_args);

    An example of how this will work on the server side:

    grpc::ServerBuilder server_builder;
// Create OTel StatsPlugin.
grpc::OpenTelemetryPluginBuilder ot_plugin_builder;
// ...set options on builder...
auto ot_plugin = ot_plugin_builder.Build();
assert(ot_plugin.ok());
// Add the StatsPlugin to both child args and the server builder.
grpc::ChannelArguments child_args;
ot_plugin->AddToChannelArguments(&child_args);
ot_plugin->AddToServerBuilder(&server_builder);
// Add the child args to the server builder.
server_builder.SetChildChannelArgs(child_args);
// Start the server.
auto server = server_builder.BuildAndStart();

Rationale

The proposed mechanism of Child Channel Options provides a targeted way to propagate configuration from a parent channel to its children. This approach is chosen because it allows configuration to be scoped to specific parent-child hierarchies, which is necessary for accurate telemetry and interceptor application without affecting unrelated channels.

Why not Global Configuration?

The primary use-case we care about is setting a StatsPlugin for one particular channel, in which case we want that same StatsPlugin to also be used for any child of that channel.

For example, let's say that we create two channels, one to target A and one to target B, both of which create their own child channel to an ext_authz server target Z. If we create the channel to target A with a specific StatsPlugin, then we want that StatsPlugin to also be used for the child channel to target Z created by the channel to target A. We do not want it to be used for the child channel to target Z created by the channel to target B, because we did not configure the StatsPlugin for the channel to target B.

We cannot achieve this using the global registry, for a couple of reasons. First, the global registry can only select channels based on parameters like the target URI. To attach our StatsPlugin to the internal target Z, we would have to select it based on target Z, which would erroneously attach it to the child channels for both A and B. Second, it is hard for the application that registers the global StatsPlugin to know what target URIs will be used for internal child channels.

Non-Goals and Future Work

Differentiating Configuration Per Child Channel

This proposal mandates that the child channel options provided by a parent are uniform across all its child channels. We considered allowing different configurations for different child channels (e.g., based on the purpose of the channel like xDS vs RLS). However, this would require a mechanism to categorize or identify the purpose of each child channel, which adds significant complexity. Since no strong need for this was identified during initial discussions, it is left for future work if and when it becomes necessary.