initial dev doc

devel/architecture/kgateway/DEV.md

@@ -0,0 +1,103 @@
+# Architecture
+Kgateway is a control plane for envoy based on the Kubernetes Gateway API. This means that we translate K8s objects like Gateways, HTTPRoutes, Services, EndpointSlices and user policy into envoy configuration.
+
+Our goals with the architecture of the project are to make it scalable and extendable.
+
+To make the project scalable, it's important to keep the computation minimal when changes occur. For example, when a pod changes, or a policy is updated, we do the minimum amount of computation to update the envoy configuration.
+
+With extendability, kgateway is the basis on top of which users can easily add their own custom logic. To that end, we have a plugin system that allows users to add their own custom logic to the control plane in a way that's opaque to the core code.
+
+
+Going down further, to enable these goals we use a [KRT](https://github.com/istio/istio/tree/master/pkg/kube/krt#krt-kubernetes-declarative-controller-runtime) based system. KRT gives us a few advantages:
+- The ability to complement controllers with custom Intermediate Representations (IRs).
+- Automatically track object dependencies and changes and only invoke logic that depends on the object that changed.
+
+# CRD Lifecycle
+
+How does a user CRD make it into envoy?
+
+We have 3 main translation lifecycles: Routes & Listeners, Clusters, and Endpoints.
+
+Let's focus on the first one - Routes and Listeners, as this is where the majority of the logic is.
+
+Envoy Routes & Listeners translate from Gateways, HTTPRoutes, and user policies (i.e. RoutePolicy, ListenerPolicy, etc).
+
+## Policies (Contributed by Plugins)
+
+To add a policy to Kgateway, it needs to be contributed through a **plugin**. Each plugin acts as an independent Kubernetes controller that provides kgateway with a **KRT collection of IR policies** (this is done by translating the policy CRD to a policy IR). The second important thing a plugin provides is a function that allocates a **translation pass** for the gateway translation phase. This will be used when doing xDS translation. More on that later.
+
+To add a policy to kgateway we need to 'contribute' it as a plugin. You can think of a plugin as an
+independent k8s controller that translates a CRD to a KRT collection of policies in an IR form kgateway can understand (it's called `ir.PolicyWrapper`).
+A plugin lifecycle is the length of the program - it doesn't reset on each translation (we will explain later where we hold translation state).
+
+When a plugin can contribute a policy to kgateway, kgateway uses policy collection to perform **policy attachment** - this is the process of assigning policies to the object they impact (like HTTPRoutes) based on their targetRefs.
+
+You can see in the Plugin interface a field called `ContributesPolicies` which is a map of GK -> `PolicyPlugin`.
+The policy plugin contains a bunch of fields, but for our discussion we'll focus on these two:
+
+```go
+type PolicyPlugin struct {
+	Policies       krt.Collection[ir.PolicyWrapper]
+	NewGatewayTranslationPass func(ctx context.Context, tctx ir.GwTranslationCtx) ir.ProxyTranslationPass
+    // ... other fields
+}
+```
+`Policies` is the collection of policies that the plugin contributes. The plugin is responsible for creating
+this collection, usually by starting with a CRD collection, and then translating to a `ir.PolicyWrapper` struct.
+
+Let's look at the important fields in the PolicyWrapper:
+
+```go
+type PolicyWrapper struct {
+	// The Group, Kind Name, Namespace to the original policy object
+	ObjectSource `json:",inline"`
+	// The IR of the policy objects. Ideally with structural errors either removed so it can be applied to envoy, or retained and returned in the translation pass (this depends on the defined fallback behavior).
+	// Opaque to us other than metadata.
+	PolicyIR PolicyIR
+	// Where to attach the policy. This usually comes from the policy CRD.
+	TargetRefs []PolicyTargetRef
+}
+```
+
+The system will make use of the targetRefs to attach the policy IR to Listeners and HTTPRoutes. You will then 
+get access to the IR during translation (more on that later).
+
+When translating a Policy to a PolicyIR, you'll want to take the translation **as close as you can** to envoy's config structures. For example, if your policy will result in an envoy
+PerFilterConfig on a route, the PolicyIR should contain the proto.Any that will be applied there. Doing most of the policy translation at this phase has advantages:
+- Policy will only re-translate when the policy CRD changes
+- We can expose errors in translation on the policy CRD status as soon as the policy is written (no need to depend on later translation phases).
+
+The second field, `NewGatewayTranslationPass` allocates a new translation state for the
+gateway/route translation. This function will be invoked during the Translation to xDS phase, so we will expand on it later.
+
+## HTTPRoutes and Gateways
+
+HTTPRoutes and Gateways are handled by kgateway. Kgateway builds an IR for HTTPRoutes and Gateways, that looks very similar to 
+the original object, but in additional has an `AttachedPolicies` struct that contains all the policies that are attached to the object.
+
+Kgateway uses the `TargetRefs` field in the PolicyWrapper (and `extensionRefs` in the Route objects) to opaquely attach the policy to the HTTPRoute or Gateway.
+
+## Translation to xDS
+
+When we reach this phase, we already have the Policy -> IR translation done; and we have all the HTTPRoutes and Gateways in IR form with the policies attached to them.
+
+The translation to xDS has 2 phases:
+- The first phase aggregates all the HTTPRoutes for a single Gateway into a single Gateway IR object with all the policies and routes inside it. This resolves delegation, and merges HTTPRoutes based on the Gateway API spec. The phase uses KRT to track the dependencies of the routes and gateways (for example, ssl certs). Recall that by the time we get here, policy attachment was already performed (i.e. policies will already be in the AttachedPolicies struct).
+- The second phase translates the Gateway IR to envoy configuration. It does not use KRT, as all the dependencies are resolved in the first phase, and everything needed for translation is in the Gateway IR. At this phase the **policy plugins** are invoked.
+
+Having these 2 phases has these advantages:
+- The second phase is simple and fast, as all the dependencies are resolved in the previous phases.
+- We can create alternative translators for the Gateway IR, for example, to translate a waypoint.
+
+
+At the beginning of the translation of the Gateway IR to xDS, we take all the contributed policies and allocate a `NewGatewayTranslationPass` for each of them. This will hold the state for the duration of the translation. 
+
+This allows us, for example, to translate a route, and in response to that hold state that tells us to add an http filter.
+
+When it translates Gateway API route rules to envoy routes, it reads the `AttachedPolicies` and calls the appropriate function in the `ProxyTranslationPass` and passes in 
+the attached policy IR. This lets the policy plugin code modify the route or listener as needed, based on the policy IR (add filters, add filter route config, etc..).
+Then policy plugins are invoked with the attached policy IR, and can modify the envoy protobufs as needed.
+
+# Diagram of the translation lifecycle
+
+![](translation.svg)

examples/plugin/main.go

@@ -0,0 +1,246 @@
+package main
+
+import (
+	"context"
+	"time"
+
+	mutation_v3 "github.com/envoyproxy/go-control-plane/envoy/config/common/mutation_rules/v3"
+	envoy_core_v3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
+	envoy_config_route_v3 "github.com/envoyproxy/go-control-plane/envoy/config/route/v3"
+	header_mutationv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/header_mutation/v3"
+	"google.golang.org/protobuf/proto"
+	"google.golang.org/protobuf/types/known/structpb"
+	"istio.io/istio/pkg/kube/kclient"
+	"istio.io/istio/pkg/kube/krt"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	gwv1 "sigs.k8s.io/gateway-api/apis/v1"
+
+	"github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/common"
+	extensionsplug "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugin"
+	"github.com/kgateway-dev/kgateway/v2/internal/kgateway/ir"
+	"github.com/kgateway-dev/kgateway/v2/internal/kgateway/plugins"
+	"github.com/kgateway-dev/kgateway/v2/internal/kgateway/setup"
+)
+
+/******
+
+An example plugin, that uses a ConfigMap as a policy. We use a targetRef annotation to attach the
+policy to an HTTPRouter. We will then add the key value pairs in the ConfigMap to the metadata of
+the envoy route route.
+
+Exmaple ConfigMap:
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: my-configmap
+  annotations:
+    targetRef: my-http-route
+data:
+  key1: value1
+  key2: value2
+
+
+To test, use this example HTTPRoute that adds the metadata to a header:
+
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: my-http-route
+spec:
+  parentRefs:
+  - name: gw
+  rules:
+  - backendRefs:
+    - name: example-svc
+      port: 8080
+    filters:
+    - type: ResponseHeaderModifier
+      responseHeaderModifier:
+        add:
+        - name: my-header-name
+          value: %METADATA(ROUTE:example.plugin:key1)%
+
+*****/
+
+var (
+	configMapGK = schema.GroupKind{
+		Group: "",
+		Kind:  "ConfigMap",
+	}
+)
+
+// Our policy IR.
+type configMapIr struct {
+	creationTime time.Time
+	metadata     *structpb.Struct
+}
+
+var _ ir.PolicyIR = &configMapIr{}
+
+// in case multiple policies attached to the same resource, we sort by policy creation time.
+func (d *configMapIr) CreationTime() time.Time {
+	return d.creationTime
+}
+
+// Equals is needed because this is in KRT collection.
+func (d *configMapIr) Equals(in any) bool {
+	d2, ok := in.(*configMapIr)
+	if !ok {
+		return false
+	}
+	return d.creationTime == d2.creationTime && proto.Equal(d.metadata, d2.metadata)
+}
+
+// convert a configmap to our IR.
+func configMapToIr(cm *corev1.ConfigMap) *configMapIr {
+	// When converting to IR, we want to take the translation as close to envoy xDS as we can.
+	// That's why our IR intentionaly uses structpb.Struct and not map[string]string.
+
+	mdStruct := &structpb.Struct{
+		Fields: map[string]*structpb.Value{},
+	}
+	for k, v := range cm.Data {
+		mdStruct.Fields[k] = structpb.NewStringValue(v)
+	}
+
+	return &configMapIr{
+		creationTime: cm.CreationTimestamp.Time,
+		metadata:     mdStruct,
+	}
+}
+
+// Configmaps don't have a target ref, so we extract it from the annotations.
+func extractTargetRefs(cm *corev1.ConfigMap) []ir.PolicyTargetRef {
+	return []ir.PolicyTargetRef{{
+		Group: gwv1.GroupName,
+		Kind:  "HTTPRoute",
+		Name:  cm.Annotations["targetRef"],
+	}}
+
+}
+
+// Create a collection of our policies. This will be done by converting a configmap collection
+// to our policy IR.
+func ourPolicies(commoncol *common.CommonCollections) krt.Collection[ir.PolicyWrapper] {
+	// We create 2 collections here - one for the source config maps, and one for the policy IR.
+	// Whenever creating a new krtCollection use commoncol.KrtOpts.ToOptions("<Name>") to provide the
+	// collection with common options and a name. It's important so that the collection appears in
+	// the krt debug page.
+
+	// get a configmap client going
+	configMapCol := krt.WrapClient(kclient.New[*corev1.ConfigMap](commoncol.Client), commoncol.KrtOpts.ToOptions("ConfigMaps")...)
+
+	// convertIt to policy IR
+	return krt.NewCollection(configMapCol, func(krtctx krt.HandlerContext, i *corev1.ConfigMap) *ir.PolicyWrapper {
+		if i.Annotations["targetRef"] == "" {
+			return nil
+		}
+
+		var pol = &ir.PolicyWrapper{
+			ObjectSource: ir.ObjectSource{
+				Group:     configMapGK.Group,
+				Kind:      configMapGK.Kind,
+				Namespace: i.Namespace,
+				Name:      i.Name,
+			},
+			Policy:     i,
+			PolicyIR:   configMapToIr(i),
+			TargetRefs: extractTargetRefs(i),
+		}
+		return pol
+	}, commoncol.KrtOpts.ToOptions("MetadataPolicies")...)
+
+}
+
+// Our translation pass struct. This holds translation specific state.
+// In our case, we check if our policy was applied to a route and if so, we add a filter.
+type ourPolicyPass struct {
+	// Add the unimplemented pass so we don't have to implement all the methods.
+	ir.UnimplementedProxyTranslationPass
+
+	// We keep track of which filter chains need our filter.
+	filterNeeded map[string]bool
+}
+
+// ApplyForRoute is called when a an HTTPRouteRule is being translated to an envoy route.
+func (s *ourPolicyPass) ApplyForRoute(ctx context.Context, pCtx *ir.RouteContext, out *envoy_config_route_v3.Route) error {
+	// get our policy IR. Kgateway used the targetRef to attach the policy to the HTTPRoute. and now as it
+	// translates the HTTPRoute to xDS, it calls our plugin and passes the policy for the plugin's translation pass to do the
+	// policy to xDS translation.
+	cmIr, ok := pCtx.Policy.(*configMapIr)
+	if !ok {
+		// should never happen
+		return nil
+	}
+	// apply the metadata from our IR to envoy's route object
+	if out.Metadata == nil {
+		out.Metadata = &envoy_core_v3.Metadata{}
+	}
+	out.Metadata.FilterMetadata["example.plugin"] = cmIr.metadata
+
+	// mark that we need a filter for this filter chain.
+	if s.filterNeeded == nil {
+		s.filterNeeded = map[string]bool{}
+	}
+	s.filterNeeded[pCtx.FilterChainName] = true
+
+	return nil
+}
+
+func (s *ourPolicyPass) HttpFilters(ctx context.Context, fc ir.FilterChainCommon) ([]plugins.StagedHttpFilter, error) {
+	if !s.filterNeeded[fc.FilterChainName] {
+		return nil, nil
+	}
+	// Add an http filter to the chain that adds a header indicating metadata was added.
+	return []plugins.StagedHttpFilter{
+		plugins.MustNewStagedFilter("example_plugin",
+			&header_mutationv3.HeaderMutation{
+				Mutations: &header_mutationv3.Mutations{
+					ResponseMutations: []*mutation_v3.HeaderMutation{
+						{
+							Action: &mutation_v3.HeaderMutation_Append{
+								Append: &envoy_core_v3.HeaderValueOption{
+									Header: &envoy_core_v3.HeaderValue{
+										Key:   "x-metadata-added",
+										Value: "true",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+			plugins.BeforeStage(plugins.AcceptedStage))}, nil
+
+}
+
+// A function that initializes our plugins.
+func pluginFactory(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin {
+	return []extensionsplug.Plugin{
+		{
+			ContributesPolicies: extensionsplug.ContributesPolicies{
+				configMapGK: extensionsplug.PolicyPlugin{
+					Name: "metadataPolicy",
+					NewGatewayTranslationPass: func(ctx context.Context, tctx ir.GwTranslationCtx) ir.ProxyTranslationPass {
+						// Return a fresh new translation pass
+						return &ourPolicyPass{}
+					},
+					// Provide a collection of our policies in IR form.
+					Policies: ourPolicies(commoncol),
+				},
+			},
+		},
+	}
+}
+
+func main() {
+
+	// TODO: move setup.StartGGv2 from internal to public.
+	// Start Kgateway and provide our plugin.
+	// This demonstrates how to start Kgateway with a custom plugin.
+	// This binary is the control plane. normally it would be packaged in a docker image and run
+	// in a k8s cluster.
+	setup.StartKgateway(context.Background(), pluginFactory, nil)
+}

internal/kgateway/controller/start.go

@@ -65,7 +65,7 @@ type StartConfig struct {
 	RestConfig *rest.Config
 	// ExtensionsFactory is the factory function which will return an extensions.K8sGatewayExtensions
 	// This is responsible for producing the extension points that this controller requires
-	ExtraPlugins []extensionsplug.Plugin
+	ExtraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin
 
 	Client istiokube.Client
 
@@ -174,11 +174,13 @@ func NewControllerBuilder(ctx context.Context, cfg StartConfig) (*ControllerBuil
 	}, nil
 }
 
-func pluginFactoryWithBuiltin(extraPlugins []extensionsplug.Plugin) extensions2.K8sGatewayExtensionsFactory {
+func pluginFactoryWithBuiltin(extraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin) extensions2.K8sGatewayExtensionsFactory {
 	return func(ctx context.Context, commoncol *common.CommonCollections) extensionsplug.Plugin {
 		plugins := registry.Plugins(ctx, commoncol)
 		plugins = append(plugins, krtcollections.NewBuiltinPlugin(ctx))
-		plugins = append(plugins, extraPlugins...)
+		if extraPlugins != nil {
+			plugins = append(plugins, extraPlugins(ctx, commoncol)...)
+		}
 		return registry.MergePlugins(plugins...)
 	}
 }

internal/kgateway/extensions2/plugin/plugin.go

@@ -83,7 +83,7 @@ type GwTranslatorFactory func(gw *gwv1.Gateway) KGwTranslator
 type ContributesPolicies map[schema.GroupKind]PolicyPlugin
 
 type Plugin struct {
-	ContributesPolicies
+	ContributesPolicies     ContributesPolicies
 	ContributesBackends     map[schema.GroupKind]BackendPlugin
 	ContributesGwTranslator GwTranslatorFactory
 	// extra has sync beyong primary resources in the collections above