diff --git a/devel/architecture/kgateway/DEV.md b/devel/architecture/kgateway/DEV.md new file mode 100644 index 00000000000..778ec09d956 --- /dev/null +++ b/devel/architecture/kgateway/DEV.md @@ -0,0 +1,103 @@ +# Architecture +Kgateway is a control plane for envoy based on the gateway-api. This means that we translate K8s objects like Gateways, HttpRoutes, Service, EndpointSlices and User policy into envoy configuration. + +Our goals with the architecture of the project are to make it scalable and extendable. + +To make the project scalable, it's important to keep the computation minimal when changes occur. For example, when a pod changes, or a policy is updated, we do the minimum amount of computation to update the envoy configuration. + +With extendability, Kgateway is the basis on-top of which users can easily add their own custom logic. to that end we have a plugin system that allows users to add their own custom logic to the control plane in a way that's opaque to the core code. + + +Going down further, to enable these goals we use [KRT](https://github.com/istio/istio/tree/master/pkg/kube/krt#krt-kubernetes-declarative-controller-runtime) based system. KRT gives us a few advantages: +- The ability to complement controllers of custom Intermediate Representation (henceforth IR). +- Automatically track object dependencies and changes and only invoke logic that depends on the object that changed. + +# CRD Lifecycle + +How does a user CRD make it into envoy? + +We have 3 main translation lifecycles: Routes & Listeners, Clusters and Endpoints. + +Let's focus on the first one - Routes and Listeners, as this is where the majority of the logic is. + +Envoy Routes & Listeners translate from Gateways, HttpRoutes, and user policies (i.e. RoutePolicy, ListenerPolicy, etc). + +## Policies (Contributed by Plugins) + +To add a policy to KGateway, it needs to be contributed through a **plugin**. Each plugin acts as an independent Kubernetes controller that provides Kgateway with a **KRT collection of IR policies** (this is done by translating the policy CRD to a policy IR). The second important thing a plugin provides is a function that allocates a **translation pass** for the gateway translation phase. This will be used when doing xDS translation. More on that later. + +To add a policy to Kgateway we need to 'contribute' it as a plugin. You can think of a plugin as an +independent k8s controller that translates a CRD to an krt-collection of policies in IR form Kgateway can understand (it's called `ir.PolicyWrapper`). +A plugin lifecycle is the length of the program - it doesn't reset on each translation (we will explain later where we hold translation state). + +When a plugin can contribute a policy to Kgateway, Kgateway uses policy collection to perform **policy attachment** - this is the process of assigning policies to the object they impact (like HttpRoutes) based on their targetRefs. + +You can see in the Plugin interface a field called `ContributesPolicies` which is a map of GK -> `PolicyPlugin`. +The policy plugin contains a bunch of fields, but for our discussion we'll focus on these two: + +```go +type PolicyPlugin struct { + Policies krt.Collection[ir.PolicyWrapper] + NewGatewayTranslationPass func(ctx context.Context, tctx ir.GwTranslationCtx) ir.ProxyTranslationPass + // ... other fields +} +``` +Policies is a the collection of policies that the plugin contributes. The plugin is responsible to create +this collection, usually by starting with a CRD collection, and then translating to a `ir.PolicyWrapper` struct. + +Lets look at the important fields in the PolicyWrapper: + +```go +type PolicyWrapper struct { + // The Group, Kind Name, Namespace to the original policy object + ObjectSource `json:",inline"` + // The IR of the policy objects. Ideally with structural errors either removed so it can be applied to envoy, or retained and returned in the translation pass (this depends on the defined fallback behavior). + // Opaque to us other than metadata. + PolicyIR PolicyIR + // Where to attach the policy. This usually comes from the policy CRD. + TargetRefs []PolicyTargetRef +} +``` + +The system will make use of the targetRefs to attach the policy IR to Listners and HttpRoutes. You will then +get access to the IR during translation (more on that later). + +When translating a Policy to a PolicyIR, you'll want to take the translation **as close as you can** to envoy's config structures. For example, if your policy will result in an envoy +PerFilterConfig on a route, the PolicyIR should contain the proto.Any that will be applied there. Doing most of the policy translation at this phase as advantages: +- Policy will only re-translate when the policy CRD changes +- We can expose errors in translation on the policy CRD status as soon as the policy is written (no need to depend on later translation phases). + +The second field, `NewGatewayTranslationPass` allocates a new translation state for the +gateway/route translation. This function will be invoked during the Translation to xDS phase, so will expand on it later. + +## HttpRotues and Gateways + +HttpRoutes and Gateways are handled by Kgateway. Kgateway builds an IR for HttpRoutes and Gateways, that looks very similar to +the original object, but in additional has an `AttachedPolicies` struct that contains all the policies that are attached to the object. + +Kgateway uses the `TargetRefs` field in the PolicyWrapper (and extensionRefs in the Route objects) to opaquely attach the policy to the HttpRoute or Gateway. + +## Translation to xDS + +When we reach this phase, we already have the Policy -> IR translation done; and we have all the HttpRoutes and Gateways in IR form with the policies attached to them. + +The translation to xDS has 2 phases. +- The first phase, aggregating all the httproutes for a single gateway into a single gateway IR object with all the policies and routes inside it. This resolves delegation, and merges http routes based on the Gateway Api spec. The phases uses KRT to track the dependencies of the routes and gateways (like for example, ssl certs). recall that by the time we get here, policy attachment was already performed (i.e. policies will already be in the AttachedPolicies struct). +- The second phase, translates the gateway IR to envoy configuration. It does not use KRT, as all the dependencies are resolved in the first phase, and everything needed for translation is in the gateway IR. At this phase the **policy plugins** are invoked. + +Having these 2 phases has these advantages: +- The second phase is simple and fast, as all the dependencies are resolved in the previous phases. +- We can create alternative translators for the gateway IR, for example, to translate a waypoint. + + +In the begining of the transaltion of the gateway IR to xDS, we take all the contributed policies and allocate a `NewGatewayTranslationPass` for each of them. This will hold the state for the duration of the translation. + +This allows us for example translate a route, and in response to that hold state that tells us to add an http filter. + +When it translates GW-api route rules to envoy routes, it reads the `AttachedPolicies` and calls the appropriate function in the `ProxyTranslationPass` and passes in +the attached policy IR. This let's the policy plugin code the modify the route or listener as needed, based on the policy IR (add filters, add filter route config, etc..). +Then policy plugins are invoked with the attached policy IR, and can modify the envoy protbufs as needed + +# Diagram of the translation lifecycle + +![](translation.svg) \ No newline at end of file diff --git a/devel/architecture/kgateway/translation.svg b/devel/architecture/kgateway/translation.svg new file mode 100644 index 00000000000..a2ba2783b58 --- /dev/null +++ b/devel/architecture/kgateway/translation.svg @@ -0,0 +1,2 @@ +eyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO1dWXPaStq+P7+CytR3M3Vgel9y51x1MDAwNWNcdTAwMWPHxmDs2DNTLlx1MDAxOcRiXHUwMDE2XHUwMDExXHUwMDEw2Hjq/PevXHUwMDFi2yCh1lx1MDAwNoqdnEAqqSCE6O15n3frt//3Ry73yZ2P7E+fc5/sp4bV7zbH1uOnP/X1mT2edJ2h+lxiLd5PnOm4sbiz47qjyed//ctcdTAwMWGNXG6rb1x1MDAxNVx1MDAxYc7g5Zt231x1MDAxZdhDd6Lu/bd6n8v9b/Gv+qTb1N/fv7tcdTAwMDO98oPjjPjxl/IheLxuNejiq4ub3lx1MDAxYTS2XHUwMDFirjVs9+3VR0+6NVRcdTAwMTTo8spcXF2BXFyQXHUwMDAyUC8sMZaAUbb8+LHbdDv6XHUwMDE2XHUwMDAy0fJix+62O666SjgoSO9recvL737OgeWViTt2evaB03fGunH/XHUwMDAwXHJBpVxcNe3eavTaY2c6bC7vccfWcDKyxmosVve1uv1+zZ0vnq7GVI3dp7XfuH5r89r1sG+pXHUwMDFmbXeG9mTi+44zslx1MDAxYV13MTxg1Vx1MDAwYt3CUbm5mJr/rto0tlx1MDAwNnZZz81w2u8vL3eHTVuP+Kfbie/Xhs3XX3ub19Wk4dcrf63abtvNxVx1MDAxY1x1MDAwMcpcdTAwMDVcdTAwMTBw1Vx1MDAxNs9cdTAwMDLDcv3qmTNcXCw2yFx1MDAxMKVUzflqTruTQ7XC3MVjW1Z/Yq/mQLetuL76vCvQt8BcXPtpNTGe9XnRPCmW683DfFWK4sV95/m8bp9/Wt731+v/VuM3XHUwMDFkNS33tZtcdTAwMDRcdTAwMTIqKUNcdTAwMWOu5qLfXHUwMDFk9tZcdTAwMDe37zR6qy784Vx1MDAxObQ1tJhbXHUwMDEzQIuvMy9AYThcYlx1MDAxNFx1MDAxOVx1MDAwM1x1MDAxNOqZiiVOMN1cdTAwMDHDXGZcZtdcZlxm392vXGJAhDKh5JAwXHUwMDAxXHUwMDAw8FBcdTAwMDBcdTAwMTAsXHUwMDE4oZhssv6zXFyiq1x1MDAxNadXmur8Ud9yXXtoN3M1u9/KXHUwMDFmOEPX6uq3JfVLj9Y8V656Zld9Wus+61x1MDAxZSHhu3pkXHK6fT1cdTAwMWTC91x1MDAwYnv9bltcdTAwMGbNp77d8oyyXHUwMDFhXHUwMDFkt6soZ/mx64xWnzZe2zAuJyFcdTAwMWFn3G13h1b/cpNcdTAwMGVZU9ep2pOXLrnjqe1cdTAwMWRK+/hccjqwgGhcdTAwMDS4u5U7jpqMnzydsWL5pnjZmFx1MDAxZsgtqFBcdTAwMDJZoJrJhKRAUoFNTGhAOMd4x4RJXHUwMDAwP92eXHRcdTAwMTnEXHUwMDA0UVx1MDAwNUODIICYrV99XHUwMDEzXHUwMDA0XGZcdTAwMDLCJGErPeaHXHUwMDEzYbc6OX6w5pfjfm0+eJidXFxPblx1MDAwNk/piJBcIki5XHUwMDE3XHUwMDFhm1x1MDAxM6G5NVx1MDAxYlx1MDAxMaGEJFx1MDAwNiZU0Fx1MDAwMlJMXHSRks1cdTAwMDCbtMdcdTAwMWQrhoFklpxcdTAwMTUpJFx1MDAxMlx1MDAwM2pkRUV9YWDAXHUwMDFjSFx1MDAwNFx1MDAxNI6yZsW06zXAiuVq7rrrdnIjNeqNrj3JWcNmTs+2PWyqN65rNTp284OJMYZ21okxbZ+y4cajyd70UVSe9stccmiT28v7k+9HVlxubvRDXHUwMDFlYVx1MDAwMVx1MDAwYlghWjCAIEHYSI1cdTAwMDaYc8hcdTAwMTc689trdctcdTAwMGX1PtTPt6dGoWxELok0SVx1MDAwMyUlwqRcdTAwMDFcdTAwMDGUYc7BO9qIe0fD88vD0ZevXHUwMDE3tdLN3VWtNb2f5VNRo+qlslx1MDAwN7KhRnNrXHUwMDEyUSNYR1x0i0FcdMIwXGKSXHUwMDFkXHUwMDE3hqHiOYWFXGJcdTAwMTHFXFww4/JHLNxHXCKRRIJTj28lXHUwMDFiNky9RFx1MDAwM2z4dFjLXeq5VKaV7sfH0l5cZqOs015o47Pht9I3i4zrpVx1MDAwYrv4MJufXHUwMDBlhjeNu0krMb8pvdWHXFzsUVveoIpcdTAwMTn1W3Urk2OJXFzEV+NcdTAwMWNcdTAwMGZdaOs/kdD9h0WbotX6XHUwMDE5YbtiooolykjmT1x1MDAxZu5cdTAwMWYqfXAzbfbFXHUwMDE4vYnv5Oi2wNachzhcIlxmXGKjNchcdTAwMDPO0jfIS8WTQGk0q7n74Yw3PkeXrN05uKtNmuCxSUtN+FxcTcN4nFx1MDAwMFx1MDAwNCT2YmVzxjO3Jlx1MDAwMeMpg8OPXHUwMDFij0G9VFx1MDAwNFx1MDAwNVx1MDAwZlx1MDAwMoWnobhcdTAwMWROcj6cwOQsqOhcdTAwMGZcdTAwMDMksVx0XHUwMDEwXHUwMDFlMlpcdTAwMDOEYkCMXHUwMDE4xZtcdTAwMDDC14rtV22AXHUwMDA0j113VHWmrv2fYS43VCP1OTfWb/Mtx9GXXvScqt36nNOhNTNJXHUwMDAy39VcdTAwMWZIkjG0tE6SXHUwMDFidy5cdTAwMWJcdTAwMTJ1j2vP0K5+pU7xaFTbm12ewP5hclx1MDAxMvXoXHUwMDFlejRcdDfou8jAmpCk8ZAmkVx1MDAwNveohe7vf1x1MDAwZmmAtrdcdTAwMTQpXHUwMDA2RCpcdTAwMWJcdTAwMDV5XHUwMDE3+dJvXHUwMDE0kFx1MDAxZG9iglx1MDAwM0KxJOxcdTAwMWQtxVx1MDAxMby/6X6ZPEwn1eLpbbXUqpXnqSzFTHnT3JokvOkxXHUwMDA1X6BcIlx1MDAwMlCBnFx1MDAxOaDiWVY74lxcm+tYqOBcdTAwMTTEiVx1MDAxMFOmOza7UkPjXG5cYlBMMUFcdTAwMWLpku9GndVpXzGMJpRcXK745NpD3Vx1MDAwNcUqL1f2X3yR6v1cdTAwMDeTZ1xmXHUwMDFkhZJn+u5lQ5/Fi+dBkTWb/dLNPu+C8y9W9/okMX1K7LdBWZA9MVx1MDAwMj7vKTBIXGLEadg9f3eBkUJcdTAwMTKQrUlcdTAwMTNcdTAwMTMgKWFGzVx1MDAxYVx1MDAwNVx1MDAxM3OW7iXMpZL+71x1MDAxYXm8s+5cdTAwMWZvnNFev1IqXG5cZlx1MDAwNbvEJ7NUpKk6iz1cIm0r0jS3Jlx1MDAwMWlK4jc2mYEzsSHmQH9cdTAwMWbKTIFcdTAwMDCanFx1MDAwYlx1MDAwNVMvZdpcdTAwMDdYb6E7hibbUFx1MDAwMlx1MDAxOYdYZFx1MDAxY1ZMv1x1MDAxOFx1MDAwM0T4moKyMrJ+XHUwMDAySzGGPNbJLqZcdTAwMGLZXHUwMDEwXHUwMDFhuz2c7Z2eWO5Fi59cZrq3+Ful7SYmNIiw3yDETFx1MDAxNoSUkiO0yJoz+Fh3XHUwMDA0l1x1MDAxMbzZ9lYhXHUwMDExXHUwMDEyLaBvYjhcdTAwMTJcdTAwMWFAQVx1MDAxMCEsXHUwMDA1Ju9oXHUwMDE2Wue1/FXvyMawan9t3ORPq1x1MDAwZuP9lGYhwVwiI4YztyZcdTAwMDHDQUT8diHmJFx1MDAwNjGIXHUwMDE4XHUwMDE4XHUwMDBmot/HvZpcdTAwMDJcdTAwMTM8RfRQiVx1MDAxNcTVgFx1MDAxYjOsg/GFpUuEYKTJMuNcdTAwMDTT9MszwHlH1rTvVnTOiYc0Rk5fv3Gtcdt+cSv63I2Tkd34/GI1tfTXtS71f1x1MDAxZsyTMZxcdTAwMTRIQ82+29lwa6PXXHUwMDFiulNye3i3X9+r0dZccibHXHUwMDA231FcYrcqXHUwMDAxu56OjiguXHUwMDAwqvPRlWWCxS6CmaXoXHUwMDEwW9OpXHUwMDEwQGdcdTAwMGIzo49cdTAwMTWEXG5cdTAwMTRlSlwiXGb4uzpZe5VcdTAwMWEq1a+6Rbt9iLFg14I0aSo2pUhZYlx1MDAxOTlZza1JwKbKTFxiYkTEYOR3j1amwIRMTqfKXGKkXHUwMDAwIXNiavDq0llcdTAwMDJcdJZqvjZTJbNcXJ9cdTAwMDE67Y5cdTAwMGK/ViQyhm/WeXOb/mVDkGLvivMvcHz+xcm3J9WT7r31ME5MkFx1MDAwNPJ18GvzM4YgofSZlobcPJgqOe/XjlWmkFx1MDAwNXtb8yMhSjhcdTAwMTNgTN1BIDRVXHUwMDAxcp3gwPg75u5U8Py6cVO5nk3PLmu972fFXHUwMDFiXHUwMDA27jz0+Kf5sS9fvnw6eOS1J159XHUwMDFjXHUwMDE0WafrVmfisOP/lbfft8Zj5zFcdTAwMTXtMqrzXGazoV1zL5PQLkJcdTAwMDHkKTM2XHUwMDFhecY82J1cdTAwMTlrxNp+ct6FkFx1MDAwMa7GNlx1MDAxOLDUY05DtU6KJYacboaqqFx1MDAwNYoo8mRcdTAwMTlkXHUwMDEzw9x73TJRed1Q8WK5/VMx2Ivddz22RiN7rC/ri6/xv1x1MDAwZqbjXHUwMDE4dktcdTAwMTDb3Lbb2bB0LX9Trd+enFTbx1x1MDAwZs1W46o+r8OzxCwtUGDPJY0l6Z2POCNBcrA1aTNMXGInQlx1MDAxYb1k4Vx1MDAxYs5cdTAwMTBcdTAwMDBCiSWvXHUwMDFm7YezNjgl+dpj64Tkm+3Bg8NKg1qrkc5FrPjIs5d3K3Y1tyZcdTAwMDG7XG5cdTAwMWPYfsniyHVcdTAwMTdcdTAwMTVNXGaJw1x1MDAxNNxcbpje4IjMYVFcdTAwMTm+25JcdTAwMTJAOMp6tyVnUinPaZQ/k037M0ZGYyjGYLm+S3C0c2mXVbf3jubFh3qRTuG306ty8uAoRFx1MDAwMVx1MDAwM1x1MDAxNWn/LJGSU1x1MDAwNiTnXHUwMDFlTX7HfVlcdTAwMDO9uH18VFxuyDg1XHUwMDA0g/QkiHDyo1x1MDAxY1x1MDAwYvV6R4duuUafrmbfO/w7XHUwMDA1g3Ll6uxrXHUwMDEx139cdTAwMWWLlXrxtzmnmnuZgFOhXCLIXHUwMDAwXHUwMDE2dTWrSCxcImnwXHUwMDE0/075uSnQdpQm8kqEQIBcdFOgXHUwMDA088DVZeRcdTAwMTVKQrWXOWtaXHUwMDA1Su1jKVaoiVZcdTAwMDNcdTAwMTaZL1x1MDAwZbmoXHUwMDA0MC+P32KO3b5rj1x1MDAwZpxhq9tW94xcdTAwMWTXKexccudcdTAwMWbMwjF0Z2DhbDudXHJpR1x1MDAwYrM1QeZcdTAwMTNcdTAwMTKMrbCxQFx1MDAwNaGvdfIg4WqJoKCEIMwgIUSavS4tIFx1MDAxYVx1MDAwMERcdTAwMGKIv4X/uJSUjlEoXHUwMDFkM1x1MDAwNKSEXHUwMDFjXHUwMDE4XHUwMDAzrDI0Y1x1MDAxZlxuTJU+vuF+7834WH6Z3u1ZXHUwMDBmw2a1dN3iNenWxF3KmniCIJDGXHUwMDE2XV5cdTAwMWU53fW2/9vTQuBtLlj+/79/XHUwMDFh7/bso9evvJBx3/BgYvFccu75RqDbfWviXHUwMDFlOINB13W1n0s1PEBcdTAwMWGuNXb31TrqXHUwMDBl2/718lo3M4m3bSE8XHUwMDFiUz0ooIA5g4xwXGZcdTAwMTHAknvCwXpxW6NcdTAwMDXyXHUwMDAza9BcdTAwMWU241x1MDAxYlx1MDAxMS1BfY2AXHUwMDA0XHUwMDAwofRLrSBcdTAwMDGlgtBgI4I40EOxp1x1MDAwNVfHtlx1MDAwMlBTLfR+ti7h7P6985hIyzKv3Vx1MDAwNFqWXHUwMDEwIFx1MDAxMFx1MDAxN6BK74pcdTAwMTSh0lBeLVVU4PeRoFcpXHUwMDEyulx0lFx1MDAxNFxiY3JcdTAwMWJcZtWwIEdK6tCNXG7DRG5t2kKavelYXHUwMDBi4TDJuU5cdTAwMTZaUkM12lx1MDAxZUfoSYNus+k16NdUpVx1MDAxOCVjXVUytD1cdTAwMWJlp342fDxcdTAwMTjtyW+lPdnM191bcc6OglBcctvOS/zZ21BCXHUwMDFhg9ade1wiKzSXk+pD4aUwtPHKpLlCKlx1MDAwZvVOQFxusDLA5HuWf+IlMmVMjO0na9+xXHUwMDBmi1x1MDAxN/v2cz2VXHUwMDFigSvtj2TkRjC3Jlx1MDAwMcFRitdcdTAwMDCj4Fx1MDAxMFxyXHUwMDE4o2d+XHUwMDE39jZC4iQ5v0miy4JcdTAwMTLjfnbEQzcsXHUwMDExXGZcdTAwMTnnm2VaRi1OSjnZit1WLu1y1e/U1u9cdTAwMTZJWZOFXHUwMDFknc+Np3371aRWXHUwMDBmyIBcZrdwXHUwMDE5xPBPuON+015mQ5vP0iHnI3Ygv2G312l9w6PjvVJy2lx1MDAxNHRNzUVcdTAwMDDRRYSOUMwhoIZC4Yj5y1wiQoPjQFxuP2+uYLZcdTAwMTNcdTAwMTJLIfF1a96kXHUwMDAyXHUwMDEzxSbcxJuYhO7rpWpelVpNM/RcIryGmah1S5tPNef2+m5+c8QnX+uPz35cdTAwMTZccnO//1x1MDAxOfXcW1p6bvR6p43v7Wr38PCxnHenXHUwMDA35ucuxFBcdTAwMWE6XHUwMDE2gHr9t1vRcXQ7o+hYrtubyqJcdTAwMDdxQDTloe3Sv01QO0vOx3oroVx1MDAwMECaffosXHUwMDE0VFxuh1xi6cJcdTAwMTdcdTAwMTkz8mJ9blWK8Vx1MDAxN806i6G2dUb+WZLMJrhcdTAwMDKcfYuLavmmXq82W1x1MDAxN8+j5GfcQKxcdTAwMGKUr6nmXHUwMDAwXHUwMDE1JNVESzkkRO5qbfw4QXG+NSdDgDlRhqkx1YagUGOWXHUwMDExXcdYoHe0ZVvfioezq7NcdTAwMTNoXfVcdTAwMGW+fKudXHLRQzUpJ2/B9fGkzIVcdTAwMWFC4MXf5qRs7mVcdTAwMDJShnqLRVx1MDAwMIs8XHUwMDA2i7tQe3K0VVLQslKIkLZPTX5gXHUwMDFh7lxiXHUwMDE2XHUwMDEyXHUwMDAwXHUwMDAx4WYlbLJcXKEmQ/lvXHUwMDEwao+hu18k1Fx1MDAxZS3MokLtOitcdTAwMWNTIZS6Lijk3F/2R+eGXHUwMDE0oLqsT3KSiDBD4F3AglRKJ0eEXHUwMDEyLiEwiFx1MDAwZkpcdTAwMTdcdTAwMGaBevdcdTAwMTBGXHUwMDA0eDZD7KJKS2lysX1cXJ5zpKZJXHUwMDA0N5t88rtMXHUwMDAzR3UhRiSQmyn/m5H3vDR4vDh8mnyvunuWqMjagPFKuri8xEpZ2USEZVx1MDAxYpfnuMBcYoSSaVx1MDAxOCBPXHUwMDAyk37lg2s/7oHhiHp5oNKmkdA1X1x1MDAwNIJCQXf1vMCYZVx1MDAxNtSPNmZy3ng6UlQnKKKYXCJcIqFYZZHl3uLpiFx1MDAxNKBnwLiggVx1MDAwNZ0oyFx1MDAxZi27fY2CTC1vRlx1MDAwNOBMXHUwMDAwQlbbJZaNXCJcdTAwMDW0mCDCXHUwMDE2QVx1MDAwNGlcdTAwMDDZ+1x1MDAwNP3NwEii7ynpva7vQYlcdTAwMTdhXHUwMDExjlx0Rlx1MDAxMu2i/rnN5XM1hbbHiUImp8SYIFx1MDAxNXpcdTAwMWNcZmRCXHUwMDEyKGTWYZFF0Fx1MDAxZm6l7f1UQf9cdTAwMTh1592C/rPjkjh1yOy5eMual/Tg9rIoXHJcdTAwMTmOiVx1MDAwZkHMs1x1MDAwMlmEXHUwMDFigGJjolx1MDAxN1FcdTAwMDCuIacgenhwd4yNXHUwMDBmtbWtPVwiXHUwMDE4S1x1MDAwMVx0MO89XG6v2C1cdTAwMTikkIHN9lx1MDAxZW2mU1x1MDAxZPXPXHUwMDFmYft+fHU/v2y0XHUwMDBlv+AyZ/jDznYytyZcdTAwMDGRXHUwMDA1jz3Mw1hgKHYrsIVfUCmFXHUwMDE0UoNcdTAwMGJxd9ZTKEhSJLRBwJDEgpurK6Hw84CVYlxidTnRTcJ20TltTCmbYKuKbVx1MDAwN9XDSa47zLlcdTAwMWQ71+hPJz56+pAjn2K4ZZ3jXCI7kFxy250+zEpcdTAwMGaD52LRqlSO0Y171LmyXHUwMDFmXHUwMDEys1x1MDAxZFnL2NFcdTAwMDfJXHUwMDE21LTp4zApJlx1MDAxMFx1MDAxYnb97Fwi9dnA29qaXHUwMDAz9VneukqFybFASHjFXHUwMDE4vcFcYiC4WY1GM1x0vizGUn5uXHUwMDAzMLu7YzdfT4etnuVcdTAwMWM9d1x1MDAxM4XU/4x67LF7+1xcYVxcXFxU9tHTbHDa6OGDvPmxqaNcdTAwMDJcdTAwMDJcbmXQZ5Q5XHUwMDE33f+okjFUrOFcdTAwMTChXHUwMDE4XHUwMDFjbm1cIv42QLtPQaNSSqzVLVx1MDAxM6LCj0zEXFxwtY5QxvFcdTAwMDBcdTAwMDF1XHUwMDFk4a0sxGXJlFx1MDAwZvbox9DUOnFcdTAwMWGanVxyXV6d3ZOLUa1cdTAwMGI7vHs3lvfNo+/F5HTJxVo+XHKFWP2eUm9cdKJYXHUwMDAwU0E1XHUwMDExW3FUZn34099cdTAwMDK1ja3pUUJcdIWAxk1cdTAwMWXe1Kf1mLnQ9VxcsiTHOFx1MDAwYpGUL/PHY3g4vKm2raeK83zSOrhISo5cdTAwMGZ1IFx1MDAxZerNa/tuXHUwMDAw2PxLb9S5fbzMjlx1MDAxY6V3x/FW5GjuZVx1MDAwMnLknuo6r6hcdTAwMTPRqDOGy3f1Xkwwa6YgR4FcdJc6XHUwMDBmxVx1MDAwNChcdTAwMTmub0JCuMJcIsy4Jrj2/+BUtVx1MDAxM1x1MDAwMvT4i8TDY1grmLX2MVx1MDAwMe9S7fRcZsyt3vPxzSV6fiw6XHUwMDBmXHUwMDFk+zZ5llx1MDAxYeRsPV+VwoWPXHRCXHRcdTAwMThDWPyc5mirZTe8XqlfUlx1MDAxMPS2T1x1MDAxY+dU+32k0Vx1MDAxY1xyXl1cdTAwMTX4Vlx1MDAxYbdcdTAwMTLpWZ4+/LJcdTAwMWXbT0cnQ5qvXzqo0WbHh/c3X8H19ubooSO7Mn/RXHUwMDFhPVxmvtaO66xU/fbNyYxxdVx1MDAxODkjxo3uf2Q9XHUwMDE4XHUwMDAxg1BkMVCEyJA6vjNJTWDrJ2ddXHUwMDAydaFAajRJvadoXHUwMDA2KlxmXG7BseQy41xuplx1MDAwMlIp8FabuYqzoTPPLUy8XHUwMDBm5tVcdTAwMTjCWudVY8Ozoc5cdTAwMGK7fWX3ru5cdTAwMWZm9e9cYnW7M1x09ocp3LgribFcdTAwMDArXHUwMDA3IFx1MDAwNqzSU59rlVGauY78t8gxXHUwMDE4ZJC/jVxiZIxjk+pcdTAwMWN+kpSQhFKdJfN+tmi/UT8/mfXqe72r0dGAOVJcdTAwMWPgk3dw1EY+d1x1MDAwYlx1MDAxYvdHMXnkc6/alfNGZ3B7QYf2hTubNjuPeZmVhoBcdTAwMTCHnnOwttJcdTAwMTDMs51AQ+BKQVx1MDAxMIviT4D662S/yFx1MDAxZspcdTAwMGIyUv5ghFx1MDAwYlg7xLDQKWDE4Fx1MDAxN9tcdTAwMWQ0YFx1MDAxMkbD5KpcdTAwMDPEjFKhdGtzgChM6kBlrDNcZkj2MWGBXHUwMDAw5nirXHUwMDEyrZX+VHHy59zeaNSfXHUwMDFmOeNcYjVcIl1UeNv0p1x1MDAxOFx1MDAwNlx1MDAwZqQ/RfcjXHUwMDFirVwiWlx1MDAxZUdloOu9o1x1MDAxY0Ksq+oyRqH/ZGbEIC0wKahaI3hRc9mAb2UuYIyVXHUwMDEwoFx1MDAwMkhpKPMgSFx1MDAwMSrgY8H1XHUwMDE2R7orXHUwMDBiZ0C7k1T1XGJNP9fbTlx1MDAxMTRb5TS81Fx1MDAxMZZCKKNjIzd4fK2jzdJcdTAwMWUzzVx1MDAxMFx1MDAwZl2g+rW+NFdcdTAwMGZcdTAwMGLwc2bZ3NEhspw3cVrxLVx1MDAxMYpjsWJRvevec1NYcbREudvRMszXXHUwMDA0TFx1MDAwMIOE60ZcYiRpMHVb0TtcdTAwMTBqkJW6QpmUXFxcdTAwMDRcdTAwMTfo+6RuRyuPUUJQXHUwMDEyWaBcdTAwMThxSVx1MDAxOFwi2lxiX1x1MDAxN4KoQCnQMWNcdTAwMDS4J/i7lIGkQDWQXHUwMDEwVbaY4mKTXGbkXHUwMDA1ZVx1MDAxNVxiKqGkVN22XHUwMDEzglx1MDAwNiE42lpcYkJcdTAwMDH0Olx1MDAxNdRU1IKG2l+SM85921x1MDAwM/9mQjBcdTAwMWa6RPUrsDjfQ1xmRsc6PDIoXHUwMDBmXG7KXHUwMDBlUraDpNokUn9BUFxu/WhBqFx1MDAxYlx1MDAwMSFhmCpxiKCAfJWou2pEXHUwMDAx6sLMklx1MDAxMiYgXHUwMDE0XHUwMDFj8I+ShdFcdTAwMDZvlCyEXHUwMDE4s4LiXHUwMDFjLFx1MDAwMMGUeDdRvFxuQ1DQ3dOHXHUwMDE1SuDNwF7uSURCjYMkXHUwMDFjXHT9sFx1MDAxNVx1MDAxMj2VgVx1MDAwYpxL9Vx1MDAwMMKV9YE8+293wnApXGa/by1cZnXKrz5Q23gmVjC4u/IoY4qU4vFcdTAwMDNShX9cdTAwMTZpXHUwMDE4tkT1K7A430NcdTAwMThGO6j9cogjzDhcdTAwMTBcdTAwMDTrXHUwMDEzcyXxbHh808h+vCwkum5cdTAwMTSAet+q3mpcYoKN+LCyvc73jnsxvaxVrkbCPW1cdTAwMWZcXO2NXHUwMDA3vaDoXHUwMDBiO6dcdTAwMGJIf1x1MDAxZZdf+klEYvztmKG4kzWFP5KdavfQb1x1MDAxM6ZcdTAwMWUnXHUwMDE1f+G+eDU/XHUwMDE4IGSsxUQj3GJISiHxZlx1MDAwMbVl81xm3uK7Sfv26qnTLcGza1wiL/acq/vTilx1MDAwN2C5zcLUP9ZcdE1cdTAwMTClXHUwMDE5XHUwMDFkXHUwMDA1XHUwMDE23f+ogtqQRINSxoHSXHUwMDEzj9nliUXibpImYo0l48y4NY/yUGNLXHUwMDFmuanUyKy32eqlXG7pVj5nT9z3JZ9qZI+PfspEsVx1MDAxOJKLXGJoJ+9YRonYkdIp0isjlJ5BlJ6BqbJOMVgrjlwiKChcYsaViVx1MDAwMoU2WoMlXHUwMDBlaVx1MDAwMVx1MDAxMVxmgTLtlSmmXHUwMDE2Klj5dVZ7d3FcdTAwMDEpZU9Zl1hbljvntEkmuFubXCJQmY3aiWg8P1x1MDAxM1x1MDAwN5w1qyNLXGJCXHUwMDAwkFx1MDAxZlSJ/ycwRcLXqH5cdTAwMDVW5+pxXHUwMDAx6s7MXHUwMDE2SWNcdTAwMDdcdTAwMDCGlKGIsUSYXHUwMDAzzmDwXGaRXHKNkWhcdTAwMDG31lxifW5cdTAwMTUjWFx1MDAxOURcdTAwMTJIjLlcZrTiPayRP15n45M1XHUwMDFh1VxctdqW3fs069qP+0ZNXb+0QrZomJZcdTAwMGX2XHUwMDAyWX/98df/XHUwMDAzZzzy+yJ9Flattened Self-Contained Gateway IRIR With policies and backends attachedxDS TranslationHttpRoute name: route-foo parentRef: httpHttpRouteRule ExtensionRef BackendRefGateway name: httpFaultPolicy name: pol targetRef: route-foo spec: fault: 50%ir.HttpRoute name: route-foo parentRef: httpHttpRouteRule AttachedPolicies *ir.PolicyWrapper *ir.Backendir.Gateway name: httpir.PolicyWrapper name: pol policyIr: filterConfig: proto.AnyPoints toir.GatewayIR name: http routes: - rule: AttachedPolicies *ir.PolicyWrapper *ir.Backendir.PolicyWrapper name: pol policyIr: filterConfig: proto.AnyPoints toCRDs in the clusterRouteRule policyIr: filterConfig: proto.AnyEvnoy RoutePlugin: ApplyForRouteEvnoy Route perFilterConfig: proto.Any \ No newline at end of file diff --git a/examples/plugin/main.go b/examples/plugin/main.go new file mode 100644 index 00000000000..69811ec10b9 --- /dev/null +++ b/examples/plugin/main.go @@ -0,0 +1,246 @@ +package main + +import ( + "context" + "time" + + mutation_v3 "github.com/envoyproxy/go-control-plane/envoy/config/common/mutation_rules/v3" + envoy_core_v3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3" + envoy_config_route_v3 "github.com/envoyproxy/go-control-plane/envoy/config/route/v3" + header_mutationv3 "github.com/envoyproxy/go-control-plane/envoy/extensions/filters/http/header_mutation/v3" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/structpb" + "istio.io/istio/pkg/kube/kclient" + "istio.io/istio/pkg/kube/krt" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + gwv1 "sigs.k8s.io/gateway-api/apis/v1" + + "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/common" + extensionsplug "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugin" + "github.com/kgateway-dev/kgateway/v2/internal/kgateway/ir" + "github.com/kgateway-dev/kgateway/v2/internal/kgateway/plugins" + "github.com/kgateway-dev/kgateway/v2/internal/kgateway/setup" +) + +/****** + +An example plugin, that uses a ConfigMap as a policy. We use a targetRef annotation to attach the +policy to an HTTPRouter. We will then add the key value pairs in the ConfigMap to the metadata of +the envoy route route. + +Exmaple ConfigMap: + +apiVersion: v1 +kind: ConfigMap +metadata: + name: my-configmap + annotations: + targetRef: my-http-route +data: + key1: value1 + key2: value2 + + +To test, use this example HTTPRoute that adds the metadata to a header: + +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: my-http-route +spec: + parentRefs: + - name: gw + rules: + - backendRefs: + - name: example-svc + port: 8080 + filters: + - type: ResponseHeaderModifier + responseHeaderModifier: + add: + - name: my-header-name + value: %METADATA(ROUTE:example.plugin:key1)% + +*****/ + +var ( + configMapGK = schema.GroupKind{ + Group: "", + Kind: "ConfigMap", + } +) + +// Our policy IR. +type configMapIr struct { + creationTime time.Time + metadata *structpb.Struct +} + +var _ ir.PolicyIR = &configMapIr{} + +// in case multiple policies attached to the same resource, we sort by policy creation time. +func (d *configMapIr) CreationTime() time.Time { + return d.creationTime +} + +// Equals is needed because this is in KRT collection. +func (d *configMapIr) Equals(in any) bool { + d2, ok := in.(*configMapIr) + if !ok { + return false + } + return d.creationTime == d2.creationTime && proto.Equal(d.metadata, d2.metadata) +} + +// convert a configmap to our IR. +func configMapToIr(cm *corev1.ConfigMap) *configMapIr { + // When converting to IR, we want to take the translation as close to envoy xDS as we can. + // That's why our IR intentionaly uses structpb.Struct and not map[string]string. + + mdStruct := &structpb.Struct{ + Fields: map[string]*structpb.Value{}, + } + for k, v := range cm.Data { + mdStruct.Fields[k] = structpb.NewStringValue(v) + } + + return &configMapIr{ + creationTime: cm.CreationTimestamp.Time, + metadata: mdStruct, + } +} + +// Configmaps don't have a target ref, so we extract it from the annotations. +func extractTargetRefs(cm *corev1.ConfigMap) []ir.PolicyTargetRef { + return []ir.PolicyTargetRef{{ + Group: gwv1.GroupName, + Kind: "HTTPRoute", + Name: cm.Annotations["targetRef"], + }} + +} + +// Create a collection of our policies. This will be done by converting a configmap collection +// to our policy IR. +func ourPolicies(commoncol *common.CommonCollections) krt.Collection[ir.PolicyWrapper] { + // We create 2 collections here - one for the source config maps, and one for the policy IR. + // Whenever creating a new krtCollection use commoncol.KrtOpts.ToOptions("") to provide the + // collection with common options and a name. It's important so that the collection appears in + // the krt debug page. + + // get a configmap client going + configMapCol := krt.WrapClient(kclient.New[*corev1.ConfigMap](commoncol.Client), commoncol.KrtOpts.ToOptions("ConfigMaps")...) + + // convertIt to policy IR + return krt.NewCollection(configMapCol, func(krtctx krt.HandlerContext, i *corev1.ConfigMap) *ir.PolicyWrapper { + if i.Annotations["targetRef"] == "" { + return nil + } + + var pol = &ir.PolicyWrapper{ + ObjectSource: ir.ObjectSource{ + Group: configMapGK.Group, + Kind: configMapGK.Kind, + Namespace: i.Namespace, + Name: i.Name, + }, + Policy: i, + PolicyIR: configMapToIr(i), + TargetRefs: extractTargetRefs(i), + } + return pol + }, commoncol.KrtOpts.ToOptions("MetadataPolicies")...) + +} + +// Our translation pass struct. This holds translation specific state. +// In our case, we check if our policy was applied to a route and if so, we add a filter. +type ourPolicyPass struct { + // Add the unimplemented pass so we don't have to implement all the methods. + ir.UnimplementedProxyTranslationPass + + // We keep track of which filter chains need our filter. + filterNeeded map[string]bool +} + +// ApplyForRoute is called when a an HTTPRouteRule is being translated to an envoy route. +func (s *ourPolicyPass) ApplyForRoute(ctx context.Context, pCtx *ir.RouteContext, out *envoy_config_route_v3.Route) error { + // get our policy IR. Kgateway used the targetRef to attach the policy to the HTTPRoute. and now as it + // translates the HTTPRoute to xDS, it calls our plugin and passes the policy for the plugin's translation pass to do the + // policy to xDS translation. + cmIr, ok := pCtx.Policy.(*configMapIr) + if !ok { + // should never happen + return nil + } + // apply the metadata from our IR to envoy's route object + if out.Metadata == nil { + out.Metadata = &envoy_core_v3.Metadata{} + } + out.Metadata.FilterMetadata["example.plugin"] = cmIr.metadata + + // mark that we need a filter for this filter chain. + if s.filterNeeded == nil { + s.filterNeeded = map[string]bool{} + } + s.filterNeeded[pCtx.FilterChainName] = true + + return nil +} + +func (s *ourPolicyPass) HttpFilters(ctx context.Context, fc ir.FilterChainCommon) ([]plugins.StagedHttpFilter, error) { + if !s.filterNeeded[fc.FilterChainName] { + return nil, nil + } + // Add an http filter to the chain that adds a header indicating metadata was added. + return []plugins.StagedHttpFilter{ + plugins.MustNewStagedFilter("example_plugin", + &header_mutationv3.HeaderMutation{ + Mutations: &header_mutationv3.Mutations{ + ResponseMutations: []*mutation_v3.HeaderMutation{ + { + Action: &mutation_v3.HeaderMutation_Append{ + Append: &envoy_core_v3.HeaderValueOption{ + Header: &envoy_core_v3.HeaderValue{ + Key: "x-metadata-added", + Value: "true", + }, + }, + }, + }, + }, + }, + }, + plugins.BeforeStage(plugins.AcceptedStage))}, nil + +} + +// A function that initializes our plugins. +func pluginFactory(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin { + return []extensionsplug.Plugin{ + { + ContributesPolicies: extensionsplug.ContributesPolicies{ + configMapGK: extensionsplug.PolicyPlugin{ + Name: "metadataPolicy", + NewGatewayTranslationPass: func(ctx context.Context, tctx ir.GwTranslationCtx) ir.ProxyTranslationPass { + // Return a fresh new translation pass + return &ourPolicyPass{} + }, + // Provide a collection of our policies in IR form. + Policies: ourPolicies(commoncol), + }, + }, + }, + } +} + +func main() { + + // TODO: move setup.StartGGv2 from internal to public. + // Start Kgateway and provide our plugin. + // This demonstrates how to start Kgateway with a custom plugin. + // This binary is the control plane. normally it would be packaged in a docker image and run + // in a k8s cluster. + setup.StartGGv2(context.Background(), pluginFactory, nil) +} diff --git a/internal/kgateway/controller/start.go b/internal/kgateway/controller/start.go index 424ffa8004c..7f178ef083a 100644 --- a/internal/kgateway/controller/start.go +++ b/internal/kgateway/controller/start.go @@ -65,7 +65,7 @@ type StartConfig struct { RestConfig *rest.Config // ExtensionsFactory is the factory function which will return an extensions.K8sGatewayExtensions // This is responsible for producing the extension points that this controller requires - ExtraPlugins []extensionsplug.Plugin + ExtraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin Client istiokube.Client @@ -174,11 +174,13 @@ func NewControllerBuilder(ctx context.Context, cfg StartConfig) (*ControllerBuil }, nil } -func pluginFactoryWithBuiltin(extraPlugins []extensionsplug.Plugin) extensions2.K8sGatewayExtensionsFactory { +func pluginFactoryWithBuiltin(extraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin) extensions2.K8sGatewayExtensionsFactory { return func(ctx context.Context, commoncol *common.CommonCollections) extensionsplug.Plugin { plugins := registry.Plugins(ctx, commoncol) plugins = append(plugins, krtcollections.NewBuiltinPlugin(ctx)) - plugins = append(plugins, extraPlugins...) + if extraPlugins != nil { + plugins = append(plugins, extraPlugins(ctx, commoncol)...) + } return registry.MergePlugins(plugins...) } } diff --git a/internal/kgateway/extensions2/plugin/plugin.go b/internal/kgateway/extensions2/plugin/plugin.go index 05ecfd53a05..7a4de36500d 100644 --- a/internal/kgateway/extensions2/plugin/plugin.go +++ b/internal/kgateway/extensions2/plugin/plugin.go @@ -83,7 +83,7 @@ type GwTranslatorFactory func(gw *gwv1.Gateway) KGwTranslator type ContributesPolicies map[schema.GroupKind]PolicyPlugin type Plugin struct { - ContributesPolicies + ContributesPolicies ContributesPolicies ContributesBackends map[schema.GroupKind]BackendPlugin ContributesGwTranslator GwTranslatorFactory // extra has sync beyong primary resources in the collections above diff --git a/internal/kgateway/ir/iface.go b/internal/kgateway/ir/iface.go index 7325211eef9..80ce70a42c6 100644 --- a/internal/kgateway/ir/iface.go +++ b/internal/kgateway/ir/iface.go @@ -49,14 +49,18 @@ func (r *RouteBackendContext) GetTypedConfig(key string) proto.Message { } type RouteContext struct { - Policy PolicyIR - In HttpRouteRuleMatchIR + FilterChainName string + Policy PolicyIR + In HttpRouteRuleMatchIR } type HcmContext struct { Policy PolicyIR } +// ProxyTranslationPass represents a single translation pass for a gateway. It can hold state +// for the duration of the translation. +// Each of the functions here will be called in the order they appear in the interface. type ProxyTranslationPass interface { // Name() string // called 1 time for each listener @@ -65,12 +69,13 @@ type ProxyTranslationPass interface { pCtx *ListenerContext, out *envoy_config_listener_v3.Listener, ) - // called 1 time per filter chain after listeners + // called 1 time per filter chain after listeners and allows tweaking HCM settings. ApplyHCM(ctx context.Context, pCtx *HcmContext, out *envoy_hcm.HttpConnectionManager) error - // called 1 time for all the routes in a filter chain. + // called 1 time for all the routes in a filter chain. Use this to set default PerFilterConfig + // No policy is provided here. ApplyRouteConfigPlugin( ctx context.Context, pCtx *RouteConfigContext, @@ -81,18 +86,16 @@ type ProxyTranslationPass interface { pCtx *VirtualHostContext, out *envoy_config_route_v3.VirtualHost, ) - // called 0 or more times + // called 0 or more times (one for each route) + // Applies policy for an HTTPRoute that has a policy attached via a targetRef. + // The output configures the envoy_config_route_v3.Route ApplyForRoute( ctx context.Context, pCtx *RouteContext, out *envoy_config_route_v3.Route) error - // runs for policy applied - ApplyForRouteBackend( - ctx context.Context, - policy PolicyIR, - pCtx *RouteBackendContext, - ) error - // no policy applied + + // no policy applied - this is called for every backend in a route. + // For this to work the backend needs to register itself as a policy. TODO: rethink this. ApplyForBackend( ctx context.Context, pCtx *RouteBackendContext, @@ -100,13 +103,20 @@ type ProxyTranslationPass interface { out *envoy_config_route_v3.Route, ) error - // called 1 time per listener - // if a plugin emits new filters, they must be with a plugin unique name. - // any filter returned from route config must be disabled, so it doesnt impact other routes. + // Applies a policy attached to a specific Backend (via extensionRef on the BackendRef). + ApplyForRouteBackend( + ctx context.Context, + policy PolicyIR, + pCtx *RouteBackendContext, + ) error + + // called 1 time per filter-chain. + // If a plugin emits new filters, they must be with a plugin unique name. + // filters added to impact specific routes should be disabled on the listener level, so they don't impact other routes. HttpFilters(ctx context.Context, fc FilterChainCommon) ([]plugins.StagedHttpFilter, error) NetworkFilters(ctx context.Context) ([]plugins.StagedNetworkFilter, error) - // called 1 time (per envoy proxy). replaces GeneratedResources + // called 1 time (per envoy proxy). replaces GeneratedResources and allows adding clusters to the envoy. ResourcesToAdd(ctx context.Context) Resources } @@ -157,18 +167,21 @@ type PolicyIR interface { } type PolicyWrapper struct { + // A reference to the original policy object ObjectSource `json:",inline"` - Policy metav1.Object + // The policy object itself. TODO: we can probably remove this + Policy metav1.Object // Errors processing it for status. // note: these errors are based on policy itself, regardless of whether it's attached to a resource. // TODO: change for conditions Errors []error - // original object. ideally with structural errors removed. + // The IR of the policy objects. ideally with structural errors removed. // Opaque to us other than metadata. PolicyIR PolicyIR + // Where to attach the policy. This usually comes from the policy CRD. TargetRefs []PolicyTargetRef } @@ -199,6 +212,8 @@ var ( ) type PolicyRun interface { + // Allocate state for single listener+rotue translation pass. NewGatewayTranslationPass(ctx context.Context, tctx GwTranslationCtx) ProxyTranslationPass + // Process cluster for a backend ProcessBackend(ctx context.Context, in BackendObjectIR, out *envoy_config_cluster_v3.Cluster) error } diff --git a/internal/kgateway/setup/ggv2setup.go b/internal/kgateway/setup/ggv2setup.go index a205cf93ea1..f1432565475 100644 --- a/internal/kgateway/setup/ggv2setup.go +++ b/internal/kgateway/setup/ggv2setup.go @@ -21,6 +21,7 @@ import ( "github.com/kgateway-dev/kgateway/v2/internal/kgateway/admin" "github.com/kgateway-dev/kgateway/v2/internal/kgateway/controller" + "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/common" extensionsplug "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugin" "github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/settings" "github.com/kgateway-dev/kgateway/v2/internal/kgateway/krtcollections" @@ -54,7 +55,7 @@ func createKubeClient(restConfig *rest.Config) (istiokube.Client, error) { func StartGGv2( ctx context.Context, - extraPlugins []extensionsplug.Plugin, + extraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin, extraGwClasses []string, // TODO: we can remove this and replace with something that watches all GW classes with our controller name ) error { logger := contextutils.LoggerFrom(ctx) @@ -99,7 +100,7 @@ func StartGGv2WithConfig( setupOpts *controller.SetupOpts, restConfig *rest.Config, uccBuilder krtcollections.UniquelyConnectedClientsBulider, - extraPlugins []extensionsplug.Plugin, + extraPlugins func(ctx context.Context, commoncol *common.CommonCollections) []extensionsplug.Plugin, extraGwClasses []string, // TODO: we can remove this and replace with something that watches all GW classes with our controller name ) error { ctx = contextutils.WithLogger(ctx, "k8s") diff --git a/internal/kgateway/translator/irtranslator/route.go b/internal/kgateway/translator/irtranslator/route.go index f181166db11..1fac102f102 100644 --- a/internal/kgateway/translator/irtranslator/route.go +++ b/internal/kgateway/translator/irtranslator/route.go @@ -201,8 +201,9 @@ func (h *httpRouteConfigurationTranslator) runRoutePlugins(ctx context.Context, } for _, pol := range pols { pctx := &ir.RouteContext{ - Policy: pol.PolicyIr, - In: in, + FilterChainName: h.fc.FilterChainName, + Policy: pol.PolicyIr, + In: in, } err := pass.ApplyForRoute(ctx, pctx, out) if err != nil {