design 10574: transformations (#10689)

Co-authored-by: Jenny Shu <[email protected]>
Co-authored-by: Yuval Kohavi <[email protected]>

Author: 3 people

design/10574.md

@@ -0,0 +1,173 @@
+<!--
+**Note:** When your Enhancement Proposal (EP) is complete, all of these comment blocks should be removed.
+
+This template is inspired by the Kubernetes Enhancement Proposal (KEP) template: https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/0000-kep-process/README.md
+
+To get started with this template:
+
+- [ ] **Create an issue in kgateway-dev/kgateway**
+- [ ] **Make a copy of this template.**
+  `EP-[ID]: [Feature/Enhancement Name]`, where `ID` is the issue number (with no
+  leading-zero padding) assigned to your enhancement above.
+- [ ] **Fill out this file as best you can.**
+  At minimum, you should fill in the "Summary" and "Motivation" sections.
+- [ ] **Create a PR for this EP.**
+  Assign it to maintainers with relevant context.
+- [ ] **Merge early and iterate.**
+  Avoid getting hung up on specific details and instead aim to get the goals of
+  the EP clarified and merged quickly. The best way to do this is to just
+  start with the high-level sections and fill out details incrementally in
+  subsequent PRs.
+
+Just because a EP is merged does not mean it is complete or approved. Any EP
+marked as `provisional` is a working document and subject to change. You can
+denote sections that are under active debate as follows:
+
+```
+<<[UNRESOLVED optional short context or usernames ]>>
+Stuff that is being argued.
+<<[/UNRESOLVED]>>
+```
+
+When editing EPS, aim for tightly-scoped, single-topic PRs to keep discussions
+focused. If you disagree with what is already in a document, open a new PR
+with suggested changes.
+
+One EP corresponds to one "feature" or "enhancement" for its whole lifecycle. Once a feature has become
+"implemented", major changes should get new EPs.
+-->
+# EP-10574: Basic Transformations for Request and Response 
+
+
+* Issue: [#10574](https://github.com/kgateway-dev/kgateway/issues/10574)
+
+## Background 
+The previous project (gloo) had a pretty widely used staged transformation concept which could be slotted into both request and response paths. This Transformation filter most critically was used to perform simple modification with inja templates and to extract data for usage in access logging. Because some applications of transformation can be rather heavy (rewrite whole responses perform regex lookups) this was built as a custom filter and packaged with a seperate repository called envoy-gloo to make sure that it is secure and performant. 
+
+
+
+## Motivation
+Being able to peform conditional header mutation and enrich access logs via dynamic metadata in a performant way is a great quality of life feature that the gloo had and we should support.
+
+
+### Goals
+* Allow for extraction of data from a header and place it in dynamic metadata or filter state such that it can be pulled into access logs
+* Allow for mutations that are based on the contents of the request / response 
+* Have transformations support be stably defined and not require a network hop
+* Have an opinionated api to keep functionality be locked to transforming envoy constructs such as filter state, headers, body, trailers and metadata
+* Keep all kgateway implementation specifics in kgateway repository.
+
+
+### Non-Goals 
+* Have full feature parity with gloo transformations (at first)
+* Keep the same user facing api as gloo
+
+## Implementation Details
+As part of this we should look at leveraging the quickly stabilizing [Dynamic Module support](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/advanced/dynamic_modules) to reduce overhead in the repository and grant us the potential to use raw envoy releases as the default standard for kgateway builds. 
+An example of how we might start to do this can be found in [poc](https://github.com/kgateway-dev/kgateway/pull/10677). This would allow for side by side of old and new functionality to make sure that as users migrate to kgateway we can expose a single user facing api with the ability to filp the underlying engine on the fly.
+
+In order to drive this side by side comparison we can use a set of environment variables as we shore up any gaps in implementation.
+Lets consider the following situation where dynamic modules are enabled as (dual filters)[https://github.com/alyssawilk/envoy/blob/main/source/docs/upstream_filters.md] and we want to transition the ability to get host metadata. In this case we could add a "UPSTREAM_TRANSFORM_USE_LEGACY" variable, shift the default configuration for upstream filter configuration to use dynamic modules with the ability to revert via setting the variable to true. After a reasonable period and based on any feedback we remove the environment variable and the code path. 
+
+Once we reach full parity and remove all temporary environment variables we could be entirely on upstream envoy and contain the entirety of kgateway functionality in the kgateway repository!
+
+At first we should start with mutating headers via rich inja templates and settinginformation in one of dynamic metadata or filter state so that we can enrich access logs.
+
+Given the above we should start with a minimal API where we expose Transformations as something like the following
+
+```
+type TransformationPolicy struct {
+	// +optional
+	Request *Transform `json:"request,omitempty"`
+	// +optional
+	Response *Transform `json:"response,omitempty"`
+}
+
+type Transform struct {
+
+	// +optional
+	// +listType=map
+	// +listMapKey=name
+	// +kubebuilder:validation:MaxItems=16
+	Set []HeaderTransformation `json:"set,omitempty"`
+
+	// +optional
+	// +listType=map
+	// +listMapKey=name
+	// +kubebuilder:validation:MaxItems=16
+	Add []HeaderTransformation `json:"add,omitempty"`
+
+	// +optional
+	// +listType=set
+	// +kubebuilder:validation:MaxItems=16
+	Remove []string `json:"remove,omitempty"`
+
+	// +optional
+	// +listType=map
+	// +listMapKey=name
+	// +kubebuilder:validation:MaxItems=16
+	SetMetdata []MetadataTransformation `json:"set,omitempty"`
+
+}
+
+type HeaderTransformation struct {
+
+	// +required
+	Name  gwv1.HeaderName `json:"name,omitempty"`
+	Value InjaTemplate    `json:"value,omitempty"`
+}
+
+type MetadataTransformation struct {
+
+  // +required
+  Namespace string      `json:"name,omitempty"`
+	// +required
+	Name  string          `json:"name,omitempty"`
+	Value InjaTemplate    `json:"value,omitempty"`
+}
+
+```
+
+We then can start building common tooling to enhance inja to support repeated use cases.
+For example the gloo supported the following inja enhancements:
+ "substring" , "trim" , "base64_encode" , "base64url_encode" , "base64_decode" , "base64url_decode" , "replace_with_random" , "raw_string" , "header" , "request_header" , "extraction" , "body" , "dynamic_metadata" , "data_source" , "host_metadata" , "cluster_metadata" , "context" , "env"
+
+In order to properly implement enriching access logs with information based on headers we would minimally need the "header" extension.
+
+Given this the first addition here should contain the following functions:
+"base64_encode", "header", "replace_with_random"
+
+## Dependency
+A main differentiator of the classic transformation filter is its ability to make complex templated substitutions based on the current state of the request.
+
+A simple version of this is where an internal system returns incompatible or non-compliant status codes and could be achieved with the proposed api something like this.
+
+ResponseTransformation.Set["status": "{% if header(status) == \"418\" %} 405 {% else %} header(status)  {% endif %}}]
+
+In order to power this we should rely on a mature inja implementation such as gloo's previous reliance on (pantor's inja)[https://github.com/solo-io/inja].
+We could call out to the same library to maintain compatibility but it would make adding bindings much harder for contributors.
+
+In order to simplify contributions we should adopt a rust based inja implementation such as (minijinja)[https://docs.rs/minijinja/latest/minijinja/]
+
+As part of this investigation we tested advanced inja functionality that has been leveraged by users in gloo and made sure that it behaves the same. Testing was not exhaustive and identified that minijinja has parity as long as adjacent_loop_items was not disabled.
+
+
+### Plugin
+Given that transformations are tightly coupled with route based decisions this new transformation policy should be part of the existing route policy and be colocated in the code base with the existing routepolicy.
+
+
+### Test Plan 
+Testing should include unit tests per Inja extension, end-to-end tests for basic inja functionality and a integration test with enriched access logs to prove out the two user stories.
+
+
+## Alternatives
+Port the existing Transformation from the old project and figure out how to migrate the custom envoy builds off of the legacy cloud builder run by Solo.io onto github actions.
+To be precise this is to mirror some version of the envoy-gloo repo to kgateway-dev and make it be able to build without Solo.io vendor lock.
+
+
+
+## Open Questions
+What is the timeline to removal of the current extended envoy image.
+How do we want to handle route level overrides. 
+
+