3737kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.0.2/v1-manifests.yaml
3838```
3939
40+ ### Install the Gateway
41+
42+ Choose one of the following options to install Gateway.
43+
44+ === "GKE"
45+
46+ Nothing to install here, you can move to the next [section](#deploy-the-inferencepool-and-endpoint-picker-extension)
47+
48+ === "Istio"
49+
50+ 1. Requirements
51+ - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
52+
53+ 2. Install Istio
54+
55+ ```
56+ TAG=$(curl https://storage.googleapis.com/istio-build/dev/1.28-dev)
57+ # on Linux
58+ wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
59+ tar -xvf istioctl-$TAG-linux-amd64.tar.gz
60+ # on macOS
61+ wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-osx.tar.gz
62+ tar -xvf istioctl-$TAG-osx.tar.gz
63+ # on Windows
64+ wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-win.zip
65+ unzip istioctl-$TAG-win.zip
66+
67+ ./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true
68+ ```
69+
70+ === "Kgateway"
71+
72+ 1. Requirements
73+
74+ - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
75+ - [Helm](https://helm.sh/docs/intro/install/) installed.
76+
77+ 2. Set the Kgateway version and install the Kgateway CRDs.
78+
79+ ```bash
80+ KGTW_VERSION=v2.1.0
81+ helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
82+ ```
83+
84+ 3. Install Kgateway
85+
86+ ```bash
87+ helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
88+ ```
89+
90+ === "Agentgateway"
91+
92+ 1. Requirements
93+
94+ - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
95+ - [Helm](https://helm.sh/docs/intro/install/) installed.
96+
97+ 2. Set the Kgateway version and install the Kgateway CRDs.
98+
99+ ```bash
100+ KGTW_VERSION=v2.1.0
101+ helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
102+ ```
103+
104+ 3. Install Kgateway
105+
106+ ```bash
107+ helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true --set agentgateway.enabled=true
108+ ```
109+
40110### Deploy the InferencePool and Endpoint Picker Extension
41111
42112 Install an InferencePool named ` vllm-llama3-8b-instruct ` that selects from endpoints with label ` app: vllm-llama3-8b-instruct ` and listening on port 8000. The Helm install command automatically installs the endpoint-picker, InferencePool along with provider specific resources.
@@ -59,7 +129,7 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
59129 See [Deploy Inference Gateways](https://cloud.google.com/kubernetes-engine/docs/how-to/deploy-gke-inference-gateway)
60130 for detailed instructions.
61131
62- 2. Deploy Inference Gateway:
132+ 2. Deploy the Inference Gateway:
63133
64134 ```bash
65135 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.2/config/manifests/gateway/gke/gateway.yaml
@@ -89,34 +159,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
89159 Please note that this feature is currently in an experimental phase and is not intended for production use.
90160 The implementation and user experience are subject to changes as we continue to iterate on this project.
91161
92- 1. Requirements
93-
94- - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
95-
96- 2. Install Istio
97-
98- ```
99- TAG=$(curl https://storage.googleapis.com/istio-build/dev/1.28-dev)
100- # on Linux
101- wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
102- tar -xvf istioctl-$TAG-linux-amd64.tar.gz
103- # on macOS
104- wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-osx.tar.gz
105- tar -xvf istioctl-$TAG-osx.tar.gz
106- # on Windows
107- wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-win.zip
108- unzip istioctl-$TAG-win.zip
109-
110- ./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing --set values.pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true
111- ```
112-
113- 3. If your EPP uses secure serving with self-signed certs (default), temporarily bypass TLS verification:
162+ 1. If your EPP uses secure serving with self-signed certs (default), temporarily bypass TLS verification:
114163
115164 ```bash
116165 kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml
117166 ```
118167
119- 4 . Deploy Gateway
168+ 2 . Deploy the Inference Gateway
120169
121170 ```bash
122171 kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml
@@ -129,13 +178,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
129178 inference-gateway inference-gateway <MY_ADDRESS> True 22s
130179 ```
131180
132- 5 . Deploy the HTTPRoute
181+ 3 . Deploy the HTTPRoute
133182
134183 ```bash
135184 kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/httproute.yaml
136185 ```
137186
138- 6 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
187+ 4 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
139188
140189 ```bash
141190 kubectl get httproute llm-route -o yaml
@@ -144,28 +193,9 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
144193=== "Kgateway"
145194
146195 [Kgateway](https://kgateway.dev/) is a Gateway API and Inference Gateway
147- [conformant](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/conformance/reports/v1.0.0/gateway/kgateway)
148- gateway. Follow these steps to run Kgateway:
149-
150- 1. Requirements
151-
152- - [Helm](https://helm.sh/docs/intro/install/) installed.
153- - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
196+ [conformant](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/conformance/reports/v1.0.0/gateway/kgateway) gateway. Follow these steps to run Kgateway:
154197
155- 2. Set the Kgateway version and install the Kgateway CRDs.
156-
157- ```bash
158- KGTW_VERSION=v2.1.0
159- helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
160- ```
161-
162- 3. Install Kgateway
163-
164- ```bash
165- helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
166- ```
167-
168- 4. Deploy the Gateway
198+ 1. Deploy the Inference Gateway
169199
170200 ```bash
171201 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.2/config/manifests/gateway/kgateway/gateway.yaml
@@ -178,13 +208,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
178208 inference-gateway kgateway <MY_ADDRESS> True 22s
179209 ```
180210
181- 5 . Deploy the HTTPRoute
211+ 2 . Deploy the HTTPRoute
182212
183213 ```bash
184214 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.2/config/manifests/gateway/kgateway/httproute.yaml
185215 ```
186216
187- 6 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
217+ 3 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
188218
189219 ```bash
190220 kubectl get httproute llm-route -o yaml
@@ -196,25 +226,7 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
196226 Agentgateway integrates with [Kgateway](https://kgateway.dev/) as it's control plane. Follow these steps to run Kgateway with the agentgateway
197227 data plane:
198228
199- 1. Requirements
200-
201- - [Helm](https://helm.sh/docs/intro/install/) installed.
202- - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
203-
204- 2. Set the Kgateway version and install the Kgateway CRDs.
205-
206- ```bash
207- KGTW_VERSION=v2.1.0
208- helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
209- ```
210-
211- 3. Install Kgateway
212-
213- ```bash
214- helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true --set agentgateway.enabled=true
215- ```
216-
217- 4. Deploy the Gateway
229+ 1. Deploy the Inference Gateway
218230
219231 ```bash
220232 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.2/config/manifests/gateway/agentgateway/gateway.yaml
@@ -227,13 +239,13 @@ kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extens
227239 inference-gateway agentgateway <MY_ADDRESS> True 22s
228240 ```
229241
230- 5 . Deploy the HTTPRoute
242+ 2 . Deploy the HTTPRoute
231243
232244 ```bash
233245 kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api-inference-extension/refs/tags/v1.0.2/config/manifests/gateway/agentgateway/httproute.yaml
234246 ```
235247
236- 6 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
248+ 3 . Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
237249
238250 ```bash
239251 kubectl get httproute llm-route -o yaml
0 commit comments