MTA-7133-LLM-Proxy-Disconnection-Network-Isolation-Procedures

docs/developer-lightspeed-guide/master.adoc

@@ -23,4 +23,6 @@ include::topics/developer-lightspeed/assembly_run-dev-lightspeed-analysis.adoc[l
 
 include::topics/developer-lightspeed/con_developer-lightspeed-logs.adoc[leveloffset=+1]
 
+include::topics/developer-lightspeed/proc_emergency-llm-proxy-shutdown.adoc[leveloffset=+1]
+
 :!mta-developer-lightspeed:

docs/topics/developer-lightspeed/proc_emergency-llm-proxy-shutdown.adoc

@@ -0,0 +1,264 @@
+// Module included in the following assemblies:
+//
+// * docs/developer-lightspeed-guide/master.adoc
+
+:_template-generated: 2026-07-05
+
+:_mod-docs-content-type: PROCEDURE
+
+[id="emergency-system-isolation-llm-proxy-shutdown_{context}"]
+= Shutting down the LLM proxy for emergency system isolation
+
+[role="_abstract"]
+You can stop all outbound large language model (LLM) traffic from developer integrated development environments (IDEs) during a security incident. The procedure differs for centralized configuration and standalone mode.
+
+[WARNING]
+====
+Perform these steps only under the direction of your security or incident-response team. Disabling the proxy immediately severs all in-progress LLM sessions for every connected developer without warning.
+====
+
+[id="centralized-config-mode-shutdown_{context}"]
+== Centralized configuration mode (Operator or Red{nbsp}Hat OpenShift Container Platform deployments)
+
+In centralized deployments, the MTA Operator manages the MTA proxy. To disable the proxy, you must set `kai_llm_proxy_enabled: false` in the Tackle custom resource (CR). The Operator then terminates the proxy and does not re-create it.
+
+[IMPORTANT]
+====
+Do not directly scale the proxy `Deployment` object to zero replicas. The MTA Operator reconciles managed resources and re-creates any manually scaled-down pods. Editing the Tackle CR is the only supported method to permanently disable the proxy.
+====
+
+[id="disable-proxy-web-console_{context}"]
+=== Disabling the LLM proxy by using the OpenShift Container Platform web console
+
+.Prerequisites
+* You are logged in to the Red{nbsp}Hat OpenShift Container Platform web console with the `cluster-admin` role or an equivalent role that grants write access to the MTA namespace.
+* You have identified the namespace where MTA is installed (for example, `mta` or `openshift-mta`).
+
+.Procedure
+
+. In the OpenShift Container Platform web console, switch to the **Administrator** perspective.
+. Click **Operators** > **Installed Operators**.
+. From the **Project** list, select the MTA namespace (for example, `openshift-mta`).
+. Click **Migration Toolkit for Applications**.
+. Click the **Tackle** tab, and then click the name of your Tackle instance (for example, `mta`).
+. Click **Actions** > **Edit Tackle**.
+. In the YAML editor, find the `spec` section and set the `kai_llm_proxy_enabled` field to `false`:
++
+[source,yaml]
+----
+spec:
+  kai_llm_proxy_enabled: false
+----
+. Click **Save**.
++
+The Operator reconciles the change and terminates the proxy pod.
+
+.Verification
+. To verify that no proxy pods are running, click **Workloads** > **Pods**.
+. From the **Project** list, select the MTA namespace.
+. Verify that no pod with `proxy` in the name has a `Running` status.
+
+[id="disable-proxy-cli_{context}"]
+=== Disabling the LLM proxy by using the CLI
+
+You can use this method for scripted incident response or when the web console is unavailable.
+
+.Prerequisites
+* You have installed the OpenShift CLI (`oc`).
+* You are logged in to the cluster with the `cluster-admin` role.
+
+.Procedure
+
+. Patch the Tackle CR to disable the LLM proxy:
++
+[source,terminal,subs="+quotes"]
+----
+$ *oc patch tackle __<tackle_instance_name>__ -n __<mta_namespace>__ \
+  --type=merge -p '{"spec":{"kai_llm_proxy_enabled":false}}'*
+----
++
+.Example
+[source,terminal,subs="+quotes"]
+----
+$ *oc patch tackle mta -n openshift-mta \
+  --type=merge -p '{"spec":{"kai_llm_proxy_enabled":false}}'*
+----
++
+.Expected output
+[source,terminal]
+----
+tackle.tackle.konveyor.io/mta patched
+----
+
+. Verify that no proxy pods are running:
++
+[source,terminal,subs="+quotes"]
+----
+$ *oc get pods -n __<mta_namespace>__ | grep proxy*
+----
++
+.Expected output
+[source,terminal]
+----
+No resources found in mta namespace.
+----
+
+[id="developer-impact_{context}"]
+=== Expected behavior on developer workstations
+
+After the Operator terminates the proxy pod:
+
+* Any in-progress LLM request from an IDE client receives an immediate `503 Service Unavailable` or `connection refused` error.
+* No further LLM requests can be sent until the proxy is re-enabled.
+* Standard IDE functionality, such as code editing, file navigation, and build tools, is unaffected. Only the Developer Lightspeed AI assistant stops responding.
+* Developers see an error indicator in the Lightspeed panel of their IDE. The exact message depends on the IDE client, but it is typically the following message:
++
+----
+Error: Unable to connect to the LLM proxy service. Contact your administrator.
+----
+
+[id="reenable-proxy_{context}"]
+=== Re-enabling the LLM proxy after an incident
+
+Re-enable the proxy only after your security team confirms that the incident is resolved and the environment is safe.
+
+.Procedure by using the OpenShift Container Platform web console
+
+. Click **Operators** > **Installed Operators**, and select the **Tackle** instance.
+. Click **Actions** > **Edit Tackle**.
+. Set the `kai_llm_proxy_enabled` field to `true`:
++
+[source,yaml]
+----
+spec:
+  kai_llm_proxy_enabled: true
+----
+. Click **Save**, and wait for the proxy pod to reach the `Running` status.
+
+.Procedure by using the CLI
+
+. Patch the Tackle CR to enable the LLM proxy:
++
+[source,terminal,subs="+quotes"]
+----
+$ *oc patch tackle __<tackle_instance_name>__ -n __<mta_namespace>__ \
+  --type=merge -p '{"spec":{"kai_llm_proxy_enabled":true}}'*
+----
+
+.Verification
+
+. Verify that a proxy pod is running:
++
+[source,terminal,subs="+quotes"]
+----
+$ *oc get pods -n __<mta_namespace>__ | grep proxy*
+----
+. Ask a developer to submit a test query from their IDE and confirm that a valid response is returned.
+
+[id="standalone-mode-shutdown_{context}"]
+== Standalone mode (manual desktop setup)
+
+In standalone mode, there is no central proxy component to disable. Each developer IDE connects directly to the configured public LLM endpoint. Network administrators must block this traffic at the corporate firewall or network policy layer.
+
+[id="firewall-egress-block_{context}"]
+=== Blocking egress traffic at the corporate firewall
+
+.Prerequisites
+* You have administrative access to the corporate firewall or network policy system.
+* You have identified which LLM provider endpoints are in use. The following table lists common endpoints.
+
+.Common LLM provider egress endpoints
+[cols="1,2,1", options="header"]
+|===
+|Provider |Hostname or hostnames |Port
+
+|OpenAI
+|`api.openai.com`
+|443
+
+zure OpenAI Service
+|`*.openai.azure.com`
+|443
+
+|IBM watsonx.ai
+|`*.watson.cloud.ibm.com` +
+`*.watsonx.ai`
+|443
+
+|Custom or self-hosted
+|As configured in your `lightspeed.yaml` file
+|As configured
+|===
+
+.Procedure
+
+. Log in to your firewall management console.
+. Create an **outbound deny** rule for each LLM provider hostname that is listed in the previous table. If the complete list is unknown, create a rule for all hostnames:
++
+** **Direction**: Egress (outbound)
+** **Source**: IP address range or ranges of developer workstations
+** **Destination**: LLM provider hostnames or IP ranges
+** **Port**: 443 (HTTPS)
+** **Action**: Deny or Drop
+. Apply the rules immediately. A scheduled maintenance window is not required for deny rules.
+. Verify the rule activation in your firewall audit log.
+
+[NOTE]
+====
+If your organization uses a DNS-based filtering solution, such as Pi-hole, Cisco Umbrella, or Zscaler, you can alternatively add the LLM provider hostnames to the blocklist. DNS blocking takes effect as soon as the TTL on cached entries expires. This expiration typically occurs within 60 seconds for most resolvers.
+====
+
+[id="validate-proxy-shutdown_{context}"]
+== Validating that the LLM connection is terminated
+
+After you apply either the centralized or standalone isolation procedure, perform the following validation to confirm that outbound LLM traffic is fully blocked.
+
+.Procedure
+
+. Open the Developer Lightspeed panel in your IDE, such as VS Code or an IntelliJ-based IDE with the MTA plugin installed.
+. In the AI assistant chat input, type any plain-language query. For example, type the following query:
++
+----
+Summarize this project for me.
+----
+. Submit the query.
+
+.Expected results
+[cols="1,2", options="header"]
+|===
+|Deployment mode |Expected response
+
+|Centralized on Red{nbsp}Hat OpenShift Container Platform
+|A `503 Service Unavailable` or `connection refused` error is displayed in the IDE panel. No LLM response is returned.
+
+|Standalone with a firewall block
+|A `Network timeout`, `connection refused`, or `Name or service not known` error is displayed in the IDE panel. No LLM response is returned.
+|===
+
+[IMPORTANT]
+====
+If a valid LLM response is returned, the isolation has not taken effect. Verify your Tackle CR configuration or firewall rules, and repeat the validation.
+====
+
+.Optional: Network-level validation by using the `curl` command
+
+* On the affected workstation, run the following command to confirm that the endpoint is unreachable:
++
+[source,terminal,subs="+quotes"]
+----
+$ *curl -v --max-time 10 https://api.openai.com/v1/models*
+----
++
+.Expected output when isolation is active
+[source,terminal,subs="+quotes"]
+----
+* connect to __<ip>__ port 443 failed: Connection refused
+curl: (7) Failed to connect to api.openai.com port 443 after __<X>__ ms: Connection refused
+----
++
+Alternatively, in centralized mode where DNS resolves but the service is down, the expected output is as follows:
++
+[source,terminal]
+----
+curl: (22) The requested URL returned error: 503
+----