APISIX error behind proxy and #35
Description
I cound not complete to install this successfully due to some proxy or connectivity error on APISIX.
Could you check if do we need some configuration ?
$ pwd
/home/intel/Enterprise-Inference/core
$ cat inventory/inference-config.cfg
cluster_url=api.example.com
cert_file=/certs/cert.pem/certs/key.pem
key_file=
keycloak_client_id=my-client-id
keycloak_admin_user=admin
keycloak_admin_password=password
vault_pass_code=place-holder-123
model=
deploy_kubernetes_fresh=on
deploy_ingress_controller=on
deploy_keycloak_apisix=on
deploy_genai_gateway=off
deploy_observability=off
deploy_llm_models=on
deploy_ceph=off
deploy_istio=off
uninstall_ceph=off
http_proxy="http://XXX.intel.com:XXX" # replaced to XXX for security
https_proxy="http://XXX.intel.com:XXX"
no_proxy="localhost,127.0.0.1,.intel.com"
HTTP_PROXY="http://XXX.intel.com:XXX"
HTTPS_PROXY="http://XXX.intel.com:XXX"
NO_PROXY="localhost,127.0.0.1,.intel.com"
$ ./inference-stack-deploy.sh --models "21" --cpu-or-gpu "cpu" --hugging-face-token $HUGGINGFACE_TOKEN
| Intel AI for Enterprise Inference |
|---|
| 1) Provision Enterprise Inference Cluster |
| 2) Decommission Existing Cluster |
| 3) Update Deployed Inference Cluster |
| 4) Brownfield Deployment of Enterprise Inference |
| --------------------------------------------------------- |
| Please choose an option (1, 2, 3 or 4): |
1
Configuration file found, setting vars!
temp_env_vars: line 18: =: command not found
Metadata configuration file found, setting vars!
Deployment configuration: yes
Proceeding with the setup of Fresh Kubernetes cluster: yes
Proceeding with the setup of Habana AI Operator: no
Proceeding with the setup of Ingress Controller: yes
Proceeding with the setup of Keycloak : yes
Proceeding with the setup of Apisix: yes
Proceeding with the setup of GenAI Gateway: no
Proceeding with the setup of Observability: no
Proceeding with the setup of Ceph cluster: no
Proceeding with the setup of Istio: no
NRI CPU Balloon Policy automatically enabled for CPU deployment
Enter the token for Huggingface: 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
Proceeding with the setup of Large Language Model (LLM): yes
Using provided models: 21
Deploying/removing CPU models: cpu-llama-8b
----- Input -----
Using provided CLUSTER URL: api.example.com
Using provided certificate file: /home/intel/certs/cert.pem
Using provided key file: /home/intel/certs/key.pem
Using provided keycloak client id: my-client-id
Using provided Keycloak admin username: admin
Using provided Keycloak admin password
cpu_or_gpu is already set to c
.......................
TASK [inference-tools : Ensure jq is installed] ****************************************************************************************
ok: [master1]
Monday 24 November 2025 11:28:13 +0000 (0:00:01.243) 0:00:07.532 *******
TASK [Create Namespace for APISIX] *****************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: Value of unknown type: <class 'urllib3.exceptions.ProxyError'>, ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))
fatal: [master1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 697, in urlopen\n self._prepare_proxy(conn)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 971, in _prepare_proxy\n conn.connect()\n File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect\n self._tunnel()\n File "/usr/lib/python3.10/http/client.py", line 925, in _tunnel\n raise OSError(f"Tunnel connection failed: {code} {message.strip()}")\nOSError: Tunnel connection failed: 403 Forbidden\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 195, in retrieve\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 319, in get\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 112, in get\n return self.request('get', path, **kwargs)\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 277, in request\n api_response = self.client.call_api(\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/rest.py", line 217, in request\n r = self.pool_manager.request(method, url,\n File "/usr/lib/python3/dist-packages/urllib3/request.py", line 74, in request\n return self.request_encode_url(\n File "/usr/lib/python3/dist-packages/urllib3/request.py", line 96, in request_encode_url\n return self.urlopen(method, url, **extra_kw)\n File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 551, in urlopen\n return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)\n File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 391, in urlopen\n response = conn.urlopen(method, u.request_uri, **kw)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen\n retries = retries.increment(\n File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 576, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/auth-apisix (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 68, in run_module\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 112, in perform_action\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 210, in retrieve\nansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Failed to retrieve requested object: HTTPSConnectionPool(host='127.0.0.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/auth-apisix (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/home/intel/.ansible/tmp/ansible-tmp-1763983693.1181862-3095140-107964064342051/AnsiballZ_k8s.py", line 107, in \n _ansiballz_main()\n File "/home/intel/.ansible/tmp/ansible-tmp-1763983693.1181862-3095140-107964064342051/AnsiballZ_k8s.py", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File "/home/intel/.ansible/tmp/ansible-tmp-1763983693.1181862-3095140-107964064342051/AnsiballZ_k8s.py", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s', _modlib_path=modlib_path),\n File "/usr/lib/python3.10/runpy.py", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File "/usr/lib/python3.10/runpy.py", line 86, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s.py", line 479, in \n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s.py", line 473, in main\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 85, in run_module\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/core.py", line 79, in fail_json\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/basic.py", line 1531, in fail_json\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/basic.py", line 1500, in _return_formatted\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/common/parameters.py", line 927, in remove_values\n File "/tmp/ansible_kubernetes.core.k8s_payload_0z8v14yg/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/common/parameters.py", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'urllib3.exceptions.ProxyError'>, ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
NO MORE HOSTS LEFT *********************************************************************************************************************
PLAY RECAP *****************************************************************************************************************************
master1 : ok=6 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0
Monday 24 November 2025 11:28:14 +0000 (0:00:01.661) 0:00:09.193 *******
inference-tools : Install Deployment Client ------------------------------------------------------------------------------------- 2.03s
Create Namespace for APISIX ----------------------------------------------------------------------------------------------------- 1.66s
inference-tools : Install Kubernetes Python SDK --------------------------------------------------------------------------------- 1.43s
inference-tools : Ensure Python pip module is installed ------------------------------------------------------------------------- 1.36s
inference-tools : Ensure jq is installed ---------------------------------------------------------------------------------------- 1.24s
kubernetes-precheck : Check if kubectl is Available ----------------------------------------------------------------------------- 0.69s
kubernetes-precheck : Check Kubernetes API server connectivity ------------------------------------------------------------------ 0.57s
kubernetes-precheck : Fail if Kubernetes API is not reachable ------------------------------------------------------------------- 0.10s
kubernetes-precheck : Fail if kubectl is not Available -------------------------------------------------------------------------- 0.07s
21
I also got the similar issue when
deploy_keycloak_apisix=off
TASK [Set default model parallelism configuration for CPU deployments] *****************************************************************
ok: [master1]
Monday 24 November 2025 11:54:24 +0000 (0:00:00.105) 0:00:07.870 *******
TASK [Create/Update Kubernetes Secret for Hugging Face Token] **************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: Value of unknown type: <class 'urllib3.exceptions.ProxyError'>, ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))
fatal: [master1]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 697, in urlopen\n self._prepare_proxy(conn)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 971, in _prepare_proxy\n conn.connect()\n File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect\n self._tunnel()\n File "/usr/lib/python3.10/http/client.py", line 925, in _tunnel\n raise OSError(f"Tunnel connection failed: {code} {message.strip()}")\nOSError: Tunnel connection failed: 403 Forbidden\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 195, in retrieve\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py", line 319, in get\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 112, in get\n return self.request('get', path, **kwargs)\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/dynamic/client.py", line 277, in request\n api_response = self.client.call_api(\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/api_client.py", line 373, in request\n return self.rest_client.GET(url,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/rest.py", line 244, in GET\n return self.request("GET", url,\n File "/usr/local/lib/python3.10/dist-packages/kubernetes/client/rest.py", line 217, in request\n r = self.pool_manager.request(method, url,\n File "/usr/lib/python3/dist-packages/urllib3/request.py", line 74, in request\n return self.request_encode_url(\n File "/usr/lib/python3/dist-packages/urllib3/request.py", line 96, in request_encode_url\n return self.urlopen(method, url, **extra_kw)\n File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 551, in urlopen\n return super(ProxyManager, self).urlopen(method, url, redirect=redirect, **kw)\n File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 391, in urlopen\n response = conn.urlopen(method, u.request_uri, **kw)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 784, in urlopen\n return self.urlopen(\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen\n retries = retries.increment(\n File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 576, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/default/secrets/hugging-face-token (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 68, in run_module\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 112, in perform_action\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py", line 210, in retrieve\nansible_collections.kubernetes.core.plugins.module_utils.k8s.exceptions.CoreException: Failed to retrieve requested object: HTTPSConnectionPool(host='127.0.0.1', port=6443): Max retries exceeded with url: /api/v1/namespaces/default/secrets/hugging-face-token (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/home/intel/.ansible/tmp/ansible-tmp-1763985264.9258165-3137443-16668619360151/AnsiballZ_k8s.py", line 107, in \n _ansiballz_main()\n File "/home/intel/.ansible/tmp/ansible-tmp-1763985264.9258165-3137443-16668619360151/AnsiballZ_k8s.py", line 99, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File "/home/intel/.ansible/tmp/ansible-tmp-1763985264.9258165-3137443-16668619360151/AnsiballZ_k8s.py", line 47, in invoke_module\n runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s', _modlib_path=modlib_path),\n File "/usr/lib/python3.10/runpy.py", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File "/usr/lib/python3.10/runpy.py", line 86, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s.py", line 479, in \n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s.py", line 473, in main\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py", line 85, in run_module\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/core.py", line 79, in fail_json\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/basic.py", line 1531, in fail_json\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/basic.py", line 1500, in _return_formatted\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/common/parameters.py", line 927, in remove_values\n File "/tmp/ansible_kubernetes.core.k8s_payload_gydbm_qp/ansible_kubernetes.core.k8s_payload.zip/ansible/module_utils/common/parameters.py", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'urllib3.exceptions.ProxyError'>, ('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}