Prometheus unreachable from NL2Kubectl

[r-rishabh-j]

I have been trying to run the agent on several scenarios, but face the following issue each time.

I obtained the observability IP via sudo kubectl --kubeconfig /app/lumyn/config get ingress -n prometheus

sre-agent@hpml:/app/lumyn$ sudo crewai run
Running the Crew
/app/lumyn/.venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py:502: UserWarning: <built-in function callable> is not a Python type (it may be an instance of an object), Pydantic will allow any object with no validation since we cannot even enforce that the input is an instance of the given type. To get rid of this error wrap the type with `pydantic.SkipValidation`.
  warn(
An unexpected error occurred: No crew found.
NL2Kubectl Tool command execution: NAME                               READY   STATUS             RESTARTS   AGE
accounting-6f7598849c-d96wd        1/1     Running            0          26m
ad-58d567556b-thtkr                1/1     Running            0          26m
cart-6b747c9bb-dfvs6               1/1     Running            0          26m
checkout-578bdd8b44-9j4bq          0/1     InvalidImageName   0          10m
currency-6b8755c876-sdpwn          1/1     Running            0          26m
email-6c7ffd58ff-xwk7m             1/1     Running            0          26m
flagd-7945576cf5-mgn6n             2/2     Running            0          26m
fraud-detection-79794d4777-8xsbw   1/1     Running            0          26m
frontend-76d888895-z59b7           1/1     Running            0          26m
frontend-proxy-745965d9d5-gk27j    1/1     Running            0          26m
image-provider-5c886fc496-5v4jt    1/1     Running            0          26m
kafka-75d698b4d4-9h6lt             1/1     Running            0          26m
load-generator-756d87d8d8-8rssx    1/1     Running            0          26m
otel-collector-55968cb6df-kpdgk    1/1     Running            0          26m
payment-84fb966bb7-7dx5l           1/1     Running            0          26m
postgresql-6c5589f779-shbh6        1/1     Running            0          26m
product-catalog-754bbb99f7-5jp4k   1/1     Running            0          26m
quote-7bcc5bf6c8-p48v2             1/1     Running            0          26m
recommendation-7944876fdf-528vr    1/1     Running            0          26m
shipping-6bd45c67cb-zcnfm          1/1     Running            0          26m
valkey-cart-cdb6b4699-x44zj        1/1     Running            0          26m

NL2Kubectl Tool command execution: 
ERROR:lumyn.tools.observability_stack.observability_stack_base_client:Request timed out after 120 seconds
Error querying Prometheus Alerts API: HTTPConnectionPool(host='10.96.230.115', port=80): Max retries exceeded with url: /prometheus/api/v1/alerts (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f63f1384740>, 'Connection to 10.96.230.115 timed out. (connect timeout=120)'))
ERROR:lumyn.tools.observability_stack.get_alerts:Error querying Prometheus Alerts API: HTTPConnectionPool(host='10.96.230.115', port=80): Max retries exceeded with url: /prometheus/api/v1/alerts (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f63f1384740>, 'Connection to 10.96.230.115 timed out. (connect timeout=120)'))

[rohanarora]

Hi @r-rishabh-j, Could you please elaborate a little on your development environment/setup?