Calls from Salesforce AgentForce die at the gateway

[davem-git]

Trying to make simple api calls from sales force, AgentForce and they die at the gateway, We get a 400 error and the only logs i see is this

{
  "searchResults": "Error: 400 Bad Request"
}

{
    "bytes_received": 0,
    "bytes_sent": 11,
    "cluster_name": "example-cluster",
    "container_name": "envoy",
    "duration": 0,
    "entity.guid": "MjM5MTMxMnxFWFR8U0VSVklDRXwtNTQ3NTc2NzA3NDY0MjI5NTg0",
    "entity.guids": "MjM5MTMxMnxFWFR8U0VSVklDRXwtNTQ3NTc2NzA3NDY0MjI5NTg0",
    "entity.name": "envoy",
    "entity.type": "SERVICE",
    "method": "null",
    "newrelic.source": "api.logs",
    "pod_labels.app.kubernetes.io/component": "proxy",
    "pod_labels.app.kubernetes.io/managed-by": "envoy-gateway",
    "pod_labels.app.kubernetes.io/name": "envoy",
    "pod_labels.gateway.envoyproxy.io/owning-gateway-name": "acme-gateway",
    "pod_labels.gateway.envoyproxy.io/owning-gateway-namespace": "envoy-gateway-system",
    "pod_labels.kubernetes.acme.com/role": "server",
    "pod_labels.kubernetes.acme.com/service": "envoy-gateway",
    "pod_labels.kubernetes.acme.com/team": "acme",
    "pod_labels.pod-template-hash": "7fc447488d",
    "pod_name": "acme-gateway-7fc447488d-qtth2",
    "pod_namespace": "envoy-gateway-system",
    "pod_node_name": "nodename",
    "pod_owner": "ReplicaSet/acme-gateway-7fc447488d",
    "response_code": 400,
    "response_flags": "DownstreamProtocolError",
    "route_name": "null",
    "service_name": "envoy",
    "source_type": "kubernetes_logs",
    "start_time": "2025-05-20T15:28:07.326Z",
    "timestamp": 1747754888299,
    "upstream_host": "null",
    "upstream_ssl_cipher": "null",
    "user_agent": "null",
    "vhost": "null",
    "x-envoy-original-path": "null",
    "x-forwarded-for": "null",
    "x-request-id": "null"
  }

When making the same call to nginx, it works and i get the following

  {
    "bytes_sent": "458",
    "cluster_name": "example-cluster",
    "container_name": "nginx-ingress-controller",
    "duration": "0.511",
    "entity.guid": "MjM5MTMxMnxFWFR8U0VSVklDRXwtNzk2ODczMjAzNDcyODAzNDk1MQ",
    "entity.guids": "MjM5MTMxMnxFWFR8U0VSVklDRXwtNzk2ODczMjAzNDcyODAzNDk1MQ",
    "entity.name": "ingress-nginx",
    "entity.type": "SERVICE",
    "http_referrer": "",
    "http_user_agent": "SFDC-Callout/63.0",
    "ingress": "default/agent-ecosystem-gateway",
    "method": "POST",
    "newrelic.source": "api.logs",
    "path": "/aeg/api/v1/aura-search",
    "pod_name": "nginx-ingress-controller-74677fcd8-d64sl",
    "pod_node_name": "nodename",
    "remote_addr": "",
    "request_length": "2400",
    "request_proto": "HTTP/1.1",
    "request_query": "",
    "service_name": "ingress-nginx",
    "source_type": "kubernetes_logs",
    "ssl_cipher": "TLS_AES_256_GCM_SHA384",
    "status": "401",
    "time": "2025-05-20T14:57:53+00:00",
    "timestamp": 1747753081323,
    "traceparent": "",
    "upstream": "default-agent-ecosystem-gateway-5000",
    "upstream_connect_time": "0.001",
    "upstream_header_time": "0.511",
    "upstream_response_time": "0.512",
    "upstream_server": "10.101.71.167:5000",
    "vhost": "example.com",
    "x_auto_test": "",
    "x_forward_for": "IPADDDRESS"
    }

Is this a cipher mismatch? I've tried setting some client traffic policys to adjust it but with no luck

[arkodg]

any idea @rudrakhp ?

[rudrakhp]

@davem-git can you share your gateway configurations (both EG and NGINX)? Access log has no info, so Envoy wasn't able to parse anything from the request:

    "response_code": 400,
    "response_flags": "DownstreamProtocolError",
    "route_name": "null",
    "service_name": "envoy",
    "upstream_host": "null",
    "upstream_ssl_cipher": "null",
    "user_agent": "null",
    "vhost": "null",
    "x-envoy-original-path": "null",
    "x-forwarded-for": "null",
    "x-request-id": "null"

I think there is a mismatch between the protocol used by the caller and your gateway.

[davem-git]

Your config must be in accordance with HTTPS config for gateway: https://github.com/kubernetes-sigs/gateway-api/blob/main/examples/standard/simple-http-https/gateway.yaml

So from this example, I don't listen on http, which nginx doesn't either, and I don't specify a hostname, which hasn't been a problem so far. I can add a wildcard for testing, but it doesn't seem like thats the problem

[rudrakhp]

@davem-git can you confirm if calling your HTTPS service via EG directly using curl works? Want to rule out any problems with the caller here.

[davem-git]

yes it works from postman, it just doesn't work from salesforce's agentforce

[davem-git]

I think we figured it out. enableing HTTP debug was hard to read so i missed

2025-05-21 20:36:34.383][44][debug][http] [source/common/http/conn_manager_impl.cc:394] [Tags: "ConnectionId":"54"] new stream
[2025-05-21 20:36:34.383][44][debug][http] [source/common/http/http1/codec_impl.cc:1411] [Tags: "ConnectionId":"54"] Rejecting request due to header name with underscores: SFDC_STACK_DEPTH
[2025-05-21 20:36:34.383][44][debug][http] [source/common/http/filter_manager.cc:1094] [Tags: "ConnectionId":"54","StreamId":"9098150007507973928"] Sending local reply with details http1.unexpected_underscore
[2025-05-21 20:36:34.383][44][debug][http] [source/common/http/conn_manager_impl.cc:1830] [Tags: "ConnectionId":"54","StreamId":"9098150007507973928"] closing connection due to connection close header
[2025-05-21 20:36:34.383][44][debug][http] [source/common/http/conn_manager_impl.cc:1896] [Tags: "ConnectionId":"54","StreamId":"9098150007507973928"] encoding headers via codec (end_stream=false):

Once we found that, looking at client traffic policies i was able to add

  headers:
    withUnderscoresAction: Allow

and now it works

[arkodg]

this is a good candidate for a Gotcha/Common Problems section

[rudrakhp]

#6146