Downstream handshake error TLSHanshakeFailure context: TLS accept() failed: unexpected EOF

[daviddesmet]

I'm testing out Pingora with a basic setup of an LB with 2 upstream services.
When I enabled TLS, I started noticing a bunch of those logs, but only in the testing environment (a docker container):

ERROR pingora_core::services::listening Downstream handshake error TLSHanshakeFailure context: TLS accept() failed: unexpected EOF

When running locally, I don't get any of those messages.

The LB is working as expected, I can reach the upstream services using HTTPS so I wonder if I did somehow miss a configuration. The upstream services are only listening for HTTP connections (no HTTPS).

Those logs are printed at the console in random intervals, it can go from 1 second to 2-3 seconds, even when no service is calling the LB.

So any hint on what may be going on to track and fix the issue is very much appreciated.

fn main() {
    let mut server = Server::new(Some(Opt::default())).unwrap();
    server.bootstrap();

    let cert_path = format!("{}/tests/keys/server.crt", env!("CARGO_MANIFEST_DIR"));
    let key_path = format!("{}/tests/keys/key.pem", env!("CARGO_MANIFEST_DIR"));
    let dynamic_cert = boringssl_openssl::DynamicCert::new(&cert_path, &key_path);
    let tls_settings = TlsSettings::with_callbacks(dynamic_cert).unwrap();
    tls_settings.enable_h2();

    let mut upstreams = LoadBalancer::try_from_iter(["192.168.1.101:8080", "192.168.1.102:8080"]).unwrap();

    // Set health check
    let hc = TcpHealthCheck::new();
    upstreams.set_health_check(hc);
    upstreams.health_check_frequency = Some(Duration::from_secs(10));

    // Set background service
    let background = background_service("health check", upstreams);
    let upstreams = background.task();

    // Set load balancer
    let mut lb = http_proxy_service(&server.configuration, LB(upstreams));
    lb.add_tcp("0.0.0.0:6188");
    lb.add_tls_with_settings("0.0.0.0:6189", None, tls_settings);

    server.add_service(background);
    server.add_service(lb);
    server.run_forever();
}

[eaufavor]

The error log suggests something attempt to connect to the listening port and disconnected before trying to perform TLS handshake. This could be certain health-check probes in your testing environment.

[daviddesmet]

That's what I first thought but I haven't yet set any health check in that Kubernetes deployment.
I will implicitly define one and see how it goes. I will keep you posted!

[daviddesmet]

Just verified and Kubernetes doesn't add any health check probe by default. I defined one for the TCP echo service I've added based on the examples, the health check works but I'm still getting that error in the console. The TLS port 6189 has not yet been consumed by an outside service, it has been added for testing and as soon as the app starts it starts showing that log.

[daviddesmet]

I've tracked down the issue and it is the Azure Load Balancer health probe rules that were created automatically.