SSL_CTX_compress_certs() failed, ignored with Certificate Compression in NGINX 1.29.1 n.wtf

[beatquantum]

NGINX Version: 2:1.29.1-2nwtf+352+13trixie1
OpenSSL Version: 3.5.2 (built with --with-openssl=/build/nginx/../openssl)
OS: Debian Trixie
Issue: When enabling ssl_certificate_compression on and ssl_certificate_cache (e.g., max=100 inactive=10s valid=30s) in server blocks, NGINX logs SSL_CTX_compress_certs() failed, ignored warnings. The compress_certificate TLS extension (type 27) is advertised in the handshake (confirmed via tshark), but openssl s_client -tls1_3 reports Compression: NONE. This occurs with both Let's Encrypt ECDSA and self-signed RSA/ECDSA certificates. Disabling compression eliminates warnings, but compression fails to work.
Steps to Reproduce:

1. Enable ssl_certificate_compression on in /etc/nginx/nginx.conf.
2. Add ssl_certificate_cache max=100 inactive=10s valid=30s to a server block.
3. Run nginx -t and check /var/log/nginx/error.log.
4. Test with openssl s_client -connect <domain>:443 -tls1_3.
   Logs:

2025/09/06 07:16:15 [warn] 12064#12064: SSL_CTX_compress_certs() failed, ignored

NGINX Build Info:
nginx -V 2>&1 | grep -i openssl
built with OpenSSL 3.5.2 5 Aug 2025
configure arguments: --with-cc-opt='-g -O2 -Werror=implicit-function-declaration ...' --with-openssl=/build/nginx/../openssl ...

Expected Behavior: Certificate compression should work, reducing handshake size.
Actual Behavior: Compression fails with warnings, and handshake shows Compression: NONE.

[msg7086]

Does this also impact official nginx builds?

Our 1.29.1-1 had an issue with OpenSSL 3.5.1+ so I reverted the API macros, that might have impacted some features.

[beatquantum]

I do not use official nginx mainline builds Because they do not have brotli and geoip modules. Sorry, cannot confirm if this is an upstream issue.

[Showfom]

We have upgrade to Nginx 1.29.3, please check if the issue still exists.