ACME does not receive certificates

[arut-plus]

I'm trying to set up a simple proxy (to get familiar with the Membrane), but I can't set up HTTPS.
Of course, I did everything according to the instructions from the official guide to setting up SSL using ACME.

What I've tried (to no avail):

• Wait until the certificates are requested (if they were requested at all) - waited more than 25 minutes
• Both directory URL's: https://acme-staging-v02.api.letsencrypt.org/directory and https://acme-v02.api.letsencrypt.org/directory

All the necessary information is provided below (* all IP addresses and domain addresses have been replaced with fake ones):

docker-compose.yml

name: membrane
services:
  membrane:
    image: predic8/membrane:6.3.10
    container_name: membrane
    volumes:
      - ./proxies.xml:/opt/membrane/conf/proxies.xml
      - ./log4j2.xml:/opt/membrane/conf/log4j2.xml
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "0.0.0.0:2000:2000"

proxies.yml

<spring:beans xmlns="http://membrane-soa.org/proxies/1/"
			  xmlns:spring="http://www.springframework.org/schema/beans"
			  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
			  xsi:schemaLocation="http://www.springframework.org/schema/beans
	                    http://www.springframework.org/schema/beans/spring-beans-4.2.xsd
	                    http://membrane-soa.org/proxies/1/ http://membrane-soa.org/schemas/proxies-1.xsd">

	<ssl id="demoSSL">
	  <acme
	    experimental="true" 
	    directoryUrl="https://acme-v02.api.letsencrypt.org/directory"
	    contacts="mailto:my@mail.com"
	    termsOfServiceAgreed="true"
	  >
	  <fileStorage dir="/tmp/acme-certs" />
	  </acme>
	</ssl>

	<router>

		<api port="80">
		  <acmeHttpChallenge />
		  <javascript>
		    exc.setResponse(Response.ok().build());
		    RETURN
		  </javascript>
		</api>

		<api host="mydomain.com" port="443">
			<log/>
		  <spring:ref bean="demoSSL" />
          <!--
            At this address, I launched nginx in docker to test redirection.
            I also tried to set the container name as the host.
          -->
		  <target host="mydomain.com" port="8888" />
		</api>

	</router>

</spring:beans>

log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{ABSOLUTE} %5p %tid %tn %c{1}:%L %X - %m%n" />
        </Console>
    </Appenders>
    <Loggers>
       <Logger name="com.predic8.membrane.core.transport" level="debug">
           <AppenderRef ref="STDOUT" />
       </Logger>
        <Root level="info">
            <AppenderRef ref="STDOUT" />
        </Root>
    </Loggers>
</Configuration>

docker compose logs --follow membrane

10:24:10,880  INFO 3 main LanguageAdapter:67 {} - Found Rhino Javascript engine.
10:24:12,319 DEBUG 3 main AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com
10:24:12,319 DEBUG 3 main AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com
10:24:12,415  INFO 3 main HttpEndpointListener:95 {} - listening at '*:80'
10:24:12,415  INFO 3 main HttpEndpointListener:95 {} - listening at '*:80'
10:24:12,431  INFO 3 main HttpEndpointListener:95 {} - listening at '*:443'
10:24:12,431  INFO 3 main HttpEndpointListener:95 {} - listening at '*:443'
10:24:12,440  INFO 3 main ApiInfo:35 {} - Started 2 APIs:
10:24:12,445  INFO 3 main ApiInfo:37 {} -   API 0.0.0.0:80
10:24:12,446  INFO 3 main ApiInfo:37 {} -   API mydomain.com:443
10:24:12,448  INFO 3 main Router:343 {} - Membrane API Gateway 6.3.10 up and running!
10:24:19,362 DEBUG 20 Connection Acceptor '*:443' HttpEndpointListener:120 {} - Accepted connection from /88.88.88.88:2042
10:24:19,362 DEBUG 20 Connection Acceptor '*:443' HttpEndpointListener:120 {} - Accepted connection from /88.88.88.88:2042
10:24:19,378 ERROR 22 router /88.88.88.88:2042 HttpServerHandler:140 {} -
java.lang.RuntimeException: ACME has not yet acquired a certificate.
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.getSocketFactory(AcmeSSLContext.java:112)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.check(AcmeSSLContext.java:164)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrap(AcmeSSLContext.java:159)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrapAcceptedSocket(AcmeSSLContext.java:154)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.setup(HttpServerHandler.java:77)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.run(HttpServerHandler.java:100)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
    at java.base/java.lang.Thread.run(Thread.java:1474)
10:24:19,378 ERROR 22 router /88.88.88.88:2042 HttpServerHandler:140 {} -
java.lang.RuntimeException: ACME has not yet acquired a certificate.
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.getSocketFactory(AcmeSSLContext.java:112)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.check(AcmeSSLContext.java:164)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrap(AcmeSSLContext.java:159)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrapAcceptedSocket(AcmeSSLContext.java:154)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.setup(HttpServerHandler.java:77)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.run(HttpServerHandler.java:100)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
    at java.base/java.lang.Thread.run(Thread.java:1474)
10:24:19,395 DEBUG 20 Connection Acceptor '*:443' HttpEndpointListener:120 {} - Accepted connection from /88.88.88.88:2046
10:24:19,395 DEBUG 20 Connection Acceptor '*:443' HttpEndpointListener:120 {} - Accepted connection from /88.88.88.88:2046
10:24:19,398 ERROR 23 router /88.88.88.88:2046 HttpServerHandler:140 {} -
java.lang.RuntimeException: ACME has not yet acquired a certificate.
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.getSocketFactory(AcmeSSLContext.java:112)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.check(AcmeSSLContext.java:164)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrap(AcmeSSLContext.java:159)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrapAcceptedSocket(AcmeSSLContext.java:154)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.setup(HttpServerHandler.java:77)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.run(HttpServerHandler.java:100)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
    at java.base/java.lang.Thread.run(Thread.java:1474)
10:24:19,398 ERROR 23 router /88.88.88.88:2046 HttpServerHandler:140 {} -
java.lang.RuntimeException: ACME has not yet acquired a certificate.
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.getSocketFactory(AcmeSSLContext.java:112)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.check(AcmeSSLContext.java:164)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrap(AcmeSSLContext.java:159)
    at com.predic8.membrane.core.transport.ssl.AcmeSSLContext.wrapAcceptedSocket(AcmeSSLContext.java:154)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.setup(HttpServerHandler.java:77)
    at com.predic8.membrane.core.transport.http.HttpServerHandler.run(HttpServerHandler.java:100)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
    at java.base/java.lang.Thread.run(Thread.java:1474)
10:24:22,335 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com
10:24:22,335 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com
10:24:32,341 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com
10:24:32,341 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for mydomain.com

I get this error in the browser:

Secure Connection Failed
An error occurred during a connection to mydomain.com. PR_END_OF_FILE_ERROR
Error code: PR_END_OF_FILE_ERROR
    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

In addition to accessing it from the browser, I also tried the following commands from the console:

$ curl https://mydomain.com
curl: (35) Send failure: Broken pipe

$ curl --insecure https://mydomain.com
curl: (35) TLS connect error: error:0A000126:SSL routines::unexpected eof while reading

$ curl -v http://mydomain.com/.well-known/acme-challenge
* Host mydomain.com:80 was resolved.
* IPv6: (none)
* IPv4: 12.34.56.78
*   Trying 12.34.56.78:80...
* Established connection to mydomain.com (12.34.56.78 port 80) from 192.168.1.12 port 33494
* using HTTP/1.x
> GET /.well-known/acme-challenge HTTP/1.1
> Host: mydomain.com
> User-Agent: curl/8.16.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 Ok
< Server: Membrane API Gateway
< Content-Length: 0
<
* Connection #0 to host mydomain.com:80 left intact

$ openssl s_client -connect mydomain.com:443
Connecting to 12.34.56.78
CONNECTED(00000003)
40C715D7AA7F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_lay
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: <NULL>
---
SSL handshake has read 0 bytes and written 1558 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
40C715D7AA7F0000:error:0A000197:SSL routines:SSL_shutdown:shutdown while in init:ssl/ssl_lib.c:2834:

$ openssl s_client -ignore_unexpected_eof -connect mydomain.com:443
Connecting to 12.34.56.78
CONNECTED(00000003)
closed
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: <NULL>
---
SSL handshake has read 0 bytes and written 1551 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Thank you in advance 🙏

[rrayst]

Please check your storage (=/tmp/acme-certs) for any errors. Check the newest files first, if you already have too many of them.

[arut-plus]

Please check your storage (=/tmp/acme-certs) for any errors. Check the newest files first, if you already have too many of them.

Yes, I'm sorry, I forgot to tell you about it, the directory was never created while I was dealing with this error.

There is no such directory:

$ docker compose exec membrane ls /tmp/acme-certs
ls: cannot access '/tmp/acme-certs': No such file or directory

I also tried to set it in proxies.xml just acme-certs instead of /tmp/acme-certs and try to find for this directory in the container:

$ docker compose exec membrane sh -c "find / -type d -name 'acme-certs' 2>/dev/null"
# nothing...

[arut-plus]

There is also no problem accessing both directory URLs from the container:

$ docker compose exec membrane curl -o /dev/null -sw "%{http_code}\n" https://acme-staging-v02.api.letsencrypt.org/directory
200
$ docker compose exec membrane curl -o /dev/null -sw "%{http_code}\n" https://acme-v02.api.letsencrypt.org/directory
200

[rrayst]

Hi @arut-plus , you should persist the directory. It will contain your account key, among other things.

[rrayst]

Hi @arut-plus , you should persist the directory. It will contain your account key, among other things.

[arut-plus]

Hi @arut-plus , you should persist the directory. It will contain your account key, among other things.

Do you mean manually initializing the certificates (for example, obtaining them using certbot)? Do I have to do it myself, and then the membrane will only handle the updates? Or what do you mean by "persist" — creating a Docker volume so that the membrane has a place to store the keys?

I created a Docker volume, so the directory already exists. I waited 15 minutes, but the keys still haven't appeared — the directory is empty.

[rrayst]

A Docker volume will work. Membrane will need write-access to the directory. (I'd guess this gets handled by the Docker Engine.)

Membrane will handle the whole process.

• The first step (long before actually requesting certificates) is to persist the contact addresses into a file called "account-contacts.txt" within the directory. If you don't see that, Membrane hasn't even started (or at least failed creating the file, but that would be logged).

Usually, there are some errors indicating a problem. Either shown in the log (=console) [usually right after startup or at the end] or as an "error" file within the directory.

Which Membrane version are you using?

[rrayst]

I created a Docker volume, so the directory already exists. I waited 15 minutes, but the keys still haven't appeared — the directory is empty.

@arut-plus the docker-compose.yml shown in the original issue report doesn't have a volume for the ACME state.

You would need something like

name: membrane
services:
  membrane:
    image: predic8/membrane:6.3.10
    container_name: membrane
    volumes:
      - ./proxies.xml:/opt/membrane/conf/proxies.xml
      - ./log4j2.xml:/opt/membrane/conf/log4j2.xml
      # Mount the named volume to the target path
      - acme_certs:/tmp/acme-certs
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "0.0.0.0:2000:2000"

# Declare the volume for the ACME state
volumes:
  acme_certs:

[arut-plus]

GI use latest predic8/membrane:6.3.10.

Of course, I wrote the same docker-compose.yml with named volume, but the result is the same.

The directory is still empty

docker compose exec membrane ls -la /tmp/certs
total 8
drwxr-xr-x 2 root root 4096 Dec  9 08:43 .
drwxrwxrwt 1 root root 4096 Dec  9 08:43 ..

But I turned on debug mode for the root and saw errors related to headers

08:58:12,027 DEBUG 3 main FileSchemaResolver:81 {} - Resolving /opt/membrane/conf/proxies.xml
08:58:12,031 DEBUG 3 main FileSchemaResolver:81 {} - Resolving /opt/membrane/conf/proxies.xml
08:58:12,035 DEBUG 3 main Router:142 {} - loading spring config: /opt/membrane/conf/proxies.xml
08:58:12,097 DEBUG 3 main TrackingFileSystemXmlApplicationContext:674 {} - Refreshing com.predic8.membrane.core.config.spring.TrackingFileSystemXmlApplicationContext@125290e5
08:58:13,003 DEBUG 3 main XmlBeanDefinitionReader:402 {} - Loaded 2 bean definitions from file [/opt/membrane/conf/proxies.xml]
08:58:13,052 DEBUG 3 main DefaultListableBeanFactory:308 {} - Creating shared instance of singleton bean 'demoSSL'
08:58:13,212 DEBUG 3 main DefaultListableBeanFactory:308 {} - Creating shared instance of singleton bean 'router'
08:58:13,355 DEBUG 3 main ServiceProxyKey:97 {} - Created host pattern match: (\Qxmpp-dev.simplex2.ru\E)
08:58:13,420 DEBUG 3 main DefaultLifecycleProcessor:581 {} - Starting beans in phase 0
08:58:13,466  INFO 3 main LanguageAdapter:67 {} - Found Rhino Javascript engine.
08:58:14,820 DEBUG 3 main AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:58:14,899  INFO 3 main HttpEndpointListener:95 {} - listening at '*:80'
08:58:14,914  INFO 3 main HttpEndpointListener:95 {} - listening at '*:443'
08:58:14,919  INFO 3 main ApiInfo:35 {} - Started 2 APIs:
08:58:14,923 DEBUG 21 hotdeploy HotDeploymentThread:69 {} - Spring Hot Deployment Thread started.
08:58:14,935  INFO 3 main ApiInfo:37 {} -   API 0.0.0.0:80
08:58:14,937  INFO 3 main ApiInfo:37 {} -   API xmpp-dev.simplex2.ru:443
08:58:14,939  INFO 3 main Router:343 {} - Membrane API Gateway 6.3.10 up and running!
08:58:14,940 DEBUG 3 main DefaultLifecycleProcessor:411 {} - Successfully started bean 'router'
08:58:24,833 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:58:34,835 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:58:41,681 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:42487
08:58:41,702 DEBUG 22 router /164.52.0.90:42487 HttpServerHandler:79 {} - New ServerThread created. 1
08:58:41,760 ERROR 22 router /164.52.0.90:42487 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu*(%79-174-83-47.cloudvps.regruhosting.ru
08:58:41,761 ERROR 22 router /164.52.0.90:42487 Header:138 {} - Header read line that caused problems:
08:58:41,761 ERROR 22 router /164.52.0.90:42487 Header:138 {} - Header read line that caused problems: #
08:58:41,762 ERROR 22 router /164.52.0.90:42487 Header:138 {} - Header read line that caused problems: 0.
08:58:44,837 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:58:45,138 DEBUG 22 router /164.52.0.90:42487 HttpServerHandler:113 {} - client socket closed
08:58:45,587 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:59841
08:58:45,589 DEBUG 23 router /164.52.0.90:59841 HttpServerHandler:79 {} - New ServerThread created. 2
08:58:45,591 DEBUG 23 router /164.52.0.90:59841 Message:168 {} - createBody
08:58:45,592 DEBUG 23 router /164.52.0.90:59841 Message:171 {} - empty body created
08:58:45,593 DEBUG 23 router /164.52.0.90:59841 HttpServerHandler:226 {} - Requested URI: /
08:58:48,861 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:56043
08:58:48,869 DEBUG 24 router /164.52.0.90:56043 HttpServerHandler:79 {} - New ServerThread created. 3
08:58:48,873 ERROR 24 router /164.52.0.90:56043 Header:138 {} - Header read line that caused problems: àù[¹;¾QïÆ>À,À0̨̩̪À+À/À$À(kÀ#À'gÀ
08:58:48,873 ERROR 24 router /164.52.0.90:56043 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu*(%79-174-83-47.cloudvps.regruhosting.ru
08:58:48,878 ERROR 24 router /164.52.0.90:56043 Header:138 {} - Header read line that caused problems:
08:58:48,879 ERROR 24 router /164.52.0.90:56043 Header:138 {} - Header read line that caused problems: #
08:58:48,881 ERROR 24 router /164.52.0.90:56043 Header:138 {} - Header read line that caused problems: 0.
08:58:51,864 DEBUG 24 router /164.52.0.90:56043 HttpServerHandler:113 {} - client socket closed
08:58:52,157 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:41259
08:58:52,158 DEBUG 25 router /164.52.0.90:41259 HttpServerHandler:79 {} - New ServerThread created. 4
08:58:52,393 DEBUG 23 router /164.52.0.90:59841 RuleManager:180 {} - Host from rule: * Host from parameter rule key: 79.174.83.47
08:58:52,396 DEBUG 23 router /164.52.0.90:59841 RuleManager:209 {} - Matching Rule 0.0.0.0:80 found for RuleKey 79.174.83.47 GET / 80 172.18.0.2
08:58:52,456 DEBUG 23 router /164.52.0.90:59841 LimitedMemoryExchangeStore:119 {api=0.0.0.0:80} - addRequest currentSize: 3749
08:58:52,479 DEBUG 23 router /164.52.0.90:59841 DispatchingInterceptor:103 {api=0.0.0.0:80} - destination: null
08:58:53,031 DEBUG 23 router /164.52.0.90:59841 FlowController:83 {api=0.0.0.0:80} - Interceptor returned RETURN. Returning!
08:58:53,032 DEBUG 23 router /164.52.0.90:59841 FlowController:83 {api=0.0.0.0:80} - Interceptor returned RETURN. Returning!
08:58:53,034 DEBUG 23 router /164.52.0.90:59841 LimitedMemoryExchangeStore:125 {api=0.0.0.0:80} - addResponse currentSize: 3981
08:58:53,034 DEBUG 23 router /164.52.0.90:59841 LimitedMemoryExchangeStore:77 {} - newSnap currentSize: 3981
08:58:53,040 DEBUG 23 router /164.52.0.90:59841 LimitedMemoryExchangeStore:98 {} - newSnap currentSize: 3981
08:58:53,041 DEBUG 23 router /164.52.0.90:59841 HttpServerHandler:113 {} - client socket closed
08:58:54,839 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:58:56,148 DEBUG 25 router /164.52.0.90:41259 HttpServerHandler:113 {} - client socket closed
08:58:56,430 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:33839
08:58:56,434 DEBUG 26 router /164.52.0.90:33839 HttpServerHandler:79 {} - New ServerThread created. 5
08:58:56,442 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems: BD7AF¼ù²2AE¡pó
08:58:56,443 DEBUG 26 router /164.52.0.90:33839 Message:168 {} - createBody
08:58:56,449 ERROR 26 router /164.52.0.90:33839 Message:192 {} - Message has no content length: ü­l?  HTTP/STOMP
08:58:56,456 DEBUG 26 router /164.52.0.90:33839 HttpServerHandler:226 {} - Requested URI:
08:58:56,460 DEBUG 26 router /164.52.0.90:33839 RuleManager:180 {} - Host from rule: * Host from parameter rule key: null
08:58:56,462 DEBUG 26 router /164.52.0.90:33839 RuleManager:180 {} - Host from rule: xmpp-dev.simplex2.ru Host from parameter rule key: null
08:58:56,464 DEBUG 26 router /164.52.0.90:33839 RuleManager:244 {} - No rule found for incoming request
08:58:56,546 DEBUG 26 router /164.52.0.90:33839 FlowController:89 {} - Interceptor returned ABORT. Aborting!
08:58:56,552 DEBUG 26 router /164.52.0.90:33839 HttpServerHandler:299 {} - exchange set completed
08:58:56,563 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems: ì¯Z»ø©Ë>À,À0̨̩̪À+À/À$À(kÀ#À'gÀ
08:58:56,565 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu
08:58:56,572 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems:
08:58:56,573 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems: #
08:58:56,574 ERROR 26 router /164.52.0.90:33839 Header:138 {} - Header read line that caused problems: 0.
08:58:56,936 DEBUG 26 router /164.52.0.90:33839 HttpServerHandler:113 {} - client socket closed
08:58:57,229 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:42349
08:58:57,231 DEBUG 27 router /164.52.0.90:42349 HttpServerHandler:79 {} - New ServerThread created. 6
08:59:00,398 DEBUG 27 router /164.52.0.90:42349 HttpServerHandler:113 {} - client socket closed
08:59:00,680 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:48343
08:59:00,682 DEBUG 28 router /164.52.0.90:48343 HttpServerHandler:79 {} - New ServerThread created. 7
08:59:00,687 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems: AÏmwæ fxcM
08:59:00,693 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems: XOV'áxÎåqJåÛ(Ø       Û©»ýe>À,À0̨̩̪À+À/À$À(kÀ#À'gÀ
08:59:00,695 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu
08:59:00,696 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems:
08:59:00,700 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems: #
08:59:00,702 ERROR 28 router /164.52.0.90:48343 Header:138 {} - Header read line that caused problems: 0.
08:59:03,715 DEBUG 28 router /164.52.0.90:48343 HttpServerHandler:113 {} - client socket closed
08:59:03,989 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:56013
08:59:03,991 DEBUG 29 router /164.52.0.90:56013 HttpServerHandler:79 {} - New ServerThread created. 8
08:59:04,841 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru
08:59:07,217 DEBUG 29 router /164.52.0.90:56013 HttpServerHandler:113 {} - client socket closed
08:59:07,510 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:50299
08:59:07,512 DEBUG 30 router /164.52.0.90:50299 HttpServerHandler:79 {} - New ServerThread created. 9
08:59:07,538 ERROR 30 router /164.52.0.90:50299 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu
08:59:07,541 ERROR 30 router /164.52.0.90:50299 Header:138 {} - Header read line that caused problems:
08:59:07,543 ERROR 30 router /164.52.0.90:50299 Header:138 {} - Header read line that caused problems: #
08:59:07,546 ERROR 30 router /164.52.0.90:50299 Header:138 {} - Header read line that caused problems: 0.
08:59:10,533 DEBUG 30 router /164.52.0.90:50299 HttpServerHandler:113 {} - client socket closed
08:59:10,807 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:58037
08:59:10,809 DEBUG 31 router /164.52.0.90:58037 HttpServerHandler:79 {} - New ServerThread created. 10
08:59:13,953 DEBUG 31 router /164.52.0.90:58037 HttpServerHandler:113 {} - client socket closed
08:59:14,219 DEBUG 19 Connection Acceptor '*:80' HttpEndpointListener:120 {} - Accepted connection from /164.52.0.90:42135
08:59:14,220 DEBUG 32 router /164.52.0.90:42135 HttpServerHandler:79 {} - New ServerThread created. 11
08:59:14,340 ERROR 32 router /164.52.0.90:42135 Header:138 {} - Header read line that caused problems: éÁÃÂËýÉ(v>À,À0̨̩̪À+À/À$À(kÀ#À'gÀ
08:59:14,342 ERROR 32 router /164.52.0.90:42135 Header:138 {} - Header read line that caused problems: À9À  À3=<5/ÿu
08:59:14,343 ERROR 32 router /164.52.0.90:42135 Header:138 {} - Header read line that caused problems:
08:59:14,344 ERROR 32 router /164.52.0.90:42135 Header:138 {} - Header read line that caused problems: #
08:59:14,345 ERROR 32 router /164.52.0.90:42135 Header:138 {} - Header read line that caused problems: 0.
08:59:14,843 DEBUG 16 Timer-0 AcmeSSLContext:203 {} - ACME: do not yet have a key for xmpp-dev.simplex2.ru

By the way, I just recently successfully used other tools (for example, docker-letsencrypt-nginx-proxy-companion, which uses the same acme with letsencrypt) to obtain certificates for this domain, but I have never had such errors and don't even understand their cause 🤷‍♂

UPD: Well, it's unlikely that this is specifically related to debug, but I caught such errors several more times when working with Membrane.

[t-burch]

@arut-plus One thing to verify is that the ACME challenge endpoint is truly reachable under port 80 and isn't blocked by a firewall or redirected from HTTP to HTTPS

[arut-plus]

@arut-plus One thing to verify is that the ACME challenge endpoint is truly reachable under port 80 and isn't blocked by a firewall or redirected from HTTP to HTTPS

The firewall does not block port 80, and HTTP is not redirected to HTTPS.

It's truly reachable, I'm checking from another machine

➜ curl -v http://xmpp-dev.simplex2.ru:80
* Host xmpp-dev.simplex2.ru:80 was resolved.
* IPv6: (none)
* IPv4: 79.174.83.47
*   Trying 79.174.83.47:80...
* Established connection to xmpp-dev.simplex2.ru (79.174.83.47 port 80) from 192.168.1.12 port 42280
* using HTTP/1.x
> GET / HTTP/1.1
> Host: xmpp-dev.simplex2.ru
> User-Agent: curl/8.16.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 Ok
< Server: Membrane API Gateway
< Content-Length: 0
<
* Connection #0 to host xmpp-dev.simplex2.ru:80 left intact

➜ curl -Lv http://xmpp-dev.simplex2.ru:80
* Host xmpp-dev.simplex2.ru:80 was resolved.
* IPv6: (none)
* IPv4: 79.174.83.47
*   Trying 79.174.83.47:80...
* Established connection to xmpp-dev.simplex2.ru (79.174.83.47 port 80) from 192.168.1.12 port 35652
* using HTTP/1.x
> GET / HTTP/1.1
> Host: xmpp-dev.simplex2.ru
> User-Agent: curl/8.16.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 Ok
< Server: Membrane API Gateway
< Content-Length: 0
<
* Connection #0 to host xmpp-dev.simplex2.ru:80 left intact

➜ curl -Lv http://xmpp-dev.simplex2.ru:80/.well-known/acme-certificate
* Host xmpp-dev.simplex2.ru:80 was resolved.
* IPv6: (none)
* IPv4: 79.174.83.47
*   Trying 79.174.83.47:80...
* Established connection to xmpp-dev.simplex2.ru (79.174.83.47 port 80) from 192.168.1.12 port 39178
* using HTTP/1.x
> GET /.well-known/acme-certificate HTTP/1.1
> Host: xmpp-dev.simplex2.ru
> User-Agent: curl/8.16.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 Ok
< Server: Membrane API Gateway
< Content-Length: 0
<
* Connection #0 to host xmpp-dev.simplex2.ru:80 left intact