SSL Cert generation with certbot

[troublestarter]

Hi all

Can someone give me a little help with the configuration of lighttpd for make the cert generation working ?
I have made this setup with apache, haproxy and other stuff but don't succeed to do it with lighthttpd...
The documentation is not up2date on this actually. (add-apt-repository ppa:certbot/certbot do not work on my side.)

Here is the commands use in CLI in debug mode and dry-run mode

sudo apt update
sudop apt install software-properties-common
sudo apt install certbot
mkdir ~/allsky/config/ssl         
chmod 775 ~/allsky/config/ssl
mkdir ~/allsky/config/ssl/allsky02.my-cosi.info
chmod 775 ~/allsky/config/ssl/allsky02.my-cosi.info
sudo certbot certonly --webroot -w allsky/config/ssl/allsky02.my-cosi.info -d allsky02.my-cosi.info --dry-run --debug

Here is the result:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Simulating a certificate request for allsky02.my-cosi.info
Performing the following challenges:
http-01 challenge for allsky02.my-cosi.info
Using the webroot path /home/admin/allsky/config/ssl/allsky02.my-cosi.info for all unmatched domains.
Waiting for verification...
Challenge failed for domain allsky02.my-cosi.info
http-01 challenge for allsky02.my-cosi.info
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==1.12.0', 'console_scripts', 'certbot')())
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1413, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1293, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 134, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: allsky02.my-cosi.info
   Type:   unauthorized
   Detail: 82.65.190.214: Invalid response from
   http://allsky02.my-cosi.info/.well-known/acme-challenge/Srmu_L63Xo2waUegP3XoTbEXohoPQyjyae3JTTlj9sQ:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

[custard2022]

I had the some issues above getting my local SSL set up, gave up and did it through Cloudflare instead.
Following the instructions, I couldn't get it working, got the same error:

$ sudo certbot certonly --webroot -w ~/allsky/config/ssl/example.com -d example.com --dry-run --debug
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Account registered.
Simulating a certificate request for example.com
Performing the following challenges:
http-01 challenge for example.com
Using the webroot path /home/pi/allsky/config/ssl/example.com for all unmatched domains.
Waiting for verification...
Challenge failed for domain example.com
http-01 challenge for example.com
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==1.12.0', 'console_scripts', 'certbot')())
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1413, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1293, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 134, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: example.com
   Type:   unauthorized
   Detail: MASKED_IP_FOR_PRIVACY: Invalid response from
   http://example.com/.well-known/acme-challenge/lLEZKNcgYQBJOEFP5KKdLgZE8YFuzz_V6obYDtGAHpw:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.```

[troublestarter]

I know we can do it with other methods but the goal is to succeed it and update the documentation 😉
Thanks for your feedback @custard2022

For me, this is because of a bad configuration in lighttpd, just maybe because the 443 port is not opened by default or something else to verify with .well-known folder

[Alex-developer]

I have done this a few times, if you give me a few days I will get you some notes.

Certbot doesn't support lighttpd very well, at least it didn't last time I looked

Cheers

Alex

[troublestarter]

Thank you @Alex-developer your help will be of great use of course!

[morrowwm]

I agree with @Alex-developer , I actually quit using lighttpd on my web site a few years ago and switched to nginx for this very reason.

Not to hijack the conversation, but has there been any discussion on making allsky web server agnostic? I assume lighttpd is used because of its light footprint?

[troublestarter]

i think for the footprint...
Or maybe Eric know well the configuration...

[troublestarter]

ok i have find it.
I do a summarize in few min.

[troublestarter]

it's seems all good !

sudo apt update
sudo apt install software-properties-common certbot
mkdir -p ~/allsky/config/ssl/allsky02.my-cosi.info
chmod 775 ~/allsky/config/ssl/allsky02.my-cosi.info
sudo nano /etc/lighttpd/conf-enabled/97-certbot.conf

    $HTTP["url"] =~ "^/.well-known/" {
        alias.url += ( "/.well-known/" => "/home/admin/allsky/config/ssl/allsky02.my-cosi.info/.well-known/" )
    }

sudo systemctl restart lighttpd.service

sudo certbot certonly --webroot -w allsky/config/ssl/allsky02.my-cosi.info -d allsky02.my-cosi.info -m xxxxxx@xxx.fr --agree-tos

sudo cat /etc/letsencrypt/live/allsky02.my-cosi.info/cert.pem /etc/letsencrypt/live/allsky02.my-cosi.info/privkey.pem | sudo tee /etc/letsencrypt/live/allsky02.my-cosi.info/fullchain.pem

sudo nano /etc/lighttpd/conf-enabled/98-allsky02.my-cosi.info.conf

server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
        ssl.engine = "enable"
        ssl.pemfile = "/home/admin/fullchain.pem"
        ssl.ca-file = "/home/admin/chain.pem"
        ssl.privkey = "/home/admin/privkey.pem"
        server.name = "allsky02.my-cosi.info"

        # Uncomment the next line if you want the web server to log access requests.
        # accesslog.filename = "/var/log/lighttpd/myallsky.com_access.log"
}

$HTTP["scheme"] == "http" {
        $HTTP["host"] == "allsky02.my-cosi.info" {
                url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
                url.redirect-code = 308
        }
}

cd ~
sudo cp /etc/letsencrypt/live/allsky02.my-cosi.info/{privkey.pem,fullchain.pem} .
sudo chown admin:admin privkey.pem fullchain.pem
sudo chmod 644 privkey.pem fullchain.pem

sudo systemctl restart lighttpd.service

[troublestarter]

Can someone try on his setup ? @Alex-developer @custard2022 ?

Replace

/home/admin/

by your home directory

Replace

allsky02.my-cosi.info

by your domain name

Of course we will discuss to optimize and i will modify the documentation to submit to @EricClaeys

[Alex-developer]

Yes will do but I will help once I have finished making all of the changes for the revised overlay system, I would like to concentrate on getting that done before doing anything else as its pretty complex :-)

Cheers

Alex

[troublestarter]

sure :)

[EricClaeys]

@troublestarter, @custard2022, did you initially try the "SSL" documentation page? I created it based on someone else's notes. I have never tried it - I'm too scared to hose something up, and I have a remote Website so don't really need to make my Pi Internet accessible.

If you have better instructions, if you detail them, i.e., step-by-step, I can replace the current instructions.

[troublestarter]

@EricClaeys
yes, i initially tried "SSL" documentation page. it is not working.

i have explained details up in the discussion: #3489 (comment)
i also modified a simple documentation "waiting approval"

SSL.zip

[custard2022]

I have not tried it yet, as I just have SSL/TLS encryption though Cloudflare right now, but will try it when I return home.

[troublestarter]

Thanks, works great on my setup

[troublestarter]

one thing that will need to be fixed:

This morning i see this message:

So i tried to disable ssl and test then enable it again:

Then i tried to update my settings with https in URL:

So a bad test or a bad config in
admin@allsky:~ $ sudo cat /etc/lighttpd/conf-enabled/98-allsky02.my-cosi.info.conf

server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
        ssl.engine = "enable"
        ssl.pemfile = "/home/admin/fullchain.pem"
#        ssl.ca-file = "/home/admin/privkey.pem"
        ssl.privkey = "/home/admin/privkey.pem"
        server.name = "allsky02.my-cosi.info"

        # Uncomment the next line if you want the web server to log access requests.
        # accesslog.filename = "/var/log/lighttpd/myallsky.com_access.log"
}

$HTTP["scheme"] == "http" {
        $HTTP["host"] == "allsky02.my-cosi.info" {
                url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
                url.redirect-code = 308
        }
}

Also tried directly with curl in CLI:

The certificate is not trusted in all web browsers. You may need to install an Intermediate/chain certificate to link it to a trusted root certificate

So i will verify the cert config in lighthttpd, that must be the config error... Ever seen that

[troublestarter]

ok works better with that !

server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
        ssl.engine = "enable"
        ssl.pemfile = "/home/admin/fullchain.pem"
        ssl.ca-file = "/home/admin/chain.pem"
        ssl.privkey = "/home/admin/privkey.pem"
        server.name = "allsky02.my-cosi.info"

        # Uncomment the next line if you want the web server to log access requests.
        # accesslog.filename = "/var/log/lighttpd/myallsky.com_access.log"
}

$HTTP["scheme"] == "http" {
        $HTTP["host"] == "allsky02.my-cosi.info" {
                url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
                url.redirect-code = 308
        }
}

Then changed settings with HTTPS in allsky webui

Then

[troublestarter]

Updated documentation:

it's seems all good ! Works on allsky map too.

Replace

/home/admin/

by your home directory

Replace

allsky02.my-cosi.info

by your domain name

sudo apt update
sudo apt install software-properties-common certbot
mkdir -p ~/allsky/config/ssl/allsky02.my-cosi.info
chmod 775 ~/allsky/config/ssl/allsky02.my-cosi.info
sudo nano /etc/lighttpd/conf-enabled/97-certbot.conf

    $HTTP["url"] =~ "^/.well-known/" {
        alias.url += ( "/.well-known/" => "/home/admin/allsky/config/ssl/allsky02.my-cosi.info/.well-known/" )
    }

sudo systemctl restart lighttpd.service

sudo certbot certonly --webroot -w allsky/config/ssl/allsky02.my-cosi.info -d allsky02.my-cosi.info -m xxxxxx@xxx.fr --agree-tos

sudo cat /etc/letsencrypt/live/allsky02.my-cosi.info/cert.pem /etc/letsencrypt/live/allsky02.my-cosi.info/privkey.pem | sudo tee /etc/letsencrypt/live/allsky02.my-cosi.info/fullchain.pem

sudo nano /etc/lighttpd/conf-enabled/98-allsky02.my-cosi.info.conf

server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
        ssl.engine = "enable"
        ssl.pemfile = "/home/admin/fullchain.pem"
        ssl.ca-file = "/home/admin/chain.pem"
        ssl.privkey = "/home/admin/privkey.pem"
        server.name = "allsky02.my-cosi.info"

        # Uncomment the next line if you want the web server to log access requests.
        # accesslog.filename = "/var/log/lighttpd/myallsky.com_access.log"
}

$HTTP["scheme"] == "http" {
        $HTTP["host"] == "allsky02.my-cosi.info" {
                url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
                url.redirect-code = 308
        }
}

cd ~
sudo cp /etc/letsencrypt/live/allsky02.my-cosi.info/{privkey.pem,fullchain.pem,chain.pem} .
sudo chown admin:admin privkey.pem fullchain.pem chain.pem
sudo chmod 644 privkey.pem fullchain.pem chain.pem

sudo systemctl restart lighttpd.service

[troublestarter]

SSL.zip

[evripidis]

Hello guys!

Does anyone see anything wrong?

Step 1

$HTTP["url"] =~ "^/.well-known/" { alias.url += ( "/.well-known/" => "/home/lubuntu/allsky/config/ssl/www.lpobservatory.gr/.well-known/" ) }

Step 2

sudo certbot certonly --webroot -w /home/lubuntu/allsky/config/ssl/www.lpobservatory.gr \
    -d www.lpobservatory.gr -m macedonas@icloud.com --agree-tos

Step 3

lubuntu@lubuntu-ami:~$ sudo nano /etc/lighttpd/conf-enabled/97-certbot.conf
lubuntu@lubuntu-ami:~$ sudo systemctl restart lighttpd.service
lubuntu@lubuntu-ami:~$ sudo certbot certonly --webroot -w /home/lubuntu/allsky/config/ssl/www.lpobservatory.gr \
    -d www.lpobservatory.gr -m macedonas@icloud.com --agree-tos
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for www.lpobservatory.gr

Certbot failed to authenticate some domains (authenticator: webroot).
The Certificate Authority reported these problems:
  Domain: www.lpobservatory.gr
  Type:   unauthorized
  Detail: 162.55.161.171: Invalid response from http://www.lpobservatory.gr/.well-known/acme-challenge/AVy0xCw9-Z-UaYJyS6qr6COt6V_ai_oQ-BP26VF6GPc: "\n\n\t\n\t\n<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Frameset//EN\" \"http://www.w3.org/TR/html4/frameset.dtd\">\n<html>\n<head>\n<meta "

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot.
Ensure that the listed domains serve their content from the provided --webroot-path/-w and that
files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org.
See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

[troublestarter]

Hi

Please try everything from start with : https://htmlpreview.github.io/?https://raw.githubusercontent.com/AllskyTeam/allsky/master/html/documentation/explanations/SSL.html

Then verify that you have port 80 and 443 opened on your allsky lie asked in pre requisites.

Thanks you.

[evripidis]

That's exactly what I am already following. Right now i'm struggling to understand if my ports are actually forwarded, cause I am actually getting back different results from the web checkers