Credhub login failure related to CA certs

[sureshgoli25]

I am trying to connect to Credhub and seem to get an error around the CA cert being invalid.

credhub api
Error connecting to the targeted API: "Get https://<domain name>>:8844/info: x509: certificate is not valid for any names, but wanted to match <<domain name>>". Please validate your target and retry your request.

if use --skip-tls-validation, i am able to login

credhub api --skip-tls-validation
Setting the target url: https://ci.ap.innogy.com:8844/
Warning: The targeted TLS certificate has not been verified for this connection.
Warning: The --skip-tls-validation flag is deprecated. Please use --ca-cert instead.

This is after exporting all the env vars and they are there. Is there something else to do with this certificate?

While trying to setting-up the credstore values, i am getting below error.

credhub set -n /concourse/***/expo-username -v value -t ***
Get https://<<domain name>>:8844/info: x509: certificate is not valid for any names, but wanted to match <<domain name>>

Anyway to resolve it quickly.

[sureshgoli25]

Found similar issue is reported in https://github.com/EngineerBetter/concourse-up/issues/72#issue-372781951

[evadinckel]

Hello @sureshgoli25 ,
Thanks for reaching out to us.
Let us look into this matter and get back to you as soon as we can.
Best regards,
Eva

[evadinckel]

Hi @sureshgoli25 ,
May I ask which version of Concourse-Up you were using when encountering the issue?
Was your deployment against AWS or GCP?
Thanks,
Eva

[crsimmons]

@sureshgoli25 That error isn't the result of the CA being invalid. Instead it appears that the certificate on the credhub is missing your domain from its CN and/or SANs.

You can check what names are in your certificate with openssl s_client -connect ci.ap.innogy.com:8844. Could you post back the (redacted) output of that command?

We generate that certificate with BOSH here so provided you are passing the --domain flag when you deploy I would expect it to be generated correctly.

[sureshgoli25]

@crsimmons here is the output

curl https://ci.ap.innogy.com:8844
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
suresh $ curl https://ci.ap.innogy.com:8844 -k
{"error":"invalid_token","error_description":"Full authentication is required to access this resource"}suresh $
suresh $
suresh $ clear
suresh $ openssl s_client -connect ci.ap.innogy.com:8844
CONNECTED(00000003)
depth=0 C = USA, O = Cloud Foundry, CN = 18.184.118.127
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = USA, O = Cloud Foundry, CN = 18.184.118.127
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/C=USA/O=Cloud Foundry/CN=18.184.118.127
   i:/C=USA/O=Cloud Foundry/CN=ConcourseCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDUjCCAjqgAwIBAgIRAKCFmxINV2S6gMOfeKVtBBgwDQYJKoZIhvcNAQELBQAw
PDEMMAoGA1UEBhMDVVNBMRYwFAYDVQQKEw1DbG91ZCBGb3VuZHJ5MRQwEgYDVQQD
EwtDb25jb3Vyc2VDQTAeFw0xOTAyMjExMjU5NTBaFw0yMDAyMjExMjU5NTBaMD8x
DDAKBgNVBAYTA1VTQTEWMBQGA1UEChMNQ2xvdWQgRm91bmRyeTEXMBUGA1UEAxMO
MTguMTg0LjExOC4xMjcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
j/QQT95H7ZS5LvFMJpqqT9mubchLc15G05x8L9YaM3/SjCOeDIMusQKeHzWSAp/l
omCtcZ0LC34QMwv2Flgwhn2gq0rmg1vcycH+0xMNIMaTVfX8rUAcNzgImDir94lw
JUVSYWWGnG7KovIIxzPgIILRkAjRZmgd0SqrGTGLtp70IfGuSMyll6JVw2+x/t10
7AR8D6t4/aVIO7+q4bNL19EuK3hq+incAplRsUV4zCID9OBVAzOk2lnekNyuhmrQ
VTT14fLYMPN2A0hQQc5hD8riueb6sxxmKBvbDXb55smlaT34aa3XBp7qZZtAFWOV
MksnPAAydIg6slIw7tglAgMBAAGjTDBKMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE
DDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMBUGA1UdEQQOMAyHBBK4dn+HBH8A
AAEwDQYJKoZIhvcNAQELBQADggEBAC386Pt6yE9CAImD3CjJPXq3xhCA3g4YpdZ4
7GhehZL5B3KT5PI2gbuMdNjxboNzW8St7KKzjKKINsMt9lw48qh+0YYpfCrHwBcf
BnqgF6hiq3g2Ia1BSz1apepjCVYOEVWuq8aWMQv2QLzzuet9DZL3tj517IFNgc90
bMikoGFZkHOcxx6SYvffxJgnTXfgt5xY5WtyJJGtNwToqrpvyk+DjVpx36C9Tg/s
Ptu0fkytkUJRjNqAgQF5RRHzCIGJgSwn2vbE1QbZlWFGQHTvQ2w1ua7DwESxpwLl
3KvI4OdWGJXXozWCkjiA77WceFuVR2+d0hwbtWTpLU0zxuhtBe8=
-----END CERTIFICATE-----
subject=/C=USA/O=Cloud Foundry/CN=18.184.118.127
issuer=/C=USA/O=Cloud Foundry/CN=ConcourseCA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: DH, 4096 bits
---
SSL handshake has read 2300 bytes and written 879 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES128-GCM-SHA256
    Session-ID: 5C6FB68830033281139B441DC042487DA0E0F2B5A5332F0DAD0545339FE4E4F6
    Session-ID-ctx:
    Master-Key: C6D0894B2734AEA4FE9E0961D7E114353EC5169BCC3DD39C6D26F065F9F257BFA9D5DE2884B100FB2A81A46CC1350C21
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1550825096
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

[sureshgoli25]

@crsimmons @evadinckel
IAAS: AWS,
Concourse-up version: Concourse-Up version 0.20.0
I could see the SAN having only one name which ci.ap.innogy.com that could be the cause of the problem.

Also i would see that the certificate which is generated is valid for only 3 months. I would assume this certificate will be auto-renewed before expiry. Is my assumption is correct?

[crsimmons]

The certificate is using the EIP of the web vm as the common name. This suggests the --domain flag isn't being respected when you did the concourse-up deploy. What appears in your config.json in S3 for domain?

[sureshgoli25]

In config.json it showing "domain":"ci.ap.innogy.com"

[crsimmons]

Sorry for the delay. We had some unrelated issues on our end.

I'm not sure why the CN is showing as the EIP. We just tried curl https://ci.engineerbetter.com:8844 both with and without -k and got the same messages as you but after running the concourse-up info --env command to export env vars credhub api works as expected.

What command did you use to deploy originally? We will try to reproduce it on Monday.

[crsimmons]

After speaking with @sureshgoli25 elsewhere we have established that this problem resulted from:

1. Deploy without specifying a domain concourse-up deploy foo
2. Deploy again this time specifying a domain concourse-up deploy --domain foo.example.com foo
3. Observe that certificate on credhub has not been regenerated to have CN/SAN for foo.example.com so credhub login fails.

This happens because that certificate is generated by BOSH then is stored in the director-creds.yml vars store.

Using the ATC IP instead of the domain in the CREDHUB_SERVER env var may work as a temporary solution. We haven't tested this.

A manual solution is to download director-creds.yml from your config bucket, deleting internal_tls, uploading it to the bucket then running concourse-up deploy --domain foo.example.com foo again. We have tested this and it works.

[evadinckel]

Hi everyone, jumping back to this conversation:

Hello @sureshgoli25,
Thanks again for reaching out to us and for contributing to the growth of our product by sharing your input.
We have made the decision to depriotrise this stream of work of our current workflow and will not focus more engineering effort on this matter; our team will keep the question you have raised nonetheless if we happen to look further into this in the future.
Please see Colin’s response on how to approach the fix of this issue.
Best regards,
Eva