Certificates generated by ACMEBot are in a incorrect order

[FunLow]

Describe the bug
Certificates being fetched by the ACME Bot are provided to the keyvault in the Wrong order. Taking any certificate being generated using the ACME Bot and exporting the certificates shows that the certificates are ending up in the wrong order in the key vault.

While this is checked and fixed for different Implementations (e.g. Application Gateway, AKS Secrets Provider) this can lead to issues in a lot of places where this check is not done before injecting the certificate. Expecially if there there are various different kinds of processes utilizing the certificates from different origins. From my point of view this should be a validation of the acmebot and something the ACME Bot should check and fix due to suit the international defined standard. Thats also the reason im creating this as bug.
https://datatracker.ietf.org/doc/html/rfc4346#section-7.4.2

I noticed that there is the MitigateChainOrder Property in the configuration ( at least in the version we are using) but it seems to not handle / work properly or im missing something.
Appsetting used on the function app:
Acmebot:MitigateChainOrder=true

Actual Output

Bag Attributes: <Empty Attributes>
subject=C=US, O=Let's Encrypt, CN=R12
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X1
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Bag Attributes
    localKeyID: <redacted>
subject=CN=my-domain.de
issuer=C=US, O=Let's Encrypt, CN=R12
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

Expected Output

-----END CERTIFICATE-----
Bag Attributes
    localKeyID: <redacted>
subject=CN=my-domain.de
issuer=C=US, O=Let's Encrypt, CN=R12
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Bag Attributes: <Empty Attributes>
subject=C=US, O=Let's Encrypt, CN=R12
issuer=C=US, O=Internet Security Research Group, CN=ISRG Root X1
-----BEGIN CERTIFICATE-----
...

To Reproduce
Steps to reproduce the behavior:

1. Generate a certificate
2. Export the certificate from the Keyvault
3. Export Certificate: openssl pkcs12 -in '.\my-cert.pfx' -nokeys -out cert.pem

Environment (please complete the following information):

• Certificate Type: Zone Apex, Sub-domain
• Certificate Deploy Target: Azure Kubernetes Cluster with App Configuration Provider to fetch certificates

Additional context
ACME Bot Version: 4.2.6.0

Thanks for the great work anyway!

[JohnLBevan]

I think this may be an issue in the AcmeSharpCore library... The same bug existed in the powershell module a while back (PKISharp/ACME-PS#129; though that was fixed pre-Core version... but if the bug were inthe upstream library and that got ported to core, it's likely still there (I couldn't find any issues in their repo for this though)...

[shibayan]

This issue originates from Key Vault, and Acmebot currently finds it difficult to address. While we plan to replace the ACME implementation with our own in the future, which should resolve bugs dependent on the library, this particular issue may prove difficult to fix.

[FunLow]

Why does this issue originates from the keyvault ? when is this caused in the process ?
The certificate after it has been created doesn't gets changed on upload to the keyvault. I also checked the CSR and it looks fine to me in the first place.

Is it possible to offer some kind of workaround as feature flag otherwise to avoid issues and allow usage in legacy contexts ?

[shibayan]

The design philosophy relies heavily on Key Vault, as it handles everything from CSR generation to PFX/PEM creation. While the PEM issued by Let's Encrypt is in the correct order, it appears to become corrupted when downloaded from Key Vault.

That said, since this is an issue I want to resolve, I plan to review the latest Key Vault implementation.

[FunLow]

The design philosophy relies heavily on Key Vault, as it handles everything from CSR generation to PFX/PEM creation. While the PEM issued by Let's Encrypt is in the correct order, it appears to become corrupted when downloaded from Key Vault.

That said, since this is an issue I want to resolve, I plan to review the latest Key Vault implementation.

What does "downloaded from key vault" exactly mean ? Does the keyvault itself download the certificate from lets encrypt or are you referring to the download of the certificate of the clients using the certificates ?
I would like to understand the issue currently present.

[shibayan]

Key Vault is actually the one that combines the private key and certificate into a PFX or PEM file; Acmebot simply submits the certificate issued by Let's Encrypt directly to the merge request, so it has no control over the order of the certificates.

Unfortunately, even now, when I assign a key from Key Vault to Application Gateway, it seems to trigger a sequencing warning. I tried several different approaches, but couldn't resolve the issue.

Azure/azure-rest-api-specs#10637

[FunLow]

Ok Thanks for the Info. Wasn't aware Keyvautl itself is already capable of doing that just with the CSR and MSR.

Still as this is an issue in context of more legacy or generic implementations that can't handle the sequence of the certificates themselves would it be possible to offer some kind of workaround ? It would be nice if the acmebot itself could restructure the certificates either immediatly after they have been generated or generate an alternative version for the certificate that is in a proper sequence e.g. using a suffixed name instead. I'm not sure if the MitigateChainOrder is supposed to do exactly already something like that but trying the version mentioned that should contain this feature it doesn't seem to work as I expected.

While we will for now stick to a custom solution to address this issue it would be nice to offer some kind of feature out of the box. The decision to rely mostly on the keyvault for the process is understandable still this would improve the general suitability for more use cases.