Certificate Count Discrepancy on CT Community Website

[Barre]

The certificate count displayed on the CT community website's landing page retrieves data from Cloudflare's endpoint:

https://ct.cloudflare.com/data/total

Reference:

certificate-transparency-community-site/scripts/update_certcount.js

Line 21 in 7ce678f

const dataUrl = "https://ct.cloudflare.com/data/total";

As the operator of Merklemap CT monitor, I've observed that our certificate count significantly exceeds Cloudflare's count:

curl -N https://api.merklemap.com/v1-webui/counters

Questions

1. Which CT logs are included in Cloudflare's total count?
2. Is there documentation explaining how Cloudflare calculates their total?
3. Could we update the community site to reference a more comprehensive count or clarify which logs are being counted?

Thanks in advance for looking into this!

[phbnf]

It's really nice to see different sources of information for this!

For MerkleTown (1, 2), @lukevalenta from this thread might have more information about this, or you can reach out to ct-logs@cloudflare.com directly from https://ct.cloudflare.com/about.

For 3: as you've seen, we get this data from Cloudflare, so it sounds like a good idea to link to this source. This number is displayed on this page to give a rough understanding of the scale of CT, and since we don't compute it ourselves, I'd rather not try and list on certificate.transparency.dev which logs someone else's pipeline is configured with. If there's a page we can link to though, that would be great. As far as I can tell, MerkleTown lists the current CT logs it ingests https://ct.cloudflare.com/logs, but not all the logs that it ever ingested.

[Barre]

It's really nice to see different sources of information for this!

For MerkleTown (1, 2), @lukevalenta from this thread might have more information about this, or you can reach out to ct-logs@cloudflare.com directly from https://ct.cloudflare.com/about.

For 3: as you've seen, we get this data from Cloudflare, so it sounds like a good idea to link to this source. This number is displayed on this page to give a rough understanding of the scale of CT, and since we don't compute it ourselves, I'd rather not try and list on certificate.transparency.dev which logs someone else's pipeline is configured with. If there's a page we can link to though, that would be great. As far as I can tell, MerkleTown lists the current CT logs it ingests https://ct.cloudflare.com/logs, but not all the logs that it ever ingested.

Thanks for the thorough reply.

That makes sense. I guess we can close this issue if it's not really actionable as of now. I had hoped for some "easy" obvious reason that I hadn't thought about.

I'll try to reach out to Luke on the Google Groups thread to try to get to the bottom of this :)

[lukevalenta]

I wasn't aware that the community site was using the https://ct.cloudflare.com/data/total endpoint--good to know! That endpoint should be considered unstable as the work (from the linked thread above) to migrate the CT monitor information to radar.cloudflare.com is ongoing.

And the number we have there is indeed an undercount. Our CT monitor data prior to 2020 is incomplete (we switched database backends, and were unable to import data for some older logs due to some database corruption in the old backend). We have tentative plans to backfill the missing data, but that's lower priority for the team.

Given the above, switching the community site to a different datasource (such as one from MerkleMap) seems prudent.

[Barre]

Thanks Luke for the backstory!

And the number we have there is indeed an undercount. Our CT monitor data prior to 2020 is incomplete (we switched database backends, and were unable to import data for some older logs due to some database corruption in the old backend). We have tentative plans to backfill the missing data, but that's lower priority for the team.

That makes more sense now.

Given the above, switching the community site to a different datasource (such as one from MerkleMap) seems prudent.

Well, I wouldn't mind that. Perhaps we could even display the count as a live feed from the SSE stream, similar to what I've implemented at https://www.merklemap.com/

[lukevalenta]

As another data point, summing the columns of https://crt.sh/cert-populations I get 17.3B total certs+percerts. @robstradling fyi.

[Barre]

As another data point, summing the columns of https://crt.sh/cert-populations I get 17.3B total certs+percerts. @robstradling fyi.

This makes sense, as Merklemap still has a backlog to process (hence the ~7M certificates ingested per hour compared to the current volume). I expect these figures to converge in approximately one to one and a half months.