Triage incidents

[mcpherrinm]

This ticket is a survey of past incidents from CAs to ensure all related incidents can be covered.

• 1726333: Network Solutions: All test CA test website certificates are expired
• 1730291: Apple: Test website certificates expired
• 1904038: Chunghwa Telecom: “Test Website - Valid" URL disclosed to CCADB is expired
• 1925239: Microsec: Expired Certificates on test Pages for Revocation
• 1947207: FNMT: Incorrect publication of information for Test Website
• 1962426: NETLOCK: CA/Browser Forum TLS BR Non-compliance
• 1962809: SSL.com: Expired certificate for a “Valid” Test Website
• 1988405: TunTrust: Issue with Valid test Certificate
• 2008847: Microsoft PKI Services: Sample Site Certificates expired
• 2009149: D-Trust: Expired certificate provided on the CA TLS test website for demonstration of valid certificates
• 2013576: Microsec: "DV valid" test website certificate issued under incorrect root
• 2025135: SHECA: Test Website certificate expired
• 2025231: Chunghwa Telecom: Test Website certificate not revoked

[mcpherrinm]

• some of these are because of no automation or monitoring at all. Some specifically cite the challenges around automating revoked/expired certs.
• two incidents related to DNS records not pointing to the right IP
• alerts going to ignored emails is a common theme
• wrong chain served due to root transitions - Check chains ourselves #20

Test-certs-site solves the automation.
Monitoring is done with boulder-observer, prometheus, and paging.

Wrong-chain issues might happen until #20 is fixed. Boulder-monitor would notice.

Wrong-IP (especially A & AAAA split) could go un-noticed if monitoring is consistently only using IPv4 or IPv6. Currently we are using a CNAME to an AWS NLB so that's unlikely to break.