Comparing Results from Tracker 1.0 and Tracker 2.0

Tracker 1.0 focused on HTTPS/SSL configuration so this comparison concerns only web results.

Validation of results was performed with the SSL Labs server tester.

Domains

Tracker 1.0 (via https-everywhere.canada.ca) has results for around 3000 domains, while at this time Tracker 2.0 is monitoring 194. There are 136 domains for which both have results. When looking at 4 key metrics (HTTPS enforcement, use of HSTS, use of weak cipher suites, and certificate validity), 69 of these domains have differing results between the two.

HTTPS Enforcement

There are 4 domains for which the two versions of Tracker differ on detection of HTTPS enforcement. 3 of the differences are a result of Tracker 2.0 having a stricter definition of what it means to "enforce" HTTPS (less tolerant of redirects). The remaining domain was a match until 13/05/2021, when it became unreachable to Tracker 2.0's scanners. This issue is being investigated.

HSTS

There are 5 domains for which the two versions of Tracker differ on detection of HSTS implementation. In 3 of these cases Tracker 2.0 was unambiguously correct. Of the remaining two domains, one produced different results when checking for HSTS with different methods and the other is the unreachable domain mentioned above.

Weak Cipher Suites

There are 44 domains for which the two versions of Tracker differ on detection of weak cipher suites in use. In 40 out of 44 cases Tracker 2.0 successfully detected the use of weak ciphers where 1.0 did not. Of the remaining 4, 2 were domains Tracker 1.0 falsely detected as using weak ciphers when they were not, and 2 domains were unreachable.

If this sample of 136 domains is representative of overall performance, Tracker 1.0 has a false negative rate of around 29.5% and a false positive rate of around 1.5% with respect to detection of weak ciphers. This suggests Tracker 1.0 has wrongly detected no weak ciphers in use for approximately 890 domains when extrapolating to its full set of 3038 monitored domains. It should however be noted that our sample of 136 domains is not only somewhat small compared to the population but also not randomly selected. Nevertheless, this demonstrates that Tracker 2.0 is significantly more trustworthy in detecting the use of non-compliant cipher suites.

Certificates

Note: Although documentation is unclear, it appears Tracker 1.0 only evaluates the signature algorithm used on a domain's certificate. Tracker 2.0 primarily relies on this as well, although it does add checks for expired or self signed certs. Per guidance, it is also required to check if a certificate has been revoked and if the name on the certificate matches the domain request.

There are 32 domains for which the two versions of Tracker differ on detection of certificate validity. WIP

ITPIN 2018-01 Compliance Determination

Both versions of Tracker are intended to monitor compliance with this ITPIN on HTTPS implementation. Tracker 2.0 differs in the determination of compliance in that it deems a domain non-compliant if it does not implement HSTS preload. This is in line with best practices but is not mentioned in the relevant policy documents so it should not be used to determine compliance status. This issue will be fixed shortly.

Since a domain is compliant if it is passing in the above metrics, Tracker 2.0's more accurate detection of weak ciphers and certificate issues will mean fewer domains are considered compliant.