Merge TLS1.3 Feature Branch into Main

[developStorm]

• Passes Go unit tests
• ZGrab works with this branch and negotiates TLS13 correctly
• Larger scale test with Censys

  Hey, wanted to report back on my testing. Once I updated a few go dependencies, I was able to integrate into our monorepo (I do need to verify the go dependencies with other services, but that is an internal issue). Once that was done, I spot checked several hosts, paying particular attention to ones that support multiple TLS versions (we support TLS 1.3, 1.2, 1.1, 1.0, and SSLv3 when doing versioning checks). I also checked TLS handshake fingerprints that we perform, like JA3S and JA4S.
  All of these tests passed!
  When we do TLS versioning, we use the zcrypto TLS 1.3 branch for these versions: TLS 1.3, 1.2, 1.1, 1.0. That is what I switched over to your refactor merge branch. For SSLv3, we use the zcrypto main, as the TLS 1.3 branch was missing SSLv3 support.
  I presume this is still the case, or does your refactor branch add back in SSLv3? (It is fine if it does not) (this is expected).

I manually inspected every failed Go official unit test after the merge with Wireshark, confirming root cause of the change in recorded traffic flow, and apply fix or adopt the new flow as appropriate.

I also tested this with zgrab. Confirmed we are using TLS 1.3 (and also seems like we supported some new signature algorithms too!)

(Left original, right with new zcrypto)

Features discontinued to support

This list may be incomplete due to the significant differences between branches. I discussed this with @phillip-stephens, and we agreed to identify any missing features on the fly after merging. Since basic TLS and testing with Censys went smoothly, I’m confident that any missing features are unlikely to cause major issues.

• HeartBeat extension / Heartbleed exploitation
• Next Protocol Negotiation (deprecated RFC, replaced by ALPN)
• SSLv3 (can workaround with old branch)
• ...

[phillip-stephens]

This is too large to reasonably review line-by-line. I've done some sanity checks and wanted to report back on why I think this is safe to merge.

After running a 1% of IPv4 space ZMap scan to ports 443 (HTTPS) and 587 (SMTP), I followed up with running a ZGrab scan on the hits with tls.

Of the over 700k targets, there was a slight uplift in success rate with TLS 1.3 included:

• master -> 67.43%
• TLS13 -> 68.03%
  With 4,206 new successful hosts able to handshake with TLS13 support.

This tracks with my observed ratio of hosts that only support TLS 1.3, I found that 0.6% of hosts only supported TLS 1.3 on 100k targets, so this explains the difference.

There are some removed fields from the handshake_log, specifically:

data.tls.result.handshake_log.key_material, data.tls.result.handshake_log.client_key_exchange, data.tls.result.handshake_log.server_key_exchange,data.tls.result.handshake_log.client_finished, data.tls.result.handshake_log.server_finished, data.tls.result.handshake_log.key_material.pre_master_secret

[zakird]

How many SSLv3 hosts were lost on those two scans?

…

On Thu, Mar 6, 2025 at 12:50 PM Phillip Stephens ***@***.***> wrote: ***@***.**** approved this pull request. This is too large to reasonably review line-by-line. I've done some sanity checks and wanted to report back on why I think this is safe to merge. After running a 1% of IPv4 space ZMap scan to ports 443 (HTTPS) and 587 (SMTP), I followed up with running a ZGrab scan on the hits with tls. Of the over 700k targets, there was a slight uplift in success rate with TLS 1.3 included: - master -> 67.43% - TLS13 -> 68.03% With 4,206 new successful hosts able to handshake with TLS13 support. This tracks with my observed ratio of hosts that only support TLS 1.3, I found that 0.6% of hosts *only* supported TLS 1.3 on 100k targets, so this explains the difference. There are some removed fields from the handshake_log, specifically: data.tls.result.handshake_log.key_material, data.tls.result.handshake_log.client_key_exchange, data.tls.result.handshake_log.server_key_exchange,data.tls.result.handshake_log.client_finished, data.tls.result.handshake_log.server_finished, data.tls.result.handshake_log.key_material.pre_master_secret These were removed in TLS1.3 and we'd need to add our own support for parsing them if provided in TLS1.2 — Reply to this email directly, view it on GitHub <#406 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABREUCP4HIH4ICUA7YTVUD2TCYKHAVCNFSM6AAAAABXLQ47OGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDMNRVGU4TMMJZHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[developStorm]

@zakird

grep "unsupported protocol version 300" smtp_and_tls_zgrab_13_branch.output | wc -l
802

Out of 705,737 targets

[phillip-stephens]

@developStorm This looks significantly better w.r.t. bringing back TLS 1.2 scan data. There are 2 things I'm noticing in the diff though:
CLI invocation - echo "100.0.59.51, , , 443" | ./zgrab2 tls | jq

 diff /tmp/master-100.0.59.51.json /tmp/branch-100.0.59.51.json
...
133c133
<               "browser_error": "x509: failed to load system roots and no roots provided"
---
>               "browser_error": "x509: unknown error"
...
159,160c159,160
<                 "signature_algorithm": "rsa",
<                 "hash_algorithm": "sha256"
---
>                 "signature_algorithm": "unknown.227",
>                 "hash_algorithm": "sha384"

Can we:

1. Bring that browser_error back if it indeed was caused by not loading the certs?
2. Figure out why that signature algorithm says unknown.227

I'm running a bulk scan to be sure there's no regressions there now.

[phillip-stephens]

A 1% scan of TLS and SMTP services yields a 1.24% increase in successful responses between master and a ZGrab branch using this version of ZCrypto.

So large-scale scans look good. I think if we can get those two fields fixed, this is good to go.

[zakird]

@developStorm once you've gotten those fields added back in, can you paste before and after scans of a few hosts to spot check?