Skip to content

Merge TLS1.3 Feature Branch into Main #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Mar 17, 2025

Conversation

developStorm
Copy link
Member

@developStorm developStorm commented Feb 18, 2025

  • Passes Go unit tests
  • ZGrab works with this branch and negotiates TLS13 correctly
  • Larger scale test with Censys

    Hey, wanted to report back on my testing. Once I updated a few go dependencies, I was able to integrate into our monorepo (I do need to verify the go dependencies with other services, but that is an internal issue). Once that was done, I spot checked several hosts, paying particular attention to ones that support multiple TLS versions (we support TLS 1.3, 1.2, 1.1, 1.0, and SSLv3 when doing versioning checks). I also checked TLS handshake fingerprints that we perform, like JA3S and JA4S.
    All of these tests passed!
    When we do TLS versioning, we use the zcrypto TLS 1.3 branch for these versions: TLS 1.3, 1.2, 1.1, 1.0. That is what I switched over to your refactor merge branch. For SSLv3, we use the zcrypto main, as the TLS 1.3 branch was missing SSLv3 support.
    I presume this is still the case, or does your refactor branch add back in SSLv3? (It is fine if it does not) (this is expected).

I manually inspected every failed Go official unit test after the merge with Wireshark, confirming root cause of the change in recorded traffic flow, and apply fix or adopt the new flow as appropriate.

I also tested this with zgrab. Confirmed we are using TLS 1.3 (and also seems like we supported some new signature algorithms too!)

(Left original, right with new zcrypto)
image
image
image

Features discontinued to support

This list may be incomplete due to the significant differences between branches. I discussed this with @phillip-stephens, and we agreed to identify any missing features on the fly after merging. Since basic TLS and testing with Censys went smoothly, I’m confident that any missing features are unlikely to cause major issues.

  • HeartBeat extension / Heartbleed exploitation
  • Next Protocol Negotiation (deprecated RFC, replaced by ALPN)
  • SSLv3 (can workaround with old branch)
  • ...

dissoupov and others added 30 commits July 15, 2021 10:17
* [DRAFT] Handshake logs

* Removed heartbleed

* Added handshake logs for ZGrab2

* Added handshake logs for ZGrab2

* TLS1.3: added cert logs

* TLS1.3: adding keyAgreement logs

* TLS1.3: log SignatureAndHashes

* TLS1.3: log SupportedCurves

* TLS1.3: log ServerKeyExchange

* TLS 1.3: adding legacy Key Agreement algs
* Extract and output session ticket lifetime hint

This restores the functionality from commit db98bd3, on the TLSv13
branch

* tls: support ForceSessionTicketExt for ticketSupported
For TLS 1.3 connections, SupportedVersions.SelectedVersions will be
present, and be 0x0304.  Add this to the HandshakeLog, if present.
Do not overwrite collected server certs when we are asked for a client cert.
#331)

* Added extension IDs for Server Hello messages to handshake log

* Added marshalling capabilities for unknown extensions with empty data

* Switched to extension extract function on serverHelloMsg

* Re-added whitespace break
…erNotice (#334)

* x509: make jsonifyExtensions() public

* Certificate Policies: add grouped user notices field

The separate fields for NoticeReferencNumbers, NoticeRefOrganization,
and ExplicitTexts introduce ambiguity since these fields are structured
and optional in the source data.

A certificate with a mixture of UserNotices that have only one of
ExplicitText or NoticeReference would previously be impossible to
reconstruct.

Add a new field, UserNotices, which preserved the original grouping of
values, leaving the old format exposed in place, so that this case can
be reconstructed without breaking existing usage.
Add option for CT log client to emit unparseable certs
Prior to the TLS 1.3 backport, there was a type assertion to make sure
that cert.PublicKey.(*rsa.PublicKey) was true.  This was lost in the
backport work, and while very rare we did recently hit a case where
this assertion is not true.  Doing it inline in the call leads to a
panic.

This restores the prior type assertion check, and returns err if it
fails.
The x509 package sets this field true when it finds a valid signature
while validating certificates; copy the behavior here for consistency.
verifier: add AppendFromPEMErr method
verifier: set ValidSignature for certificates in the graph
@developStorm developStorm self-assigned this Feb 18, 2025
@developStorm developStorm marked this pull request as ready for review March 4, 2025 22:11
Copy link
Contributor

@phillip-stephens phillip-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too large to reasonably review line-by-line. I've done some sanity checks and wanted to report back on why I think this is safe to merge.

After running a 1% of IPv4 space ZMap scan to ports 443 (HTTPS) and 587 (SMTP), I followed up with running a ZGrab scan on the hits with tls.

Of the over 700k targets, there was a slight uplift in success rate with TLS 1.3 included:

  • master -> 67.43%
  • TLS13 -> 68.03%
    With 4,206 new successful hosts able to handshake with TLS13 support.

This tracks with my observed ratio of hosts that only support TLS 1.3, I found that 0.6% of hosts only supported TLS 1.3 on 100k targets, so this explains the difference.

There are some removed fields from the handshake_log, specifically:

data.tls.result.handshake_log.key_material, data.tls.result.handshake_log.client_key_exchange, data.tls.result.handshake_log.server_key_exchange,data.tls.result.handshake_log.client_finished, data.tls.result.handshake_log.server_finished, data.tls.result.handshake_log.key_material.pre_master_secret

@zakird
Copy link
Member

zakird commented Mar 6, 2025 via email

@developStorm
Copy link
Member Author

developStorm commented Mar 6, 2025

@zakird

grep "unsupported protocol version 300" smtp_and_tls_zgrab_13_branch.output | wc -l
802

Out of 705,737 targets

@phillip-stephens phillip-stephens self-requested a review March 11, 2025 17:00
Copy link
Contributor

@phillip-stephens phillip-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@developStorm This looks significantly better w.r.t. bringing back TLS 1.2 scan data. There are 2 things I'm noticing in the diff though:
CLI invocation - echo "100.0.59.51, , , 443" | ./zgrab2 tls | jq

 diff /tmp/master-100.0.59.51.json /tmp/branch-100.0.59.51.json
...
133c133
<               "browser_error": "x509: failed to load system roots and no roots provided"
---
>               "browser_error": "x509: unknown error"
...
159,160c159,160
<                 "signature_algorithm": "rsa",
<                 "hash_algorithm": "sha256"
---
>                 "signature_algorithm": "unknown.227",
>                 "hash_algorithm": "sha384"

Can we:

  1. Bring that browser_error back if it indeed was caused by not loading the certs?
  2. Figure out why that signature algorithm says unknown.227

I'm running a bulk scan to be sure there's no regressions there now.

@phillip-stephens
Copy link
Contributor

A 1% scan of TLS and SMTP services yields a 1.24% increase in successful responses between master and a ZGrab branch using this version of ZCrypto.

So large-scale scans look good. I think if we can get those two fields fixed, this is good to go.

@zakird
Copy link
Member

zakird commented Mar 12, 2025

@developStorm once you've gotten those fields added back in, can you paste before and after scans of a few hosts to spot check?

…thms

Dropped an incorrect comment that suggested signature algorithm constants aligned with RFC 5246, as this was a long-standing misunderstanding. Added support for additional signature algorithms introduced by the updated TLS library.
@developStorm
Copy link
Member Author

developStorm commented Mar 16, 2025

@phillip-stephens Thanks for pointing this out!

  • Browser Error: This was due to the new TLS library introducing a dedicated error constant for self-signed certs. I've added the correct error string, so it now shows up as:
"browser_error": "x509: certificate is self-signed and not a trusted root"
  • Signature Algorithm: The "unknown.227" issue happened because we previously assumed our signature algorithm constants aligned with RFC 5246. Turns out, that wasn’t actually the case - the values were explicitly made distinct in the new TLS library exposed this mismatch. I've updated the mappings now, and it correctly shows:
"signature_algorithm": "pkcs1v15",
"hash_algorithm": "sha384"

It seems we never directly exposes the signature_algorithm to outside of the package in its integer form, so using Go's internal int representation should be fine.

For the before-and-after scans, I conducted five scans on five different hosts identified through Censys for each of TLS 1.0, 1.1, and 1.2. I didn't observe any changes beyond what we already know. These include: randoms, secret values obviously changing; cipher suites occasionally negotiating to different ones due to upstream preference changes; the next_protocol_negotiation extension being dropped; unknown_extensions being renamed to extension_identifiers; and a stricter criteria for secure_renegotiation to be true (following upstream updates)*.

Complete scan logs: diffs.tar.gz

* For the secure renegotiation, the criteria changed from m.secureRenegotiationSupported to m.secureRenegotiationSupported && len(m.secureRenegotiation) > 0 (where m.secureRenegotiation is a newly introduced []byte field from upstream. I'm not sure if we should be this strict or stay consistent with previous version is better? It's a minor patch to make if we want this reverted.

A summarized report on differences, with these keys ignores: any(ignore in diff for ignore in [".random", ".digest", ".y.value", ".x.value", "master_secret.value", "verify_data", ".signature.raw", ".session_id", "client_private.value", ".encrypted_pre_master_secret", ".timestamp"])

Found differences in 15 entries.
Sample differences for key ('147.135.97.40', 443) (TLS version: TLSv1.2):
In branch:
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: pkcs1v15)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 49200)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: unknown.7)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xC030)
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256)
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsa)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha512)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 49199)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xC02F)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('13.56.239.180', 443) (TLS version: TLSv1.2):
In branch:
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: pkcs1v15)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.extension_identifiers
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha384)
In master:
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsa)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha256)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('185.253.212.22', 443) (TLS version: TLSv1.2):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate is self-signed and not a trusted root)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.id (value: 29)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.name (value: x25519)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 49200)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: C=PL, ST=Mazowieckie, L=Warszawa, O=AfterMarket.pl, CN=banned.ssl.vhost, [email protected])
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)
  data.tls.result.handshake_log.server_hello.extension_identifiers
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.name (value: x25519)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xC030)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=PL, ST=Mazowieckie, L=Warszawa, O=AfterMarket.pl, CN=banned.ssl.vhost, [email protected])
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsapss)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.id (value: 29)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha384)
In master:
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.id (value: 23)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.client_public.y
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256)
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsa)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.id (value: 23)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.server_public.y
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: [email protected], C=PL, ST=Mazowieckie, L=Warszawa, O=AfterMarket.pl, CN=banned.ssl.vhost, [email protected])
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=PL, ST=Mazowieckie, L=Warszawa, O=AfterMarket.pl, CN=banned.ssl.vhost, [email protected])
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.name (value: secp256r1)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: unknown error)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha256)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 49199)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xC02F)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.name (value: secp256r1)
Sample differences for key ('177.52.61.83', 443) (TLS version: TLSv1.1):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate is self-signed and not a trusted root)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_AES_128_CBC_SHA)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x002F)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=CH, ST=HU, L=WU, O=FH, OU=SF, CN=PON, [email protected])
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 47)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: C=CH, ST=HU, L=WU, O=FH, OU=SF, CN=PON, [email protected])
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_RC4_128_SHA)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=CH, ST=HU, L=WU, O=FH, OU=SF, CN=PON, [email protected])
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: [email protected], C=CH, ST=HU, L=WU, O=FH, OU=SF, CN=PON, [email protected])
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x0005)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 5)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('79.42.45.114', 443) (TLS version: TLSv1.0):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate is self-signed and not a trusted root)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.signature.valid (value: True)
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_certificates.certificate.parsed.signature.valid (value: False)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('78.46.212.149', 8443) (TLS version: TLSv1.0):
In branch:
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x000A)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 10)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_3DES_EDE_CBC_SHA)
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_RC4_128_SHA)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x0005)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 5)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
Sample differences for key ('139.196.195.35', 443) (TLS version: TLSv1.0):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_trusted (value: True)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.signature.valid (value: True)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.signature.valid (value: False)
  data.tls.result.handshake_log.server_certificates.validation.browser_error
  data.tls.result.handshake_log.server_certificates.validation.browser_trusted (value: False)
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('186.92.127.142', 443) (TLS version: TLSv1.1):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "zx"))
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: [email protected], C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
Sample differences for key ('190.94.160.13', 443) (TLS version: TLSv1.0):
In branch:
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 47)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_AES_128_CBC_SHA)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x002F)
In master:
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_RSA_WITH_RC4_128_SHA)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0x0005)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 5)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
Sample differences for key ('45.162.168.254', 443) (TLS version: TLSv1.2):
In branch:
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: pkcs1v15)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.extension_identifiers
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha384)
In master:
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsa)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha256)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
Sample differences for key ('38.165.69.193', 443) (TLS version: TLSv1.2):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate is self-signed and not a trusted root)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.id (value: 29)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.name (value: x25519)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 52392)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.name (value: x25519)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.id (value: 29)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsapss)
  data.tls.result.handshake_log.server_hello.extension_identifiers
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha384)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xCCA8)
In master:
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.id (value: 23)
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.client_public.y
  data.tls.result.handshake_log.server_hello.cipher_suite.name (value: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256)
  data.tls.result.handshake_log.server_hello.unknown_extensions
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.signature_algorithm (value: rsa)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.id (value: 23)
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.server_public.y
  data.tls.result.handshake_log.server_key_exchange.ecdh_params.curve_id.name (value: secp256r1)
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: unknown error)
  data.tls.result.handshake_log.server_hello.cipher_suite.value (value: 49199)
  data.tls.result.handshake_log.server_key_exchange.signature.signature_and_hash_type.hash_algorithm (value: sha256)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.client_key_exchange.ecdh_params.curve_id.name (value: secp256r1)
  data.tls.result.handshake_log.server_hello.cipher_suite.hex (value: 0xC02F)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)
Sample differences for key ('186.92.19.245', 443) (TLS version: TLSv1.1):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "zx"))
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: [email protected], C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
Sample differences for key ('190.201.134.226', 443) (TLS version: TLSv1.0):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=cn, ST=JiangSu, L=NanJing, O=ZTE, OU=zte, CN=Kevin, [email protected])
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=cn, ST=JiangSu, L=NanJing, O=ZTE, OU=zte, CN=Kevin, [email protected])
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
Sample differences for key ('186.94.36.97', 443) (TLS version: TLSv1.1):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "zx"))
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: [email protected], C=cn, ST=gd, L=sz, O=hw, OU=nsd, CN=zx, [email protected])
  data.tls.result.handshake_log.server_certificates.chain (list content differs)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: [email protected], C=cn, ST=gd, O=hw, OU=nsd, CN=zxserver, [email protected])
Sample differences for key ('93.40.27.154', 443) (TLS version: TLSv1.1):
In branch:
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: certificate is self-signed and not a trusted root)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: name=AR169-Self-Signed-Certificate-21500102152SJ5600624)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: name=AR169-Self-Signed-Certificate-21500102152SJ5600624)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: False)
  data.tls.result.handshake_log.server_hello.extension_identifiers
In master:
  data.tls.result.handshake_log.server_certificates.certificate.parsed.subject_dn (value: name=AR169-Self-Signed-Certificate-21500102152SJ5600624, name=AR169-Self-Signed-Certificate-21500102152SJ5600624)
  data.tls.result.handshake_log.server_hello.next_protocol_negotiation
  data.tls.result.handshake_log.server_certificates.validation.browser_error (value: x509: failed to load system roots and no roots provided)
  data.tls.result.handshake_log.server_certificates.certificate.parsed.issuer_dn (value: name=AR169-Self-Signed-Certificate-21500102152SJ5600624, name=AR169-Self-Signed-Certificate-21500102152SJ5600624)
  data.tls.result.handshake_log.server_hello.secure_renegotiation (value: True)

@phillip-stephens phillip-stephens self-requested a review March 17, 2025 17:15
Copy link
Contributor

@phillip-stephens phillip-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-ran a diff and everything looks good from my end. Really awesome work @developStorm, thanks for getting this over the finish line.

$ diff <(echo "100.0.59.51, , , 443" | ./zgrab2 tls | jq) <(echo "100.0.59.51, , , 443" | zgrab2 tls | jq)

15,16c15,16
<             "random": "fcH+wn0LMYiS6NjFDCbzP7Z8Iebk+kUakIT9NR7bQn4=",
<             "session_id": "9k1bb7f46Z7Kol4RIjnHbfAj59vObIVqR2lqQ4RoAGU=",
---
>             "random": "eHAvFs01oxcjSWBVf4ubrC3ovGNJN0lFQxGMrJn20gU=",
>             "session_id": "gxi4/ka1or04J1oQse4IhXQkzSChTPhMM+pin4Vp27E=",
18,20c18,20
<               "hex": "0xC030",
<               "name": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
<               "value": 49200
---
>               "hex": "0xC014",
>               "name": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
>               "value": 49172
29c29
<             "secure_renegotiation": false,
---
>             "secure_renegotiation": true,
32,34c32,34
<             "extension_identifiers": [
<               65281,
<               11
---
>             "next_protocol_negotiation": false,
>             "unknown_extensions": [
>               "AAsABAMAAQI="
144c144
<                   "value": "NoW+tp1PHiDR/ykjWQ6xUKTPDzqboE1SciISWLFqA2A=",
---
>                   "value": "LHWllfWINJHJ1ehdwRhPKgOltoWSMqK2rPpsWRgHW4E=",
148c148
<                   "value": "3ND5YwkH/gsp6H3DRQ1EbYVnwSedLOXJZHsfHlEuluk=",
---
>                   "value": "g9JgBjuU9cJO/5gy7UeJnM9523DxH65c1NJ19vJmEiU=",
153c153
<             "digest": "mJLUUhEOdAP/5W/k4hlMRG83jn+wxTccPwpeN/Y5DGg=",
---
>             "digest": "SIA7NWBgxBv+QVbqh8g5RlG47DHLnbMyw1iJWYGE4OY=",
155c155
<               "raw": "uNWGNPU180tjPJhI1KjCKMxVLr3nGbr3QA3Z0xG6ahroer1Q9O2UHOfCSKjo1L/t/+ms1RvqPABKrkqF4Qhu+JdcUWKtemwxh5h/I8anZGuYvBbshGm00oRTrihSS8pJkUG/XGB3LaO69DojJf9RUZjlwQBPJ63M5AQnaoai3Spuculv//q6wA02pVPpkO2Z9QF2/tdyET6vF2tgK84JQvDs+1v/F8VYqp38EAnWzMxIqcPuBKpcmEsHSlKf5IGkkqdTJ8Mdy8wk8Eyx/C7SNCWApAueXZTpWxm4nsIsPm/0QdrnAoRCsIEVhgVa00u3MUP6UJ/fYkTqhUTLOuWoVw==",
---
>               "raw": "EcbDUZkg+A0T91R2Z8Vhowo8FjJu2Faj7DIq/ygMbxWb9B64+jo//d/jG5cz9pyKeoQCVM8+EUOG4j70qWYgpHEcOa11ulRZMHl1xYDuo+vEPqf5VuiGejd32zyKEwaQ7IbdE3IZR8+uMZagndg+JbIo1c9lOdDwOKg0mGBLZzaWecmmCoqgFBjdyIRrbP0t8YCYdhqh6gCLNRaROLRIMqkpntV7pDmDIoZD1ZZn2Cf1/IQBYqwveQwfd6q0otjHI6eruF8g34aBMzjWL6WPnwTU8sLncVyJsVIEEP/FoI3VzLoJ/2RudD8f2yrrZT9Uj3CogD6Nuww82j/lxidNfA==",
159,160c159,160
<                 "signature_algorithm": "pkcs1v15",
<                 "hash_algorithm": "sha384"
---
>                 "signature_algorithm": "rsa",
>                 "hash_algorithm": "sha256"
176c176
<                   "value": "qMQYWjwTEMlDWIc+rocNikrg1LK88ZVGUy0KAX1V/EI=",
---
>                   "value": "TEmoshEVL7wgVF3VEjIoRGS4HLpeGlmRQPXVO07n5Ok=",
180c180
<                   "value": "JB5gMO7g2q6yKuN2vDVsSZCqewIXCx/4PjbfQ/W0NLw=",
---
>                   "value": "N44UUAVsrlRzr+ztKGJq7IIEzedYHWnLMlkQ58jMBWQ=",
185c185
<                 "value": "EgIzKSqWrevlJCszuLk9v1CIQyhv0/vt4fFGqDO1Ol0=",
---
>                 "value": "+95xj/t7dpOWdk/v2vtTz3cT+hQ9mJnWnweINW/tJes=",
191c191
<             "verify_data": "lYAtPXXIvS7EEfqR"
---
>             "verify_data": "LgoKuhPH6KJkOEiH"
194c194
<             "verify_data": "qUt2n5b3qb6CSQ/G"
---
>             "verify_data": "4ZSbs1/N4RFtN59H"
198c198
<               "value": "iDV8OjB8vcFhqXFvEsyvsL+U32t723OmD87xQHH683L03E6s4rDFKwuI26OD088T",
---
>               "value": "IuntMVx1NM5zfKaKrCJJpGoQTlofmPT8CQPDcA6vhnPlfUHjQG3xpPbnMH7WfHrc",
202c202
<               "value": "1dBXoAQ4U3IxqrzDO2oJDQHvvxAzzLpQFbkM9k16bGk=",
---
>               "value": "r2q6+7KMEHCcwC0qLYiXHf2FtczwzQ735QCP7LEcJyQ=",
208c208
<       "timestamp": "2025-03-17T10:14:34-07:00"
---
>       "timestamp": "2025-03-17T10:14:33-07:00"

@phillip-stephens
Copy link
Contributor

I've tagged current master with the tag SSLv3 so if people need that it's easily findable.

@phillip-stephens phillip-stephens merged commit 58d1d32 into master Mar 17, 2025
5 checks passed
@phillip-stephens phillip-stephens deleted the refactor/tls13-merge-attempt-3 branch March 17, 2025 17:27
@zakird zakird mentioned this pull request Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants