Merge TLS1.3 Feature Branch into Main #406
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* [DRAFT] Handshake logs * Removed heartbleed * Added handshake logs for ZGrab2 * Added handshake logs for ZGrab2 * TLS1.3: added cert logs * TLS1.3: adding keyAgreement logs * TLS1.3: log SignatureAndHashes * TLS1.3: log SupportedCurves * TLS1.3: log ServerKeyExchange * TLS 1.3: adding legacy Key Agreement algs
Co-authored-by: Denis Issoupov <[email protected]>
* Extract and output session ticket lifetime hint This restores the functionality from commit db98bd3, on the TLSv13 branch * tls: support ForceSessionTicketExt for ticketSupported
For TLS 1.3 connections, SupportedVersions.SelectedVersions will be present, and be 0x0304. Add this to the HandshakeLog, if present.
Do not overwrite collected server certs when we are asked for a client cert.
#331) * Added extension IDs for Server Hello messages to handshake log * Added marshalling capabilities for unknown extensions with empty data * Switched to extension extract function on serverHelloMsg * Re-added whitespace break
…erNotice (#334) * x509: make jsonifyExtensions() public * Certificate Policies: add grouped user notices field The separate fields for NoticeReferencNumbers, NoticeRefOrganization, and ExplicitTexts introduce ambiguity since these fields are structured and optional in the source data. A certificate with a mixture of UserNotices that have only one of ExplicitText or NoticeReference would previously be impossible to reconstruct. Add a new field, UserNotices, which preserved the original grouping of values, leaving the old format exposed in place, so that this case can be reconstructed without breaking existing usage.
Add option for CT log client to emit unparseable certs
Prior to the TLS 1.3 backport, there was a type assertion to make sure that cert.PublicKey.(*rsa.PublicKey) was true. This was lost in the backport work, and while very rare we did recently hit a case where this assertion is not true. Doing it inline in the call leads to a panic. This restores the prior type assertion check, and returns err if it fails.
Expose 'validSignature' field
The x509 package sets this field true when it finds a valid signature while validating certificates; copy the behavior here for consistency.
verifier: set ValidSignature field
Run go fmt to fix CI
verifier: add AppendFromPEMErr method
verifier: set ValidSignature for certificates in the graph
This reverts commit 90dd94c.
A change in ECDSA signature generation made old flow data incompatible with newer Go versions: golang/go@08f2091
phillip-stephens
approved these changes
Mar 6, 2025
There was a problem hiding this comment.
This is too large to reasonably review line-by-line. I've done some sanity checks and wanted to report back on why I think this is safe to merge.
After running a 1% of IPv4 space ZMap scan to ports 443 (HTTPS) and 587 (SMTP), I followed up with running a ZGrab scan on the hits with tls.
Of the over 700k targets, there was a slight uplift in success rate with TLS 1.3 included:
- master -> 67.43%
- TLS13 -> 68.03%
With 4,206 new successful hosts able to handshake with TLS13 support.
This tracks with my observed ratio of hosts that only support TLS 1.3, I found that 0.6% of hosts only supported TLS 1.3 on 100k targets, so this explains the difference.
There are some removed fields from the handshake_log, specifically:
data.tls.result.handshake_log.key_material, data.tls.result.handshake_log.client_key_exchange, data.tls.result.handshake_log.server_key_exchange,data.tls.result.handshake_log.client_finished, data.tls.result.handshake_log.server_finished, data.tls.result.handshake_log.key_material.pre_master_secret
|
How many SSLv3 hosts were lost on those two scans?
…On Thu, Mar 6, 2025 at 12:50 PM Phillip Stephens ***@***.***> wrote:
***@***.**** approved this pull request.
This is too large to reasonably review line-by-line. I've done some sanity
checks and wanted to report back on why I think this is safe to merge.
After running a 1% of IPv4 space ZMap scan to ports 443 (HTTPS) and 587
(SMTP), I followed up with running a ZGrab scan on the hits with tls.
Of the over 700k targets, there was a slight uplift in success rate with
TLS 1.3 included:
- master -> 67.43%
- TLS13 -> 68.03%
With 4,206 new successful hosts able to handshake with TLS13 support.
This tracks with my observed ratio of hosts that only support TLS 1.3, I
found that 0.6% of hosts *only* supported TLS 1.3 on 100k targets, so
this explains the difference.
There are some removed fields from the handshake_log, specifically:
data.tls.result.handshake_log.key_material,
data.tls.result.handshake_log.client_key_exchange,
data.tls.result.handshake_log.server_key_exchange,data.tls.result.handshake_log.client_finished,
data.tls.result.handshake_log.server_finished,
data.tls.result.handshake_log.key_material.pre_master_secret
These were removed in TLS1.3 and we'd need to add our own support for
parsing them if provided in TLS1.2
—
Reply to this email directly, view it on GitHub
<#406 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABREUCP4HIH4ICUA7YTVUD2TCYKHAVCNFSM6AAAAABXLQ47OGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDMNRVGU4TMMJZHE>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Out of 705,737 targets |
phillip-stephens
requested changes
Mar 11, 2025
There was a problem hiding this comment.
@developStorm This looks significantly better w.r.t. bringing back TLS 1.2 scan data. There are 2 things I'm noticing in the diff though:
CLI invocation - echo "100.0.59.51, , , 443" | ./zgrab2 tls | jq
diff /tmp/master-100.0.59.51.json /tmp/branch-100.0.59.51.json
...
133c133
< "browser_error": "x509: failed to load system roots and no roots provided"
---
> "browser_error": "x509: unknown error"
...
159,160c159,160
< "signature_algorithm": "rsa",
< "hash_algorithm": "sha256"
---
> "signature_algorithm": "unknown.227",
> "hash_algorithm": "sha384"
Can we:
- Bring that
browser_errorback if it indeed was caused by not loading the certs? - Figure out why that signature algorithm says
unknown.227
I'm running a bulk scan to be sure there's no regressions there now.
|
A 1% scan of TLS and SMTP services yields a 1.24% increase in successful responses between So large-scale scans look good. I think if we can get those two fields fixed, this is good to go. |
|
@developStorm once you've gotten those fields added back in, can you paste before and after scans of a few hosts to spot check? |
…thms Dropped an incorrect comment that suggested signature algorithm constants aligned with RFC 5246, as this was a long-standing misunderstanding. Added support for additional signature algorithms introduced by the updated TLS library.
|
@phillip-stephens Thanks for pointing this out!
"browser_error": "x509: certificate is self-signed and not a trusted root"
"signature_algorithm": "pkcs1v15",
"hash_algorithm": "sha384"It seems we never directly exposes the For the before-and-after scans, I conducted five scans on five different hosts identified through Censys for each of TLS 1.0, 1.1, and 1.2. I didn't observe any changes beyond what we already know. These include: randoms, secret values obviously changing; cipher suites occasionally negotiating to different ones due to upstream preference changes; the Complete scan logs: diffs.tar.gz * For the secure renegotiation, the criteria changed from A summarized report on differences, with these keys ignores: |
phillip-stephens
approved these changes
Mar 17, 2025
There was a problem hiding this comment.
I re-ran a diff and everything looks good from my end. Really awesome work @developStorm, thanks for getting this over the finish line.
$ diff <(echo "100.0.59.51, , , 443" | ./zgrab2 tls | jq) <(echo "100.0.59.51, , , 443" | zgrab2 tls | jq)
15,16c15,16
< "random": "fcH+wn0LMYiS6NjFDCbzP7Z8Iebk+kUakIT9NR7bQn4=",
< "session_id": "9k1bb7f46Z7Kol4RIjnHbfAj59vObIVqR2lqQ4RoAGU=",
---
> "random": "eHAvFs01oxcjSWBVf4ubrC3ovGNJN0lFQxGMrJn20gU=",
> "session_id": "gxi4/ka1or04J1oQse4IhXQkzSChTPhMM+pin4Vp27E=",
18,20c18,20
< "hex": "0xC030",
< "name": "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
< "value": 49200
---
> "hex": "0xC014",
> "name": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
> "value": 49172
29c29
< "secure_renegotiation": false,
---
> "secure_renegotiation": true,
32,34c32,34
< "extension_identifiers": [
< 65281,
< 11
---
> "next_protocol_negotiation": false,
> "unknown_extensions": [
> "AAsABAMAAQI="
144c144
< "value": "NoW+tp1PHiDR/ykjWQ6xUKTPDzqboE1SciISWLFqA2A=",
---
> "value": "LHWllfWINJHJ1ehdwRhPKgOltoWSMqK2rPpsWRgHW4E=",
148c148
< "value": "3ND5YwkH/gsp6H3DRQ1EbYVnwSedLOXJZHsfHlEuluk=",
---
> "value": "g9JgBjuU9cJO/5gy7UeJnM9523DxH65c1NJ19vJmEiU=",
153c153
< "digest": "mJLUUhEOdAP/5W/k4hlMRG83jn+wxTccPwpeN/Y5DGg=",
---
> "digest": "SIA7NWBgxBv+QVbqh8g5RlG47DHLnbMyw1iJWYGE4OY=",
155c155
< "raw": "uNWGNPU180tjPJhI1KjCKMxVLr3nGbr3QA3Z0xG6ahroer1Q9O2UHOfCSKjo1L/t/+ms1RvqPABKrkqF4Qhu+JdcUWKtemwxh5h/I8anZGuYvBbshGm00oRTrihSS8pJkUG/XGB3LaO69DojJf9RUZjlwQBPJ63M5AQnaoai3Spuculv//q6wA02pVPpkO2Z9QF2/tdyET6vF2tgK84JQvDs+1v/F8VYqp38EAnWzMxIqcPuBKpcmEsHSlKf5IGkkqdTJ8Mdy8wk8Eyx/C7SNCWApAueXZTpWxm4nsIsPm/0QdrnAoRCsIEVhgVa00u3MUP6UJ/fYkTqhUTLOuWoVw==",
---
> "raw": "EcbDUZkg+A0T91R2Z8Vhowo8FjJu2Faj7DIq/ygMbxWb9B64+jo//d/jG5cz9pyKeoQCVM8+EUOG4j70qWYgpHEcOa11ulRZMHl1xYDuo+vEPqf5VuiGejd32zyKEwaQ7IbdE3IZR8+uMZagndg+JbIo1c9lOdDwOKg0mGBLZzaWecmmCoqgFBjdyIRrbP0t8YCYdhqh6gCLNRaROLRIMqkpntV7pDmDIoZD1ZZn2Cf1/IQBYqwveQwfd6q0otjHI6eruF8g34aBMzjWL6WPnwTU8sLncVyJsVIEEP/FoI3VzLoJ/2RudD8f2yrrZT9Uj3CogD6Nuww82j/lxidNfA==",
159,160c159,160
< "signature_algorithm": "pkcs1v15",
< "hash_algorithm": "sha384"
---
> "signature_algorithm": "rsa",
> "hash_algorithm": "sha256"
176c176
< "value": "qMQYWjwTEMlDWIc+rocNikrg1LK88ZVGUy0KAX1V/EI=",
---
> "value": "TEmoshEVL7wgVF3VEjIoRGS4HLpeGlmRQPXVO07n5Ok=",
180c180
< "value": "JB5gMO7g2q6yKuN2vDVsSZCqewIXCx/4PjbfQ/W0NLw=",
---
> "value": "N44UUAVsrlRzr+ztKGJq7IIEzedYHWnLMlkQ58jMBWQ=",
185c185
< "value": "EgIzKSqWrevlJCszuLk9v1CIQyhv0/vt4fFGqDO1Ol0=",
---
> "value": "+95xj/t7dpOWdk/v2vtTz3cT+hQ9mJnWnweINW/tJes=",
191c191
< "verify_data": "lYAtPXXIvS7EEfqR"
---
> "verify_data": "LgoKuhPH6KJkOEiH"
194c194
< "verify_data": "qUt2n5b3qb6CSQ/G"
---
> "verify_data": "4ZSbs1/N4RFtN59H"
198c198
< "value": "iDV8OjB8vcFhqXFvEsyvsL+U32t723OmD87xQHH683L03E6s4rDFKwuI26OD088T",
---
> "value": "IuntMVx1NM5zfKaKrCJJpGoQTlofmPT8CQPDcA6vhnPlfUHjQG3xpPbnMH7WfHrc",
202c202
< "value": "1dBXoAQ4U3IxqrzDO2oJDQHvvxAzzLpQFbkM9k16bGk=",
---
> "value": "r2q6+7KMEHCcwC0qLYiXHf2FtczwzQ735QCP7LEcJyQ=",
208c208
< "timestamp": "2025-03-17T10:14:34-07:00"
---
> "timestamp": "2025-03-17T10:14:33-07:00"
|
I've tagged current |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I manually inspected every failed Go official unit test after the merge with Wireshark, confirming root cause of the change in recorded traffic flow, and apply fix or adopt the new flow as appropriate.
I also tested this with zgrab. Confirmed we are using TLS 1.3 (and also seems like we supported some new signature algorithms too!)
(Left original, right with new zcrypto)
Features discontinued to support
This list may be incomplete due to the significant differences between branches. I discussed this with @phillip-stephens, and we agreed to identify any missing features on the fly after merging. Since basic TLS and testing with Censys went smoothly, I’m confident that any missing features are unlikely to cause major issues.