Skip to content

Commit a841fca

Browse files
authored
fix: update ja4 compliance (#4773)
1 parent 003412f commit a841fca

22 files changed

+362
-176
lines changed

api/unstable/fingerprint.h

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ S2N_API int s2n_fingerprint_get_hash_size(const struct s2n_fingerprint *fingerpr
106106
*
107107
* JA4: A string consisting of three parts, separated by underscores: the prefix,
108108
* and the hex-encoded truncated SHA256 hashes of the other two parts of the raw string.
109-
* - See https://github.com/FoxIO-LLC/ja4/blob/v0.18.2/technical_details/JA4.md
109+
* - See https://github.com/FoxIO-LLC/ja4/blob/df3c067/technical_details/JA4.md
110110
* - Example: "t13i310900_e8f1e7e78f70_1f22a2ca17c4"
111111
*
112112
* @param fingerprint The s2n_fingerprint to be used for the hash
@@ -145,7 +145,7 @@ S2N_API int s2n_fingerprint_get_raw_size(const struct s2n_fingerprint *fingerpri
145145
* 156-61-60-53-47-255,11-10-35-22-23-13-43-45-51,29-23-30-25-24,0-1-2"
146146
*
147147
* JA4: A string consisting of three parts: a prefix, and two lists of hex values.
148-
* - See https://github.com/FoxIO-LLC/ja4/blob/v0.18.2/technical_details/JA4.md
148+
* - See https://github.com/FoxIO-LLC/ja4/blob/df3c067/technical_details/JA4.md
149149
* - Example: "t13i310900_002f,0033,0035,0039,003c,003d,0067,006b,009c,009d,009e,
150150
* 009f,00ff,1301,1302,1303,c009,c00a,c013,c014,c023,c024,c027,c028,
151151
* c02b,c02c,c02f,c030,cca8,cca9,ccaa_000a,000b,000d,0016,0017,0023,

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.txt renamed to compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.txt

Lines changed: 32 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
JA4 looks at the TLS Client Hello packet and builds a fingerprint of the client based on attributes within the packet.
66

77
### JA4 Algorithm:
8-
(QUIC=”q” or TCP=”t”)
8+
(QUIC=”q”, DTLS="d", or Normal TLS=”t”)
99
(2 character TLS version)
1010
(SNI=”d” or no SNI=”i”)
1111
(2 character count of ciphers)
@@ -22,22 +22,29 @@ t13d1516h2_8daaf6152771_b186095e22b6
2222
## Details:
2323
The program needs to ignore GREASE values anywhere it sees them: (https://datatracker.ietf.org/doc/html/draft-davidben-tls-grease-01#page-5)
2424

25-
### QUIC:
25+
### QUIC and DTLS:
26+
“q”, "d" or “t”, denotes whether the hello packet is for QUIC, DTLS, or normal TLS.
27+
2628
https://en.wikipedia.org/wiki/QUIC
27-
“q” or “t”, which denotes whether the hello packet is for QUIC or TCP. QUIC is the protocol which the new HTTP/3 standard utilizes, encapsulating TLS 1.3 into UDP packets. As QUIC was developed by Google, if an organization heavily utilizes Google products, QUIC could make up half of their network traffic, so this is important to capture.
29+
QUIC is the protocol which the new HTTP/3 standard utilizes, encapsulating TLS 1.3 into UDP packets. As QUIC was developed by Google, if an organization heavily utilizes Google products, QUIC could make up half of their network traffic, so this is important to capture.
30+
31+
https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security
32+
DTLS is a version of TLS that can operate over UDP or SCTP.
2833

29-
If the protocol is QUIC then the first character of the fingerprint is “q” if not, it’s “t”.
34+
If the protocol is QUIC then the first character of the fingerprint is “q”, if DTLS it is "d", else it is “t”.
3035

31-
### TLS Version:
32-
TLS version is shown in 3 different places. If extension 0x002b exists (supported_versions), then the version is the highest value in the extension. Remember to ignore GREASE values. If the extension doesn’t exist, then the TLS version is the value of the Protocol Version. Handshake version (located at the top of the packet) should be ignored.
36+
### TLS and DTLS Version:
37+
The TLS version is shown in 3 different places. If extension 0x002b exists (supported_versions), then the version is the highest value in the extension. Remember to ignore GREASE values. If the extension doesn’t exist, then the TLS version is the value of the Protocol Version. Handshake version (located at the top of the packet) should be ignored.
3338

3439
0x0304 = TLS 1.3 = “13”
3540
0x0303 = TLS 1.2 = “12”
3641
0x0302 = TLS 1.1 = “11”
3742
0x0301 = TLS 1.0 = “10”
3843
0x0300 = SSL 3.0 = “s3”
39-
0x0200 = SSL 2.0 = “s2”
40-
0x0100 = SSL 1.0 = “s1”
44+
0x0002 = SSL 2.0 = “s2”
45+
0xfeff = DTLS 1.0 = "d1"
46+
0xfefd = DTLS 1.2 = "d2"
47+
0xfefc = DTLS 1.3 = "d3"
4148

4249
Unknown = “00”
4350

@@ -51,16 +58,21 @@ If the SNI extension (0x0000) exists, then the destination of the connection is
5158
Same as counting ciphers. Ignore GREASE. Include SNI and ALPN.
5259

5360
### ALPN Extension Value:
54-
The first and last characters of the ALPN (Application-Layer Protocol Negotiation) first value.
61+
The first and last alphanumeric characters of the ALPN (Application-Layer Protocol Negotiation) first value.
5562
List of possible ALPN Values (scroll down): https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml
5663

57-
58-
59-
In the above example, the first ALPN value is h2 so the first and last characters to use in the fingerprint are “h2”. IF the first ALPN listed was http/1.1 then the first and last characters to use in the fingerprint would be “h1”.
64+
In the above example, the first ALPN value is h2 so the first and last characters to use in the fingerprint are “h2”. If the first ALPN listed was http/1.1 then the first and last characters to use in the fingerprint would be “h1”.
6065

6166
In Wireshark this field is located under tls.handshake.extensions_alpn_str
6267

63-
If there are no ALPN values or no ALPN extension then we print “00” as the value in the fingerprint.
68+
If there is no ALPN extension, no ALPN values, or the first ALPN value is empty, then we print "00" as the value in the fingerprint. If the first ALPN value is only a single character, then that character is treated as both the first and last character.
69+
70+
If the first or last byte of the first ALPN is non-alphanumeric (meaning not `0x30-0x39`, `0x41-0x5A`, or `0x61-0x7A`), then we print the first and last characters of the hex representation of the first ALPN instead. For example:
71+
* `0xAB` would be printed as "ab"
72+
* `0xAB 0xCD` would be printed as "ad"
73+
* `0x30 0xAB` would be printed as "3b"
74+
* `0x30 0x31 0xAB 0xCD` would be printed as "3d"
75+
* `0x30 0xAB 0xCD 0x31` would be printed as "01"
6476

6577
### Cipher hash:
6678
A 12 character truncated sha256 hash of the list of ciphers sorted in hex order, first 12 characters. The list is created using the 4 character hex values of the ciphers, lower case, comma delimited, ignoring GREASE.
@@ -73,10 +85,13 @@ Is sorted to:
7385
002f,0035,009c,009d,1301,1302,1303,c013,c014,c02b,c02c,c02f,c030,cca8,cca9 = 8daaf6152771
7486
```
7587

88+
If there are no ciphers in the sorted cipher list, then the value of JA4_b is set to `000000000000`
89+
We do this rather than running a sha256 hash of nothing as this makes it clear to the user when a field has no values.
90+
7691
### Extension hash:
7792
A 12 character truncated sha256 hash of the list of extensions, sorted by hex value, followed by the list of signature algorithms, in the order that they appear (not sorted).
7893

79-
The extension list is created using the 4 character hex values of the extensions, lower case, comma delimited, sorted (not in the order they appear). Ignore the SNI extension (0000) and the ALPN extension (0010) as we’ve already captured them in the _a_ section of the fingerprint. These values are omitted so that the same application would have the same _b_ section of the fingerprint regardless of if it were going to a domain, IP, or changing ALPNs.
94+
The extension list is created using the 4 character hex values of the extensions, lower case, comma delimited, sorted (not in the order they appear). Ignore the SNI extension (0000) and the ALPN extension (0010) as we’ve already captured them in the _a_ section of the fingerprint. These values are omitted so that the same application would have the same _c_ section of the fingerprint regardless of if it were going to a domain, IP, or changing ALPNs.
8095

8196
For example:
8297
```
@@ -112,6 +127,9 @@ For example:
112127
0005,000a,000b,000d,0012,0015,0017,001b,0023,002b,002d,0033,4469,ff01 = 6d807ffa2a79
113128
```
114129

130+
If there are no extensions in the sorted extensions list, then the value of JA4_c is set to `000000000000`
131+
We do this rather than running a sha256 hash of nothing as this makes it clear to the user when a field has no values.
132+
115133
### Example
116134

117135
JA4 fingerprint:
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#alpn-extension-value"
2+
3+
# ### ALPN Extension Value:
4+
#
5+
# The first and last alphanumeric characters of the ALPN (Application-Layer Protocol Negotiation) first value.
6+
# List of possible ALPN Values (scroll down): https://www.iana.org/assignments/tls-extensiontype-values/tls-extensiontype-values.xhtml
7+
#
8+
# In the above example, the first ALPN value is h2 so the first and last characters to use in the fingerprint are “h2”. If the first ALPN listed was http/1.1 then the first and last characters to use in the fingerprint would be “h1”.
9+
#
10+
# In Wireshark this field is located under tls.handshake.extensions_alpn_str
11+
#
12+
# If there is no ALPN extension, no ALPN values, or the first ALPN value is empty, then we print "00" as the value in the fingerprint. If the first ALPN value is only a single character, then that character is treated as both the first and last character.
13+
#
14+
# If the first or last byte of the first ALPN is non-alphanumeric (meaning not `0x30-0x39`, `0x41-0x5A`, or `0x61-0x7A`), then we print the first and last characters of the hex representation of the first ALPN instead. For example:
15+
# * `0xAB` would be printed as "ab"
16+
# * `0xAB 0xCD` would be printed as "ad"
17+
# * `0x30 0xAB` would be printed as "3b"
18+
# * `0x30 0x31 0xAB 0xCD` would be printed as "3d"
19+
# * `0x30 0xAB 0xCD 0x31` would be printed as "01"
20+
#
21+
22+
[[spec]]
23+
level = "MUST"
24+
quote = '''
25+
The first and last alphanumeric characters of the ALPN (Application-Layer Protocol Negotiation) first value.
26+
'''
27+
28+
[[spec]]
29+
level = "MUST"
30+
quote = '''
31+
If there is no ALPN extension, no ALPN values, or the first ALPN value is empty, then we print "00" as the value in the fingerprint.
32+
'''
33+
34+
[[spec]]
35+
level = "MUST"
36+
quote = '''
37+
If the first ALPN value is only a single character, then that character is treated as both the first and last character.
38+
'''
39+
40+
[[spec]]
41+
level = "MUST"
42+
quote = '''
43+
If the first or last byte of the first ALPN is non-alphanumeric (meaning not `0x30-0x39`, `0x41-0x5A`, or `0x61-0x7A`), then we print the first and last characters of the hex representation of the first ALPN instead.
44+
'''
Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#cipher-hash"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#cipher-hash"
22

33
# ### Cipher hash:
4-
#
54
# A 12 character truncated sha256 hash of the list of ciphers sorted in hex order, first 12 characters. The list is created using the 4 character hex values of the ciphers, lower case, comma delimited, ignoring GREASE.
65
# Example:
76
# ```
@@ -12,6 +11,9 @@ target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_deta
1211
# 002f,0035,009c,009d,1301,1302,1303,c013,c014,c02b,c02c,c02f,c030,cca8,cca9 = 8daaf6152771
1312
# ```
1413
#
14+
# If there are no ciphers in the sorted cipher list, then the value of JA4_b is set to `000000000000`
15+
# We do this rather than running a sha256 hash of nothing as this makes it clear to the user when a field has no values.
16+
#
1517

1618
[[spec]]
1719
level = "MUST"
@@ -25,3 +27,8 @@ quote = '''
2527
The list is created using the 4 character hex values of the ciphers, lower case, comma delimited, ignoring GREASE.
2628
'''
2729

30+
[[spec]]
31+
level = "MUST"
32+
quote = '''
33+
If there are no ciphers in the sorted cipher list, then the value of JA4_b is set to `000000000000`
34+
'''

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/details.toml renamed to compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4/details.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#details"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#details"
22

33
# ## Details:
44
#

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/example.toml renamed to compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4/example.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#example"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#example"
22

33
# ### Example
44
#
Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#extension-hash"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#extension-hash"
22

33
# ### Extension hash:
44
#
55
# A 12 character truncated sha256 hash of the list of extensions, sorted by hex value, followed by the list of signature algorithms, in the order that they appear (not sorted).
66
#
7-
# The extension list is created using the 4 character hex values of the extensions, lower case, comma delimited, sorted (not in the order they appear). Ignore the SNI extension (0000) and the ALPN extension (0010) as we’ve already captured them in the _a_ section of the fingerprint. These values are omitted so that the same application would have the same _b_ section of the fingerprint regardless of if it were going to a domain, IP, or changing ALPNs.
7+
# The extension list is created using the 4 character hex values of the extensions, lower case, comma delimited, sorted (not in the order they appear). Ignore the SNI extension (0000) and the ALPN extension (0010) as we’ve already captured them in the _a_ section of the fingerprint. These values are omitted so that the same application would have the same _c_ section of the fingerprint regardless of if it were going to a domain, IP, or changing ALPNs.
88
#
99
# For example:
1010
# ```
@@ -40,6 +40,9 @@ target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_deta
4040
# 0005,000a,000b,000d,0012,0015,0017,001b,0023,002b,002d,0033,4469,ff01 = 6d807ffa2a79
4141
# ```
4242
#
43+
# If there are no extensions in the sorted extensions list, then the value of JA4_c is set to `000000000000`
44+
# We do this rather than running a sha256 hash of nothing as this makes it clear to the user when a field has no values.
45+
#
4346

4447
[[spec]]
4548
level = "MUST"
@@ -70,3 +73,9 @@ level = "MUST"
7073
quote = '''
7174
If there are no signature algorithms in the hello packet, then the string ends without an underscore and is hashed.
7275
'''
76+
77+
[[spec]]
78+
level = "MUST"
79+
quote = '''
80+
If there are no extensions in the sorted extensions list, then the value of JA4_c is set to `000000000000`
81+
'''
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#ja4-algorithm"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#ja4-algorithm"
22

33
# ### JA4 Algorithm:
44
#
5-
# (QUIC=”q” or TCP=”t”)
5+
# (QUIC=”q”, DTLS="d", or Normal TLS=”t”)
66
# (2 character TLS version)
77
# (SNI=”d” or no SNI=”i”)
88
# (2 character count of ciphers)
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#ja4-tls-client-fingerprinting"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#ja4-tls-client-fingerprinting"
22

33
# # JA4: TLS Client Fingerprinting
44
#
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#number-of-ciphers"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#number-of-ciphers"
22

33
# ### Number of Ciphers:
44
#
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#number-of-extensions"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#number-of-extensions"
22

33
# ### Number of Extensions:
44
#
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#quic-and-dtls"
2+
3+
# ### QUIC and DTLS:
4+
# “q”, "d" or “t”, denotes whether the hello packet is for QUIC, DTLS, or normal TLS.
5+
#
6+
# https://en.wikipedia.org/wiki/QUIC
7+
# QUIC is the protocol which the new HTTP/3 standard utilizes, encapsulating TLS 1.3 into UDP packets. As QUIC was developed by Google, if an organization heavily utilizes Google products, QUIC could make up half of their network traffic, so this is important to capture.
8+
#
9+
# https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security
10+
# DTLS is a version of TLS that can operate over UDP or SCTP.
11+
#
12+
# If the protocol is QUIC then the first character of the fingerprint is “q”, if DTLS it is "d", else it is “t”.
13+
#
14+
15+
[[spec]]
16+
level = "MUST"
17+
quote = '''
18+
If the protocol is QUIC then the first character of the fingerprint is “q”, if DTLS it is "d", else it is “t”.
19+
'''
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#raw-output"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#raw-output"
22

33
# ### Raw Output
44
#

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/sni.toml renamed to compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4/sni.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#sni"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#sni"
22

33
# ### SNI:
44
#

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/tls-version.toml renamed to compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4/tls-and-dtls-version.toml

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
1-
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4.md#tls-version"
1+
target = "https://raw.githubusercontent.com/FoxIO-LLC/ja4/df3c067/technical_details/JA4.md#tls-and-dtls-version"
22

3-
# ### TLS Version:
4-
#
5-
# TLS version is shown in 3 different places. If extension 0x002b exists (supported_versions), then the version is the highest value in the extension. Remember to ignore GREASE values. If the extension doesn’t exist, then the TLS version is the value of the Protocol Version. Handshake version (located at the top of the packet) should be ignored.
3+
# ### TLS and DTLS Version:
4+
# The TLS version is shown in 3 different places. If extension 0x002b exists (supported_versions), then the version is the highest value in the extension. Remember to ignore GREASE values. If the extension doesn’t exist, then the TLS version is the value of the Protocol Version. Handshake version (located at the top of the packet) should be ignored.
65
#
76
# 0x0304 = TLS 1.3 = “13”
87
# 0x0303 = TLS 1.2 = “12”
98
# 0x0302 = TLS 1.1 = “11”
109
# 0x0301 = TLS 1.0 = “10”
1110
# 0x0300 = SSL 3.0 = “s3”
12-
# 0x0200 = SSL 2.0 = “s2”
13-
# 0x0100 = SSL 1.0 = “s1”
11+
# 0x0002 = SSL 2.0 = “s2”
12+
# 0xfeff = DTLS 1.0 = "d1"
13+
# 0xfefd = DTLS 1.2 = "d2"
14+
# 0xfefc = DTLS 1.3 = "d3"
1415
#
1516
# Unknown = “00”
1617
#

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/alpn-extension-value.toml

Lines changed: 0 additions & 26 deletions
This file was deleted.

compliance/specs/raw.githubusercontent.com/FoxIO-LLC/ja4/v0.18.2/technical_details/JA4/quic.toml

Lines changed: 0 additions & 15 deletions
This file was deleted.

tests/pcap/data/no_extensions.pcap

3.02 KB
Binary file not shown.

0 commit comments

Comments
 (0)