-
Notifications
You must be signed in to change notification settings - Fork 7
Description
The Thunderbolt Bridge driver on macOS does not have loop protection. It only functions as a flat L2 fabric and if a ring is naively routed all devices IPs end up on the same subnet resulting in a broadcast storm where MDNS, ARP and other packets are forwarded indefinitely in the loop, growing in size until the entire throughput is filled and all future requests are lost, meaning full loss of connectivity to both LAN and WLAN, DNS, ARP, high CPU utilization and temperature, etc. until one of the TBs is disconnected.
Below is the CPU under normal load but with a fully connected TB ring:
A break can be seen whenever any of the TB cables are disconnected and the ring broken:
On macOS 15.3.1 -- full network crash
On older version the storm escalates and results in having no connection to LAN or WAN. I only captured a smaller tcpdump of the bridge0 interface taken during the early stage of the storm. It shows about 80k packets/s and rising:
|------------------------------|
| Interval | Frames | Bytes |
|------------------------------|
| 0 <> 1 | 76249 | 30006253 |
| 1 <> 2 | 95122 | 31746752 |
| 2 <> 3 | 99798 | 32255080 |
| 3 <> 4 | 99558 | 32177093 |
| 4 <> 5 | 100161 | 32371754 |
Number of packets: 524 k
Capture duration: 6.451817 seconds
Data byte rate: 27 MBps
Average packet size: 335.34 bytes
Average packet rate: 81 kpackets/s
On macOS Beta 26.2
The storm stabilizes around 100k packets/s, probably due to a rate-limit update in the driver, resulting only in connections delays but not a complete loss:
Number of packets: 6,917 k
Capture duration: 69.722017 seconds
Data byte rate: 33 MBps
Average packet size: 338.75 bytes
Average packet rate: 99 kpackets/s
|------------------------------|
| Interval | Frames | Bytes |
|------------------------------|
| 0 <> 1 | 9 | 4493 | <== Baseline
| 1 <> 2 | 1 | 213 |
| 2 <> 3 | 1 | 213 |
| 3 <> 4 | 2 | 2990 |
| 4 <> 5 | 6 | 3230 |
| 5 <> 6 | 0 | 0 |
| 6 <> 7 | 0 | 0 |
| 7 <> 8 | 34060 | 16080754 | <== Last cable connected
| 8 <> 9 | 103319 | 45295802 |
| 9 <> 10 | 115833 | 48190858 |
| 10 <> 11 | 109716 | 44041404 |
| 11 <> 12 | 105955 | 42650075 |
| 12 <> 13 | 107182 | 43143676 |
| 13 <> 14 | 111384 | 44835336 |
| 14 <> 15 | 108720 | 43762165 |
| 15 <> 16 | 108989 | 43871459 |
| 16 <> 17 | 108004 | 43475207 |
| 17 <> 18 | 113107 | 44604846 |
| 18 <> 19 | 104005 | 38541778 |
| 19 <> 20 | 111194 | 40068920 |
Reproduction:
- Connect 3+ Macs with TB4/5 in a loop (A -> B -> C -> A) but keep one cable disconnected
- Allow for automatic IP assignment or manually assign all devices on same subnet (10.10.0.0/24)
- Trace all packets with
tcpdump -i bridge0or donetstat -I bridge0 1for full-interface bytes/s stats - Connect the last TB cable and see traffic increase