Akamai and EDNS Client Subnet issues

[trmdi]

Issue Details

$ CLASS=CH RRTYPE="TXT" dnslookup id.server tls://dns.adguard.com                                      
dnslookup v1.9.1                                        
dnslookup result (elapsed 276.594529ms):
;; opcode: QUERY, status: NXDOMAIN, id: 60293           
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, 
AUTHORITY: 1, ADDITIONAL: 6                                            
;; QUESTION SECTION:                                    
;id.server.     CH       TXT                            ;; 
AUTHORITY SECTION:
.       1237    IN      SOA     a.root-servers.net. 
nstld.verisign-grs.com. 2024012202 1800 900 
604800 86400    
;; ADDITIONAL SECTION:
client-ip.adguard-dns.com.      3600    CH      TXT    "14.226.99.x"                                          
server-ip.adguard-dns.com.      3600    CH      TXT    "94.140.14.14"
country.adguard-dns.com.        3600    CH      TXT    "VN"
asn.adguard-dns.com.    3600    CH      TXT     "45899"
subdivision.adguard-dns.com.    3600    CH      TXT    "HP"                                                     resp.res- 
type.adguard-dns.com.  3600    CH      TXT    "normal"

Expected Behavior

When you are in the North of Vietnam, your ECS should be set to ones in the North, because the client should connect to CDNs in Hanoi/HCM/Hongkong for shorter distances and latency, instead of Singapore.

Actual Behavior

$ RRTYPE=TXT dnslookup o-o.myaddr.l.google.com tls://dns.adguard.com
dnslookup v1.9.1                                        
dnslookup result (elapsed 285.909138ms):
;; opcode: QUERY, status: NOERROR, id: 18760            
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:                                    ;o- 
o.myaddr.l.google.com.       IN       TXT

;; ANSWER SECTION:
o-o.myaddr.l.google.com.        60      IN      TXT    "edns0-client-subnet 222.255.238.0/24"
o-o.myaddr.l.google.com.        60      IN      TXT    "138.199.46.250"

$ ipinfo.sh 222.255.238.0           [11:14:28]
{
"ip": "222.255.238.0",
"hostname": "static.vnpt.vn",
"city": "Ho Chi Minh City",
"region": "Ho Chi Minh",
"country": "VN",
"loc": "10.8230,106.6296",
"org": "AS45899 VNPT Corp",
"postal": "71606",
"timezone": "Asia/Ho_Chi_Minh"
}

[trmdi]

From a screenshot in the video below, I see subdivision is applied to CN, IN... but not to VN. Is it right?
#659 (comment)

[trmdi]

Adguard DNS:

$ dig starling-oversea.byteoversea.com @dns.adguard.com +short
starling-oversea.byteoversea.com.ttdns2.com.
starling-oversea.byteoversea.com.edgekey.net. 
e11942.a.akamaiedge.net.
23.215.7.32                                             23.215.7.28
23.215.7.19                                             23.215.7.16
23.215.7.22                                             23.215.7.23
23.215.7.20                                             23.215.7.25
23.215.7.18                                             

$ ping -c3 23.215.7.15             [11:49:13]
PING 23.215.7.15 (23.215.7.15) 56(84) bytes of data.
64 bytes from 23.215.7.15: icmp_seq=1 ttl=57 time=47.5 ms
64 bytes from 23.215.7.15: icmp_seq=2 ttl=57 time=46.1 ms
64 bytes from 23.215.7.15: icmp_seq=3 ttl=57 time=48.9 ms

--- 23.215.7.15 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 46.087/47.489/48.892/1.145 ms

Google DNS:

$ dig starling-oversea.byteoversea.com @dns.google +short
starling-oversea.byteoversea.com.ttdns2.com.
starling-oversea.byteoversea.com.edgekey.net.
e11942.a.akamaiedge.net.
113.171.10.152                                       113.171.10.144
113.171.10.162                                       113.171.10.160
113.171.10.155                                       113.171.10.154
113.171.10.146                                       113.171.10.139
113.171.10.145

$ ping -c3 113.171.10.155           [11:49:33]
PING 113.171.10.155 (113.171.10.155) 56(84) bytes of data.
64 bytes from 113.171.10.155: icmp_seq=1 ttl=58 time=6.05 ms
64 bytes from 113.171.10.155: icmp_seq=2 ttl=58 time=4.58 ms
64 bytes from 113.171.10.155: icmp_seq=3 ttl=58 time=3.22 ms

--- 113.171.10.155 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.220/4.618/6.052/1.156 ms

[ameshkov]

Right, the subdivision logic at this moment is enabled for 4 countries only: US, Russia, China and India.

Interesting data, thank you! We'll experiment with it a bit, I am not yet sure that it should be solved. We should first figure out why Akamai returns Singapore addresses for this subnet, it's unclear from the BGP information what's so different with this prefix compared to other prefixes announced by AS45899.

[trmdi]

We should first figure out why Akamai returns Singapore addresses for this subnet

Because that ECS belongs to HCM which is near Singapore? I guess a ECS in HN should be better. Since my subnet is in Hanoi (the north of Vietnam)

[ameshkov]

@trmdi HCM?

As I see both subnets (yours and the one that AG DNS uses for ECS) belongs to the same AS45899.

• ECS - 222.255.238.0/24
• Your subnet - 14.226.99.0/24

[ameshkov]

Ah okay I got it, HCM = Ho Chi Minh.

Interesting idea indeed.

[trmdi]

Sorry, HCM means Ho Chi Minh city. (In the south of Vietnam)

[ameshkov]

Maybe you're right, we should check how Akamai nameserver responses change depending on what location the subnet belongs to.

Internal task ID: AGDNS-1881

[ameshkov]

Just got one more idea. Vietnam has quite a lot of different subdivisions in MaxMind (it uses ISO 3166-2). According to the ISO 3166-2 standard, Vietnam is divided into 58 provinces and 5 municipalities, but for network routing it is probably not that important and it'd be enough to do a "north-south" split.

@trmdi does that sound reasonable to you? If so, how would you split the subdivisions between north/south?

I think we could probably do the same for large countries as well, split subdivisions between west/east/north/south/central in order to improve cache efficiency.

[trmdi]

Yes, I think splitting between west/east/north/south/central is fine.
Specifically, Vietnam area is S-shaped, narrow and stretched from the north to the south, so I think splitting to north/south (the upper 50% area is the north) is enough for network routing.

[ameshkov]

Okay it's not that easy.

Something's strange with Akamai. They indicate in responses that they don't rely on ECS:

dig -t a starling-oversea.byteoversea.com. @8.8.8.8 +subnet=14.226.99.0/24

...
CLIENT-SUBNET: 14.226.99.0/24/0
...

Now one more strange thing.

There's another domain that I tested with: store.akamai.steamstatic.com.

For some reason via Google DNS Akamai responds very accurately with Cyprus IP addresses.

dig -t a store.akamai.steamstatic.com. @8.8.8.8

...
;; ANSWER SECTION:
store.akamai.steamstatic.com. 20 IN	A	195.14.151.138
store.akamai.steamstatic.com. 20 IN	A	195.14.151.155
...

What's strange is that when I pass a custom subnet via dig to Google DNS, it continues to respond with accurate Cyprus IP addresses.

dig -t a store.akamai.steamstatic.com. @8.8.8.8 +subnet=14.226.99.0/24

...
;; ANSWER SECTION:
store.akamai.steamstatic.com. 20 IN	A	195.14.151.155
store.akamai.steamstatic.com. 20 IN	A	195.14.151.138
...

I honestly do not understand what's going on there and how on earth could Akamai learn about my location considering that I am being routed to Google DNS servers located in Switzerland (and Google DNS does not have any servers in Cyprus anyways).

Checking what Google DNS server is being used and what ECS it sends to authoritative nameservers:

dig -t txt o-o.myaddr.l.google.com. @8.8.8.8 +subnet=14.226.99.0/24

...
;; ANSWER SECTION:
o-o.myaddr.l.google.com. 60	IN	TXT	"172.253.237.133"
o-o.myaddr.l.google.com. 60	IN	TXT	"edns0-client-subnet 14.226.99.0/24"
...

[ameshkov]

Here's what I suspect:

1. In the case of Akamai nameservers, Google DNS sends my real subnet in ECS even if I specify the subnet in the query.
2. Akamai only accepts ECS from a select list of DNS servers. They accept it from Google DNS and OpenDNS, but not from us or Quad9.