In a CDN'd world, OpenDNS is the enemy!

While many people are happy with using DNS service providers such as OpenDNS, Google, etc... I will show you here why they may not produce optimal results. The way most CDNs work is by using DNS routing. When a user attempts to resolve a hostname, the CDN's DNS server responds with an IP which is closest based on the IP address of the requester. A more detailed insight into the workings of a CDN can be found on an earlier post "Make your own cheap charlie CDN" For my test here, I tested from the following locations listed below :-
  1. True - Thailand : My personal internet connection provided by the ISP called True Internet.
  2. Softlayer - United States : A server hosted at Softlayer's Washington DC Datacenter.
  3. EC2 - United States : An EC2 instance at Amazons us-east-1c availability zone.
  4. EC2-EU - Ireland : An EC2 instance at Amazons eu-west-1 availability zone. - Thanks Luke
  5. EC2-APAC - Singapore : An EC2 instance at Amazons ap-southeast-1a availability zone.
  6. Com Hem - Sweden : An ISP in sweden. - Thanks Adam
  7. Tata - India : An ISP in India. - Thanks Angsuman
The following DNS servers were used to resolve the domains :-
  1. OpenDNS (208.67.222.222 , 208.67.220.220 )- Has different caches in multiple locations(Anycasted) - Chicago, Illinois, USA; Dallas, Texas, USA; Los Angeles, California, USA; Miami, Florida, USA; New York, New York, USA; Palo Alto, California, USA; Seattle, Washington, USA; Washington, DC, USA; Amsterdam, The Netherlands and London, England, UK
  2. Google Public DNS (8.8.8.8 , 8.8.4.4 ) - "Google Public DNS servers are available worldwide" . I think Google has their DNS servers in all countries where they have hosting infrastructure.
  3. Local DNS - The ISP provided DNS in the different locations.
The test was done to the following CDN providers :-
  1. Internap ( cdn.thaindian.com ) - Uses DNS routing. POPs (Point Of Presence) in the following locations : Atlanta; Boston; Chicago; Dallas; Denver; El Segundo; Houston; Miami; New York; Philadelphia; Phoenix; San Jose; Seattle; Washington, DC; Sydney; Tokyo; Singapore; Hong Kong; Amsterdam; London
  2. Akamai ( profile.ak.fbcdn.net ) - AFAIK they have a POP in almost all countries including Thailand. Note: Akamai does not entertain sales queries from Thai companies.

Results:-

1) Internap ( using cdn.thaindian.com )
Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 64.94.126.65 256 74.201.0.130 365 203.190.126.131 152
Softlayer (US-East Coast) 69.88.152.250 1.253 74.201.0.130 25.69 69.88.152.250 1.388
EC2 (US-East Coast) 69.88.152.250 2.144 74.201.0.130 20.229 69.88.152.250 2.094
EC2 (Europe) 77.242.194.130 13.331 64.7.222.130 159.422 77.242.194.130 12.504
EC2 (Singapore) 64.94.126.65 202 74.201.0.130 228 202.58.12.98 37.260
Com Hem (Sweden) 77.242.194.130 40.035 64.7.222.130 189.647 69.88.148.130 36.310
Tata (India) 64.7.222.130 313.2 64.74.124.65 304.1 203.190.126.131 150
2) Akamai ( using profile.ak.fbcdn.net )
Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 208.50.77.112 239.4 60.254.185.83 138.9 58.97.45.59 18.88
Softlayer (US-East Coast) 72.246.31.57 1.312 72.246.31.42 1.262 24.143.196.88 0.877
EC2 (US-East Coast) 72.246.31.73 2.581 72.246.31.25 1.792 72.247.242.51 1.941
EC2 (Europe) 195.59.150.139 13.449 92.122.207.177 29.022 195.59.150.138 13.516
EC2 (Singapore) 208.50.77.94 202 60.254.185.73 71.7 124.155.222.10 7.052
Com Hem (Sweden) 217.243.192.8 51.73 92.123.69.82 35.972 92.123.155.139 13.212
Tata (India) 209.18.46.113 300 203.106.85.33 196 125.252.226.58 100.5
The ping timings represent the lag to the destination server from the location in question. I will try to update the results from more locations if I can get shell access to a server or PC in other countries. If you are willing to run the tests for me please contact me(or post in comments)

Conclusion

Using OpenDNS or Google Public DNS may be fast in resolving the DNS, but they do not give the ideal results. In the case of Global DNS providers, the IP of the original requester is not passed along to the CDN's DNS servers so they are unable to route the user to the nearest POP. As you can see in the result tables above, when using OpenDNS from Thailand, trying to access static assets of Facebook, I am directed to a server in the USA whereas when using Google's DNS i am directed to a server in Japan and when using my ISP's DNS server I access content locally, hosted within my own ISPs network! While the effect on large websites using CDN is significant, smaller non-CDN'd websites are also effected. Most websites embed widgets, advertising and other assets which are likely to be CDN'd. The solution would be to use your ISPs DNS server rather than these Global providers. If they really suck so bad, its fairly simple to set up BIND as a caching recursive resolver to resolve hostnames directly bypassing the ISPs crappy service. Bill Fumerola, ex-director of network engineering at OpenDNS confirms this problem on OpenDNS forums. You can run the tests from your own computer using this simple script: dnstest.py Here is the named.conf for a recursive server. Set your computer to use 127.0.0.1 as the DNS. - config may differ for you, RTFM and adapt accordingly.
options {
        directory "/var/named";
        listen-on {
		127.0.0.1;
        };
        auth-nxdomain yes;
        allow-recursion {
                127.0.0.1;
        };
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

};

//
// a caching only nameserver config
//
zone "." {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";

include "/etc/named.dnssec.keys";
include "/etc/pki/dnssec-keys/dlv/dlv.isc.org.conf";
EDIT 1: Inverted the axis added test data from Europe EDIT 2: Added test data from Singapore EDIT 3: Added test data from Sweden EDIT 4: Added test data from India EDIT 5: Added link to Bill Fumerola’s explanation of the problem.

Shameless plug

Turbobytes - Faster content delivery using multiple CDNs
Tags: Akamai benchmark bind CDN dns google OpenDNS pagespeed site performance Softlayer true
Categories: Webmaster Things