In a CDN’d world, OpenDNS is the enemy!

May 18th, 2010 | by Sajal Kayan |

While many people are happy with using DNS service providers such as OpenDNS, Google, etc… I will show you here why they may not produce optimal results.

The way most CDNs work is by using DNS routing. When a user attempts to resolve a hostname, the CDN’s DNS server responds with an IP which is closest based on the IP address of the requester. A more detailed insight into the workings of a CDN can be found on an earlier post “Make your own cheap charlie CDN”

For my test here, I tested from the following locations listed below :-

  1. True - Thailand : My personal internet connection provided by the ISP called True Internet.
  2. Softlayer - United States : A server hosted at Softlayer’s Washington DC Datacenter.
  3. EC2 - United States : An EC2 instance at Amazons us-east-1c availability zone.
  4. EC2-EU - Ireland : An EC2 instance at Amazons eu-west-1 availability zone. - Thanks Luke
  5. EC2-APAC - Singapore : An EC2 instance at Amazons ap-southeast-1a availability zone.
  6. Com Hem - Sweden : An ISP in sweden. - Thanks Adam
  7. Tata - India : An ISP in India. - Thanks Angsuman

The following DNS servers were used to resolve the domains :-

  1. OpenDNS (208.67.222.222 , 208.67.220.220 )- Has different caches in multiple locations(Anycasted) - Chicago, Illinois, USA; Dallas, Texas, USA; Los Angeles, California, USA; Miami, Florida, USA; New York, New York, USA; Palo Alto, California, USA; Seattle, Washington, USA; Washington, DC, USA; Amsterdam, The Netherlands and London, England, UK
  2. Google Public DNS (8.8.8.8 , 8.8.4.4 ) - “Google Public DNS servers are available worldwide” . I think Google has their DNS servers in all countries where they have hosting infrastructure.
  3. Local DNS - The ISP provided DNS in the different locations.

The test was done to the following CDN providers :-

  1. Internap ( cdn.thaindian.com ) - Uses DNS routing. POPs (Point Of Presence) in the following locations : Atlanta; Boston; Chicago; Dallas; Denver; El Segundo; Houston; Miami; New York; Philadelphia; Phoenix; San Jose; Seattle; Washington, DC; Sydney; Tokyo; Singapore; Hong Kong; Amsterdam; London
  2. Akamai ( profile.ak.fbcdn.net ) - AFAIK they have a POP in almost all countries including Thailand. Note: Akamai does not entertain sales queries from Thai companies.

Results:-

1) Internap ( using cdn.thaindian.com )

Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 64.94.126.65 256 74.201.0.130 365 203.190.126.131 152
Softlayer (US-East Coast) 69.88.152.250 1.253 74.201.0.130 25.69 69.88.152.250 1.388
EC2 (US-East Coast) 69.88.152.250 2.144 74.201.0.130 20.229 69.88.152.250 2.094
EC2 (Europe) 77.242.194.130 13.331 64.7.222.130 159.422 77.242.194.130 12.504
EC2 (Singapore) 64.94.126.65 202 74.201.0.130 228 202.58.12.98 37.260
Com Hem (Sweden) 77.242.194.130 40.035 64.7.222.130 189.647 69.88.148.130 36.310
Tata (India) 64.7.222.130 313.2 64.74.124.65 304.1 203.190.126.131 150

2) Akamai ( using profile.ak.fbcdn.net )

Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 208.50.77.112 239.4 60.254.185.83 138.9 58.97.45.59 18.88
Softlayer (US-East Coast) 72.246.31.57 1.312 72.246.31.42 1.262 24.143.196.88 0.877
EC2 (US-East Coast) 72.246.31.73 2.581 72.246.31.25 1.792 72.247.242.51 1.941
EC2 (Europe) 195.59.150.139 13.449 92.122.207.177 29.022 195.59.150.138 13.516
EC2 (Singapore) 208.50.77.94 202 60.254.185.73 71.7 124.155.222.10 7.052
Com Hem (Sweden) 217.243.192.8 51.73 92.123.69.82 35.972 92.123.155.139 13.212
Tata (India) 209.18.46.113 300 203.106.85.33 196 125.252.226.58 100.5

The ping timings represent the lag to the destination server from the location in question. I will try to update the results from more locations if I can get shell access to a server or PC in other countries. If you are willing to run the tests for me please contact me(or post in comments)

Conclusion

Using OpenDNS or Google Public DNS may be fast in resolving the DNS, but they do not give the ideal results.

In the case of Global DNS providers, the IP of the original requester is not passed along to the CDN’s DNS servers so they are unable to route the user to the nearest POP.

As you can see in the result tables above, when using OpenDNS from Thailand, trying to access static assets of Facebook, I am directed to a server in the USA whereas when using Google’s DNS i am directed to a server in Japan and when using my ISP’s DNS server I access content locally, hosted within my own ISPs network!

While the effect on large websites using CDN is significant, smaller non-CDN’d websites are also effected. Most websites embed widgets, advertising and other assets which are likely to be CDN’d.

The solution would be to use your ISPs DNS server rather than these Global providers. If they really suck so bad, its fairly simple to set up BIND as a caching recursive resolver to resolve hostnames directly bypassing the ISPs crappy service.

Bill Fumerola, ex-director of network engineering at OpenDNS confirms this problem on OpenDNS forums.

You can run the tests from your own computer using this simple script: dnstest.py

Here is the named.conf for a recursive server. Set your computer to use 127.0.0.1 as the DNS. - config may differ for you, RTFM and adapt accordingly.

options {
        directory "/var/named";
        listen-on {
		127.0.0.1;
        };
        auth-nxdomain yes;
        allow-recursion {
                127.0.0.1;
        };
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

};

//
// a caching only nameserver config
//
zone "." {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";

include "/etc/named.dnssec.keys";
include "/etc/pki/dnssec-keys/dlv/dlv.isc.org.conf";

EDIT 1: Inverted the axis added test data from Europe
EDIT 2: Added test data from Singapore
EDIT 3: Added test data from Sweden
EDIT 4: Added test data from India
EDIT 5: Added link to Bill Fumerola’s explanation of the problem.

  1. 24 Responses to “In a CDN’d world, OpenDNS is the enemy!”

  2. By owen on May 29, 2010 | Reply

    Nice work. It would seem to me that the conclusion should be “use a CDN blocker.”

  3. By Jason on May 29, 2010 | Reply

    You’re using ping as a representation of whether or not the route is optimal. I disagree, however. Ping, while an excellent troubleshooting tool, does not always represent a true latency to a remote service. It could very well be that ping has been given lower priority than other traffic as is often the case with routers.

    A better tool, or perhaps one that may provide additional insight in addition to ping, would be a traceroute to the destination host. Determining the number of hops along the path, as well as the latency to each hop (again using ping), would provide additional data points to prove your hypothesis. For instance, you may find that ping times between you and the router prior to your destination are very low while the destination is much higher. This may indicate a priority issue with ICMP traffic.

    You can also use a simple script to time the connection to the remote server and download the main html page. This introduces additional complexity as a portion of the response time will ultimately be the server response time as opposed to the network, but this may lead to additional insights into the failure of CDN in general. That is, while CDN theoretically means you get a connection to the closest server, the closest server may not be able to fulfill your request in the quickest time.

  4. By Tom Limoncelli on May 30, 2010 | Reply

    If you are going to post something like this, you should also point out that Google has proposed a solution to this problem:

    http://tech.slashdot.org/story/10/01/28/183208/Google-Proposes-DNS-Extension

    How do your tests hold up with this DNS extension implemented?

  5. By Remco on May 30, 2010 | Reply

    From The Netherlands, using online.nl as ISP:

    ====== testing : cdn.thaindian.com Internap ===========
    testing : OpenDNS ( 208.67.222.222 )
    64.7.222.130
    PING 64.7.222.130 (64.7.222.130) 56(84) bytes of data.

    — 64.7.222.130 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 188.971/189.555/190.654/0.822 ms
    ———————————-
    testing : Google ( 8.8.8.8 )
    74.201.26.2
    PING 74.201.26.2 (74.201.26.2) 56(84) bytes of data.

    — 74.201.26.2 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 190.621/191.457/192.995/0.982 ms
    ———————————-
    testing : default ( 192.168.1.1 )
    69.88.148.130
    PING 69.88.148.130 (69.88.148.130) 56(84) bytes of data.

    — 69.88.148.130 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4005ms
    rtt min/avg/max/mdev = 35.600/36.230/37.232/0.674 ms
    ———————————-
    =====================================
    ====== testing : profile.ak.fbcdn.net Akamai ===========
    testing : OpenDNS ( 208.67.222.222 )
    80.239.205.72
    PING 80.239.205.72 (80.239.205.72) 56(84) bytes of data.

    — 80.239.205.72 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 46.428/48.487/53.945/2.811 ms
    ———————————-
    testing : Google ( 8.8.8.8 )
    92.122.217.58
    PING 92.122.217.58 (92.122.217.58) 56(84) bytes of data.

    — 92.122.217.58 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 34.676/35.493/37.218/0.999 ms
    ———————————-
    testing : default ( 192.168.1.1 )
    92.122.217.122
    PING 92.122.217.122 (92.122.217.122) 56(84) bytes of data.
    ====== testing : cdn.twitbooth.com Amazon Cloudfront ===========
    testing : OpenDNS ( 208.67.222.222 )
    216.137.59.136
    PING 216.137.59.136 (216.137.59.136) 56(84) bytes of data.

    — 216.137.59.136 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4006ms
    rtt min/avg/max/mdev = 36.253/37.372/38.707/0.998 ms
    ———————————-
    testing : Google ( 8.8.8.8 )
    216.137.59.239
    PING 216.137.59.239 (216.137.59.239) 56(84) bytes of data.

    — 216.137.59.239 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4005ms
    rtt min/avg/max/mdev = 36.147/37.854/39.869/1.436 ms
    ———————————-
    testing : default ( 192.168.1.1 )
    216.137.59.130
    PING 216.137.59.130 (216.137.59.130) 56(84) bytes of data.

    — 216.137.59.130 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 36.497/37.293/38.062/0.553 ms
    ———————————-
    =====================================

    — 92.122.217.122 ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4007ms
    rtt min/avg/max/mdev = 33.835/35.762/37.770/1.374 ms
    ———————————-
    =====================================

  6. By Vincent on May 30, 2010 | Reply

    Thanks for the article. I’m using ActivePython on Windows: How can I install the required “dnspython” package?

  7. By Michael on May 30, 2010 | Reply

    How about adding Canadian data?

  8. By Russell on May 30, 2010 | Reply

    Interesting analysis - I wouldn’t have expected such a dramatic change in performance!

    I was curious how the data would look when charted, so I plopped the Akamai table into a chart. I’m not sure if it will show up correctly as a comment here but it makes the difference abundantly clear:

    I can make another for Internap if you like. It appears we here in the US luck out again…

  9. By Olivier Mignault on May 30, 2010 | Reply

    It’s ridiculous to favor your local DNS server over OpenDNS or Google when they returned the same address (or addresses in the same /24 block - probably servers sitting in the same rack!!)
    At that point, you’re just skewing your data to illustrate your point. Those results should be considered equal.
    The local DNS would still win overall.

  10. By Nick on May 30, 2010 | Reply

    I brought this to Google’s attention just after it released their DNS solution:

    http://www.noodles.net.nz/2009/12/04/google-dns-and-opendns-vs-cdns/

    At the time it was being looked it, but don’t know if anything ever came of it.

  11. By Jeff on May 30, 2010 | Reply

    Use this nice utility for Windows to benchmark DNS servers

    http://www.grc.com/dns/benchmark.htm

    .

  12. By Marco van den Berg on May 30, 2010 | Reply

    We noted the phenomenon described above as well. We’re using the enterprise version of OpenDNS mainly to block ‘certain’ categories of web traffic (including malware and botnet traffic), so changing DNS servers would have meant getting a new filtering product as well. Turns out that it’s fairly easy to change the DNS forwarders for the most popular CDN destinations, especially if you use a central in-house DNS resolver. Just exclude the CDN domains and change their resolution to a more optimal server. In our case, pointing l.google.com to forward to Google’s DNS servers at 8.8.8.8/8.8.4.4 caused significant performance improvements. Some of the main CDN’s that can be optimized are akamai.net / akadns.org / akadns.net / akamaiedge.net / akamaitech.net / edgesuite.net / edgekey.net / nsatc.net / llnwd.net / l.google.com / b.yahoo.com / yahoodns.com

    If you want to optimize these I suggest you run a couple of tests to find the best forwarder.

    Cheers
    Marco van den Berg
    International Rice Research Institute

  13. By Richard Hartmann on May 30, 2010 | Reply

    141.1.1.1 is another publically-accessible, geo-routed DNS resolver. In case you run more tests, you should add that, as well.

  14. By Richard on May 30, 2010 | Reply

    We at GeoEdge.com are willing to help you perform world wide tests, we have servers in more than 55 countries world wide.

  15. By Angsuman Chakraborty on May 30, 2010 | Reply

    A better solution for SMB and tech enthusiasts is to host your own local caching DNS server for optimal performance as well as help CDN’s with location awareness.

  16. By Chris on May 30, 2010 | Reply

    If you live in Thailand, this may be true. But if you live in the US I doubt that it will make much difference in most cases. There are also other benefits to using OpenDNS or GoogleDNS - I have used several ISPs and their DNS always goes down for a few minutes to hours a year, whereas a computer set to OpenDNS has never had problems (this may just be fortitude, but oh well.)

    While the issue identified is valid and has implications for bandwidth, I don’t think that bandwidth is the bottleneck. It is more likely to be your browser. You say “Most websites embed widgets, advertising and other assets which are likely to be CDN’d.” I block advertising, so it doesn’t matter. I tend to ignore widgets. In any case, the substance of the page loads first so that I can begin reading even if the CDN-stuff hasn’t loaded yet. I can even stop them loading if I don’t want to see your widgets.

  17. By Sajal Kayan on May 30, 2010 | Reply

    @owen : Nice point if you dont visit large websites, don’t mind having no CSS, Javascripts, Images on most sites.

    @Jason : Agree PING is not the optimal test, but it gives a fairly accurate indication of distance. When I have time I will set up a dummy file on both CDNs tested and compare time to download those files. Any objections to that?

    @Tom Limoncelli: Didn’t know about that extension. Once/if that is implemented, my blogpost would become irrelevant.

    @Remco : Thanks for your test data, ill include it in my tables soon. could you report ping timing to 92.122.217.122 ? it seems to be missing.

    @Vincent : I have no knowledge about running Python on Windows. dnspython can be obtained from http://www.dnspython.org/ . I think you need to also replace “ping -c 5 -q ” with “ping -n 5 ” in the script. The arguments for ping in Windows is different than Linux. Unfortunately I have nowhere to test the script with Windows.

    @Michael : I’d welcome test data from a Canada

    @Olivier Mignault : Thanks for accusing me of skewing the data. Please see the table Internap data for “Softlayer (US-East Coast)” and Akamai data for “EC2 (US-East Coast)” and “EC2 (Europe)” gave better ping times for non local DNS and i have shown it as it is. I ran the script and posted whatever I got. Non Local resolvers gave better results than local only by few milliseconds and the result IPs were in the same range. The better ping times were a mere coincidence. See only the non-datacenter locations (Singapore, Sweden, India and Thailand) to notice the difference. Those are purely home connections.

    @Nick : Cool, in fact after I posted I noticed few others noticed similar thing some as far back in 2008. - http://blog.goolamabbas.org/2008/02/24/impact-of-opendns-on-cdn-services-particularly-when-used-in-asia/

    @Marco van den Berg : All clients need to be in the same network as the central in-house DNS resolver to get the optimal results…

    @Richard : Thanks, you could post the results somewhere and Ill link to it from my tables, or if you don’t want to do that, I can post it on my blog too. Your results would be highly appreciated. you can contact me at sajal at thaindian dot com

    @Angsuman Chakraborty: Agreed!

    @Chris: The bandwidth is a problem in Asia (esp Thailand) with intercontinental links running at its saturation point. I understand that you block advertising (viewing free stuff while wasting the websites bandwidth and resources and not allowing them to monetize), but do you also block css, javascripts, flash, videos and images? These are also usually CDN’d. Without loading the stylesheet usually the page doesn’t start rendering.

  18. By Gary on May 31, 2010 | Reply

    Tom wrote:

    > How do your tests hold up with this DNS extension implemented?

    Uh, you tell us once you get it approved and rolled out to BIND and all the other name servers in use.

    -Gary

  19. By Gerardo Lisboa on Jun 14, 2010 | Reply

    Are the results from http://www.akamai.com/html/technology/products/personalization.html consistent with your problem?

  20. By Sajal Kayan on Jun 14, 2010 | Reply

    @Gerardo : It seems like http://www.akamai.com/html/technology/products/personalization.html does not use DNS for detecting the information under “What We Know About You”. They r simply using the IP of the requester to determine that. Similar to sites such as http://www.ipaddresslocation.org/

  21. By Billy Vierra on Aug 18, 2010 | Reply

    This only true if the CDN is using older technology (such as returning a different IP depending on what DNS server you hit). If they are using Anycast IP’s this actually is not an issue at all.

  22. By Alaor on Aug 25, 2010 | Reply

    By your response, I sincerely suggest you read the article again. You REALLY didn’t understand it.

  1. 3 Trackback(s)

  2. May 30, 2010: Why GoogleDNS (or OpenDNS) isn’t Always Faster - Dave Naffziger’s Blog
  3. Jun 9, 2010: TheInnerLayer — where SL’ers come to rant » Blog Archive » DNS from All Angles
  4. Jul 4, 2010: Sajal Kayan » ATTN Big Media: Web page speed matters!

Post a Comment