In a CDN’d world, OpenDNS is the enemy!

May 18th, 2010 | by Sajal Kayan |

While many people are happy with using DNS service providers such as OpenDNS, Google, etc… I will show you here why they may not produce optimal results.

The way most CDNs work is by using DNS routing. When a user attempts to resolve a hostname, the CDN’s DNS server responds with an IP which is closest based on the IP address of the requester. A more detailed insight into the workings of a CDN can be found on an earlier post “Make your own cheap charlie CDN

For my test here, I tested from the following locations listed below :-

  1. True – Thailand : My personal internet connection provided by the ISP called True Internet.
  2. Softlayer – United States : A server hosted at Softlayer’s Washington DC Datacenter.
  3. EC2 – United States : An EC2 instance at Amazons us-east-1c availability zone.
  4. EC2-EU – Ireland : An EC2 instance at Amazons eu-west-1 availability zone. – Thanks Luke
  5. EC2-APAC – Singapore : An EC2 instance at Amazons ap-southeast-1a availability zone.
  6. Com Hem – Sweden : An ISP in sweden. – Thanks Adam
  7. Tata – India : An ISP in India. – Thanks Angsuman

The following DNS servers were used to resolve the domains :-

  1. OpenDNS ( , )- Has different caches in multiple locations(Anycasted) – Chicago, Illinois, USA; Dallas, Texas, USA; Los Angeles, California, USA; Miami, Florida, USA; New York, New York, USA; Palo Alto, California, USA; Seattle, Washington, USA; Washington, DC, USA; Amsterdam, The Netherlands and London, England, UK
  2. Google Public DNS ( , ) – “Google Public DNS servers are available worldwide” . I think Google has their DNS servers in all countries where they have hosting infrastructure.
  3. Local DNS – The ISP provided DNS in the different locations.

The test was done to the following CDN providers :-

  1. Internap ( ) – Uses DNS routing. POPs (Point Of Presence) in the following locations : Atlanta; Boston; Chicago; Dallas; Denver; El Segundo; Houston; Miami; New York; Philadelphia; Phoenix; San Jose; Seattle; Washington, DC; Sydney; Tokyo; Singapore; Hong Kong; Amsterdam; London
  2. Akamai ( ) – AFAIK they have a POP in almost all countries including Thailand. Note: Akamai does not entertain sales queries from Thai companies.


1) Internap ( using )

Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 256 365 152
Softlayer (US-East Coast) 1.253 25.69 1.388
EC2 (US-East Coast) 2.144 20.229 2.094
EC2 (Europe) 13.331 159.422 12.504
EC2 (Singapore) 202 228 37.260
Com Hem (Sweden) 40.035 189.647 36.310
Tata (India) 313.2 304.1 150

2) Akamai ( using )

Location Opendns Google Local
IP Returned Ping to IP (ms) IP Returned Ping to IP (ms) IP Returned Ping to IP (ms)
True (Thailand) 239.4 138.9 18.88
Softlayer (US-East Coast) 1.312 1.262 0.877
EC2 (US-East Coast) 2.581 1.792 1.941
EC2 (Europe) 13.449 29.022 13.516
EC2 (Singapore) 202 71.7 7.052
Com Hem (Sweden) 51.73 35.972 13.212
Tata (India) 300 196 100.5

The ping timings represent the lag to the destination server from the location in question. I will try to update the results from more locations if I can get shell access to a server or PC in other countries. If you are willing to run the tests for me please contact me(or post in comments)


Using OpenDNS or Google Public DNS may be fast in resolving the DNS, but they do not give the ideal results.

In the case of Global DNS providers, the IP of the original requester is not passed along to the CDN’s DNS servers so they are unable to route the user to the nearest POP.

As you can see in the result tables above, when using OpenDNS from Thailand, trying to access static assets of Facebook, I am directed to a server in the USA whereas when using Google’s DNS i am directed to a server in Japan and when using my ISP’s DNS server I access content locally, hosted within my own ISPs network!

While the effect on large websites using CDN is significant, smaller non-CDN’d websites are also effected. Most websites embed widgets, advertising and other assets which are likely to be CDN’d.

The solution would be to use your ISPs DNS server rather than these Global providers. If they really suck so bad, its fairly simple to set up BIND as a caching recursive resolver to resolve hostnames directly bypassing the ISPs crappy service.

Bill Fumerola, ex-director of network engineering at OpenDNS confirms this problem on OpenDNS forums.

You can run the tests from your own computer using this simple script:

Here is the named.conf for a recursive server. Set your computer to use as the DNS. – config may differ for you, RTFM and adapt accordingly.

options {
        directory "/var/named";
        listen-on {;
        auth-nxdomain yes;
        allow-recursion {
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";


// a caching only nameserver config
zone "." {
        type hint;
        file "";

include "/etc/named.rfc1912.zones";

include "/etc/named.dnssec.keys";
include "/etc/pki/dnssec-keys/dlv/";

EDIT 1: Inverted the axis added test data from Europe
EDIT 2: Added test data from Singapore
EDIT 3: Added test data from Sweden
EDIT 4: Added test data from India
EDIT 5: Added link to Bill Fumerola’s explanation of the problem.

Shameless plug

Turbobytes – Faster content delivery using multiple CDNs

  • owen

    Nice work. It would seem to me that the conclusion should be “use a CDN blocker.”

  • Jason

    You’re using ping as a representation of whether or not the route is optimal. I disagree, however. Ping, while an excellent troubleshooting tool, does not always represent a true latency to a remote service. It could very well be that ping has been given lower priority than other traffic as is often the case with routers.

    A better tool, or perhaps one that may provide additional insight in addition to ping, would be a traceroute to the destination host. Determining the number of hops along the path, as well as the latency to each hop (again using ping), would provide additional data points to prove your hypothesis. For instance, you may find that ping times between you and the router prior to your destination are very low while the destination is much higher. This may indicate a priority issue with ICMP traffic.

    You can also use a simple script to time the connection to the remote server and download the main html page. This introduces additional complexity as a portion of the response time will ultimately be the server response time as opposed to the network, but this may lead to additional insights into the failure of CDN in general. That is, while CDN theoretically means you get a connection to the closest server, the closest server may not be able to fulfill your request in the quickest time.

  • Tom Limoncelli

    If you are going to post something like this, you should also point out that Google has proposed a solution to this problem:

    How do your tests hold up with this DNS extension implemented?

  • Remco

    From The Netherlands, using as ISP:

    ====== testing : Internap ===========
    testing : OpenDNS ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 188.971/189.555/190.654/0.822 ms
    testing : Google ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 190.621/191.457/192.995/0.982 ms
    testing : default ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4005ms
    rtt min/avg/max/mdev = 35.600/36.230/37.232/0.674 ms
    ====== testing : Akamai ===========
    testing : OpenDNS ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 46.428/48.487/53.945/2.811 ms
    testing : Google ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 34.676/35.493/37.218/0.999 ms
    testing : default ( )
    PING ( 56(84) bytes of data.
    ====== testing : Amazon Cloudfront ===========
    testing : OpenDNS ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4006ms
    rtt min/avg/max/mdev = 36.253/37.372/38.707/0.998 ms
    testing : Google ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4005ms
    rtt min/avg/max/mdev = 36.147/37.854/39.869/1.436 ms
    testing : default ( )
    PING ( 56(84) bytes of data.

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4004ms
    rtt min/avg/max/mdev = 36.497/37.293/38.062/0.553 ms

    — ping statistics —
    5 packets transmitted, 5 received, 0% packet loss, time 4007ms
    rtt min/avg/max/mdev = 33.835/35.762/37.770/1.374 ms

  • Vincent

    Thanks for the article. I’m using ActivePython on Windows: How can I install the required “dnspython” package?

  • Michael

    How about adding Canadian data?

  • Russell

    Interesting analysis – I wouldn’t have expected such a dramatic change in performance!

    I was curious how the data would look when charted, so I plopped the Akamai table into a chart. I’m not sure if it will show up correctly as a comment here but it makes the difference abundantly clear:

    I can make another for Internap if you like. It appears we here in the US luck out again…

  • Olivier Mignault

    It’s ridiculous to favor your local DNS server over OpenDNS or Google when they returned the same address (or addresses in the same /24 block – probably servers sitting in the same rack!!)
    At that point, you’re just skewing your data to illustrate your point. Those results should be considered equal.
    The local DNS would still win overall.

  • Nick

    I brought this to Google’s attention just after it released their DNS solution:

    At the time it was being looked it, but don’t know if anything ever came of it.

  • Jeff

    Use this nice utility for Windows to benchmark DNS servers


  • Marco van den Berg

    We noted the phenomenon described above as well. We’re using the enterprise version of OpenDNS mainly to block ‘certain’ categories of web traffic (including malware and botnet traffic), so changing DNS servers would have meant getting a new filtering product as well. Turns out that it’s fairly easy to change the DNS forwarders for the most popular CDN destinations, especially if you use a central in-house DNS resolver. Just exclude the CDN domains and change their resolution to a more optimal server. In our case, pointing to forward to Google’s DNS servers at caused significant performance improvements. Some of the main CDN’s that can be optimized are / / / / / / / / / / /

    If you want to optimize these I suggest you run a couple of tests to find the best forwarder.

    Marco van den Berg
    International Rice Research Institute

  • Richard Hartmann is another publically-accessible, geo-routed DNS resolver. In case you run more tests, you should add that, as well.

  • Why GoogleDNS (or OpenDNS) isn’t Always Faster – Dave Naffziger’s Blog

    [...] circles the belief that consumer ISP DNS servers are terrible is widely accepted. So, I found this study fascinating. The quick takeaway is that Content Distribution Networks (CDNs) like Akamai rely [...]

  • Richard

    We at are willing to help you perform world wide tests, we have servers in more than 55 countries world wide.

  • Angsuman Chakraborty

    A better solution for SMB and tech enthusiasts is to host your own local caching DNS server for optimal performance as well as help CDN’s with location awareness.

  • Chris

    If you live in Thailand, this may be true. But if you live in the US I doubt that it will make much difference in most cases. There are also other benefits to using OpenDNS or GoogleDNS – I have used several ISPs and their DNS always goes down for a few minutes to hours a year, whereas a computer set to OpenDNS has never had problems (this may just be fortitude, but oh well.)

    While the issue identified is valid and has implications for bandwidth, I don’t think that bandwidth is the bottleneck. It is more likely to be your browser. You say “Most websites embed widgets, advertising and other assets which are likely to be CDN’d.” I block advertising, so it doesn’t matter. I tend to ignore widgets. In any case, the substance of the page loads first so that I can begin reading even if the CDN-stuff hasn’t loaded yet. I can even stop them loading if I don’t want to see your widgets.

  • Sajal Kayan

    @owen : Nice point if you dont visit large websites, don’t mind having no CSS, Javascripts, Images on most sites.

    @Jason : Agree PING is not the optimal test, but it gives a fairly accurate indication of distance. When I have time I will set up a dummy file on both CDNs tested and compare time to download those files. Any objections to that?

    @Tom Limoncelli: Didn’t know about that extension. Once/if that is implemented, my blogpost would become irrelevant.

    @Remco : Thanks for your test data, ill include it in my tables soon. could you report ping timing to ? it seems to be missing.

    @Vincent : I have no knowledge about running Python on Windows. dnspython can be obtained from . I think you need to also replace “ping -c 5 -q ” with “ping -n 5 ” in the script. The arguments for ping in Windows is different than Linux. Unfortunately I have nowhere to test the script with Windows.

    @Michael : I’d welcome test data from a Canada

    @Olivier Mignault : Thanks for accusing me of skewing the data. Please see the table Internap data for “Softlayer (US-East Coast)” and Akamai data for “EC2 (US-East Coast)” and “EC2 (Europe)” gave better ping times for non local DNS and i have shown it as it is. I ran the script and posted whatever I got. Non Local resolvers gave better results than local only by few milliseconds and the result IPs were in the same range. The better ping times were a mere coincidence. See only the non-datacenter locations (Singapore, Sweden, India and Thailand) to notice the difference. Those are purely home connections.

    @Nick : Cool, in fact after I posted I noticed few others noticed similar thing some as far back in 2008. –

    @Marco van den Berg : All clients need to be in the same network as the central in-house DNS resolver to get the optimal results…

    @Richard : Thanks, you could post the results somewhere and Ill link to it from my tables, or if you don’t want to do that, I can post it on my blog too. Your results would be highly appreciated. you can contact me at sajal at thaindian dot com

    @Angsuman Chakraborty: Agreed!

    @Chris: The bandwidth is a problem in Asia (esp Thailand) with intercontinental links running at its saturation point. I understand that you block advertising (viewing free stuff while wasting the websites bandwidth and resources and not allowing them to monetize), but do you also block css, javascripts, flash, videos and images? These are also usually CDN’d. Without loading the stylesheet usually the page doesn’t start rendering.

  • Gary

    Tom wrote:

    > How do your tests hold up with this DNS extension implemented?

    Uh, you tell us once you get it approved and rolled out to BIND and all the other name servers in use.


  • TheInnerLayer — where SL’ers come to rant » Blog Archive » DNS from All Angles

    [...] but the public DNS providers are confusing CDN location-based algorithms. The article is here: and I recommend strongly that both content providers and content consumers both read [...]

  • Gerardo Lisboa

    Are the results from consistent with your problem?

  • Sajal Kayan

    @Gerardo : It seems like does not use DNS for detecting the information under “What We Know About You”. They r simply using the IP of the requester to determine that. Similar to sites such as

  • Sajal Kayan » ATTN Big Media: Web page speed matters!

    [...] single post page on my [...]

  • Billy Vierra

    This only true if the CDN is using older technology (such as returning a different IP depending on what DNS server you hit). If they are using Anycast IP’s this actually is not an issue at all.

  • Alaor

    By your response, I sincerely suggest you read the article again. You REALLY didn’t understand it.

  • Jason Doller

    Google’s proposal is just that – a proposal. Until it is accepted asking people to test with it is a bit lame, especially as you will gett the exact same results as without using the extension unless the CDN has implemented the solution.

  • Ronin

    By your response, I sincerely suggest you read the article again.

  • Mohammed Shomam

    @Olivier Mignault : Thanks for accusing me of skewing the data. Please see the table Internap data for “Softlayer (US-East Coast)” and Akamai data for “EC2 (US-East Coast)

  • Sajal Kayan » Evaluating few CDN options

    [...] Anycast/BGP routing : No way bad DNS server can mess up [...]

  • Affordable Philadelphia SEO

    Other on-site steps that you can take as part of your efforts on SEO for small business include the need to decide if our website is in need of meta-description and the need to include the company address and the phone number. The contact information should be made visible on all pages of the website and this should be part of your SEO for small business. And finally, your SEO for small business should include offsite activities. Your SEO for small business on this stage will include the need to include the website to major industry-related directories. This stage of SEO for small business means getting more links plus deciding if the social media sites will deliver more benefits for you. And more importantly don’t stop your SEO for small business without tracking its improvement- or non improvement.

  • Website Designer Riyadh

    Very informative post.. You have mentioned very useful and profitable message for us.. I will share with my friends also..
    Web Designer Riyadh | Riyadh website design company

  • Anonymous

    I think you should also replace the “ping-c 5-q” with “ping-n 5″ in the script. The arguments for the ping on Windows is different from Linux. Unfortunately, I have nowhere to test the script with Windows.

    SEO India

  • Jeremy

    I would recommend, where the different DNS returns the same IP address, to show only one timing for the IP and highlight the IP with the shortest ping time, not the ping time itself.

    This will clarify which cases are simply due to link variability to the same IP address.  Obviously if the DNS result is the same, the result should look the same on the table (from a color coding perspective).

    I don’t think this would change the overall result or tenor of the results, but I do think it would be more accurate this way…

  • Mehdi Daoudi

    Why pick on OpenDNS? I would also argue that employees of IBM will be mis-mapped, or AOL or any major ISP like timewarner or verizon… Google, OpenDNS and other CDNs like edgecast have implemented EDNS subnet

  • Jean-Tiare Le Bigot

    CDN implementation is awkward. Why don’t they implement anycast themselves directly instead of relying on the DNS’ anycast ?

    By the way, this is how OVH did its own CDN:

    Could you add it to your tests to see the impact ?

  • Anonymous

    Hi Mehdi,
    The goal is not to pick on OpenDNS. Just wanted to show that as off May 2010, using OpenDNS or Google’s DNS would make your internet slower. At that time CDNs did not have a way to figure out client’s IP and would be sending users to faraway places.

  • Ilya Grigorik

    The Google Public DNS team built a great tool for benchmarking DNS servers:

    Instead of relying on a few pings, give that a try. It can run a test against hundrends-thousands of sites and give you objective data on which DNS provider will give you the best bang for the buck.

  • Anonymous

    Did you even read the article?

  • Anonymous

    If i access Facebook’s akamai account, resolving thru my ISP I get a server 8ms RTT away from me. but resolving thru that DNS I get an IP with RTT 360ms!

  • Ryan

    “The goal is not to pick on opendns”
    “OpenDNS is the enemy”

    Interesting choice.

  • Bugmenot

    The problem with this is that you are using PING which is using ICMP, ICMP is often put at a lower priority then other traffic such as a UDP DNS request. The best thing to do is to use a DNS benchmark tool ( or to use a tool that does a UDP ping not a ICMP ping.

  • D9u

    My ISP issues dynamically allocated IP addresses which resolve to locations which are literally thousands of miles away from my actual location.

    How many others have the same situation?

  • djernie

    Also, why do you promote to use the lease secure available caching recursive DNS server software (BIND)? There are far better and easier configurable caching DNS servers like Unbound, djbdns, tinydns, PowerDNS, etc …

  • Anonymous

    Isn’t this what anycast is for?

  • Frank Denis

    Also see on this issue, and how OpenDNS and Google are actually trying to solve it.

  • David

    Did YOU even look at that link? Namebench “hunts down the fastest DNS servers available for your computer to use.” It also takes into account your local ISP’s name servers and will recommend those if they are actually faster for you.

  • Anonymous

    Please tell me how is DNS performance relevant to the point I am making here?

    Please show me how Namebench will show me if the response for the CDNs hostname is ideal based on my location?

    Maybe the article misleaded you, I apologize.

  • Nate

    No, the users of a typical ISP (Time Warner, Verizon) will not receive bad results unless DNS is mis-configured.

    The problem happens when the DNS server is not close to the client. When using your ISP’s servers, they are close to you by definition, so CDN addresses resolve to nearby servers. Most national ISPs also use regional-level DNS servers, not just one centralized set of servers, to ensure better resolution.

    (In this discussion, “close” means close *network-wise*, which is usually, but not always, close geographically. As long as your DNS servers have a similar route to Internet destinations as you do, you will get good results.)

    OpenDNS can and probably is addressing this by placing servers in data centers around the country, especially in or near data centers used by major ISPs, and then using an anycast IP address. If they don’t have a presence in Thailand, those users may not get the best results.

  • Bachsau

    I dont like being censored based on my country. If soneone just wants to optimize traffic, he should use anycast IPs.

blog comments powered by Disqus