Censorship Protecting me from bad stuff – Interwebs has a lot of “bad” things out there. My ISP takes good care of me by not letting me access things I am not supposed to see… Even sites not explicitly blocked by court-order. I don’t know what I would do without them. My head would probably explode if I saw porn, and my feelings would get hurt if I came across certain types of political messages..

Transparent proxy Web slow down machine – Buddha teaches us “The greatest prayer is patience”. A very special offering of True ISP is that it reminds us to be patient in this fast-paced world. True’s Web slow down machine sits between users’ connection to other servers. One of its features is to slow down access… It employs several brilliant methods to accomplish this :-

• Not keeping connections alive – This is the most important factor in slowing down pageloads. True does not keep connections to remote hosts alive, thus making sure you have to establish a fresh connection with each request to a server overseas. No matter how small the file, a request to the USA will take a minimum 500ms.
• Making TCP optimizations useless, since the slow down machine is the one that actually makes connections to remote hosts.
• Overriding destination IP – True doesn’t care about what IP your computer wanted to connect to, your computer could be wrong. It sees the Host header from the request, does its own DNS lookups and routes you to the correct server. Even if you wanted to override this for development, True correctly sends you to production server. True knows development/staging servers are full of bugs, so requests should always go to production.
• 512 kbps upload speed is sufficient for everyone. If you need to upload something big, why not get your lazy ass out, buy a CD and mail it!
• Sharing is caring – My ISP oversells available bandwidth by a huge margin. Teaches us the importance of sharing

Privacy People impersonator – Your life is boring? True has a solution! It will automagically log you in as someone else so you can get a glimpse into their exciting lives.

Incompetence Motivator – Living in Thailand, I am ashamed that I don’t read Thai yet, in part due to my own laziness. True gives you an incentive.

edited by Michael van Poppel

(The tests above were run on webpagetest.org on IE8 at Dulles, VA)

Thank you Google! You just made my day.

Here is a simple way to check if you are behind one.

sajal@sajal-laptop:~$ ping -c 4 www.cdnplanet.com
PING www.cdnplanet.com (107.20.181.99) 56(84) bytes of data.
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=1 ttl=42 time=314 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=2 ttl=42 time=313 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=3 ttl=42 time=312 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=4 ttl=42 time=312 ms

--- www.cdnplanet.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 312.195/313.229/314.137/0.889 ms

sajal@sajal-laptop:~$ ab http://www.cdnplanet.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.cdnplanet.com (be patient).....done

Server Software:        Apache
Server Hostname:        www.cdnplanet.com
Server Port:            80

Document Path:          /
Document Length:        13084 bytes

Concurrency Level:      1
Time taken for tests:   0.944 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      13296 bytes
HTML transferred:       13084 bytes
Requests per second:    1.06 [#/sec] (mean)
Time per request:       943.539 [ms] (mean)
Time per request:       943.539 [ms] (mean, across all concurrent requests)
Transfer rate:          13.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       21   21   0.0     21      21
Processing:   922  922   0.0    922     922
Waiting:      611  611   0.0    611     611
Total:        944  944   0.0    944     944
sajal@sajal-laptop:~$ 

My ping time to CDN Planet is 312ms, but the connection was established in just 21ms !!!!11!!1

Reasons for doing so involve : Censorship, big brother snooping, caching, hijacking users sessions , and probably more …

Some things to note:-

• We would serve CSS, JS, Images referred to from stylesheets and some website images thru the CDN.
• Due to the nature(trust me) of the site we expect a higher than normal miss rate. The access is spread across a wide number of urls, some may get lower hits.
• Speed is important. Website should be fast(er) from everywhere. The most important regions in order of priority are US, EU, AU and ROTW with US being most important.
• Anything on the site can change anytime, there is no build process as such. Any object anywhere can change, and change must be visible ASAP.
• Developers/designers shouldn’t be harassed to purge a file if they change something.
• CDN data usage : ~100 GB per month

The providers we looked at :-

MaxCDN: Almost sealed the deal.

Pros:-

• Cheap : $40 for first TB (must use in a year) + $99 per additional TB . On current usage rates this comes to say 0.04+ /GB.
• * Anycast/BGP routing : No way bad DNS server can mess up routing.
• Nice control panel, has a purge all option for just in case. Purges take effect almost instantly.
• Handles gzip well.
• Can have separate cache timings for caching in Browser and caching at CDN. – i.e. We can say not cache a file in browser level, but cache at CDN and purge when theres a change made.

Cons:-

• Poor global coverage. – No POP/Edge in Asia/AU – Deal Breaker
• Pages loaded same speed when testing from AU with or without CDN.

EdgeCast: Looked good at first, but poor gziping.

Pros:-

• Impressive list of networks
• Highly configurable control panel.
• Can have separate cache timings for caching in Browser and caching at CDN. – i.e. We can say not cache a file in browser level, but cache at CDN and purge when theres a change made.

Cons:-

• Gzippable files will not be gzipped for cache misses. – Deal Breaker
• Request from Edge server to origin is uncompressed.
• Expensive and wants higher commitments.
• DNS Based routing

CDNetworks: Didn’t look past price

Cons:-

• Ridiculously high price – Dealbreaker

Amazon CloudFront: WIN

Pros:-

• Testing showed our pages to be fastest from all regions when using cloudfront. YMMV
• No commitments – $0.15 – $0.2/GB (depends on where user accesses from) + negligible per request fee
• Client is already AWS user, one less account to maintain.
• No need to send gazillion emails to gazillions of people to get started. No bargaining.

Cons:-

• No POP/Edge in AU (but has in Singapore, Hong Kong and Tokyo)
• DNS based routing.
• Charges fee per request and per invalidation(purge) request.
• No control panel, invalidation requests need to be done by API only.
• Does not do gzipping, but honours Vary header and serves correct version based on what user asks.
• Can’t use querystring parameters for CDN cachebusting. CloudFront ignores querystrings.

Sidenote : Requests from CloudFront to origin are HTTP 1.0 . Nginx by default does not serve gzip to 1.0 request. gzip_http_version setting must be changed in order to use nginx as origin for CloudFront.

The system we architected adds something based on the file mtime as a part of the URL, so now we don’t need to any purges at the CDN. Also now we can have far future expires on all CDN’d objects cause if something changes, the URL would automagically change.

For us, the price and features are important, but whats more important is the results. We went with the provider with lesser features just because our pages loaded fastest with them.

One of the first things I do when looking into a new site is look at Google Analytics to understand a little about the sites visitors. This helps in prioritizing changes which affects the majority of users.

An interesting observation :-

Pageviews for all visitors

Average Pageviews for all visitors

Adsense revenue also followed a similar path. — Screenshot not available

The reason for this effect was a simple database indexing tweak which sped up the backend performance of a very important action page on the site.

The site in question is a web application where user inputs something and gets some result. It is not a content site where pageview/user could have increased due to some interesting content being added or something. The only change was indexing of a column which should have an index right from the start.

DISCLAIMER: I did not have any role in the above improvement. This was done before my involvement. Perhaps this convinced the client to hire me. Screenshots/info shared with clients permission.

8th May 2011: This seems to be having problems, will investigate and update when i have time. Pls revert to normal DFP tags for now.

One of the biggest challenges when optimizing performance of websites is third party content – specifically advertisements.

Most ad networks and servers use evil document.write() in their JavaScript(and even nested document.writes) which block further rendering of the page until their code has completed execution. In this blogpost, I’ll show how you can use DFP’s iframe tagging(read warnings there) combined with LABjs and little bit of JavaScript hackery to make any ad load asynchronously with negligible impact on rest of the pageload.

Attention Deficit Disorder version : Before – After

NOTE: Use the method below entirely at your own risk! Use only if you know what you are doing…

Current blocking method

DFP has an experimental method to load ads called iframe tagging. The JS looks like this :-

The Bootstrap: In <head> (Does not have to be in <head> but before the first GA_googleFillSlotWithSize call) :-

<script type='text/javascript' src='http://partner.googleadservices.com/gampad/google_service.js'>
</script>
<script type='text/javascript'>
GS_googleAddAdSenseService("ca-pub-7046344781760701");
GS_googleEnableAllServices();
</script>
<script type='text/javascript'>
GA_googleUseIframeRendering();
</script>

Then wherever we want the ads to display, we put something like this :-

<script type='text/javascript'>
GA_googleFillSlotWithSize("ca-pub-7046344781760701", "test_async_lb", 728, 90);
</script>

With this method, the bootstrap does some blocking. First it loads a JavaScript then the following functions document.write another <script> tag which must load sequentially again. The GA_googleFillSlotWithSize function is relatively inexpensive. All it seems to do is document.write an iframe with various targeting information as parameters in the URL and does not block further rendering. The advantage of iframe tagging is that slow ad networks don’t fuck up your page. But the bootstrap is very expensive as shown in this waterfall chart.

This is what it looks like.

The hack

Last few days, I’ve been playing a little with LABjs, specifically its complete async loader.

After playing with LABjs, ive come up with the following LABjs snippet :-

      // intercepts the script inserted via document.write and loads it via LABjs
      function docwrt(str){
        var script = str.replace(/(.*)\=\"/g, '').replace(/\"(.*)/g, '');
        $LAB.script(script).wait(function(){
          GA_googleUseIframeRendering();
          // following function makes the magic happen!
          function Wrapper_googleFillSlotWithSize(pubid, slotname, width, height, target){
            var docwrttemp = function(str){
              target = document.getElementById(target);
              target.innerHTML = str;
            };
            document.write = docwrttemp;
            GA_googleFillSlotWithSize(pubid, slotname, width, height);
          }
          // usage of the new wrapper here "leaderboard" and "skyscraper" are target div ids
          Wrapper_googleFillSlotWithSize("ca-pub-7046344781760701", "test_async_lb", 728, 90, "leaderboard");
          Wrapper_googleFillSlotWithSize("ca-pub-7046344781760701", "test_async_sky", 160, 600, "skyscraper");
        });
      }

      document.write = docwrt; //intercepts document.write from below script
      $LAB.script("http://partner.googleadservices.com/gampad/google_service.js").wait(function(){
        GS_googleAddAdSenseService("ca-pub-7046344781760701");
        GS_googleEnableAllServices();
      });

(note: Since I’m lazy, I haven’t restored document.write back to its original glory.)

Here Wrapper_googleFillSlotWithSize is a wrapper around GA_googleFillSlotWithSize which takes a 5th argument – target – This is the id of <div> where we want to show the ad.

Here is a sample page using this hack. Id appreciate it if I get some feedback about this method via comments below. As I said earlier, use your own better judgment before using this snippet in production. I welcome criticism but will not accept blame if this doesn’t work for you.

In my simple example, it may seem it takes longer to fully load the page, but if you have many other things on the page, the overall effect will be better with this hack. Moreover, if you are already using LABjs on your site, this is a no-brainer. With this method, even if Google is inaccessible(for whatever reasons) it wont SPOF your page.

Slow motion video of pageloads on IE8:-

Generated via webpagetest.org

Left is normal method, right is hacked version.

Currently tested on IE(7 thru 9), Firefox 3.6.11, Chrome 10.0.648.45 dev and an unknown version of Safari.

Conclusions…

The world would be a much better place without the evil document.write(). Google should know better. They should make a function like Wrapper_googleFillSlotWithSize by default.

It probably had an attendance of > 4k participants… Probably one of the largest BarCamps ever, despite the fact that the internet connectivity in Myanmar sucks big time…. Really unusable…

I used internet at the free “wifi” internet at the hotel and the wifi arranged by the organizers. Both places, the speeds I’ve seen range from 0 kbps to ~50 kbps…. most of the time around 0 kbps

Internet access

Most locals use internet at cafes/internet shops. The average shop would have 512 kbps connection shared between 15 to 20 computers charging 300 to 500 MKY ( US$ 0.3 to 0.5 ) per hour for usage. Some give discount if you bring your own laptop, some don’t. I didn’t get a chance to visit a cafe, this is based on what people told me.

Most geeks have a computer at home, very few have laptops. ADSL is available but is very expensive. It costs > $1000 setup fee + $120/month for a mere 512 kbps. Only the very rich people can afford it.

Mobile internet was unheard of until few weeks ago. They don’t have gprs there, but recently they launched 3G and CDMA for wireless internet. CDMA would charge around $0.10/minute for 1 or 2 mbps (don’t remember clearly).

Out of the few sites i tried opening, only Facebook seemed to be usable. This is due to the enormous effort they invest in frontend performance. Google search was also fast, but none of the sites in the results seemed to open.

The average DNS query took me 2+ seconds to resolve. DNS access is limited to the ISPs crappy nameserver, I couldn’t use opendns or any external nameservers.

Almost all local sites are hosted outside of Myanmar cause domestic connectivity is as bad as international routes. Server co-location fees are 10x in Burma than in America.

Censorship

I didn’t use the internet much… but here is what I found.

Among the sites/services blocked :-

• My secret project – Details/screenshot below
• Twitter – Blocked normally, but works using https or apps.
• Gmail – works on browser but IMAP doesn’t work at all… The most surprising aspect of it is that 99% of ppl there use gmail.
• SSH – connections to port 22 are completely blocked, but I’ve been told that setting ssh server on port 443 works fine. I used DNS tunnel and it kinda worked occasionally for emergency usage – after trying for 10 – 15 mins i could get 1 min or so of connectivity.
• gtalk – Using Pidgin – doesn’t work at all.

I am building a new webservice called www.{secret}.com . Hardly anyone knows about it. Its got only 3 users, and has nothing to do with Burma. That site was somehow blocked in Burma (only one ISP). I cant understand how {secret} could trigger any phrase based blocklists or something.. Makes me feel important if governments block my site…

My moment of pride :-

Summary

BarCamp Yangoon 2011 was probably the biggest barcamp ever with about 4500 to 5000 participants (with about 20 – 30 foreigners traveling in from abroad). An interesting activity was file swapping. There were about 20 – 30 computers setup with shared folders which participants used to exchange software/music/etc.

Based on a simple survey, everyone uses Firefox. Very few people use Linux. Windows is very popular due to rampant piracy.

I was considering giving a talk on web performance optimization on day 2 of the event, but didn’t go thru with it cause an American expat covered the topic well on day 1.

One thing that I should have done would be to download all the Velocity Conference videos and pass it around… unfortunately it occurred to me to late. Ill try to arrange it now…

The country has great people… It can succeed in this digital age if only America and other western countries lift sanctions on Burma..

Please wait for a site operator to respond.

Chat session has been terminated by the site operator.

Don’t get it? couple of days to provision a damn VPS? I am used to getting physical dedicated servers provisioned at Softlayer in < 4 hours. Amazon EC2 instances in < 1 minute…

The only reason im using DreamHost is cause this is a clients project where fancy panels for domains, settings, etc would be easily user maintainable.

To know more about what a CDN is, please read this post.

The reason given by them :-

Dear SimpleCDN Customer,

I am writing this letter to update you on a situation that has been developing for the past 72 hours between SimpleCDN and our technology and infrastructure providers, SoftLayer and Hosting Services, Inc.

Two days ago these organizations decided to immediately terminate our contract and suspend service on much of our infrastructure in Dallas, Seattle and Washington, D.C. This infrastructure constitutes the majority of our delivery network for our value services, including on-demand and live streaming services.

[...]

Their full statement can be read at http://admin.simplecdn.com/. Quite likely this won’t be the permanent url for their rant.

My first thought was that they didn’t pay their bills.. but this doesn’t seem to be the case here. I’d speculate that this is related to DMCA or even some connection to Wikileaks. We need Softlayer’s side of the story to make an opinion. I’m a Softlayer user for few years, I refuse to believe it that they did it for competitive advantage. There is more to it!

MaxCDN has stepped in to help stranded SimpleCDN customers to get their sites up asap at lower costs.

So… let me be Captain Hindsight here and say what SimpleCDN users should have done all along.

Keep a hot spare CDN ready to be deployed at a moments notice.

There are many CDN services, like Softlayer, CloudFront, etc which have pay as you go plans, no upfront or monthly costs. Sign up for them, set up your zones, and keep the required CNAMEs handy. If your prime CDN provider goes under, has high latency, or any other issue, switching to these alternate CDNs is simply a change in the DNS zone. This could be automated with Amazon Route 53.

Very easy to implement for origin pull, for uploaded content it doesn’t hurt to store your content on multiple services for redundancy.

Had the users kept a hot spare CDN provider ready, it would have taken them 5 mins (plus the DNS propagation time) to switch to another provider.

In most cases, where only css, javascript, images are served by the CDN, frequent users such as admins would have these files in their browser cache and may not feel that anything is broken. For such situations, the answer is passive monitoring.

Bottomline: Everything FAILs…. eventually…

Notes:-

• This blog is hosted at softlayer
• This blog uses MaxCDN
• This blog uses Softlayer’s CDN service
• Before today i hadn’t heard about SimpleCDN

UPDATED: 11:20 (EST) Dec 15

The only 2 statements by SoftLayer on this issue comes in the form of tweets.

FYI, our privacy policy prevents us from discussing customer issues in public. We definitely can’t discuss customers of customers. – twitter

@simplecdn happy to have you as a direct customer. Send over your requirements so we can price it out for you. ^SK – twitter

IMHO the second tweet is rubbing salt on SimpleCDN’s wound.

It is common for companies under legal threat to not make any statements which may or maynot be used against them during litigation.