Posts on Sajal Kayan

Figuring out MAC address of a fitbit tracker

Sat, 24 Sep 2016 15:30:00 +0000

Last week I posted about potentially using my fitbit Charge HR for presence, and was looking for some way to figure out its MAC address. Here are some ways to do it.

BLE scan

First make sure the tracker is not connected to your phone. The simplest way is to turn off bluetooth on the phone (or force-close the fitbit app). Once this is done, the tracker not connected to any host app, starts advertising using bluetooth low energy.

Then from some linux host with BLE enabled adapter (in my case raspberry pi 3 ) run a lescan.

pi@raspberrypi:~$ sudo hcitool lescan
LE Scan ...
xx:xx:xx:xx:xx:xx (unknown)
yy:yy:yy:yy:yy:yy (unknown)
12:34:56:78:9A:BC (unknown)
12:34:56:78:9A:BC Charge HR
^Cpi@raspberrypi:~$

Since there is only 1 Charge HR detected, I can be fairly confident that 12:34:56:78:9A:BC belongs to me.

Syncing using Galileo

Galileo is project that allows you to sync your tracker using the bundled dongle that comes with fitbit. It’s python and useful if you use linux since fitbit does not provide software for it.

sajal@sajal-lappy:~/path/to/galileo$ ./run --fitbit-server client.fitbit.com --force -v
2016-09-24 22:07:11,549:INFO: Disconnecting from any connected trackers
2016-09-24 22:07:13,555:INFO: Got an I/O Timeout (> 2000ms) while reading!
2016-09-24 22:07:13,559:INFO: Discovering trackers to synchronize
2016-09-24 22:07:13,565:INFO: Ignoring message: StartDiscovery
2016-09-24 22:07:17,569:INFO: 1 trackers discovered
2016-09-24 22:07:17,569:INFO: Attempting to synchronize tracker BC9A78563412
2016-09-24 22:07:17,575:INFO: Starting new HTTPS connection (1): client.fitbit.com
2016-09-24 22:07:24,401:INFO: Getting data from tracker
2016-09-24 22:07:25,752:INFO: Sending tracker data to Fitbit
2016-09-24 22:07:25,753:INFO: Starting new HTTPS connection (1): client.fitbit.com
2016-09-24 22:07:27,165:INFO: Successfully sent tracker data to Fitbit
2016-09-24 22:07:27,165:INFO: Passing Fitbit response to tracker
Tracker: BC9A78563412: Synchronisation successful

Note how 12:34:56:78:9A:BC turns into BC9A78563412. The bytes are reversed.

Using Fitbit API

Fetch the list of devices associated to your account using the Fitbit devices API.

GET https://api.fitbit.com/1/user/-/devices.json

[
  {
    "battery": "High",
    "deviceVersion": "Charge HR",
    "features": [
      
    ],
    "id": "xxxxxxxxx",
    "lastSyncTime": "2016-09-24T22:07:26.000",
    "mac": "BC9A78563412",
    "type": "TRACKER"
  }
]

This again shows the reversed-byte MAC address in the mac field.

I have updated my presence script to continuously run hcitool lescan and keep a map of devices available. This way if for some reason my phone’s bluetooth drops out, I can keep track of myself using the Charge HR since it restarts advertisements once its link to the phone is broken.

Presence server upgrade

package main

import (
    "flag"
    "log"
    "net/http"
    "os/exec"
    "strings"
    "sync"
    "time"
)

var (
    blereg  = make(map[string]time.Time)
    blesync = &sync.RWMutex{}
)

func updateble() {
    log.Println("updating ble table")
    cmd := exec.Command("timeout", "--signal", "SIGINT", "5", "hcitool", "lescan")
    b, _ := cmd.Output()
    blesync.Lock()
    defer blesync.Unlock()
    for _, s := range strings.Split(string(b), "\n") {
        sp := strings.Split(s, " ")
        if len(sp) > 1 {
            log.Println(sp[0])
            blereg[sp[0]] = time.Now()
        }
    }
}

func l2ping(mac string) bool {
    log.Println("Checking ", mac)
    cmd := exec.Command("l2ping", "-c", "1", mac)
    err := cmd.Run()
    if err != nil {
        log.Println(err)
        return false
    }
    return true

}

func main() {
    var addr = flag.String("addr", ":8081", "address/port to listen on")
    go func() {
        for {
            updateble()
            time.Sleep(time.Second * 10)
        }
    }()
    flag.Parse()
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        mac := strings.Split(r.URL.Path, "/")[1]
        if mac == "" {
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        //Check blereg
        blesync.RLock()
        dur := time.Since(blereg[mac])
        blesync.RUnlock()
        log.Println(dur)
        if dur < time.Minute {
            //BLE registry saw this mac within last minute
            w.WriteHeader(http.StatusNoContent)
            return
        }
        if l2ping(mac) {
            w.WriteHeader(http.StatusNoContent)
            return
        }
        w.WriteHeader(http.StatusNotFound)
    })
    s := &http.Server{
        Addr:           *addr,
        ReadTimeout:    10 * time.Second,
        WriteTimeout:   10 * time.Second,
        MaxHeaderBytes: 1 << 20,
    }
    log.Fatal(s.ListenAndServe())
}

This runs hcitool lescan in a loop making note of MACs discovered. When receiving a query it checks if the MAC was seen recently, if not then it tries to ping it on bluetooth classic. I could have use the gatt library to continuously scan over BLE, but on initializing gatt, it takes over the HCI device completely making me unable to run the l2ping command. gatt does not have the capability to do Bluetooth stuff.

Detecting presence using Bluetooth

Sun, 18 Sep 2016 15:53:00 +0000

How to programmatically detect whether I am at home or not? - That’s the problem I am currently trying to solve. In this post I will outline what methods I am using and what I have tried/considered.

I use this presence information to automatically switch off lights when I leave my home, and turn them on when I return. By return I mean when my state changes from not-present to present

Previous Solution (Wi-Fi)

For about a year, I used my phone being present on my Wi-Fi network as an indicator of my presence at home. This approach did not work well for me.

My router would assign a static IP to my phone using DHCP, and the result of a ping test would indicate my presence.

#!/bin/bash

ping -c 3 192.168.x.x > /dev/null
if [ $? -eq  0 ]; then
    #Sajal is at home...
else
    #Sajal is not at home...
fi

I have set my phone to stay on Wi-Fi even if it thinks Internet connectivity is not available. In spite of this, the phone would occasionally disconnect from Wi-Fi (or fail the ping test). Often, when I am at home the lights would go off, and I would need to fiddle with my phone to get it back on Wi-Fi. Almost every morning I would wake up and realize my living room lights are turned on, at some point during the night my code thought I sleep-walked out of my home and returned.

Also, it would take anywhere between 0 to 100 seconds for the phone to get back onto Wi-Fi once I returned home. This is annoying at night, especially if I have my hands full with groceries.

I altered this a bit by using arp instead of ping.

#!/bin/bash

if [ $(arp -n | grep "xx:xx:xx:xx:xx:xx" | wc -l) -eq  1 ]; then
    #Sajal is at home...
else
    #Sajal is not at home...
fi

This provides some relief from intermittent Wi-Fi issues because it takes time for arp cache to flush out the information, but most of the problems of the ping method are still applicable.

Current Solution

Currently I have a Raspberry Pi pinging my phone over Bluetooth to detect presence. All you need for this to work is your phone’s Bluetooth MAC address. The phone does not need to be in detectable/scanning mode, just that Bluetooth should be turned on.

I was originally skeptical about using Bluetooth. I did not expect the signal to cover my entire (1-bedroom) apartment, but turns out it works very well. Now my lights turn on even before I am done opening my door.

I have the following Go code running on a Raspberry Pi 3 (which has Bluetooth builtin).

package main

import (
    "flag"
    "log"
    "net/http"
    "os/exec"
    "strings"
    "time"
)

func l2ping(mac string) bool {
    log.Println("Checking ", mac)
    cmd := exec.Command("l2ping", "-c", "1", mac)
    err := cmd.Run()
    if err != nil {
        log.Println(err)
        return false
    }
    return true

}

func main() {
    var addr = flag.String("addr", ":8081", "address/port to listen on")
    flag.Parse()
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        mac := strings.Split(r.URL.Path, "/")[1]
        if mac == "" {
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        if l2ping(mac) {
            //TODO: Use 204 not 200
            w.WriteHeader(http.StatusOK)
            return
        }
        w.WriteHeader(http.StatusNotFound)
    })
    s := &http.Server{
        Addr:           *addr,
        ReadTimeout:    10 * time.Second,
        WriteTimeout:   10 * time.Second,
        MaxHeaderBytes: 1 << 20,
    }
    log.Fatal(s.ListenAndServe())
}

I need to use this http API because the bash script runs on a device without Bluetooth support. My bash script evolved to call the Go code using curl.

#!/bin/bash

if [ $(curl --write-out %{http_code} --silent --output /dev/null http://192.168.x.x:8081/xx:xx:xx:xx:xx:xx) -eq 200 ]; then
    #Sajal is at home...
else
    #Sajal is not at home...
fi

Figuring out my phone’s Bluetooth MAC address was relatively straightforward.

Put your phone in detectable mode.
Scan available devices.
Test if it is pingable. Phone does not need to be detectable any more.

pi@raspberrypi:~$ hcitool scan
Scanning ...
    yy:yy:yy:yy:yy:yy   KDL-50W800C
    xx:xx:xx:xx:xx:xx   G4
pi@raspberrypi:~$ sudo l2ping -c 4 xx:xx:xx:xx:xx:xx
Ping: xx:xx:xx:xx:xx:xx from zz:zz:zz:zz:zz:zz (data size 44) ...
0 bytes from xx:xx:xx:xx:xx:xx id 0 time 5.90ms
0 bytes from xx:xx:xx:xx:xx:xx id 1 time 7.35ms
0 bytes from xx:xx:xx:xx:xx:xx id 2 time 28.66ms
0 bytes from xx:xx:xx:xx:xx:xx id 3 time 28.64ms
4 sent, 4 received, 0% loss
pi@raspberrypi:~$

Apparently KDL-50W800C is a Sony TV, perhaps belonging to a neighbour, I can’t ping it because it’s probably too far. The G4 is my phone.

This method has been in production for the last week, and only once did it give a false negative. My phone was running some upgrades and for some reason Bluetooth stopped working until I rebooted.

Whats next?

In order to make the presence detection more robust, I am considering the following in addition to the above…

Somehow detect the presence of my Fitbit Charge HR tracker. I ~~can’t seem to~~ can figure out the MAC address of the tracker. ~~If someone knows of a way let me know.~~ Someone else blogged about using the bundled dongle for this, but it is very unreliable method. If the Fitbit is connected with the phone, galileo will not be able to detect it.
Get a Bluetooth tag and attach to keychain. I see some commercially available tags (example Tile), but these tags are meant to work with apps on phones. I don’t know if works on regular computers with Bluetooth dongles (Raspberry Pi).
Motion sensors. These would obviously not work when I am asleep or still… But it’s something to look into, especially since I plan to use it for bathroom lights. Motion sensors might give me some sense of which room I am in, should I need that information for future projects.
Keycard holders. The type they use in hotels as a master switch for the room. The difference is in my case I would only use it as a signal for presence and not hardwire my mains thru it. Could be a simple mechanical switch wired to GPIO pins of a pi.
Door sensors. Might have some other useful applications as well.

Discarded ideas

Timer based presence. Assume me to be present from some specified time to another specified time.
GPS+phone based presence solution. At home the GPS is very inaccurate. I would need to add a huge error margin(few hundred meters - or higher if its rain-ey), effectively my system would think I am at home even if I am only in the general vicinity.

UPDATE

I investigated more about the possibility of using my Fitbit Charge HR for presence. It is not a practical option. The pi can detect it, but if I open the Fitbit app on my phone, the Fitbit will stop advertising until I either force close the Fitbit app, or turn off (and on) the Bluetooth functionality on the phone.

pi@raspberrypi:~$ sudo timeout --signal SIGINT 5 hcitool lescan
LE Scan ...
xx:xx:xx:xx:xx:xx (unknown)
xx:xx:xx:xx:xx:xx Charge HR
yy:yy:yy:yy:yy:yy (unknown)
zz:zz:zz:zz:zz:zz (unknown)
pi@raspberrypi:~$ #Now open app on phone and sync
pi@raspberrypi:~$ sudo timeout --signal SIGINT 5 hcitool lescan
LE Scan ...
zz:zz:zz:zz:zz:zz (unknown)
yy:yy:yy:yy:yy:yy (unknown)
pi@raspberrypi:~$

Sign web content using PGP

Sat, 06 Feb 2016 18:18:00 +0000

A lot of web-content these days passes thru untrusted intermediaries, especially plain text traffic which is often intercepted by ISP proxies for caching (and other purposes ;) ). A compromise at these places can subject your users to malicious payload, mostly in the form of javascript.

The obvious solution to these issues is to use TLS i.e. https:// sites, which is more accessible these days thanks to Lets Encrypt. But even this does not give complete end-to-end coverage because many sites use a CDN who might unknowingly or maliciously tamper with the contents.

One way to make such tampering detectable is to sign textual web-content using PGP. As a PoC, I have signed all html pages of this blog with a pgp signature. Go ahead, view source of this page, I’ll wait…

Bash script (signhtml.sh) to perform the signing :-

#!/bin/sh

tmpfile=$(mktemp)
echo "using $tmpfile for $1"
echo "https://keybase.io/sajal -->" > $tmpfile #Optional text in commented area
cat $1 >> $tmpfile
echo "
<!--" >> $tmpfile
gpg --digest-algo SHA256 --default-key BF15828F  --clearsign $tmpfile #Because im signing with non-default key
echo "<!--" > $1
cat "$tmpfile.asc" >> $1
echo "-->" >> $1
rm $tmpfile
rm "$tmpfile.asc"

Usage : ./signhtml.sh /path/to/file.html . Obviously remove or adjust the --default-key BF15828F portion. This overwrites the existing html file without taking a backup… YOLO.

Verify the contents using:-

$ gpg --recv-keys BF15828F
gpg: requesting key BF15828F from hkp server keys.gnupg.net
gpg: key BF15828F: public key "Sajal Kayan <sajal83@gmail.com>" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
$ curl http://www.sajalkayan.com/ | gpg --verify
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29916  100 29916    0     0  1577k      0 --:--:-- --:--:-- --:--:-- 1623k
gpg: Signature made Fri 05 Feb 2016 02:44:27 PM UTC using RSA key ID BF15828F
gpg: Good signature from "Sajal Kayan <sajal83@gmail.com>"
gpg:                 aka "<sajal@turbobytes.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A668 BBFE 438C BEDA 7BB6  3925 3964 90AC BF15 828F
$

Now if your ISP is messing with the html body, the signature will not match. There is one caveat, if the injected contents is before or after the signed portion.

Lets take this html payload

<html>
<head>
    <title>Hello World</title>
</head>
<body>
Hello World
</body>
</html>

After signing it becomes :-

<!--
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

https://keybase.io/sajal -->
<html>
<head>
	<title>Hello World</title>
</head>
<body>
Hello World
</body>
</html>

<!--
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJWtil3AAoJEDlkkKy/FYKPuawQAIAZ72rb/B+W1d1XGkXfxE0m
eUr3lE39FCLGJbroQyWzJFl3384EpDo9nSToN8y0ln6h1nohgykAma9YFAHMrRb1
0+f8FvUzAMnyaT1xSVmke6zgA2/X0sPIhMDHTUDCgvSFOtk21RgVySpTJ584013u
foroZxzloZz6vFAFh/OQhtoyaA8Br3dk0YleO5N/ApPsZZjC9hSiyfHh/kJr+71y
d6Y+EWyR+XQDpjQtyZtQIu34zJYUCTn+0iWPTLmB3pn1jgWg7dfxqJq5XNHRE2Sj
79vRQmQhzps3IYaWU+Ogauf59mVcgGV3GytL/xt5o9PsVi6g+Yo4l2xF8oC2EKwM
JYqdsvWtFAk7guxf8v9kP5aUcuA0TnW/H9VVH6oWqHgQKqWYkOcMMrZDGr3aLRiV
8mDQPP/iZgTlhI0s5Yrn7jBubHbM19qdqADHp+7Jr72qzQzDa0Qiblk4nGyEiYIg
xGGbRfHfKThVajhx6y3ggdEP6DTHTcCNLItS7OQY3pocXszCGYd1IuLRPFjKoaGh
td18ycpL2Dhq/HyOjIDcvyzliyU8YcqHFBQaWIhBw03hNFlgjUOedI/glU9IT6hY
nPdDtji6rkfL55KbZrCbYQL6Ai4LQxLOJTCrzr8tu8tEfzK1lry9ztDgmn4R9XDv
pssWJDftlfXtU4ncmdF2
=MhMl
-----END PGP SIGNATURE-----
-->

Anything injected before the first  will not be validated, but that portion easy to visually inspect, or write some code to check if something has been added or not.

Example of malicious stuff included which passes gpg verification.

<script>
alert("all your head is belong to us");
</script>
<!--
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

https://keybase.io/sajal -->
<html>
<head>
	<title>Hello World</title>
</head>
<body>
Hello World
</body>
</html>

<!--
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBCAAGBQJWtil3AAoJEDlkkKy/FYKPuawQAIAZ72rb/B+W1d1XGkXfxE0m
eUr3lE39FCLGJbroQyWzJFl3384EpDo9nSToN8y0ln6h1nohgykAma9YFAHMrRb1
0+f8FvUzAMnyaT1xSVmke6zgA2/X0sPIhMDHTUDCgvSFOtk21RgVySpTJ584013u
foroZxzloZz6vFAFh/OQhtoyaA8Br3dk0YleO5N/ApPsZZjC9hSiyfHh/kJr+71y
d6Y+EWyR+XQDpjQtyZtQIu34zJYUCTn+0iWPTLmB3pn1jgWg7dfxqJq5XNHRE2Sj
79vRQmQhzps3IYaWU+Ogauf59mVcgGV3GytL/xt5o9PsVi6g+Yo4l2xF8oC2EKwM
JYqdsvWtFAk7guxf8v9kP5aUcuA0TnW/H9VVH6oWqHgQKqWYkOcMMrZDGr3aLRiV
8mDQPP/iZgTlhI0s5Yrn7jBubHbM19qdqADHp+7Jr72qzQzDa0Qiblk4nGyEiYIg
xGGbRfHfKThVajhx6y3ggdEP6DTHTcCNLItS7OQY3pocXszCGYd1IuLRPFjKoaGh
td18ycpL2Dhq/HyOjIDcvyzliyU8YcqHFBQaWIhBw03hNFlgjUOedI/glU9IT6hY
nPdDtji6rkfL55KbZrCbYQL6Ai4LQxLOJTCrzr8tu8tEfzK1lry9ztDgmn4R9XDv
pssWJDftlfXtU4ncmdF2
=MhMl
-----END PGP SIGNATURE-----
-->
<script>
alert("all your base is belong to us");
</script>

Here is a complete verification script that includes test for tampering portions not covered by PGP. – verifyhtml.sh. Warning awk black magic ahead – copy/pasted snippets from the interwebs.

#!/bin/sh

tmpfile=$(mktemp)
#Download the file
curl -o $tmpfile $1
# checking if -----END PGP SIGNATURE----- armor is present
# Need to check for this cause gpg still validates without it.
result=`awk 'BEGIN{ found=0} /-----END PGP SIGNATURE-----/{found=1}  {if (found) print }' $tmpfile | wc -c`
if [ "$result" -eq "0" ]; then
	echo "ABORTING: -----END PGP SIGNATURE----- has been removed!!!"
	exit 1
else
	echo "-----END PGP SIGNATURE----- check passed"
fi

#Check if end has been tampered
result=`awk 'BEGIN{ found=0} /-----END PGP SIGNATURE-----/{found=1}  {if (found) print }' $tmpfile | sed -e 's/-----END PGP SIGNATURE-----//g' | sed -e 's/-->//g' | tr -d "[:space:]" | wc -c`
if [ "$result" -eq "0" ]; then
	echo "End not tampered"
else
	echo "ABORTING: Tampered at the end!!!"
	exit 1
fi
#Check if beginning has been tampered with.
result=`sed -n '1,/BEGIN PGP SIGNED MESSAGE/p' $tmpfile | sed -e 's/-----BEGIN PGP SIGNED MESSAGE-----//g' | sed -e 's/<!--//g' | tr -d "[:space:]" | wc -c`
if [ "$result" -eq "0" ]; then
	echo "Begining not tampered"
else
	echo "ABORTING: Tampered at the beginning!!!"
	exit 1
fi
echo "checking signature"
gpg --verify $tmpfile
rm $tmpfile #Perhaps keep it for debugging purpose if gpg fails to verify.

Usage: ./verifyhtml.sh http://www.sajalkayan.com/ .

Warning: I haven’t tested this enough. This is not rock solid, i.e. the interceptor could edit the payload and sign it using another key, which could pass validations…

Similar signing techniques could be used easily for .js and .css files. In my opinion popular third party embedded javascript files should be signed using PGP and users should verify and report if any discrepancy is found.

PS: I am aware of the irony that I am talking about integrity of web payloads while not serving this blog over https. It is currently hosted using github pages which does not support HTTPS for custom domains. I will perhaps move it elsewhere to play with Let’s Encrypt and http/2.

Building binary executables for Android in Go

Tue, 26 Jan 2016 16:00:00 +0000

I have a use-case to run an external Go binary from within an Android app. By external i mean something that was not bundled inside the APK, but rather (in my case) downloaded from the Internet. The reason for not bundling in-APK is that I need to be able to auto-upgrade the binary without upgrading the APK. APK updates either require user-action or play store or root - all three are not possible for my use-case. I spent an entire day on the issue(android n00b here), which turned out to be a very simple ~~problem~~ solution.

First thing I tried was building normal linux/arm binaries that I use for normal arm devices.

sajal@sajal-lappy:~$ GOARCH="arm" go build /path/to/filewithmain.go

The generated binary works in general… until you try to access the net… all socket communications are blocked. After trying few random things, I realized its due to me not using the NDK to build it… The binary needs to be built with android/arm target. gomobile to the rescue.

gomobile allows us to either generate an .aar library or an .apk, both are not applicable here. Solution - use the toolchain gomobile installed but compile code by hand.

My compile command :-

sajal@sajal-lappy:~$ GOMOBILE="/home/sajal/go/pkg/gomobile" GOOS=android GOARCH=arm CC=$GOMOBILE/android-ndk-r10e/arm/bin/arm-linux-androideabi-gcc CXX=$GOMOBILE/android-ndk-r10e/arm/bin/arm-linux-androideabi-g++ CGO_ENABLED=1 GOARM=7 go build -p=8 -pkgdir=$GOMOBILE/pkg_android_arm -tags="" -ldflags="-extldflags=-pie" -o minion -x ~/go/src/github.com/turbobytes/pulse/minion.go
sajal@sajal-lappy:~$ file minion
minion: ELF 32-bit LSB  shared object, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), not stripped
sajal@sajal-lappy:~$ ls -lh minion
-rwxr-xr-x 1 sajal sajal 9.3M Jan 26 16:52 minion
sajal@sajal-lappy:~$

It took me a while to figure out the -ldflags="-extldflags=-pie" portion, without it my phone complains about binary not being in PIE format.

Now need to wait for android/368 or android/amd64 support in gomobile so I can play with it in the emulator instead of real device…

PS: I know what I am doing is probably an anti-pattern, but this is not a normal end user app. It would run on devices dedicated to this and I will sign and validate downloads.

PSS: I figured this out by mucking around with gomobile using the -x option.

Writing Android apps with Go bindings

Tue, 28 Jul 2015 15:50:09 +0000

tl;dr version

One of the new features of upcoming Go 1.5 release is the awesome tooling around building Android (and iOS) apps using Go code.

Recently after playing with Ivy - an app made by Go mobile team - I wanted to give it a go (pun intended). Ivy is supposedly written in pure Go, unfortunately the source of it’s mobile implementation is not open yet.

Since I don’t know Java, my original goal was to try and make the whole app in Go, but thats not really possible. So I ended up using Java for UI and here is the result. No iOS version yet because i don’t own a mac.

First lets get our tooling in order… (Instructions for Ubuntu - adapt for your distro/OS)

I downloaded and extracted go1.5beta2 to /home/sajal/gobeta because I didnt want to mess up my existing 1.4.2 environment.

Now install gomobile

export PATH=/home/sajal/gobeta/bin:$PATH
go get golang.org/x/mobile/cmd/gomobile
gomobile init  #This expects go 1.5 to be in PATH which we solved in the first step.

Then I played with some example apps.

export PATH=/home/sajal/gobeta/bin:$PATH
cd $GOPATH/src/golang.org/x/mobile/example/basic/
gomobile build .
adb install basic.apk

Works on mobile… the only problem is that this is OpenGL stuff which I am totally clueless about.. So until the source of Ivy is released, I’d have to make the UI in Java…

First project DNS debugger. Much of the Go code was written for TurboBytes Pulse already… Lets reuse this….

export PATH=/home/sajal/gobeta/bin:$PATH
$ ANDROID_HOME="/home/sajal/Android/Sdk/" gomobile bind github.com/turbobytes/pulse/utils
panic: unsupported seqType: interface{} / *types.Interface

goroutine 1 [running]:
golang.org/x/mobile/bind.seqType(0x7ff3da1cd498, 0xc8261c8e60, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/bind/seq.go:72 +0xaf8
golang.org/x/mobile/bind.(*goGen).genStruct(0xc82ab31730, 0xc8261ba7d0, 0xc8261c8dc0)
    /home/sajal/go/src/golang.org/x/mobile/bind/gengo.go:185 +0xfd4
golang.org/x/mobile/bind.(*goGen).gen(0xc82ab31730, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/bind/gengo.go:424 +0x98b
golang.org/x/mobile/bind.GenGo(0x7ff3da1cd220, 0xc82ab66338, 0xc82015f780, 0xc8201ea780, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/bind/bind.go:47 +0x195
main.(*binder).GenGo.func1(0x7ff3da1cd220, 0xc82ab66338, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/bind.go:158 +0x4d
main.writeFile(0xc82ab88740, 0x35, 0xc82ab31910, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/bind.go:211 +0x35b
main.(*binder).GenGo(0xc82ab6d8c0, 0xc8201526a0, 0x1c, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/bind.go:160 +0x414
main.goAndroidBind(0xc820176000, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/bind_androidapp.go:31 +0x12d
main.runBind(0xc8cba0, 0x0, 0x0)
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/bind.go:89 +0x26c
main.main()
    /home/sajal/go/src/golang.org/x/mobile/cmd/gomobile/main.go:63 +0x495

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/sajal/gobeta/src/runtime/asm_amd64.s:1696 +0x1
sajal@sajal-lappy:/tmp$

Woops… any package exporting interface{} (and uint16 which the dns library needs) types cant be used by gomobile bind ….FAIL

Solution: I wrote package github.com/turbobytes/pulse/digdroid with a proxy function that returns a struct with only strings. No interfaces.

package digdroid
type DNSResult struct {
    Err    string
    Output string
    Rtt    string
}

func RunDNS(host, target, qtypestr string, norec bool) *DNSResult

$ ANDROID_HOME="/home/sajal/Android/Sdk/" gomobile bind  github.com/turbobytes/pulse/digdroid
$ ls -lh
total 2.8M
-rw-rw-r-- 1 sajal sajal 2.8M Jul 28 22:04 digdroid.aar
$ file digdroid.aar 
digdroid.aar: Zip archive data, at least v2.0 to extract
$ unzip -l digdroid.aar
Archive:  digdroid.aar
  Length      Date    Time    Name
---------  ---------- -----   ----
       99  1980-00-00 00:00   AndroidManifest.xml
       25  1980-00-00 00:00   proguard.txt
     9009  1980-00-00 00:00   classes.jar
  9395848  1980-00-00 00:00   jni/armeabi-v7a/libgojni.so
        0  1980-00-00 00:00   R.txt
        0  1980-00-00 00:00   res/
---------                     -------
  9404981                     6 files
$

This results in creation of a file digdroid.aar which can be used as dependency in Android Studio. jni/armeabi-v7a/libgojni.so inside digdroid.aar is the actual Go library.

Next we move to Java territory.

I installed Android Studio, setup a new project, made basic UI and an Activity.

Include digdroid.aar into the project. Instructions from the docs:-

For example, in Android Studio (1.2+), an AAR file can be imported using the module import wizard (File > New > New Module > Import .JAR or .AAR package), and setting it as a new dependency (File > Project Structure > Dependencies). This requires ‘javac’ (version 1.7+) and Android SDK (API level 9 or newer) to build the library for Android. The environment variable ANDROID_HOME must be set to the path to Android SDK.

Once thats done, I could access the Go library from anywhere simply by importing go.digdroid.Digdroid

Screenshot of steps involved in adding digdroid.aar into Android Studio : http://imgur.com/a/dEewm

Example usage of above Go code from Java

Digdroid.DNSResult result = Digdroid.RunDNS("www.example.com.", "8.8.8.8:53", "A", false);
String output = result.getOutput();
String rtt = result.getRtt();
String err = result.getErr();

gomobile bind created the getters and setters for free…

One downside is gomobile bind only created binary for armv7. So the project no longer works on the emulator which is x86. But almost all android devices are arm, so its not really a big issue.

To update digdroid.aar simply build a new one and replace the digdroid.aar file within the Android Studio source tree.

One thing… For some reason the built apk was including extra permissions that I didn’t really need. Solution declare those extra permissions in the manifest in a special manner so it gets removed when being built.

<uses-permission
    android:name="android.permission.WRITE_EXTERNAL_STORAGE"
    android:maxSdkVersion="18"
    tools:node="remove"/>
<uses-permission android:name="android.permission.READ_PHONE_STATE" tools:node="remove"/>
<uses-permission
    android:name="android.permission.READ_EXTERNAL_STORAGE"
    android:maxSdkVersion="18"
    tools:node="remove"/>

Since I added these with tools:node="remove" the builder will remove it from the apk. If I did not do that then the builder would add them into the final apk for some reason… Nobody asked for these permissions and they are not relevant to the code.

PS: I understand there is a cost associated with jumping language boundaries. This is just a fun little project. I am desperately waiting for some easier way to write apps completely in Go without bindings. I am even willing to dabble with QT stuff (or similar) if someone can show me how to build it in Go for mobile. Besides, for networked functions, few microseconds cost for jumping languages is negligible compared to the cost of making the actual network requests which can be 10s of milliseconds or more.

Play store listing: https://play.google.com/store/apps/details?id=com.turbobytes.dig
Android Project Source : https://github.com/sajal/digdroid
Go package : http://godoc.org/github.com/turbobytes/pulse/digdroid
Awesomest DNS library : https://github.com/miekg/dns
TurboBytes Pulse : https://pulse.turbobytes.com/

http/2 is stricter than http/1.1

Sun, 15 Mar 2015 20:43:00 +0000

I am starting to read the draft http/2 spec and realized its quite strict and leaves little to interpreter’s imagination.

Here are the wordcounts for various specs for following words : MUST, REQUIRED, SHALL, SHOULD, RECOMMENDED, MAY, OPTIONAL

RFC	Total	MUST	SHOULD	MAY	SHALL	REQUIRED	RECOMMENDED	OPTIONAL
rfc2616 (http/1.1)	799	348 (43.55%)	256 (32.04%)	180 (22.53%)	2	5	1	7
rfc723[0-5] (superseeds rfc2616)	668	344 (51.50%)	163 (24.40%)	126 (18.86%)	12	6	7	10
http2-17 (http/2)	296	219 (73.99%)	31 (10.47%)	41 (13.65%)	2	1	1	1

To be fair… http/2 spec does not divulge into header behavior, caching, etc. SHALL, REQUIRED, RECOMMENDED and OPTIONAL were only found in the section defining the keywords.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].

I did not include HTTP/1 for this comparison because the spec does not seem to use upper cased keywords.

Quick and dirty python script to do the word counts.

import re
from httplib2 import Http
from collections import defaultdict

h = Http()

WORDS = ["MUST", "REQUIRED", "SHALL",
   "SHOULD", "RECOMMENDED", "MAY", "OPTIONAL"]

def get_counts(string, counts):
    for w in re.split('\W', string):
        if w:
            for W in WORDS:
                if W == w:
                    counts[w] += 1
    return counts


def count_url(url, counts):
    r,c = h.request(url)
    return get_counts(c, counts)

def summarize(counts):
    total = 0
    for k in counts.keys():
        total += counts[k]
    counts = dict(counts)
    counts["total"] = total
    counts["mustpct"] = counts["MUST"] * 100.0 / total
    counts["shouldpct"] = counts["SHOULD"] * 100.0 / total
    counts["maypct"] = counts["MAY"] * 100.0 / total
    return counts 


if __name__ == "__main__":
    print "rfc2616" , summarize(count_url("http://tools.ietf.org/html/rfc2616", defaultdict(int)))
    counts = count_url("https://tools.ietf.org/html/rfc7230", defaultdict(int))
    counts = count_url("https://tools.ietf.org/html/rfc7231", counts)
    counts = count_url("https://tools.ietf.org/html/rfc7232", counts)
    counts = count_url("https://tools.ietf.org/html/rfc7233", counts)
    counts = count_url("https://tools.ietf.org/html/rfc7234", counts)
    counts = count_url("https://tools.ietf.org/html/rfc7235", counts)
    print "rfc723[0-5]", summarize(counts)
    print "draft-ietf-httpbis-http2-17", summarize(count_url("https://tools.ietf.org/id/draft-ietf-httpbis-http2-17.txt", defaultdict(int)))

Running Go programs on $15 device - Beyond Hello World

Wed, 25 Feb 2015 18:14:00 +0000

I recently purchased a WT1520 router for $15 from Aliexpress to play with. I have a project in mind which would require few nodes running my custom Go code spread out throughout the world. A Raspberry Pi (almost $40 if you include SD card, etc) fits perfectly for my purpose, but I am looking to be cheap. Not to be dissing on the pi, its awesome and LOT more powerful than the WT1520, I’m just trying to find the cheapest device for my purpose.

Raspberry Pi ($35+) and WT1520 ($15 shipped) doing the same thing

Having no experience with OpenWrt, this blog post (sidenote: our blogs look similar) helped a lot to get Hello World running.

My Build Steps

I couldn’t use the gccgo fork directly because support for my architecture was added at a later stage, so I had to clone the upstream master and patch it.

$ git clone git://git.openwrt.org/openwrt.git
$ cd openwrt
$ curl https://github.com/GeertJohan/openwrt-go/compare/add-gccgo-and-libgo.diff | patch -p1
$ make menuconfig
$ make -j8

My resulting config : https://gist.github.com/sajal/f509183ac691a32e6065 Ive removed usb and wifi related things to keep the image small. It seems eglibc uses lot more space.

This builds gccgo 4.8.3 (Go 1.1.2 implementation). gcc 4.9.x is also available in menuconfig but build fails. Even then Go 1.2 is still ancient.

Building hello world is simple

$ export PATH=/home/sajal/src/openwrt/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_eglibc-2.19/bin:$PATH
$ alias gccgo='mipsel-openwrt-linux-gccgo -Wl,-R,/home/sajal/src/openwrt/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_eglibc-2.19/lib/gcc/mipsel-openwrt-linux-gnu/4.8.3 -L /home/sajal/src/openwrt/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_eglibc-2.19/lib'
$ gccgo -o hello ~/hello.go -static-libgo

^ resulting in 2.6 MB binary…

So far so good… But my real code is not so simple. It is a file with main, which imports another package which imports another package.

In the following example, the project in question is a very rough draft, and the code is not public at the moment. Sorry.

I have a file called minion.go which id like to build.

Lets try to build it the same way as before.

$ gccgo -o minion minion.go -static-libgo
minion.go:9:38: error: import file 'github.com/turbobytes/dnsdebug/utils' not found
  "github.com/turbobytes/dnsdebug/utils"
                                      ^
minion.go:93:18: error: reference to undefined name 'dnsdebug'
  resolver := new(dnsdebug.Resolver)
                  ^
minion.go:93:26: error: expected type
  resolver := new(dnsdebug.Resolver)
                          ^
minion.go:101:10: error: reference to undefined name 'dnsdebug'
   cfg := dnsdebug.GetTLSConfig(caFile, certificateFile, privateKeyFile)
          ^
$

Hmm. gccgo does not resolve packages, and we cant make the Go provided toolchain to build using our MIPS gccgo… so lets see what the Go toolchain does when using normal gccgo.

sajal@sajal-lappy:~/go/src/github.com/turbobytes/dnsdebug$ go build -x -compiler=gccgo minion.go 
WORK=/tmp/go-build071420589
mkdir -p $WORK/github.com/miekg/dns/_obj/
mkdir -p $WORK/github.com/miekg/
cd /home/sajal/go/src/github.com/miekg/dns
gccgo -I $WORK -c -g -m64 -fgo-pkgpath=github.com/miekg/dns -fgo-relative-import-path=_/home/sajal/go/src/github.com/miekg/dns -o $WORK/github.com/miekg/dns/_obj/dns.o ./client.go ./clientconfig.go ./defaults.go ./dns.go ./dnssec.go ./edns.go ./format.go ./keygen.go ./kscan.go ./labels.go ./msg.go ./nsecx.go ./privaterr.go ./rawmsg.go ./scanner.go ./server.go ./sig0.go ./singleinflight.go ./tlsa.go ./tsig.go ./types.go ./udp.go ./udp_linux.go ./update.go ./xfr.go ./zgenerate.go ./zscan.go ./zscan_rr.go
ar cru $WORK/github.com/miekg/libdns.a $WORK/github.com/miekg/dns/_obj/dns.o
mkdir -p $WORK/github.com/turbobytes/dnsdebug/utils/_obj/
mkdir -p $WORK/github.com/turbobytes/dnsdebug/
cd /home/sajal/go/src/github.com/turbobytes/dnsdebug/utils
gccgo -I $WORK -I /home/sajal/go/pkg/gccgo_linux_amd64 -c -g -m64 -fgo-pkgpath=github.com/turbobytes/dnsdebug/utils -fgo-relative-import-path=_/home/sajal/go/src/github.com/turbobytes/dnsdebug/utils -o $WORK/github.com/turbobytes/dnsdebug/utils/_obj/dnsdebug.o ./rpc.go ./tls.go
ar cru $WORK/github.com/turbobytes/dnsdebug/libutils.a $WORK/github.com/turbobytes/dnsdebug/utils/_obj/dnsdebug.o
mkdir -p $WORK/command-line-arguments/_obj/
cd /home/sajal/go/src/github.com/turbobytes/dnsdebug
gccgo -I $WORK -I /home/sajal/go/pkg/gccgo_linux_amd64 -c -g -m64 -fgo-relative-import-path=_/home/sajal/go/src/github.com/turbobytes/dnsdebug -o $WORK/command-line-arguments/_obj/main.o ./minion.go
ar cru $WORK/libcommand-line-arguments.a $WORK/command-line-arguments/_obj/main.o
cd .
gccgo -o minion $WORK/command-line-arguments/_obj/main.o -Wl,-( -m64 $WORK/github.com/turbobytes/dnsdebug/libutils.a $WORK/github.com/miekg/libdns.a -lpthread -Wl,-E -Wl,-)
sajal@sajal-lappy:~/go/src/github.com/turbobytes/dnsdebug$

Using the hints from there… This is what I translated it to after a lot of trial and error.

WORK=`mktemp -d`
mkdir -p $WORK/obj
mkdir -p $WORK/github.com/miekg/
cd /home/sajal/go/src/github.com/miekg/dns
gccgo -I $WORK -c -g -fgo-pkgpath=github.com/miekg/dns -fgo-relative-import-path=_/home/sajal/go/src/github.com/miekg/dns -o $WORK/obj/dns.o ./client.go ./clientconfig.go ./defaults.go ./dns.go ./dnssec.go ./edns.go ./format.go ./keygen.go ./kscan.go ./labels.go ./msg.go ./nsecx.go ./privaterr.go ./rawmsg.go ./scanner.go ./server.go ./singleinflight.go ./tlsa.go ./tsig.go ./types.go ./udp.go ./udp_linux.go ./update.go ./xfr.go ./zgenerate.go ./zscan.go ./zscan_rr.go
mipsel-openwrt-linux-gnu-objcopy -j .go_export $WORK/obj/dns.o $WORK/github.com/miekg/dns.gox
mkdir -p $WORK/github.com/turbobytes/dnsdebug/
cd /home/sajal/go/src/github.com/turbobytes/dnsdebug/utils
gccgo -I $WORK  -c -g -fgo-pkgpath=github.com/turbobytes/dnsdebug/utils -fgo-relative-import-path=_/home/sajal/go/src/github.com/turbobytes/dnsdebug/utils -o $WORK/obj/dnsdebug.o ./rpc.go ./tls.go
mipsel-openwrt-linux-gnu-objcopy -j .go_export $WORK/obj/dnsdebug.o $WORK/github.com/turbobytes/dnsdebug/utils.gox
cd /home/sajal/go/src/github.com/turbobytes/dnsdebug
gccgo -I $WORK  -c -g  -o $WORK/obj/minion.o ./minion.go
gccgo -o minion $WORK/obj/minion.o $WORK/obj/dns.o $WORK/obj/dnsdebug.o -static-libgo

It took me a while to figure out that I needed to export the .gox files to be able to build code that depended on other packages.

Note: I had to adjust code a bit to support the ancient Go implementation… Specifically the TLS implementation and cipher suits.

$ file minion
minion: ELF 32-bit LSB  executable, MIPS, MIPS32 rel2 version 1, dynamically linked (uses shared libs), for GNU/Linux 2.6.16, not stripped
$ ls -lh minion
-rwxrwxr-x 1 sajal sajal 9.3M Feb 26 00:25 minion

OpenWrt creates a 14MB tempfs, so this barely fits into /tmp of the WT1520, but works as expected. Striping it makes it unusable… The binary is too big to persist on the device, but it could be programmed to download binary fresh from some server on reboot. Not entirely sure I’d use this approach for production.

I think this is the cheapest off the shelf device that a Go program can run on productively.

Dear Gophers: Please implement MIPS architecture within the gc toolchain so that I can build apps for these cheap devices as easily as for ARM.

Next up, I will try to get my hands on pogoplug . Amazon has it for $13.69, but after including shipping and taxes it comes out to $51.30. And it doesn’t seem to be something that will always be readily available at such low prices.

Using MultiPath TCP to enhance home networks

Sat, 22 Nov 2014 12:46:11 +0000

Over the last few months I’ve been playing with MultiPath TCP and in this post I will show how I use it to leverage my humble True ADSL line at home.

For performance and security reasons, I tunnel all my traffic thru a VPN. This is not necessarily to circumvent censorship, but to circumvent the evil transparent proxies my ISP puts in middle. The total bandwidth available is ~10 mbps down / ~1 mbps up.

Introduction to MultiPath TCP

MultiPath TCP is an interesting effort to use multiple interfaces/networks for any single TCP connection. A Linux kernel implementation is being developed at multipath-tcp.org. Its main use cases are for mobile (transition between Wi-Fi and 3G) and datacenters. I exploit it to get better Internet browsing experience.

Old way

SSH tunnel to EC2 instance in Singapore. Browser configured to use this tunnel as proxy
SSH tunnel to EC2 instance in us-east (for accessing geo-blocked services)

Main drawback : If my ISP has issues talking to AWS, then I’m totally screwed. This happened a month or so ago where most links coming into True was severely limited, however link from Digital Ocean to True was healthy. I had to manually change my tunnels to a $5 Digital Ocean instance.

New way

Note: This is a constantly evolving setup as I find new things to play with.

Infrastructure involved :-

PC Engines APU system board - Replaces router. All magic happens here. gateway
ADSL modem in bridge mode.
EC2 instance in Singapore - The main proxy endpoint. Runs shadowsocks server over MPTCP kernel. destination, jumpbox
EC2 instance in us-west - The proxy endpoint for US geo blocked traffic. Runs shadowsocks server over MPTCP kernel. destination
Digital Ocean instance in Singapore - An alternate path to reach the EC2 instance(s) jumpbox
VPS in CAT datacenter in Thailand - Another alternate path. All Thai ISPs usually have good connectivity to CAT. jumpbox
Android phone - With Dtac 3G for extra boost when needed. USB tethering. Bandwidth fluctuates a lot. I typically use it to get a boost in my upload bandwidth which is generally 100 kbps to 8 mbps.

All TCP Traffic is intercepted by the APU using iptables, diverted to redsocks, which sends it to the shadowsocks client, which sends it to the shadowsocks server running in EC2 Singapore. This socks connection has several ways to communicate with the EC2 instance.

APU <--> True ADSL Directly <--> EC2  
APU <--> True ADSL Directly over OpenVPN/UDP <--> EC2  
APU <--> True ADSL <--> via CAT VPS over OpenVPN/UDP <--> EC2  
APU <--> True ADSL <--> via DO Singapore over OpenVPN/UDP <--> EC2  
APU <--> Dtac 3G Directly <--> EC2 (Optional/ondemand)

Now I have 5 possible paths. MPTCP kernel creates a TCP connection over each available path and bonds them together and exposes it as a single TCP connection to the application. Packets are sent over paths that currently have the lowest delay. Now my available bandwidth is not impacted by congestion over some of these paths. All paths need to be congested for me to have a bad day… Also some path might have good uplink, some might have good downlink, with MPTCP you mix the best of both…

Example bmon stats when downloading a large file (I removed irreverent interfaces.)

  #   Interface                RX Rate         RX #     TX Rate         TX #
─────────────────────────────────────────────────────────────────────────────
xxx (source: local)
  0   tun1                     621.28KiB        628      38.82KiB        636
  3   tun3                     200.22KiB        198       9.42KiB        149
  5   ppp0                       1.07MiB       1018     119.42KiB        980
  9   tun0                      90.06KiB         90       5.94KiB         97

Configurations

Jumpbox

Jumpbox is pretty basic setup. It’s role is to provide additional gateways which MPTCP uses to build additional paths.

OpenVPN server configured normally. Set to not redirect default gateway. In my current setup I need to ensure that the server assigns the same IP to my client. This is not really that crucial, but it keeps things simple. Its important to configure each jumpbox to use a different IP range.

net.ipv4.ip_forward needs to be set to 1 to allow forwarding. In fact almost all boxes in the setup need this.

iptables rules needed :-

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t filter -A FORWARD -i tun0 -j ACCEPT
iptables -t filter -A FORWARD -o tun0 -j ACCEPT

Replace the tun0 and eth0 to suit your environment.

Destination

A destination server is remote end of our socks tunnel. It’s job is to service the socks connections patching them to the real destination.

This needs to run a MultiPath TCP kernel. On EC2 it is pretty simple. Launch an Ubuntu 14.04 instance with a pv-grub AKI. Then follow the apt-repository installation method. And ensure the grub loads the MPTCP kernel as its first choice.

Next we also need shadowsocks server running. RTFM its pretty simple. Before using shadowsocks I was using a simple ssh -D tunnel, but I found it to be inefficient. Often times one large transfer would make all other TCP streams stuck. Perhaps this has something to do with the fact that with SSH everything is happening over a single TCP stream whereas shadowsocks makes a new socks connection dedicated to each TCP connection.

Gateway

The gateway is the most complicated component. Running stock Debian wheezy with MPTCP kernel installed via their apt repository. A lot of services run here. I will not elaborate on some of them.

dbcpd - Assign LAN users with IP

bind - For DNS recursion. Since we tunnel most traffic to Singapore, I also set bind to send DNS queries thru OpenVPN.

iptables - I use iptables to do the NAT. NAT all UDP packets to OpenVPN. Send all outgoing TCP connections to redsocks.

# Generated by iptables-save v1.4.14 on Sat Nov 22 00:37:10 2014
*nat
:PREROUTING ACCEPT [378881:57485495]
:INPUT ACCEPT [210208:17266788]
:OUTPUT ACCEPT [4099955:310913862]
:POSTROUTING ACCEPT [3239510:252587265]
:REDSOCKS - [0:0]
-A PREROUTING -i br0 -p tcp -j REDSOCKS
-A PREROUTING -i br0 -j REDSOCKS
-A POSTROUTING -o tun1 -j MASQUERADE
-A POSTROUTING -o tun0 -j MASQUERADE
-A POSTROUTING -o tun2 -j MASQUERADE
-A POSTROUTING -o eth0 -j MASQUERADE
-A REDSOCKS -d 0.0.0.0/8 -j RETURN
-A REDSOCKS -d 10.0.0.0/8 -j RETURN
-A REDSOCKS -d 127.0.0.0/8 -j RETURN
-A REDSOCKS -d 169.254.0.0/16 -j RETURN
-A REDSOCKS -d 172.16.0.0/12 -j RETURN
-A REDSOCKS -d 192.168.0.0/16 -j RETURN
-A REDSOCKS -d 224.0.0.0/4 -j RETURN
-A REDSOCKS -d 240.0.0.0/4 -j RETURN
-A REDSOCKS -d a.b.c.d/32 -j RETURN
-A REDSOCKS -d e.f.g.h/32 -j RETURN
-A REDSOCKS -d i.j.k.l/32 -j RETURN
-A REDSOCKS -d m.n.o.p/32 -j RETURN
-A REDSOCKS -d q.r.s.t/32 -j RETURN
-A REDSOCKS -s 192.168.5.1/32 -j RETURN
-A REDSOCKS -s 192.168.5.1/32 -j RETURN
-A REDSOCKS -s 192.168.1.2/32 -j RETURN
-A REDSOCKS -s 192.168.5.32/27 -p tcp -j REDIRECT --to-ports 12345
COMMIT
# Completed on Sat Nov 22 00:37:10 2014
# Generated by iptables-save v1.4.14 on Sat Nov 22 00:37:10 2014
*filter
:INPUT ACCEPT [115657469:73738905421]
:FORWARD ACCEPT [64078:47442189]
:OUTPUT ACCEPT [122121802:63701527314]
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -o eth1 -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
-A FORWARD -o br0 -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-A FORWARD -o eth0 -j ACCEPT
COMMIT
# Completed on Sat Nov 22 00:37:10 2014
# Generated by iptables-save v1.4.14 on Sat Nov 22 00:37:10 2014
*mangle
:PREROUTING ACCEPT [118734314:74507536093]
:INPUT ACCEPT [115635709:73734306355]
:FORWARD ACCEPT [3100331:759352048]
:OUTPUT ACCEPT [122104929:63698976437]
:POSTROUTING ACCEPT [125198421:64456746251]
-A PREROUTING ! -d 192.168.5.0/24 -i br0 -j MARK --set-xmark 0x1/0xffffffff
-A PREROUTING -d 10.8.0.10/32 -i br0 -j MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 192.168.10.1/32 -i br0 -j MARK --set-xmark 0x2/0xffffffff
COMMIT
# Completed on Sat Nov 22 00:37:10 2014

Note: a.b.c.d, e.f.g.h, i.j.k.l, m.n.o.p and q.r.s.t are public internet ips that I don’t want redsocks to intercept.

Interfaces

br0 - LAN
tun[0-3] - Various Jumpboxes. OpenVPN tunnels.

Also, each interface is maintains its own routing tables using if-up scripts. For example this is what gets executed when one of the tunnels comes alive.

#!/bin/sh
ip rule add from 10.8.0.20 table 2 || true
ip route add 10.8.0.0/24 dev tun0 scope link table 2 || true
ip route add default via 10.8.0.21 dev tun0 table 2 || true
ip rule add fwmark 3 table 2 || true

The fwmark is added so if in future I want to pipe different traffic to go thru this interface I can set the corresponding iptables rule.

All local services are scoped to listen only on local interfaces to avoid random people connecting to local services.

redsocks - Accepts intercepted connections and pipes it off to shadowsocks client

shadowsocks client - sends all TCP connections to shadowsocks servers running in EC2 Singapore and US. By default intercepted traffic is sent to Singapore, however any application on any computer in the network could be set to explicitly use any of the available proxies.

wvdial - To dial the ADSL connection which is itself behind a Carrier-grade NAT. Sometimes the connection stops working while pppd things its still connected. Am ugly CRON script to test the network and flip it if needed.

#!/bin/bash

IP=`/sbin/ip route | grep -v default | grep ppp0 | cut -d " " -f 1`
COUNT=`echo $IP | wc -l`

echo $IP $COUNT
if [ $COUNT -eq 1 ] 
then
  ping -c 10 $IP > /dev/null
  if [ $? -eq 0 ]
  then
    echo "ppp0 is up"
  else
    echo "ppp0 is down"
    kill -SIGHUP `pgrep pppd`
    beep -l 25
  fi
else
  echo "ppp0 not found?!?!?!?1"
fi

The script above tries to ping the default gateway of the ppp0 interface to find out if it is really up. SIGHUP signals pppd to handup and redial. I don’t bother maintaining the pid of pppd because currently I use ppp0 exclusively for the ADSL modem.

Missing parts

There are some issues I am having that I need to sort out work-around for.

Default connection unstable. If the initial syn packet for ppp0 (my default interface) fails, then the connection cant be established. I need to look deeper into MPTCP docs to figure out how to make it such that if the initial TCP connection setup fails on ppp0 then make it try tun0, tun1 and so on.
Connection fairness. Sometimes if I am doing a big upload, everything else (like browsing websites) seems too slow. The upload is hogging all the available uplink, which is already too tiny. I have my suspicions on buffer bloat…
Higher uplink usage. When downloading something, I see ~10% upload traffic corresponding to it. This is lot higher than a simple setup. I need to investigate deeper whats causing it. Perhaps MPTCP or the socks setup or OpenVPN. The example bmon stats above show this as well 119.42KiB uplink while downloading @ 1.07MiB.
EC2 bandwidth is expensive. I would like to use Digital Ocean boxes for socks proxies. This is a little tricky since DO does not allow loading custom kernels. I need to figure out kexec to make this possible. Unsure if this way is stable…
Advanced routing. I would like to programatically decide which external IP should go thru which proxy. For example most Thai IPs I would like to go direct. Some American destinations should go thru the US proxy, rest thru Singapore proxy. Perhaps in future add an European proxy.. Currently the only way to use the American proxy is to explicitly configure a particular application to use socks proxy.
UDP tunneling. UDP traffic currently goes directly using a single OpenVPN session. There is no load-balancing being performed on it. I would like to switch to a different socks client/server. One that does UDP associate.

Future path enhancements

More paths can be added to get better throughput.

3G dongles from various providers.
The shitty Wi-Fi that your apartment/office provides.
More ADSL/Cable connections from diverse providers with different backbones.

Conclusion

MPTCP is a fantastic piece of technology. Hat-tip to everyone who contributed to it. The PC Engines ALU box is also awesome. Decent x86_64 box consuming only about 6 to 12W power.

In the future I will do a walk-thru type post on how to setup a Raspberry Pi as a one-arm gateway doing a subset of what I described above. The most challenging part is getting a MPTCP enabled kernel on the pi, which requires kernel patching and compiling. The throughput will likely be very limited because MPTCP has a higher CPU overhead than regular TCP.

Moved blog to github pages

Sun, 16 Nov 2014 18:43:03 +0000

After 2 years of neglecting this blog, I’ve moved away from wordpress and am hosting it on Github Pages.

The site is built using Hugo.

Source repo : https://github.com/sajal/sajalblog
Built pages : https://github.com/sajal/sajal.github.io

Some links, especially RSS URLs might be broken currently.

Disco + EC2 spot instance = WIN

Tue, 30 Oct 2012 13:13:59 +0000

tl;dr version : This is how I spent a Saturday evening.

<blink>Warning: If you like to write Java, stop reading now. Go back to using Hadoop. It's a much more mature project.</blink>

As a part of my job, I do a lot of number processing. Over the course of last few weeks, I shifted to doing most of it using MapReduce using Disco. Its a wonderful approach to processing big data where the time to process data is directly proportional to the amount of hardware you throw at it and the quantity of data. The amount of data to be processed can (in theory) be unlimited. While I don't do anything of Google scale, I deal with Small Big Data. My datasets for an individual job would probably not exceed 1 GB. I can currently afford to continue not use MapReduce, but as my data set grows, I would have to do distributed computing, so better start early.

Getting started with Disco

If you, like me, had given up on MapReduce in the past after trying to deal with administrating Hadoop, now is a great time to look into Disco. Installation is pretty easy. Follow the docs. Within 5 minutes I was writing Jobs in python to process data, would have been faster if I knew before-hand that SSH daemon should be listening on port 22.

Python for user scripts + Erlang for backend == match made in heaven

Enter disposable Disco

I made a set of python scripts to launch and manage Disco clusters on EC2 where there is no need for any data to be stored. In my usecase, the input is read from Amazon S3 and output goes back into S3.

There are some issues with running disco on EC2.

Must have ssh/keys setup such that Master can ssh into slaves.
Must have a file with erlang cookie with same contents on all slaves
Must inform master the hostnames of the slaves. FQDN or anything with a dot gets rejected
The default root directories have very limited storage space, usually 8GB

disposabledisco takes care of the above things and more. Everything needed to run the cluster is defined in a config file. First generate a sample config file.

python create_config.py > config.json

This creates a new file with some pre-populated values. For my case the config file looks like this(some info masked)

{
    "AWS_SECRET": "SNIPPED", 
    "ADDITIONAL_PACKAGES": [
        "git", 
        "libwww-perl", 
        "mongodb-clients", 
        "python-numpy", 
        "python-scipy", 
        "libzmq-dev", 
        "s3cmd", 
        "ntp", 
        "libguess1", 
        "python-dnspython", 
        "python-dateutil", 
        "pigz"
    ], 
    "SLAVE_MULTIPLIER": 1, 
    "PIP_REQUIREMENTS": [
        "iso8601",
        "pygeoip"
    ], 
    "MASTER_MULTIPLIER": 1, 
    "MGMT_KEY": "ssh-rsa SNIPPED\n", 
    "SECURITY_GROUPS": ["disco"], 
    "BASE_PACKAGES": [
        "python-pip", 
        "python-dev", 
        "lighttpd"
    ], 
    "TAG_KEY": "disposabledisco", 
    "NUM_SLAVES": 30, 
    "KEY_NAME": "SNIPPED", 
    "AWS_ACCESS": "SNIPPED", 
    "INSTANCE_TYPE": "c1.medium", 
    "AMI": "ami-6d3f9704", 
    "MAX_BID": "0.04",
    "POST_INIT": "echo \"[default]\naccess_key = SNIPPED\nsecret_key = SNIPPED\" > /tmp/s3cfg\ncd /tmp\ns3cmd -c /tmp/s3cfg get s3://SNIPPED/GeoIPASNum.dat.gz\ns3cmd -c /tmp/s3cfg get s3://SNIPPED/GeoIP.dat.gz\ns3cmd -c /tmp/s3cfg get s3://SNIPPED/GeoLiteCity.dat.gz\ns3cmd -c /tmp/s3cfg get s3://SNIPPED/GeoIPRegion.dat.gz\ngunzip *.gz\nchown disco:disco *.dat\n\n"
}

This tells disposabledisco that I want a cluster with 1 master and 30 slaces all of type c1.medium, and use ami-6d3f9704 as the starting point. It lists out the packages to be installed via apt-get and python dependencies to be installed using PIP. You can link to external tar, git repo, etc. Basically anything pip allows after pip install

The POST_INIT portion is bash script that runs as root after rest of the install. In my case I am downloading and uncompressing different GeoIP databases archived in a S3 bucket for use from within disco jobs.

Once the config file is ready run the following command many times. The output is fairly verbose.

python create_cluster.py config.json

Why many times? Cause there is no state stored in the system. All state is managed using EC2 tags. This is what the script does on each run

Check if master is running. If not request a spot instance for it (and kill any zombie slaves lying around from previous runs).
If master us up and running.
- Print the ssh command needed to setup port forwarding. After running the given ssh command you can see http://localhost:8090 on the browser to see disco's UI in all its glory.
- print the command to export DISCO_PROXY so you can create jobs locally
- Check inventory of slaves. A slave can have 3 statuses. 1) pending - spot instance requested. 2) running - the instance is running. 3) bootstrapped - slave is completely setup and can be added to master.
- If total number of slaves is less than NUM_SLAVES launch the remaining
- Try and bootstrap any running instances. If bootstrap was successful, change the EC2 tag.
Finally, update the master's disco config. Telling it hostnames of instances to use and number of workers.
???
Profit

Many steps involve EC2 provisioning spot instances, waiting for instance to get initialized, etc..

To help with shipping output to S3, I made some output classes for Disco

S3Output - Each key, value returned creates a new file in S3 with the key as S3 key and value as String thats dumped inside it. So, one key should be yielded only once from reduce.
S3LineOutput Similar to S3Output, but now it stores the output, and joins the output as one big file. has options for sorting, unique, etc.

Both these functions can be configured gzip the contents before uploading.

As far as input is concerned, I send it a list of signed S3 urls. (Sidenote: It seems disco cannot handle https inputs at the moment, so I use http). A sample job run might look like..

def get_urls():
    urls = []
    for k in bucket.list(prefix="processed"):
        if k.name.endswith("gz"):
            urls += [k.generate_url(3660).replace("https", "http")]
    return urls

MyExampleJob().run(
	input=get_urls(),
	params={
		"AWS_KEY": "SNIP",
		"AWS_SECRET": "SNIP",
		"BUCKET_NAME": "SNIP",
		"gzip": True
	},
	partitions=10,
	required_files=["s3lineoutput.py"],
	reduce_output_stream=s3_line_output_stream
	).wait()

Bonus - MagicList - Memory efficient way to store and process potentially infinite lists.

We used Disco to compute numbers for a series of blogposts on CDN Planet. For this analysis it was painful process for me to manually launch Disco clusters, which lead me to create the helper scripts.

Part 1 : Google DNS, OpenDNS and CDN performance
Part 2 : Which CDNs support edns-client-subnet?
Part 3 : Real-world CDN performance for Google DNS and OpenDNS users

Shameless plug

Turbobytes, multi-CDN made easy

Have your static content delivered by 6 global content delivery networks, not just 1. Turbobytes' platform closely monitors CDN performance and makes sure your content is always delivered by the fastest CDN, automatically.

4 reasons why I love my ISP

Mon, 28 Nov 2011 18:52:26 +0000

I've been using True ADSL for years, and I absolutely love their service (and especially the transparent proxy). Here are some of the reasons why :-

~~Censorship~~ Protecting me from bad stuff - Interwebs has a lot of "bad" things out there. My ISP takes good care of me by not letting me access things I am not supposed to see... Even sites not explicitly blocked by court-order. I don't know what I would do without them. My head would probably explode if I saw porn, and my feelings would get hurt if I came across certain types of political messages..

~~Transparent proxy~~ Web slow down machine - Buddha teaches us "The greatest prayer is patience". A very special offering of True ISP is that it reminds us to be patient in this fast-paced world. True's Web slow down machine sits between users' connection to other servers. One of its features is to slow down access... It employs several brilliant methods to accomplish this :-

Not keeping connections alive - This is the most important factor in slowing down pageloads. True does not keep connections to remote hosts alive, thus making sure you have to establish a fresh connection with each request to a server overseas. No matter how small the file, a request to the USA will take a minimum 500ms.
Making TCP optimizations useless, since the slow down machine is the one that actually makes connections to remote hosts.
Overriding destination IP - True doesn't care about what IP your computer wanted to connect to, your computer could be wrong. It sees the Host header from the request, does its own DNS lookups and routes you to the correct server. Even if you wanted to override this for development, True correctly sends you to production server. True knows development/staging servers are full of bugs, so requests should always go to production.
512 kbps upload speed is sufficient for everyone. If you need to upload something big, why not get your lazy ass out, buy a CD and mail it!
Sharing is caring - My ISP oversells available bandwidth by a huge margin. Teaches us the importance of sharing

~~Privacy~~ People impersonator - Your life is boring? True has a solution! It will automagically log you in as someone else so you can get a glimpse into their exciting lives.

~~Incompetence~~ Motivator - Living in Thailand, I am ashamed that I don't read Thai yet, in part due to my own laziness. True gives you an incentive.

edited by Michael van Poppel

DFP now officially supports asynchronous rendering!

Thu, 27 Oct 2011 12:29:52 +0000

Yesterday, DFP launched asynchronous ad loading. For the past few months ive been trying to load ads in a manner where it doesn't affect rest of the page load, this new development is like a dream come true. (The tests above were run on webpagetest.org on IE8 at Dulles, VA) Thank you Google! You just made my day.

Check if you are behind a transparent proxy

Tue, 11 Oct 2011 19:38:50 +0000

Many Asian ISPs do not provide clean internet. They route all HTTP sessions thru a transparent proxy. Here is a simple way to check if you are behind one.

sajal@sajal-laptop:~$ ping -c 4 www.cdnplanet.com
PING www.cdnplanet.com (107.20.181.99) 56(84) bytes of data.
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=1 ttl=42 time=314 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=2 ttl=42 time=313 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=3 ttl=42 time=312 ms
64 bytes from ec2-107-20-181-99.compute-1.amazonaws.com (107.20.181.99): icmp_req=4 ttl=42 time=312 ms

--- www.cdnplanet.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 312.195/313.229/314.137/0.889 ms

sajal@sajal-laptop:~$ ab http://www.cdnplanet.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.cdnplanet.com (be patient).....done


Server Software:        Apache
Server Hostname:        www.cdnplanet.com
Server Port:            80

Document Path:          /
Document Length:        13084 bytes

Concurrency Level:      1
Time taken for tests:   0.944 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      13296 bytes
HTML transferred:       13084 bytes
Requests per second:    1.06 [#/sec] (mean)
Time per request:       943.539 [ms] (mean)
Time per request:       943.539 [ms] (mean, across all concurrent requests)
Transfer rate:          13.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       21   21   0.0     21      21
Processing:   922  922   0.0    922     922
Waiting:      611  611   0.0    611     611
Total:        944  944   0.0    944     944
sajal@sajal-laptop:~$

My ping time to CDN Planet is 312ms, but the connection was established in just 21ms !!!!11!!1 Reasons for doing so involve : Censorship, big brother snooping, caching, hijacking users sessions , and probably more ...

Evaluating few CDN options

Fri, 10 Jun 2011 21:10:53 +0000

Recently, I was evaluating CDN options for a client with some unique challenges. We ended up using Amazon CloudFront, but ill detail the options we looked at and what let us to this decision. Some things to note:-

We would serve CSS, JS, Images referred to from stylesheets and some website images thru the CDN.
Due to the nature(trust me) of the site we expect a higher than normal miss rate. The access is spread across a wide number of urls, some may get lower hits.
Speed is important. Website should be fast(er) from everywhere. The most important regions in order of priority are US, EU, AU and ROTW with US being most important.
Anything on the site can change anytime, there is no build process as such. Any object anywhere can change, and change must be visible ASAP.
Developers/designers shouldn't be harassed to purge a file if they change something.
CDN data usage : ~100 GB per month

The providers we looked at :- MaxCDN: Almost sealed the deal. Pros:-

Cheap : $40 for first TB (must use in a year) + $99 per additional TB . On current usage rates this comes to say 0.04+ /GB.
* Anycast/BGP routing : No way bad DNS server can mess up routing.
Nice control panel, has a purge all option for just in case. Purges take effect almost instantly.
Handles gzip well.
Can have separate cache timings for caching in Browser and caching at CDN. - i.e. We can say not cache a file in browser level, but cache at CDN and purge when theres a change made.

Cons:-

Poor global coverage. - No POP/Edge in Asia/AU - Deal Breaker
Pages loaded same speed when testing from AU with or without CDN.

EdgeCast: Looked good at first, but poor gziping. Pros:-

Impressive list of networks
Highly configurable control panel.
Can have separate cache timings for caching in Browser and caching at CDN. - i.e. We can say not cache a file in browser level, but cache at CDN and purge when theres a change made.

Cons:-

Gzippable files will not be gzipped for cache misses. - Deal Breaker
Request from Edge server to origin is uncompressed.
Expensive and wants higher commitments.
DNS Based routing

CDNetworks: Didn't look past price Cons:-

Ridiculously high price - Dealbreaker

Amazon CloudFront: WIN Pros:-

Testing showed our pages to be fastest from all regions when using cloudfront. YMMV
No commitments - $0.15 - $0.2/GB (depends on where user accesses from) + negligible per request fee
Client is already AWS user, one less account to maintain.
No need to send gazillion emails to gazillions of people to get started. No bargaining.

Cons:-

No POP/Edge in AU (but has in Singapore, Hong Kong and Tokyo)
DNS based routing.
Charges fee per request and per invalidation(purge) request.
No control panel, invalidation requests need to be done by API only.
Does not do gzipping, but honours Vary header and serves correct version based on what user asks.
Can't use querystring parameters for CDN cachebusting. CloudFront ignores querystrings.

Sidenote : Requests from CloudFront to origin are HTTP 1.0 . Nginx by default does not serve gzip to 1.0 request. gzip_http_version setting must be changed in order to use nginx as origin for CloudFront. The system we architected adds something based on the file mtime as a part of the URL, so now we don't need to any purges at the CDN. Also now we can have far future expires on all CDN'd objects cause if something changes, the URL would automagically change. For us, the price and features are important, but whats more important is the results. We went with the provider with lesser features just because our pages loaded fastest with them.

Attention skeptics: Web Performance Optimization works!

Sat, 14 May 2011 14:39:14 +0000

Today, I was looking into web performance issues for a new client who wishes to remain anonymous and saw an interesting example of value provided by improving performance. One of the first things I do when looking into a new site is look at Google Analytics to understand a little about the sites visitors. This helps in prioritizing changes which affects the majority of users. An interesting observation :- Pageviews for all visitors Average Pageviews for all visitors Adsense revenue also followed a similar path. -- Screenshot not available The reason for this effect was a simple database indexing tweak which sped up the backend performance of a very important action page on the site. The site in question is a web application where user inputs something and gets some result. It is not a content site where pageview/user could have increased due to some interesting content being added or something. The only change was indexing of a column which should have an index right from the start. DISCLAIMER: I did not have any role in the above improvement. This was done before my involvement. Perhaps this convinced the client to hire me. Screenshots/info shared with clients permission.