Michael Crosby

Creating containers - Part 1

2014-11-16T00:00:00+00:00

This is part one of a series of blog posts detailing how docker creates containers. We will dig deep into the various pieces that are stitched together to see what it takes to make docker run ... awesome.

First, what is a container?

I think the various pieces of technology that goes into creating a container is fairly commonplace. You should have seen a few cool looking charts in various presentations about Docker where you get a quick "Docker uses namespaces, cgroups, chroot, etc." to create containers. But why does it take all these pieces to create a contaienr?
Why is it not a simple syscall and it's all done for me? The fact is that container's don't exist, they are made up. There is no such thing as a "linux container" in the kernel. A container is a userland concept.

Namespaces

In part one I'll talk about how to create Linux namespaces in the context of how they are used within docker. In later posts we will look into how namespaces are combined with other features like cgroups and an isolated filesystem to create something useful.

First off we need a high level explanation of what a namespace does and why it's useful. Basically, a namespace is a scoped view of your underlying Linux system. There are a few different types of namespaces implemented inside the kernel. As we dig into each of the different namespaces below you can follow along by running docker run -it --privileged --net host crosbymichael/make-containers. This has a few preloaded files and configuration to get your started. Even though we will be creating namespaces inside an container that docker runs for us, don't let that trip you up. I opted for this approach as providing a container preloaded with all the dependencies that you need to run the examples is why we are doing this in the first place. To make things a little easier, I'm using the --net host flag so that we are able to see your host's network interfaces within our demo container. This will be useful in the network examples. We also need to provide the --privilged flag so that we have the correct permissions to create new namespaces within our container.

If you are interested in what the Dockerfile looks like then here it is:

FROM debian:jessie

RUN apt-get update && apt-get install -y \
    gcc \
    vim \
    emacs

COPY containers/ /containers/
WORKDIR /containers
CMD ["bash"]

I'll be doing the examples in C, as it's sometimes easier to explain the lower level details better than the abstractions that Go provides. So lets start...

NET Namespace

The network namespaces provides your own view of the network stack of your system. This can include your very own localhost. Make sure you are in the crosbymichael/make-containers and run the command ip a to view all the network interfaces of your host machine.

> ip a
root@development:/containers# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:19:ca:f2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe19:caf2/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:20:84:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.103/24 brd 192.168.56.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe20:8447/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::5484:7aff:fefe:9799/64 scope link
       valid_lft forever preferred_lft forever

Ok cool, so this is all the network interfaces currently on my host system. Yours may look a little different but you get the idea. Now let's write some code to create a new network namespace. For this we will write a skeleton of a small C binary that uses the clone syscall. We will start by using clone to run binaries that are already installed inside our demo container. The file skeleton.c should be in the working directory of the demo container. We will use this file as the basis of all our examples. Here is the code incase you don't feel like running the container right now.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <sys/wait.h>
#include <errno.h>

#define STACKSIZE (1024*1024)
static char child_stack[STACKSIZE];

struct clone_args {
        char **argv;
};

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

int main(int argc, char **argv)
{
        struct clone_args args;
        args.argv = &argv[1];

        int clone_flags = SIGCHLD;

        // the result of this call is that our child_exec will be run in another
        // process returning it's pid
        pid_t pid =
            clone(child_exec, child_stack + STACKSIZE, clone_flags, &args);
        if (pid < 0) {
                fprintf(stderr, "clone failed WTF!!!! %s\n", strerror(errno));
                exit(EXIT_FAILURE);
        }
        // lets wait on our child process here before we, the parent, exits
        if (waitpid(pid, NULL, 0) == -1) {
                fprintf(stderr, "failed to wait pid %d\n", pid);
                exit(EXIT_FAILURE);
        }
        exit(EXIT_SUCCESS);
}

This is a small C binary that will allow you to run processes like ./a.out ip a. It uses the arguments that you pass on the cli as the arguments to whatever process you want to use. Don't worry about the specific implementation too much as it's the changes we will be making that are the interesting aspects. Remember, this will execute the binary and arguments of whatever program you want, this means if you want to run one of these demos below and have it spawn a shell session so that you can poke around in your new namespace then go ahead. It is a great way to explore and inspect these different namespaces at your own pace. So to get started let's make a copy of this file to start working with the network namespace.

> cp skeleton.c network.c

Ok, within this file there is a very special var called clone_flags. This is where most of our changes will happen throughout this post. Namespaces are primarily controlled via the clone flags. The clone flag for the network namespace is CLONE_NEWNET. We need to change the line in the file int clone_flags = SIGCHLD; to int clone_flags = CLONE_NEWNET | SIGCHLD; so that the call to clone creates a new network namespace for our process. Make this change in network.c then compile and run.

> gcc -o net network.c
> ./net ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

The result of this run now looks very different from the first time we ran the ip a command. We only see a loopback interface in the output. This is because our process that was created only has a view of its network namespace and not of the host. And that's it. That is how you create a new network namespace.

Right now this is pretty useless as you don't have any usable interfaces. Docker uses this new network namespace to setup a veth interface so that your container has it's own ip address allocated on a bridge, usually docker0. We won't go down the path of how to setup interfaces in namespaces at this time. We can save that for another post.

So now that we know how a network namespace is created lets look at the mount namespace.

MNT Namespace

The mount namespace gives you a scoped view of the mounts on your system. It's often confused with jailing a process inside a chroot or similar. This is not true! The mount namespaces does not equal a filesystem jail. So the next time you hear someone say that a container uses the mount namespace to "jail" the process inside it's own root filesystem you can call bullshit because they don't know what they are talking about. Do it, it's fun :)

Let's start by making a copy of skeleton.c again for our mount related changes. We can do a quick build and run to see what our current mount points looks like with the mount command.

> cp skeleton.c mount.c
> gcc -o mount mount.c
> ./mount mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

This is what the mount points look like from within my demo container, yours may look different. In order to create a new mount namespace we use the flag CLONE_NEWNS. You may notice something may look weird here. Why is this flag not CLONE_NEWMOUNT or CLONE_NEWMNT? This is because the mount namespace was the first Linux namespace introduced and the name was just an undersight. If you write code you will understand that as you start building a feature or an application, you don't often have a full picture of the end result. Anyway, let's add CLONE_NEWNS to our clone_flags variable. It should look something like int clone_flags = CLONE_NEWNS | SIGCHLD;.

Lets go ahead and build mount.c again and run the same command.

> cp skeleton.c mount.c
> gcc -o mount mount.c
> ./mount mount

Nothing changed. Whaaat??? This is because the process that we run inside the new mount namespace still has a view on /proc of the underlying system. The result is that the new process sort of inherits a view on the underlying mounts. There are a few ways that we can prevent this, like using pivot_root, but we will leave that for an additional post on filesystem jails and how chroot / pivot_root interact with the container's mount namespace.

However, one way that we can try out our new mount namespace is to, well, mount something. Let's create a new tmpfs mount in /mytmp for this demo. We will do this mount in C and continue to run our same mount command as the args to our mount binary. In order to do a mount inside of our mount namespace we need to add the code to the child_exec function, before the call to execvp. The code within the child_exec function is run inside the newly created process, i.e., inside our new namespace. The code in child_exec should look like this:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (mount("none", "/mytmp", "tmpfs", 0, "") != 0) {
                fprintf(stderr, "failed to mount tmpfs %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

We need to first create a directory /mytmp before we compile and run with the changes above.

> mkdir /mytmp
> gcc -o mount mount.c
> ./mount mount
# cutting out the common output...
none on /mytmp type tmpfs (rw,relatime)

I cut out the common output above from the first time I ran mount.
The result is that you should see a new mount point for our tmpfs mount. Nice! Go ahead and run mount in the current shell just for comparison.
Notice how the tmpfs mount is not displayed? That is because we created the mount inside our own mount namespace, not in the parent's namespace.

Remember how I said that the mount namepaces does not equal a filesystem jail? Go ahead and run our ./mount binary with the ls command. Everything is there. Now you have proof!

UTS Namespace

The next namespace is the UTS namespace that is responsible for system identification. This includes the hostname and domainname. It allows a container to have it's own hostname independently from the host system along with other containers. Let's start by making a copy of skeleton.c and running the hostname command with it.

> cp skeleton.c uts.c
> gcc -o uts uts.c
> ./uts hostname
development

This should display your system's hostname (development in my case). Like earlier, let's add the clone flag for the UTS namespace to the clone_flags variable. The flag should be CLONE_NEWUTS. If you compile and run then you should see the exact same output. This is totally fine. The values in the UTS namespace are inherited from the parent. However, within this new namespace, we can change the hostname without it affecting the parent or other container's that have a separate UTS namespace.

Let's modify the hostname in the child_exec function. To do that, you will need to add the #include <unistd.h> header to gain access to the sethostname function, as well as the #include <string.h> header to use strlen needed by sethostname. The new body of the child_exec function should look like the following:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        const char * new_hostname = "myhostname";
        if (sethostname(new_hostname, strlen(new_hostname)) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

Ensure that clone_flags within your main look like int clone_flags = CLONE_NEWUTS | SIGCHLD; then compile and run the binary with the same args. You should now see the value that we set returned from the hostname command. To verify that this change did not affect our current shell go ahead and run hostname and make sure that you have your original value back.

> gcc -o uts uts.c
> ./uts hostname
myhostname
> hostname
development

Awesome! We are doing good.

IPC Namespace

The IPC namespace is used for isolating interprocess communication, things like SysV message queues. Let's make a copy of skeleton.c for this namespace.

> cp skeleton.c ipc.c

The way we are going to test the IPC namespace is by creating a message queue on the host, and ensuring that we cannot see it when we spawn a new process inside it's own IPC namespace. Let's first create a message queue in our current shell then compile and run our copy of the skeleton code to view the queue.

> ipcmk -Q
Message queue id: 65536
> gcc -o ipc ipc.c
> ./ipc ipcs -q
------ Message Queues --------
key        msqid      owner      perms      used-bytes   message
0xfe7f09d1 65536      root       644        0            0

Without a new IPC namespace you can see the same message queue that was created. Now let's add the CLONE_NEWIPC flag to our clone_flags var to create a new IPC namespace for our process. The clone_flags var should look like int clone_flags = CLONE_NEWIPC | SIGCHLD;. Recompile and run the same command again:

> gcc -o ipc ipc.c
> ./ipc ipcs -q
------ Message Queues --------
key        msqid      owner      perms      used-bytes   message

Done! The child process is now in a new IPC namespace and has completely separate view and access to message queues.

PID Namespace

This one is fun. The PID namespace is a way to carve up the PIDs that one process can view and interact with. When we create a new PID namespace the first process will get to be the loved PID 1. If this process exits the kernel kills everyone else within the namespace. Let's start by making a copy of skeleton.c for our changes.

> cp skeleton.c pid.c

To create a new PID namespace, we will have to set the clone_flags with CLONE_NEWPID. The variable should look like int clone_flags = CLONE_NEWPID | SIGCHLD;. Let's test by running ps aux in our shell and then compile and run our pid.c binary with the same arguments.

> ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
root       147  0.0  0.1  17492  2088 ?        R+   22:49   0:00 ps aux
> gcc -o pid pid.c
> ./pid ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
root       153  0.0  0.0   5092   728 ?        S+   22:50   0:00 ./pid ps aux
root       154  0.0  0.1  17492  2064 ?        R+   22:50   0:00 ps aux

WTF??? We expected ps aux to be PID 1 or atleast not see any other pids from the parent. Why is that? proc. The process that we spawned still has a view of /proc from the parent, i.e. /proc mounted on the host system. So how do we fix this? How to we ensure that our new process can only view pids within it's namespace? We can start by remounting /proc.
Because we will be dealing with mounts, we can take the opportunity to take what we learned from the MNT namespace and combine it with our PID namespace so that we don't mess with the /proc of our host system.

We can start by including the clone flag for the mount namespace along side the clone flag for pid. It should look something like int clone_flags = CLONE_NEWPID | CLONE_NEWNS | SIGCHLD;. We need to edit the child_exec function and remount proc. This will be a simple unmount and mount syscall for the proc filesystem. Because we are creating a new mount namespace we know that this will not mess up our host system. The result should look like this:

// child_exec is the func that will be executed as the result of clone
static int child_exec(void *stuff)
{
        struct clone_args *args = (struct clone_args *)stuff;
        if (umount("/proc", 0) != 0) {
                fprintf(stderr, "failed unmount /proc %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (mount("proc", "/proc", "proc", 0, "") != 0) {
                fprintf(stderr, "failed mount /proc %s\n",
                        strerror(errno));
                exit(-1);
        }
        if (execvp(args->argv[0], args->argv) != 0) {
                fprintf(stderr, "failed to execvp argments %s\n",
                        strerror(errno));
                exit(-1);
        }
        // we should never reach here!
        exit(EXIT_FAILURE);
}

Build and run this again to see what happens.

> gcc -o pid pid.c
> ./pid ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   9076   784 ?        R+   23:05   0:00 ps aux

Perfect! Our new PID namespace is now fully operational with the help of the mount namespace!

USER Namespace

The last namespace is the user namespace. This namespace is the new kid on the block and allows you to have users within this namespace that are not equal users outside of the namespace. This is accomplished via GID and UID mappings.

This one has a simple demo application without specifying a mapping, even though it's a totally useless demo. If we add the flag CLONE_NEWUSER to our clone_flags then run something like id or ls -la you will notice that we get nobody within the user namespace. This is because the current user is undefined right now.

> cp skeleton.c user.c
# add the clone flag
> gcc -o user user.c
> ./user ls -la
total 84
drwxr-xr-x 1 nobody nogroup 4096 Nov 16 23:10 .
drwxr-xr-x 1 nobody nogroup 4096 Nov 16 22:17 ..
-rwxr-xr-x 1 nobody nogroup 8336 Nov 16 22:15 mount
-rw-r--r-- 1 nobody nogroup 1577 Nov 16 22:15 mount.c
-rwxr-xr-x 1 nobody nogroup 8064 Nov 16 21:52 net
-rw-r--r-- 1 nobody nogroup 1441 Nov 16 21:52 network.c
-rwxr-xr-x 1 nobody nogroup 8544 Nov 16 23:05 pid
-rw-r--r-- 1 nobody nogroup 1772 Nov 16 23:02 pid.c
-rw-r--r-- 1 nobody nogroup 1426 Nov 16 21:59 skeleton.c
-rwxr-xr-x 1 nobody nogroup 8056 Nov 16 23:10 user
-rw-r--r-- 1 nobody nogroup 1442 Nov 16 23:10 user.c
-rwxr-xr-x 1 nobody nogroup 8408 Nov 16 22:40 uts
-rw-r--r-- 1 nobody nogroup 1694 Nov 16 22:36 uts.c

This is a very simple example of the user namespace but you can go much deeper with it. We will save this for another post but the idea and hopes for the user namespace is that this will allow us to run as "root" within the container but not as "root" on the host system. Don't forget you can always change ls -la to bash and have a shell inside the namespace to poke around and learn more.

In the end...

So to recap we went over the mount, network, user, PID, UTS, and IPC Linux namespaces. The majority of the code that we changed was not much, just adding a flag most of the time.
The "hard work" is mostly managing the interactions between the various kernel subsystems in order to meet our requirements. Like most of the descriptions before this, namespaces are just one of the tools that we use to make a container. I hope the PID example is a glimpse of how we use multiple namespaces together in order isolate and begin the creation of a container.

In future posts we will go into detail on how we jail the container's processes inside a root filesystem, aka a docker image, as well as using cgroups and Linux capabilities. By the end we should be able to pull all these things together to create a container.

Also a thanks to tibor and everyone that helps review my brain dump of a first draft ;)

the lost packages of docker

2014-04-16T00:00:00+00:00

Docker is a large project that touches many different layers of the system. Everything from a REST api to low level filesystem and execution calls. Docker is also open source so where are all the awesome packages that make docker dock? Why do the docker developers not contribute back reusable packages that I can import in my own projects? Well we do and they are located in pkg.

While working on docker we try to keep the core as small as possible and break code out into general purpose packages that can be used in many different projects. The hard part about this is that we cannot make separate git repos for the packages because of packaging reasons. Right now it is still hard for distro maintainers to package and build Go applications because of the static nature of Go. Everyone is doing the best that they can right now so while we work on the best way for distros to handle packaging our dependencies we place resuable Go packages in a /pkg folder in the docker repository. So what does docker have to offer?

netlink

We have a pure go implementation of netlink that you can use in your projects. You can use this package to create veth interfaces, bridges, set ip, mtu, and other settings on network interfaces, move network interfaces into different linux namespaces, bla bla bla. This is the same code that handles creating the veth pairs and assigning an ip to each of your docker containers.

Lets use this package to create bridge and assign an ip:

package main

import (
        "github.com/dotcloud/docker/pkg/netlink"
        "log"
        "net"
)

func main() {
        // create a new bridge
        if err := netlink.CreateBridge("mydocker0", false); err != nil {
                log.Fatal(err)
        }
        // get the bridge
        bridge, err := net.InterfaceByName("mydocker0")
        if err != nil {
                log.Fatal(err)
        }

        ip, ipNet, err := net.ParseCIDR("10.0.41.1/16")
        if err != nil {
                log.Fatal(err)
        }

        // add an ip to the bridge
        if err := netlink.NetworkLinkAddIp(bridge, ip, ipNet); err != nil {
                log.Fatal(err)
        }
        // bring the interface up
        if err := netlink.NetworkLinkUp(bridge); err != nil {
                log.Fatal(err)
        }
}

We can run the app above then run ip a to see our new interface:

2: mydocker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether ae:1f:cb:af:f2:54 brd ff:ff:ff:ff:ff:ff
    inet 10.0.41.1/16 scope global mydocker0
    valid_lft forever preferred_lft forever
    inet6 fe80::ac1f:cbff:feaf:f254/64 scope link
    valid_lft forever preferred_lft forever

The netlink package is not feature complete but hopefully not that you know it exists we can all work together on it to make it better.

user

Linux user and group functions that do not rely on external libs.

graphdb

Graphdb is a minimal graph database built ontop of sqlite for use with names and how they relate with one another. It implements a small subset of what most graph databases have but provides a simple interface for expressing relationships between nodes. This is used for container naming and linking inside docker.

package main

import (
        "github.com/dotcloud/docker/pkg/graphdb"
        "log"
)

func main() {
        db, err := graphdb.NewSqliteConn("/root/links.db")
        if err != nil {
                log.Fatal(err)
        }
        defer db.Close()

        if _, err := db.Set("/parent", "momma"); err != nil {
                log.Fatal(err)
        }

        child, err := db.Set("/child", "matt")
        if err != nil {
                log.Fatal(err)
        }
        otherKid, err := db.Set("/otherKid", "koye")
        if err != nil {
            log.Fatal(err)
        }

        // set entity with id of matt to child of the parent
        if _, err := db.Set("/parent/child", child.ID()); err != nil {
                log.Fatal(err)
        }
        if _, err := db.Set("/parent/kidbyaothername", otherKid.ID()); err != nil {
                log.Fatal(err)
        }

        // get all children for the key /parent
        for _, e := range db.List("/parent", -1) {
                log.Println(e.ID())
        }
}

:::bash
~|⇒ go run graph.go
2014/04/14 16:16:19 matt
2014/04/14 16:16:19 koye

listenbuffer

Listenbuffer allows you to immediately start listening on a unix, tcp, or udp socket but wait for your application to signal that it is ready before accepting connections.

This is used in docker so that you can start the daemon and a client request from an init script can be sent before the daemon has completed loading. This ensures that the init script does not receive an error and your server can start accepting connections when it is ready. It uses the kernel backlog for buffering the connections and is a super simple package that offers a lot of value if you have a long boot.

mount

Mount is a package to help you work with mounting lots of things. It allows options to be passed fstab style and also offers mount table parsing functions so that your applications can easily find mountpoints.

// do a readonly bind mount
if err := mount.Mount("/myfile", "/myotherfile", "none", "bind,ro"); err != nil {
    log.Fatal(err)
}
defer mount.Unmount("/myotherfile")

term

Term handles terminal sizing, raw mode, and other settings for interacting with a unix terminal on linux, darwin, and bsd.

cgroups

This package is currently in heavy development right now but it provides the cgroup functionality for libcontainer via the raw cgroup filesystem api and also the systemd api where supported.

libcontainer

Pkg is also where libcontainer resides. It is the default execution driver for running docker containers. It sets up namespaces, networking, mounts, drops caps, and manages the container processes. You provide a root filesystem and a configuration on how libcontainer is supposed to execute a container and it does the rest. It allows spawning new containers or attaching to an existing container.

So in the end why did I write this post? I wanted to bring some visibility to these packages. While other projects are able to have separate git repositories for the different dependencies that they use and develop, docker has to keep these in it's main repository causing many people to not know that they exist. So if some of these packages seem interesting to you, feel free to use them in your projects, contribute, write docs, tests, and make them awesome.

If you know of an interesting area currently in the docker code base that is not in a reusable package, submit a PR to make it one. We are always looking for ways to make the docker core smaller and our work useful to others.

Dockerfile Best Practices - take 2

2014-03-09T00:00:00+00:00

Much has changed since my first Dockerfile best practices post. I'll leave the original post up for posterity and this post will include what has change and what you should do now.

1: Don't boot init

Containers model processes not machines. Even if you think that you need to do this you are probably wrong. Next...

2: Trusted builds

Even if you don't like the name it is an awesome feature. I have most of my github repositories added to trusted builds so that when I push a commit my new image is waiting on the index shortly after. Also, I don't have to have separate repositories with the Dockerfile to share with others, it is all in one place.

Just remember that this is not your playground for trying things out. Build locally first before you push. Docker ensures that what you build and run locally will build and run the same way when pushed elsewhere. Develop and test locally, commit and push, and wait for the official image on the index with trusted builds.

3: Don't upgrade in builds

Updates will be baked into the based images you don't need to apt-get upgrade your containers. Because of the isolation that happens this can often fail if something is trying to modify init or make device changes inside a container. It also produces inconsistent images because you no longer have one source of truth of how your application should run and what versions of dependencies are included in the image.

If there are security updates that a base image needs, let upstream know so that they can update it for everyone and ensure that your builds are consistent again.

4: Use small base images

Some images are more bloated than others. I suggest using debian:jessie as your base. If you are coming from ubuntu, you will find a more lightweight and happy home on debian. It's small and does not contain any unneeded bloat.

5: Use specific tags

This is important for your base images. Your FROM should always contain the full repository and tag for the base image that you are depending on. FROM debian:jessie not just FROM debian.

6: Group common operations

Your apt-get update should be grouped with your apt-get install. Also, you should take advantage of the \ to span multiple lines on your installs.

# Dockerfile for https://index.docker.io/u/crosbymichael/python/ 
FROM debian:jessie

RUN apt-get update && apt-get install -y \
    git \
    libxml2-dev \
    python \
    build-essential \
    make \
    gcc \
    python-dev \
    locales \
    python-pip

RUN dpkg-reconfigure locales && \
    locale-gen C.UTF-8 && \
    /usr/sbin/update-locale LANG=C.UTF-8

ENV LC_ALL C.UTF-8

Just remember that layers and the cache are good. Don't be scared of having many layers because the cache is a life saver. Also you should use upstream packages when you can.

7: Use your own base images

I'm not talking about running debbootstrap and making your own debian, you're not tianon and no one wants your shitty base image with 500mb of shit that you think ppl need. I'm talking about if you are running python applications have a python base image. The Dockerfile in the previous example is used to build crosbymichael/python that is used in many other images for python applications.

FROM crosbymichael/python

RUN pip install butterfly
RUN echo "root\nroot\n" | passwd root

EXPOSE 9191
ENTRYPOINT ["butterfly.server.py"]
CMD ["--port=9191", "--host=0.0.0.0"]

Or another:

FROM crosbymichael/python

RUN pip install --upgrade youtube_dl && mkdir /download
WORKDIR /download
ENTRYPOINT ["youtube-dl"]
CMD ["--help"]

As you can see this makes Dockerfiles that use your base very small and focused on the application that we are installing install.

Let me know what you think or if you have any other questions.

Go Dynamic Plugins

2013-09-23T00:00:00+00:00

For me, my favorite feature of Go is the static binary. I love compiling once and just dropping the binary into a container running the 2mb busybox image and having my app up and running. No pip, no dlls, no problem. However, this does come with a small price. If you want to dynamically extend your Go app at runtime with plugins your SOL.

If you look at the internals of the database/sql package in Go you can see how they handle different database driver support but that is still resolved at build time. A few people are handling plugins for Go via RPC, shelling out to another process, or what not, but all of those solutions still have their own issues to work through. If you want to keep your plugins in the same process and have a natural feel then I think this solution is worth a look.

otto

There is a really cool open source project called otto that implements a javascript runtime in pure Go. This allows you to embed js in your Go apps very easily. Yes, yes, I know it's javascript but it's not that bad when used in moderation.

Anyways, otto allows you to create a js runtime in your Go application and execute js, allow js to call go functions that were added to the runtime, and pass data to and from a javascript function. In the example below I will be making a simple http server that returns a string depending if you are authorized or not. The interesting part is that we will be passing the http.Request to a javascript function to do runtime checks. This allows us to change the code checking the request without stopping and rebuilding our Go code.

The app

Lets start by looking at our main func that will handle the web requests.

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if checkRequest(r) {
            fmt.Fprintf(w, "Welcome in\n")
        } else {
            w.WriteHeader(http.StatusUnauthorized)
            fmt.Fprintf(w, "Your not allowed!\n")
        }
    })

    if err := http.ListenAndServe(":8080", nil); err != nil {
        panic(err)
    }
}

This is very standard and simple. In the handler we make a call to checkRequest to tell us if the request is valid or not. Now lets look at the checkRequest func.

func checkRequest(r *http.Request) bool {
    runtime := loadPluginRuntime("plugins.js")

    // If we don't have a runtime all requests are accepted
    if runtime == nil {
        return true
    }
    v, err := runtime.ToValue(*r)
    if err != nil {
        log.Fatal(err)
    }

    // By convention we will require plugins have a set name
    result, err := runtime.Call("checkRequest", nil, v)
    if err != nil {
        log.Fatal(err)
    }
    // If the js function did not return a bool error out
    // because the plugin is invalid
    out, err := result.ToBoolean()
    if err != nil {
        log.Fatalf("\"checkRequest\" must return a boolean. Got %s", err)
    }
    return out
}

This is where we interface with the js runtime. We load the runtime in another func that I will show next. If we don't have any plugins we will receive a nil runtime so we just return true. If we do have a runtime then we need to convert our Go type ( the request ) into a js Value/Object so that we can pass it to the plugin method. Right now otto has a issue with pointer values as js Objects so I dereference the request when passing to the ToValue method.

Next we call the js function checkRequest and receive it's return value. In order to call js functions we need to know the name of the function in our Go code. I don't think that this is an issue because when people want to implement plugins for your application they will need to stick to certain naming conventions and interfaces in order for the two applications to interact together.

Now we need to take the return value from the js function and convert it back into a Go type that our program can understand. We expect the checkRequest function to return a bool so we use the ToBoolean method to do the conversion. If we get an err, it is probably because the js function did not return the correct type or nothing at all.

Let's quickly look at the loadPluginRuntime func.

func loadPluginRuntime(name string) *otto.Otto {
    f, err := os.Open(name)
    if err != nil {
        if os.IsNotExist(err) {
            return nil
        }
        log.Fatal(err)
    }
    defer f.Close()
    buff := bytes.NewBuffer(nil)

    if _, err := buff.ReadFrom(f); err != nil {
        log.Fatal(err)
    }
    runtime := otto.New()
    // Load the plugin file into the runtime before we
    // return it for use
    if _, err := runtime.Run(buff.String()); err != nil {
        log.Fatal(err)
    }
    return runtime
}

In the load func we load a file with all the js functions and set them in the runtime. Because this is an example I am run reading the file on each request. In a prod system you will want to do something fancy but for this example it is fine. Anyways, we load the file and then create the new runtime. We use the Run method and pass the contents of the file as a string so that our functions are loaded.

The Plugin

Lets checkout the plugin.js file to see what the javascript implementation is to check the request.

function checkRequest(r) {
    // Only allow GET requests to the API
    return r.Method == "GET";
}

And that is it. We access the Method on the http.Request and check to make sure it is a GET request and not allow anything else. You can change the plugin.js file and watch the responses change as you toggle between true and false.

Otto is still new so I hope people will check out the project and contribute bug fixes and improvements so that otto can be a viable option for dynamic plugins in Go. Of course you can also embed lua or v8 into your Go binary instead. There are many options and this is only one.

I am also interested in embedding lua into Go. I have notice a few large open source projects like Nginx and Redis embedding lua so why not give it a try. Thanks and let me know what you think in the comments below.

Full App

package main

import (
    "bytes"
    "fmt"
    "github.com/robertkrimen/otto"
    "log"
    "net/http"
    "os"
)

// Load the plugin file.  If the file does not exist
// then return a nil runtime
func loadPluginRuntime(name string) *otto.Otto {
    f, err := os.Open(name)
    if err != nil {
        if os.IsNotExist(err) {
            return nil
        }
        log.Fatal(err)
    }
    defer f.Close()
    buff := bytes.NewBuffer(nil)

    if _, err := buff.ReadFrom(f); err != nil {
        log.Fatal(err)
    }
    runtime := otto.New()
    // Load the plugin file into the runtime before we
    // return it for use
    if _, err := runtime.Run(buff.String()); err != nil {
        log.Fatal(err)
    }
    return runtime
}

func checkRequest(r *http.Request) bool {
    runtime := loadPluginRuntime("plugins.js")

    // If we don't have a runtime all requests are accepted
    if runtime == nil {
        return true
    }
    v, err := runtime.ToValue(*r)
    if err != nil {
        log.Fatal(err)
    }

    // By convention we will require plugins have a set name
    result, err := runtime.Call("checkRequest", nil, v)
    if err != nil {
        log.Fatal(err)
    }
    // If the js function did not return a bool error out
    // because the plugin is invalid
    out, err := result.ToBoolean()
    if err != nil {
        log.Fatalf("\"checkRequest\" must return a boolean. Got %s", err)
    }
    return out
}

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if checkRequest(r) {
            fmt.Fprintf(w, "Welcome in\n")
        } else {
            w.WriteHeader(http.StatusUnauthorized)
            fmt.Fprintf(w, "Your not allowed!\n")
        }
    })

    if err := http.ListenAndServe(":8080", nil); err != nil {
        panic(err)
    }
}

Advanced Docker Volumes

2013-09-14T00:00:00+00:00

I have noticed that many people have a hard time understanding what volumes are and how to use them effectively.

So what is a volume? A volume can be a directory that is located outside of the root filesystem of your container. This allows you to import this directory in other containers. You can also use volumes to mount directories from your host machine inside a container.

# create a new volume in /www
docker run -v /www ubuntu echo yo

# create a new volume mounted to your host machine
docker run -v /host/logs:/container/logs ubuntu echo momma

So pretty simple, right? So what actually happens under the hood when you specify a volume?

Under the hood

So with the command docker run -v /www ubuntu echo yo a container is created in /var/lib/docker/containers with the rootfs setup for COW from the ubuntu image. Then a new volume is created in /var/lib/docker/volumes for the data that is placed in /www for the container. We can run docker inspect to see the volume configuration.

... stufff
    "HostnamePath": "/var/lib/docker/containers/831d4ecbdf2e096475365b019b211f74aff7a3cea3c443570e1fd2f2bb0dc843/hostname",
    "HostsPath": "/var/lib/docker/containers/831d4ecbdf2e096475365b019b211f74aff7a3cea3c443570e1fd2f2bb0dc843/hosts",
    "Volumes": {
        "/www": "/var/lib/docker/volumes/ec3c543bc92f114c2c568733541e89381881e5a62996d7084e07793f86280535/layer"
    },
    "VolumesRW": {
        "/www": true
    }
}

By inspecting the container we can see that we have a new volume and the full path to the volume's data on disk. You will also notice the field VolumesRW denoting that this volume is Read/Write.

This simple from the user's perspective. You just specify a -v with the directory that you want to use as a volume and your done. Now what's up with mounting directories from the host system?

Host mounts

There are a few use cases where you want to mount a directory from your host system into the container. Just remember that this is not portable. You may have a specific directory structure on your development machine that is not on your production servers causing issues when you try to move the container. Many people also feel like they need to mount the data directory of their database container to the host machine so that it can be backed up. However, you can see all the files for your "normal" volumes are stored in /var/lib/docker/volumes on the host filesystem. Docker containers are not like VM's where your data is not locked into some disk image. So usually mounting to the host is not needed.

Volumes From

Ok, so now we have a container running with a volume specified. What can we do with it? For this example lets say that we have container A that is a web frontend and it writes logs to /logs/webapp. When we ran the frontend container we specified -v /logs/webapp to create a volume. Next we need some type of process to collect the logs and push them into a central repository. We can have container B start and reference the volume from A to gain access to it's logs.

docker run -volumes-from A log-daemon collect

We use the -volumes-from flag and pass the container's ID to gain access to it's volumes. Now our daemon can cd into /logs/webapp and start processing the logs. Easy, simple, and boring. Now lets do something cool.

The cool stuff

So how can we push volumes to the limit? I currently have docker running a minecraft server. I have a minecraft server instance running and the directory with all the world data is located in /minecraft. There is another cool project called Overviewer that will generate a "google map" like, uhh, map of your world as an html site. You give overviewer.py the path to your world files and an output directory to write the result to.

So this stack has three components. The actual game server, a worker that takes data from the server and generates html, and a web server to serve the static file output of the map. What docker and volumes allows us to do is to separate our concerns so that the web server does not have to be in the same container as the game server and the worker that generates the map does not have to run in the same container as the web or game servers. Just like functions and classes in your code, containers should do one thing and do that well.

WebServer (web frontend)

So lets look at how the web server is built and configured. It is a static file server that just serves files out of a directory to a specified port and ip. If you want to see the web server code look here.

FROM busybox

ADD server server

VOLUME /www
USER daemon
EXPOSE 8000

ENTRYPOINT ["/server", "-h", "0.0.0.0", "-p", "8000", "-dir", "/www"]

The server knows nothing about the stack that it is running in or what it is serving. it just serves files located in /www ( which is a volume ) and nothing more. Simple and clean.

Map Generator (worker)

Now lets look at our worker.

FROM ubuntu

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get upgrade -y

RUN echo "deb http://overviewer.org/debian ./" >> /etc/apt/sources.list
RUN apt-get update
RUN apt-get install -y --force-yes minecraft-overviewer wget

ENV VERSION 1.6.2

RUN wget --no-check-certificate https://s3.amazonaws.com/Minecraft.Download/versions/${VERSION}/${VERSION}.jar -P ~/.minecraft/versions/${VERSION}/

ENTRYPOINT ["overviewer.py"]
CMD ["/minecraft/world", "/www"]

The worker has a few more dependencies but this container does it's one thing. Get world files from /minecraft/world, create the map, and write the output to /www. This worker knows nothing about where or how it gets the files to process, it just processes world files into maps from it's local filesystem.

Minecraft server

Finally where the fun happens.

FROM ubuntu

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get upgrade -y

RUN apt-get install -y python-software-properties
RUN add-apt-repository ppa:webupd8team/java -y

RUN apt-get update
RUN echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections

RUN apt-get install -y oracle-java7-installer
RUN apt-get install -y wget

RUN mkdir /minecraft
RUN wget -O /minecraft/minecraft.jar https://s3.amazonaws.com/Minecraft.Download/versions/1.6.2/minecraft_server.1.6.2.jar
RUN chmod +x /minecraft/minecraft.jar

VOLUME /minecraft
WORKDIR /minecraft
EXPOSE 25565

CMD java -Xmx1600M -Xms768M -jar minecraft.jar nogui

We run the minecraft.jar and use /minecraft as our current working dir.

Now we have our containers created and they have clear boundaries. They are only concerned about doing their one job and have no frame of reference where they are located in the stack.

Runtime

So now how do we glue this stack together with volumes? We can start by running the minecraft and web servers.

MINECRAFT=$(docker run -d minecraft)
MAPSERVER=$(docker run -d mapserver)

Now that we have our servers running we use the worker to bridge both of them together with volumes. Because we exposed /minecraft with the world data from the MINECRAFT container and /www where static files are served from the MAPSERVER container we need to tell the worker/generator to use the -volumes-from both containers.

docker run -volumes-from $MINECRAFT -volumes-from $MAPSERVER mapgenerator

Just like magic the mapgenerator will pull the data out of /minecraft/world and write the output into /www, and once it is done we can remove the generator container because it contains zero data that we need. It is an ephemeral processes that takes input and produces an output with no state in the middle.

If you want the map to be generated every hour ( like I do ) you can just run the same command in a cron job.

@hourly docker run -volumes-from $MINECRAFT -volumes-from $MAPSERVER mapgenerator

Using volumes with this stack allows me to not worry about the generator process locking up the main game server, causing crashes, leaking memory and leaving artifacts on disk. The game server runs and serves up super-fun-times. The web server just serves files from one directory and does not need to be bothered. And the generator just takes input and produces output.

Docker does all the work while each container interacts with it's own filesystem and does not know the difference. We don't have to mess with or restart the web or game servers. It's almost like dependency injection but with an entire filesystem which is really cool.

Warning: this post has features of docker that are not in a current release. Look for these features in 0.6.2

Docker Events

2013-08-02T00:00:00+00:00

The proper way to do port allocation in docker is to EXPOSE a private port in a container and let docker map the public port on the host. This can be a pain for reverse proxies, load balances, etc but with the events endpoint in docker you can listen for containers to start and inspect their port mappings.

Here is a very simple Go app as an example on how to utilize this endpoint.

Code:

The first thing that we want to do is to add a few imports.

package main

import (
    "encoding/json"
    "io"
    "log"
    "net/http"
)

We will need json for decoding the information from the api, io for io.EOF, log to print to stdout, and http for the Client.

Now lets create a few types to store our information.

type Event struct {
    Id     string `json:"id"`
    Status string `json:"status"`
}

type Config struct {
    Hostname string
}

type NetworkSettings struct {
    IpAddress   string
    PortMapping map[string]map[string]string
}

type Container struct {
    Id              string
    Image           string
    Config          *Config
    NetworkSettings *NetworkSettings
}

The api will return much more container information but I only care about a few fields and types in this example.

func main() {
    c := http.Client{}
    res, err := c.Get("http://localhost:4243/events")
    if err != nil {
        log.Fatal(err)
    }
    defer res.Body.Close()

    // Read the streaming json from the events endpoint
    // http://docs.docker.io/en/latest/api/docker_remote_api_v1.3/#monitor-docker-s-events
    d := json.NewDecoder(res.Body)
    for {
        var event Event
        if err := d.Decode(&event); err != nil {
            if err == io.EOF {
                break
            }
            log.Fatal(err)
        }
        if event.Status == "start" {
            // We only want to inspect the container if it has started
            if container := inspectContainer(event.Id, c); container != nil {
                notify(container)
            }
        }
    }
}

This is my main() func. We create a new http.Client that will be used to make GET requests to the docker api. The first request that we make is to the events endpoint. This will open a stream to our app writing json objects when events happen in the docker daemon. We create a new json decoder to read from the Body of the response and decode Event types. After we decode the event we only care if the container has been started so we use the status field.

func inspectContainer(id string, c http.Client) *Container {
    // Use the container id to fetch the container json from the Remote API
    // http://docs.docker.io/en/latest/api/docker_remote_api_v1.4/#inspect-a-container
    res, err := c.Get("http://localhost:4243/containers/" + id + "/json")
    if err != nil {
        log.Println(err)
        return nil
    }
    defer res.Body.Close()

    if res.StatusCode == http.StatusOK {
        d := json.NewDecoder(res.Body)

        var container Container
        if err = d.Decode(&container); err != nil {
            log.Fatal(err)
        }
        return &container
    }
    return nil
}

Now that a container has started we use the id field from the event to then make another request to the docker api for the container information. We use another json decoder but this time we decode the json result returned from the api into the Container type.

func notify(container *Container) {
    settings := container.NetworkSettings

    if settings != nil && settings.PortMapping != nil {
        // I only care about Tcp ports but you can also view Udp mappings
        if ports, ok := settings.PortMapping["Tcp"]; ok {

            log.Printf("Ip address allocated for: %s", container.Id)

            // Log the public and private port mappings
            for privatePort, publicPort := range ports {
                // I am just writing to stdout but you can use this information to update hipache, redis, etc...
                log.Printf("%s -> %s", privatePort, publicPort)
            }
        }
    }
}

So now that we have the container information that we care about we need to get that data and make it useful. All we need to do is to look at the NetworkSettings for the PortMappings to inspect Tcp ports. You will receive a map where the key is the private port and the value is the public port. I am just writing the values to stdout but you could use this data to update hipache for a new container, write nginx config changes, etc... The possibilities are endless.

Let me know if you have any questions or comments below.

Full code:

package main

import (
    "encoding/json"
    "io"
    "log"
    "net/http"
)

type Event struct {
    Id     string `json:"id"`
    Status string `json:"status"`
}

type Config struct {
    Hostname string
}

type NetworkSettings struct {
    IpAddress   string
    PortMapping map[string]map[string]string
}

type Container struct {
    Id              string
    Image           string
    Config          *Config
    NetworkSettings *NetworkSettings
}

func inspectContainer(id string, c http.Client) *Container {
    // Use the container id to fetch the container json from the Remote API
    // http://docs.docker.io/en/latest/api/docker_remote_api_v1.4/#inspect-a-container
    res, err := c.Get("http://localhost:4243/containers/" + id + "/json")
    if err != nil {
        log.Println(err)
        return nil
    }
    defer res.Body.Close()

    if res.StatusCode == http.StatusOK {
        d := json.NewDecoder(res.Body)

        var container Container
        if err = d.Decode(&container); err != nil {
            log.Fatal(err)
        }
        return &container
    }
    return nil
}

func notify(container *Container) {
    settings := container.NetworkSettings

    if settings != nil && settings.PortMapping != nil {
        // I only care about Tcp ports but you can also view Udp mappings
        if ports, ok := settings.PortMapping["Tcp"]; ok {

            log.Printf("Ip address allocated for: %s", container.Id)

            // Log the public and private port mappings
            for privatePort, publicPort := range ports {
                // I am just writing to stdout but you can use this information to update hipache, redis, etc...
                log.Printf("%s -> %s", privatePort, publicPort)
            }
        }
    }
}

func main() {
    c := http.Client{}
    res, err := c.Get("http://localhost:4243/events")
    if err != nil {
        log.Fatal(err)
    }
    defer res.Body.Close()

    // Read the streaming json from the events endpoint
    // http://docs.docker.io/en/latest/api/docker_remote_api_v1.3/#monitor-docker-s-events
    d := json.NewDecoder(res.Body)
    for {
        var event Event
        if err := d.Decode(&event); err != nil {
            if err == io.EOF {
                break
            }
            log.Fatal(err)
        }
        if event.Status == "start" {
            // We only want to inspect the container if it has started
            if container := inspectContainer(event.Id, c); container != nil {
                notify(container)
            }
        }
    }
}

Go Helpful Commands

2013-08-01T00:00:00+00:00

Here are a few go commands that I find helpful that are hidden in documentation.

List all installed packages

go list all

Build entire pkg directory

go build . ./...

Install entire pkg directory

go install . ./...

Test entire pkg

go test . ./...

Format and simplify code

go fmt -s -w <file>

Detect races

go test -race .

Go toolbox

2013-07-20T00:00:00+00:00

I have been asked a few times about my Go development environment. Because Go is still a very new language many people do not know where to start. Here are a few tools that I have in my Go toolbox.

Tools

GoCode - Code completion daemon.
gofmt - Format your code
godoc - Stdlib and third party documentation
lint - linter for Go
doc - Tool to search the docs via the cmd line
gocov - Code coverage tool
ctags - If your using vim you can use ctags for method and variable outlines
gdb - I usually just use print statements for debugging but the new version of gdb allows you to step through Go code
race detector - Check for race conditions in your code
cross compile - Compile go for different platforms

Editors

Although I use vim for Go I have used Sublime Text 2 and SublimeGo. It provides a very complete Go development setup and code completion if you don't want to make the investment in vim.

Let me know if I missed anything.

Dockerfile Best Practices

2013-07-14T00:00:00+00:00

Dockerfiles provide a simple syntax for building images. The following are a few tips and tricks to help you get the most out of Dockerfiles.

1: Use the cache

Each instruction in a Dockerfile commits the change into a new image which will then be used as the base of the next instruction. If an image exists with the same parent and instruction ( except for ADD ) docker will use the image instead of executing the instruction, i.e. the cache.

In order to effectively utilize the cache you need to keep your Dockerfiles consistent and only add the alterations at the end. All my Dockerfiles start with the same 5 lines.

FROM ubuntu
MAINTAINER Michael Crosby <michael@crosbymichael.com>

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get upgrade -y

Changing MAINTAINER instruction will force docker to execute the proceeding RUN instructions to update apt instead of hitting the cache.

1. Keep common instructions at the top of the Dockerfile to utilize the cache.

2: Use tags

Unless you are experimenting with docker you should always pass the -t option to docker build so that the resulting image is tagged. A simple human readable tag will help you manage what each image was created for.

docker build -t="crosbymichael/sentry" .

2. Always pass -t to tag the resulting image.

3: EXPOSE-ing ports

Two of the core concepts of docker are repeatability and portability. Images should able to run on any host and as many times as needed. With Dockerfiles you have the ability to map the private and public ports, however, you should never map the public port in a Dockerfile. By mapping to the public port on your host you will only be able to have one instance of your dockerized app running.

# private and public mapping
EXPOSE 80:8080

# private only
EXPOSE 80

If the consumer of the image cares what public port the container maps to they will pass the -p option when running the image, otherwise, docker will automatically assign a port for the container.

3. Never map the public port in a Dockerfile.

4: CMD and ENTRYPOINT syntax

Both CMD and ENTRYPOINT are straight forward but they have a hidden, err, "feature" that can cause issues if you are not aware. Two different syntaxes are supported for these instructions.

CMD /bin/echo
# or
CMD ["/bin/echo"]

This may not look like it would be an issues but the devil in the details will trip you up. If you use the second syntax where the CMD ( or ENTRYPOINT ) is an array, it acts exactly like you would expect. If you use the first syntax without the array, docker pre-pends /bin/sh -c to your command. This has always been in docker as far as I can remember.

Pre-pending /bin/sh -c can cause some unexpected issues and functionality that is not easily understood if you did not know that docker modified your CMD. Therefore, you should always use the array syntax for both instructions because both will be executed exactly how you intended.

4. Always use the array syntax when using CMD and ENTRYPOINT.

5. CMD and ENTRYPOINT better together

In case you don't know ENTRYPOINT makes your dockerized application behave like a binary. You can pass arguments to the ENTRYPOINT during docker run and not worry about it being overwritten ( unlike CMD ). ENTRYPOINT is even better when used with CMD. Let's checkout my Rethinkdb Dockerfile and see how to use this.

# Dockerfile for Rethinkdb 
# http://www.rethinkdb.com/

FROM ubuntu

MAINTAINER Michael Crosby <michael@crosbymichael.com>

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
RUN apt-get update
RUN apt-get upgrade -y

RUN apt-get install -y python-software-properties
RUN add-apt-repository ppa:rethinkdb/ppa
RUN apt-get update
RUN apt-get install -y rethinkdb

# Rethinkdb process
EXPOSE 28015
# Rethinkdb admin console
EXPOSE 8080

# Create the /rethinkdb_data dir structure
RUN /usr/bin/rethinkdb create

ENTRYPOINT ["/usr/bin/rethinkdb"]

CMD ["--help"]

This is everything that is required to get Rethinkdb dockerized. We have my standard 5 lines at the top to make sure the base image is updated, ports exposed, etc... With the ENTRYPOINT set, we know that whenever this image is run, all arguments passed during docker run will be arguments to the ENTRYPOINT ( /usr/bin/rethinkdb ).

I also have a default CMD set in the Dockerfile to --help. What this does is incase no arguments are passed during docker run, rethinkdb's default help output will display to the user. This is same functionality that you would expect interacting with the rethinkdb binary.

docker run crosbymichael/rethinkdb

Output

Running 'rethinkdb' will create a new data directory or use an existing one,
  and serve as a RethinkDB cluster node.
File path options:
  -d [ --directory ] path           specify directory to store data and metadata
  --io-threads n                    how many simultaneous I/O operations can happen
                                    at the same time

Machine name options:
  -n [ --machine-name ] arg         the name for this machine (as will appear in
                                    the metadata).  If not specified, it will be
                                    randomly chosen from a short list of names.

Network options:
  --bind {all | addr}               add the address of a local interface to listen
                                    on when accepting connections; loopback
                                    addresses are enabled by default
  --cluster-port port               port for receiving connections from other nodes
  --driver-port port                port for rethinkdb protocol client drivers
  -o [ --port-offset ] offset       all ports used locally will have this value
                                    added
  -j [ --join ] host:port           host and port of a rethinkdb node to connect to
  .................

Now lets run the container with the --bind all argument.

docker run crosbymichael/rethinkdb --bind all

Output

info: Running rethinkdb 1.7.1-0ubuntu1~precise (GCC 4.6.3)...
info: Running on Linux 3.2.0-45-virtual x86_64
info: Loading data from directory /rethinkdb_data
warn: Could not turn off filesystem caching for database file: "/rethinkdb_data/metadata" (Is the file located on a filesystem that doesn't support direct I/O (e.g. some encrypted or journaled file systems)?) This can cause performance problems.
warn: Could not turn off filesystem caching for database file: "/rethinkdb_data/auth_metadata" (Is the file located on a filesystem that doesn't support direct I/O (e.g. some encrypted or journaled file systems)?) This can cause performance problems.
info: Listening for intracluster connections on port 29015
info: Listening for client driver connections on port 28015
info: Listening for administrative HTTP connections on port 8080
info: Listening on addresses: 127.0.0.1, 172.16.42.13
info: Server ready
info: Someone asked for the nonwhitelisted file /js/handlebars.runtime-1.0.0.beta.6.js, if this should be accessible add it to the whitelist.

And there it is, a full Rethinkdb instance running with access to the db and admin console by, interacting with the image the same way you interact with the binary. Very powerful and yet extremely simple. I love simple.

5. ENTRYPOINT and CMD are better together.

I hope this post helps you to get started working with Dockerfiles and building images that we all can use and benefit from. Going forward, I believe that Dockerfiles will be a very important part of what makes docker so simple and easy to use whether you are consuming or producing images. I plan to invest much of my time to provide a complete, powerful, yet simple solution to building docker images via the Dockerfile.

Docker Wallpapers

2013-07-06T00:00:00+00:00

So we finally have the new docker logo but I have not seen any wallpapers or anything else. The image posted is not very high resolution so I did some quick editing to fill in the background. All credit goes to the original post and author, I just added more gray to avoid stretching on my Retina display.

Downloads

Retina

1920x1080

Dockerfile deep dive

2013-07-02T00:00:00+00:00

Docker is an amazing project. There is plenty of great documentation and tutorials online to get you up and running with docker, building images, and deploying your apps. There is not much that I can say that others have not said better. Therefore, I am going to start a series of posts on the docker internals. How to hack on the Dockerfiles, adding new instructions, and how other features are implemented in the code base.

Getting started

The first thing that you need to do to begin hacking on docker is to fork the repo on github. If you already have a fork you need to make sure you stay up-to-date with the master branch. Docker development is moving very fast, make sure that you have an upstream set in your fork so that you can pull and rebase changes for the master repository.

origin  git@github.com:crosbymichael/docker.git (fetch)
origin  git@github.com:crosbymichael/docker.git (push)
upstream        git://github.com/dotcloud/docker.git (fetch)
upstream        git://github.com/dotcloud/docker.git (push)

Next make sure that I have all the upstream changes pulled down before creating a dev branch.

╭─michael@codeassistant  ~GOPATH/src/github.com/dotcloud/docker  ‹master›
╰─<<: git pull upstream master
remote: Counting objects: 133, done.
remote: Compressing objects: 100% (65/65), done.
remote: Total 74 (delta 49), reused 34 (delta 9)
Unpacking objects: 100% (74/74), done.
From git://github.com/dotcloud/docker
 * branch            master     -> FETCH_HEAD
 19 files changed, 203 insertions(+), 50 deletions(-)

Volumes

So now that we are ready to start coding we need to know what we will be implementing. Currently you can only specify volumes for your container when calling docker run -v. We want to be able to specify volumes inside a Dockerfile instead of on the cmd line. This is what we will be implementing. I'll start by creating a branch called buildfile-volumes.

There are a few files that we will be focusing on. buildfile.go and builder.go. The buildfile.go is where we parse the Dockerfile and build an image and builder.go is where we take the image config and the config from the commandline to build the container.

Buildfile.go

So lets jump in and see how the Dockerfiles work. So down near the bottom of the file you will see the Build method. Parsing the Dockerfile is very simple, like it should be. It starts by iterating over all the lines in the file and skipping comments and empty lines.

line, err := file.ReadString('\n')
if err != nil {
    if err == io.EOF && line == "" {
        break
    } else if err != io.EOF {
        return "", err
    }
}
line = strings.Trim(strings.Replace(line, "\t", " ", -1), " \t\r\n")
// Skip comments and empty line
if len(line) == 0 || line[0] == '#' {
    continue
}

Next we get the instruction, i.e. RUN and the arguments to the instruction. After we have the instruction and the arguments docker uses reflection to call the correct method in the buildfile.

instruction := strings.ToLower(strings.Trim(tmp[0], " "))
arguments := strings.Trim(tmp[1], " ")
stepN += 1

method, exists := reflect.TypeOf(b).MethodByName("Cmd" + strings.ToUpper(instruction[:1]) + strings.ToLower(instruction[1:]))
if !exists {
    fmt.Fprintf(b.out, "# Skipping unknown instruction %s\n", strings.ToUpper(instruction))
    continue
}
ret := method.Func.Call([]reflect.Value{reflect.ValueOf(b), reflect.ValueOf(arguments)})[0].Interface()
if ret != nil {
    return "", ret.(error)
}

To get the method of the buildfile by name, we prepend 'Cmd' and make sure that the first char of the instruction is upper case and the rest of the instruction is lowercase. Therefore, if we want to add a new instruction VOLUME we need a new method on the buildfile called CmdVolume.

func (b *buildFile) CmdVolume(args string) error {

}

Now we need to think about what type of args will be passed to the VOLUME instruction. We want to support one to many volumes passed as the args. The standard is to pass a json array just like you down with CMD.

VOLUME /redis/data
# or
VOLUME ["/redis/data", "/redis/tmp"]

So the method for CmdVolume needs to look like this.

func (b *buildFile) CmdVolume(args string) error {
        if args == "" {
                return fmt.Errorf("Volume cannot be empty")
        }

        var volume []string
        if err := json.Unmarshal([]byte(args), &volume); err != nil {
                volume = []string{args}
        }
        if b.config.Volumes == nil {
                b.config.Volumes = NewPathOpts()
        }
        for _, v := range volume {
                b.config.Volume[v] = struct{}{}
        }
        if err := b.commit("", b.config.Cmd, fmt.Sprintf("VOLUME %s", args)); err != nil {
                return err
        }
        return nil
}

Basically if we are able to Unmarshal the value into a json array we will append those values to the config else we will just use the single value.

Now we need to make sure that when a container is built we get any volumes added to the image config copied over to the container config. Containers are built in the builder.go file and we need to make sure the image config is merged with the user config. One thing to note is that the user config ( values that come in from the cmd line on docker run ) take precidence over image config values.

The MergeConfig function is located in utils.go. This is where we will add the new lines for merging the Volumes if no userConf Volumes exist.

func MergeConfig(userConf, imageConf *Config) {
        if userConf.Hostname == "" {
                userConf.Hostname = imageConf.Hostname
        }
        // ...
        if userConf.Cmd == nil || len(userConf.Cmd) == 0 {
                userConf.Cmd = imageConf.Cmd
        }
        if userConf.Dns == nil || len(userConf.Dns) == 0 {
                userConf.Dns = imageConf.Dns
        }
        if userConf.Entrypoint == nil || len(userConf.Entrypoint) == 0 {
                userConf.Entrypoint = imageConf.Entrypoint
        }
        //New code
        if userConf.Volumes == nil || len(userConf.Volumes) == 0 {
                userConf.Volumes = imageConf.Volumes
        }
}

Unit tests and Docs

The last changes that we have to make are to update the unit tests and documentation with the new instruction. Obviously you will need to execute the unit tests to make sure that your changes did not break anything.

Now it is time to build your docker binary and start user testing.

I wrote a small Dockerfile to test the new instruction.

FROM ubuntu

RUN mkdir /test

VOLUME ["/test"]

There are a few tests that we can run to verify this change. The first thing that we need to do is run docker build . to make sure the instruction is processed.

docker build .
Uploading context 10240 bytes
Step 1 : FROM ubuntu
---> 8dbd9e392a96
Step 2 : RUN mkdir /test
---> Using cache
---> 5f7c55f3749c
Step 3 : VOLUME ["/test"]
---> Running in 5c453826366d
---> f738fc81dc78
Successfully built f738fc81dc78

Next we can run docker inspect on the image id to make sure that the volume is added to the config.

//docker inspect <imageid>
{   "Volumes": {
       "/test": {}
    },
}

Now run a container from the image, make sure the volume mounted, and inspect the container for the volume mount point.

//docker inspect <containerid>
{"Volumes": {
    "/test": "/var/lib/docker/volumes/da262abbea2ee31ad27287155246cc00e8ec66fc87a503bf230514cb1e8cb3ca/layer"
},
"VolumesRW": {
    "/test": true
}}

If everything looks good and your tests complete successfully, review your diff carefully, commit, push, and open a pull request. You can view this the pull request on github. Hopefully the maintainers of the project will not require too many changes to your pull request but always be open to changes and suggestions.

Happy hacking.

Remote Development

2013-06-07T00:00:00+00:00

In the past I have not been a huge fan of "the cloud" for all of the work that I need to do. I still prefer native over web apps but I also like the convenience of having my data accessible everywhere. However, over the past few months I have been doing more work in web apps and doing development over ssh on my own server. This post will go over some of the necessary tools that is needed to make remote development awesome.

Cloud9

I used cloud9 ide for over a week. I was able to get the app running on my server with a lot of pain and dependencies. All of the nodejs app that I have tried to setup have 100's of dependencies and any one of them can fail on install. Anyways, I was able to get the app running and was able to do small development tasks on it. It is mostly geared towards web development but has syntax support for many other languages. It ran fine for a while but it would always throw a few errors and I had an issue with data loss when I encountered an error on save. After that I felt like I was tying a paper for college where I was paranoid about data loss and saving the Word doc ever few minutes.

I wanted to install cloud9 on a LXC container so that it is properly sandboxed but the dependencies kept failing to install until I gave up. I think a web ide is still premature for the type of development that I need to do and the responsivneness that I am used to.

Remote Stack

So if I am not using a web ide what am I using? I am running an Ubuntu 12.04 64bit server that I ssh into. Ssh is secure, fast, and just works. It provides the exact same shell that is on my local machine via my .dotfiles. Here is a quick list of my setup:

Setup Zsh ( oh-my-zsh ) tmux vim git vagrant docker rdiff-backup irssi python 2.7 Go 1.1 * compilers, libs, etc...

With these tools I have a consistent and persistent development environment from any location. The one thing that I do not like to do on the cmd prompt is managing sql databases. I like to have a gui when working with a sql db.

Shell Zsh plus oh-my-zsh is very powerful and fast. It provides everything that I need and works the same on linux and osx. The plugins for many of the apps that I already use and tab completion make my workflow very efficient.

Editor An ssh workflow is not going to work if you use a heavy ide to write code. Working as a .NET dev by day I couldn't imaging working with C# and .NET without Visual Studio because the framework is so large and the namespaces and inheritance chains can be very deep. Being able to remember not only the large, OO and pattern heavy applications that are written in .NET along with the entire .NET frameworks (+ ASP stack) it is nearly impossible to navigate the projects without Visual Studio doing it for you. However, the code that I write for open source, freelancing, and myself is Python, Go, and your standard web shit (html, css, js) I prefer a text editor over an ide.

My editor of choice is vim. I have used vim for a long time. I took a small six month break from vim and used Sublime Text 2 but it started to slow down after a while and I will still using vim when I ssh'd into a server. So my editing skills were split between two editors and it was just time for me to come back home to hjkl. Plus Sublime Text 2 does not work over ssh. It has good code completion for the languages that I work in so with my setup it is basically a very lightweight ide.

Tmux When working over ssh the biggest issue is persisting your state when you need to logout. It is no fun when you have a few vim tabs, splits, and other processes running while coding then after you logout, then log back in you have to recreate everything to get back where you were. I used to use screen for this but I have recently switched to tmux because of the added features. Tmux allows you to persist your sessions, have multiple windows ( ssh sessions) and terminal splits to provide an entire workspace on one ssh connection. You can logout and back in and start coding right away.

You can even have an irc client running in your terminal while working and easily switch back and forth with a keystroke. Tmux (or screen) is a must have when working over ssh, it is what makes a productive workflow possible.

Virtualization Installing dependencies on your development boxes is not a wise idea when virtualization is so easy. I use both vagrant with puppet and docker for running additional boxes on my machine. This allows me to install and run postgres, redis, etc and take these resources down when not in use without having to junk my box up with all the dependencies that are required for modern development.

Backup Even though you are "developing in the cloud" that does not mean you do not need to backup your workspace. Backups are really simple with rdiff-backup. It does incremental backups and is fast and efficient. I do daily backups of my development folder to another hard drive via a cron job to fire rdiff-backup and a weekly backup to S3 of the files.

That's about it. Any computer with a terminal allows me to securely connect to my workspace and resume on whatever I was working on. This setup also allows me to have a smaller and more portable computer because all the processing is done on my server that has fast cpus, ssd, and lots of ram. It has taken me a while to get used to working on the server and not on my local file system. I had to delete my development folder on my laptop (initially) so that I wouldn't be tempted to change files locally.

I do run the risk of not having internet access and not being able to access my files so I sill sync my development folder to my laptop. Hopefully more people will be able to look at hardware as a keyboard and a screen to connect to an always online workspace that is where ever you are.

Golang JSON RPC

2013-04-23T00:00:00+00:00

Go's rpc package has an example for the default rpc implementation but nothing for the jsonrpc package. Here is how to get it work. The rpc example uses some helpers built into the rpc package, I just looked at that code in Go's stdlib to figure out the proper way to get the jsonrpc package to work.

package main

import (
    "fmt"
    "log"
    "net"
    "net/rpc"
    "net/rpc/jsonrpc"
    "test/server"
)

func startServer() {
    arith := new(server.Arith)

    server := rpc.NewServer()
    server.Register(arith)

    server.HandleHTTP(rpc.DefaultRPCPath, rpc.DefaultDebugPath)

    l, e := net.Listen("tcp", ":8222")
    if e != nil {
        log.Fatal("listen error:", e)
    }

    for {
        conn, err := l.Accept()
        if err != nil {
            log.Fatal(err)
        }

        go server.ServeCodec(jsonrpc.NewServerCodec(conn))
    }
}

func main() {
    go startServer()

    conn, err := net.Dial("tcp", "localhost:8222")

    if err != nil {
        panic(err)
    }
    defer conn.Close()

    args := &server.Args{7, 8}
    var reply int

    c := jsonrpc.NewClient(conn)

    for i := 0; i < 1; i++ {

        err = c.Call("Arith.Multiply", args, &reply)
        if err != nil {
            log.Fatal("arith error:", err)
        }
        fmt.Printf("Arith: %d*%d=%d", args.A, args.B, reply)
    }
}

Go Lang

2013-04-12T00:00:00+00:00

If you don't know about Go it's a programming language for now. Most of the languages that we use everyday are old, 10+ years old. After so many years languages need to start to think about news ways to solve the same problems. Go is a new language that takes a different stance on how you write and deploy code. Go is a dynamically typed but statically compiled language. I have "played" with Go pre 1.0 a few times and at the 1.0 release but it never really stuck. I felt like I was writing to much code for what I wanted to accomplish. However, this last time working with Go 1.1beta I have started to do a few projects and packages with Go and I am really impressed with the language.

Abstractions

I write a lot of Python code and I still enjoy writing Python more than any other language so I am used to a fairly high level of abstraction from the language. Even working in C# with tools like LINQ you get trained to expect abstractions for working with collections. Also when used correctly, I really like having generics available in my language.

You miss these things with Go. Go is at a lower level of abstraction in many areas. Looping and working with arrays and slices ( variable sized collections ) is often back to the for i loop. Going from LINQ to anything less hurts. I find my self working with []byte more in Go than I have in any other language. The Go stdlib does provide you with some abstractions in the io packages but you still are working with byte arrays often.

Lets look at reading a file line by line in both Python and Go.

with open('names.txt') as f:
    for line in f.readlines():
        print line

Python is pretty straight forward. Very simple to remember and can be done with no imports from the stdlib. Now lets look at the Go example.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("names.txt")
    if err != nil {
        panic(err)
    }

    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
}

So the go example is a little more than just 3 lines of code. We have the package declaration, 3 imports from the stdlib, and the go error handling code in the main func. Even though Go has more code it is still simple and elegant. You do have to learn the stdlib to do simple tasks but once you do, you have a very powerful set of tools. The go example is every more amazing once you learn about implicit interface implementation.

No Classes

I like OOP and modeling classes and their interaction with each other to create a fluent and straight forward API. However, I also feel like OOP is taken too far. People will insist that you have to design a polymorphic interface, provide factories, and strategies when all you really need to a simple function. I believe in design patterns for solving problems in software but I also believe in simplicity, clarity, and elegance in the code that I write. For me, simple wins over "over engineering." Knowing when to design and when to bang out a function comes with experience.

With Go you have types and interfaces but no classes and inheritance. At the beginning this was hard for me. However, after working with packages, types, and interfaces in Go I do not miss classes. I feel like my code is simple and straight forward with only types and packages. It is OOP at it's purest, data with actions. You can still attach methods to your types and you have public and internal access via upper and lower case names. If a type and/or function begins with an upper case it is exported to be used outside of that package, if it is lower case it is internal to that package.

Interfaces in Go are also very modern. You can have a type and implement an interface without explicitly stating that you implement that interface. If we have an interface that has a

Write([]byte) int

then in Go, any type that has a method Write that takes a byte array and returns an int implements that interface, which is exactly what you want. With interfaces you are basically saying that you don't care about the type or other methods, all I care about is that whatever I have passed to me has a Write method that takes bytes and returns an int.

Why Go

So why am I writing a post on a language with no classes and missing certain abstractions that we are used to? Because their some home run features that make Go an amazing language to work with.

Concurrency

There are many articles about concurrency in Go with gorutines and channels so I am not going to say much here except that they are awesome.

Compiled binary

Deploying a python application takes automation. You need to to create a virtual environment, have a requirements file with all of your dependencies, and then finally copy your application code up to the server. There are many tools for deploying python code and it really is not that bad but it is also not simple and there are many places that things can go wrong.

With Go you have a compiled binary with all the dependencies compiled into the binary. You do not even have to have the Go tool chain installed on the computer that you are deploying your go application to run the code. You just drop the executable on the machine and your good to go, nothing else to do. This is amazing.

Also being compiled into native machine code makes go very fast.

Testing and Packages

Testing is simple and easy. Automating testing is simple and easy. Have you tried to automate C# unit tests? Oh, you don't own VS Pro with unit testing?

Installing dependencies is very easy with "go get". You install dependencies exactly the same way as you have your own packages.

Go Doc

All of Go's documentation is auto generated by the commends in the code. This includes packages in the stdlib and your own packages in your GOPATH. I have been doing a lot of work in an area with no Internet access to I really appreciate being able to run the package documentation and other general Go documentation on my local machine.

godoc -http=:8080

This provides me with all the documentation and examples available if I was connected online to the golang.org website.

Parting words

Hopefully this has peaked your interest in Go and maybe you can give it a try. I am going to start using Go as my main language for backend services and processes.

SecureString How To

2013-03-09T00:00:00+00:00

Working with the SecureString class in .NET is fairly easy. This class allows you to keep string data encrypted in memory. Maybe you load connection strings, passwords, or financial data into memory when your application starts or loads an object so the least you can do is use the SecureString class to keep this information encrypted while the data stays in memory.

Adding your data

Adding a string content to the class is very easy.

public static void Main(string[] args)
{
    var hushhush = "Something I need to keep on the down low";

    using (var secureString = new SecureString())
    {
        foreach (var chr in hushhush.ToCharArray())
        {
            secureString.AppendChar(chr);
        }
        secureString.MakeReadOnly();
    }
    Console.ReadLine();
}

All you need to do is convert your string into a char array and append each char to the secure string. After you have finished writing all data to the secure string you will want to call the MakeReadOnly method so that the instance becomes immutable.

Getting your data back out

Now this is where things get weird. It's nice not to have a convenient method on the SecureString class to get to your data for security reasons but you actually have to work with pointers and unmanaged resources to get to the data.

Because SecureString is a sealed class I made a little extension class that adds a GetString() method for reading the string data from the class.

static class SecureStringExtensions
{
    public static string GetString(
        this SecureString source)
    {
        string result = null;
        int length = source.Length;
        IntPtr pointer = IntPtr.Zero;
        char[] chars = new char[length];

        try
        {
            pointer = Marshal.SecureStringToBSTR(source);
            Marshal.Copy(pointer, chars, 0, length);

            result = string.Join("", chars);
        }
        finally
        {
            if (pointer != IntPtr.Zero)
            {
                Marshal.ZeroFreeBSTR(pointer);
            }
        }

        return result;
    }
}

The Marshal class actually has a BSTR method for use with the SecureString. Just remember when working with unmanaged resources you will need to manually release and free the pointers that you used.

Basically if you have not worked with Marshal and pointers in C# you will have to get the pointer to the char[] from the secure string. Then we use Marshal.Copy to marshal the *char[] to our managed char[] using the length of the string. A simple string.Join and we are home free. Just remember to zero out our pointer to release the resources.

F# Url Shortener

2013-01-27T00:00:00+00:00

Every year I try to learn a new language. If you are a one trick pony you are not only limiting your self in opportunities but you are also limiting your thinking to the one paradigm that you know.

So this year I wanted to learn something very different from what I am used to in the OO world ( Python, C#, Objective - C). I decided to learn a functional language this year called F#. I looked at a few different functional languages ( Scala, Erlang ) before choosing F#. I felt that it would be the most practical language for what I do but still provide me with the paradigm shift I was seeking. It does have OO parts being part of the .NET family but the functional paradigm still dominates it's landscape.

Usually when I learn a new language I rewrite a program that I have already done before. This helps me to focus on the language and not worry about the problem at hand because I already know the solution. So this program is a url shortener that makes a request to the Google api via JSON. I have written this url shortener in Python, Go, Objective - C, and C# so I know what I need to do and I have a reference. So lets get started.

To the code!

The first thing we need to do to create this program in F# is to "open" our dependencies. So at the top of a blank .fs file add these lines:

#light

open System.Net
open System.IO
open System.Text
open System.Web.Script.Serialization

We need the Net package for the WebRequest, IO for a StreamReader, Text for Encoding, and the Web.Script.Serialization for the JavaScriptSerializer so that we can convert the JSON response into a type that we can use. You will have to add the System.Web.Extensions dll reference to your F# project to use the JavaScriptSerializer.

Next we need to create a new type that the JavaScriptSerializer will deserialize the response to. The response JSON looks like this:

{"id":"short url", "kind": "kind of request", "longUrl": "long url"}

We will have to make this type a class with a default constructor so that the JavaScriptSerializer can create the instance. The members of the type; id, kind, and longUrl will have to be mutable so that the serializer can set the values. By default all types in F# are immutable unless set to mutable.

type Response = class
    val mutable id : string
    val mutable kind : string
    val mutable longUrl : string
    new() = {id = ""; kind = ""; longUrl = ""}
end

So there is the Response type with the default constructor. Next I will create another type that will hold all the request data that is needed. This is more for convenience and not really needed.

type Request = { url: string; requestMethod:string; contentType : string; data : string }

Now lets get the data required and populate a Request type. We will receive the long url via the command line args and the other data such as request method and content type can be hard coded along with the url to the Google api.

let args = System.Environment.GetCommandLineArgs()
let request = {
    url = "https://www.googleapis.com/urlshortener/v1/url";
    requestMethod = "POST";
    data = args.[args.Length - 1];
    contentType = "application/json" }

Functions

We will need a few helper functions that can convert a string in to a byte array, read a Stream into a string, and deserialize the response string into our Response type.

let get_bytes (str:string) =
    Encoding.UTF8.GetBytes(str)

let read_to_string (stream:Stream) =
    use reader = new StreamReader(stream)
    reader.ReadToEnd()

let deserialize_response (stream) =
    let serializer = new JavaScriptSerializer()
    serializer.Deserialize<Response>(read_to_string stream)

These three functions are very straight forward and to the point. One issue that I have ran into is that using classes from the .NET framework often requires a type annotation to be added to the function parameter(s) type inference cannot be used with overloaded methods. It kinda sucks but it is what it is. You will also notice the "use" keyword in the read_to_string function. This is the same as the "using" keyword in C#. Other than that everything is pretty obvious to a .NET developer looking at this code.

Now we will need to create a WebRequest, add the default parameters, write our data to the request stream, and then read the response.

let add_params (webRequest:WebRequest) =
    webRequest.ContentType <- request.contentType
    webRequest.Method <- request.requestMethod
    webRequest

let write_to_stream (webRequest:WebRequest) =
    let stream = webRequest.GetRequestStream()
    let data = get_bytes (sprintf "{\"longUrl\":\"%s\"}" request.data)
    stream.Write(data, 0, data.Length)
    webRequest

let get_response (webRequest:WebRequest) =
    webRequest.GetResponse().GetResponseStream()

let print_response (o:Response) =
    printf "%s" o.id

Easy. One of the things that you will notice is the lack of a return keyword. This is because whatever is on the last line of a function is automatically returned. You can see that in the write_to_stream function, even though it takes a webRequest as an input parameter, I set it on the last line so it can be returned. In C# ( and other languages ) this is often known as a "Fluent api". This will also help us a little bit later when we chain all of these functions together to process the request.

You can also see the "<-" operator when I assign values from our request type to the webRequest. You do not use the "=" operator for member assignment.

Lastly, you may have noticed that I never created the actual webRequest in the above code snippet. That is because in the final function we will create the webRequest and chain all of these functions together.

let get_short_url() =
    HttpWebRequest.Create(request.url)
        |> add_params
        |> write_to_stream
        |> get_response
        |> deserialize_response
        |> print_response

So get_short_url is the final function that ties everything together. We create the webRequest and use it's value to chain all the remaining functions. Each function consumes the result of the previous function this way we can use the "|>" to chain them together. This is very LINQ like but taken to a whole new level in functional programming.

So now all that is left is to get_short_url when a user passes in a long url on the command line. For this we will use a pattern match in F#.

match request.data.StartsWith("http") with
    | true -> get_short_url()
    | false -> printfn "No url passed"

System.Console.ReadLine()

Because our Request type's data member is a string we can use the standard .NET string function "StartsWith" to see if the first chars are "http". Then by using the "match with" we can check for each boolean value to determine our control flow. If true, we will call our get_short_url function, if false, we will tell the user that they did not pass a url. If you are on Windows you will want to add the "Console.ReadLine" call at the end of the file or the command prompt will exit without you seeing your results.

Complete Code

#light

open System.Net
open System.IO
open System.Text
open System.Web.Script.Serialization

type Response = class
    val mutable id : string
    val mutable kind : string
    val mutable longUrl : string
    new() = {id = ""; kind = ""; longUrl = ""}
end

type Request = { url: string; requestMethod:string; contentType : string; data : string }

let args = System.Environment.GetCommandLineArgs()
let request = {
    url = "https://www.googleapis.com/urlshortener/v1/url";
    requestMethod = "POST";
    data = args.[args.Length - 1];
    contentType = "application/json" }

let get_bytes (str:string) =
    Encoding.UTF8.GetBytes(str)

let read_to_string (stream:Stream) =
    use reader = new StreamReader(stream)
    reader.ReadToEnd()

let print_response (o:Response) =
    printf "%s" o.id

let deserialize_response (stream) =
    let serializer = new JavaScriptSerializer()
    serializer.Deserialize<Response>(read_to_string stream)

let add_params (webRequest:WebRequest) =
    webRequest.ContentType <- request.contentType
    webRequest.Method <- request.requestMethod
    webRequest

let write_to_stream (webRequest:WebRequest) =
    let stream = webRequest.GetRequestStream()
    let data = get_bytes (sprintf "{\"longUrl\":\"%s\"}" request.data)
    stream.Write(data, 0, data.Length)
    webRequest

let get_response (webRequest:WebRequest) =
    webRequest.GetResponse().GetResponseStream()

let get_short_url() =
    HttpWebRequest.Create(request.url)
        |> add_params
        |> write_to_stream
        |> get_response
        |> deserialize_response
        |> print_response

match request.data.StartsWith("http") with
    | true -> get_short_url()
    | false -> printfn "No url passed"

System.Console.ReadLine()

So that's it. I am still very fresh to F# and functional programming in general so my code does not look very elegant. It feels a little verbose working with the .NET classes in a functional language but hopefully I will learn more and be able to remove some of the excess and make this program more idiomatic to functional programming.

Infrastructure - Part 1

2012-09-29T00:00:00+00:00

Servers

Running your own infrastructure is awesome, if your a hacker like me. If your a square and you don't want to mess with your own hardware you have two choices if you are going to host your website or services online. If you don't feel like spending $1k a year you will probably have a VPS or EC2 instance with less than 1GB of RAM.

You can get a shared hosting provider and use what they give you. Basically PHP and Mysql. I run a heavy Python stack and Postgesql. PHP is not an option for me. Also 1GB of ram is not enough. I have to run a Minecraft server for myself and a few friends and I need at least 3-4GB of RAM just for Minecraft. ( This is also a test of how robust my setup is because if Minecraft would go down my friend my hurt himself. ) I also like running other services like jenkins as a build server. You still have to spend some money on things like a battery backup or upgrade your internet connection.

The trick to running your website on your own hardware and internet connection is to host all your content, images, js, css, etc.. on a CDN and only have your servers render the HTML and do database operations. Leave any content on S3. I run my websites on my own server and have S3 host all the images and content. You cannot tell the difference loading my page.

This is the first post of all the shit I have running. I'll try to give a high level overview and for some posts I will go into detail on the setup and settings. Keep an eye out for some of my shell scripts and other cool downloads.

Ajenti

Admin Console

Ajenti is an easy to install Python app that provides information about services running on your server. It also provides a nice UI for reading logs, managing cron jobs and users, and many other tasks.

On the main dashboard I can see the load and memory usage for my main node and the build server.

Ajenti has a nice plugin system. I have not dug into developing a plugin but it looks fairly straight forward. The default selection is great and plugins are fast and easy to install.

Munin

The charts that you are seeing are from Munin. It is a system monitoring application that can monitory local and remote systems. You need one computer as the server and then install munin-node on each computer (including the server) that you want to monitor. Munin is in apt and you only have a few lines to modify in the munin configs after install to get it up and running. You will have to have a HTTP server to serve the munin graphs so that ajenti can read them into the dashboard.

Check out the rest of these screenshots and view the links at the bottom of the page.

Logs

Users

Health

Usage Patterns

2012-09-28T00:00:00+00:00

Update: I will have updated graphs in a day or two. My time tracking app was only exporting the top 10 applications and I just found the setting to export the top 25.

I always have an app running on my computer that logs all the time I spend in each application. It helps to give some interesting insights how, what, how long, what days, and times do I do certain tasks. You can see what hours of the day that you are productive and what hours you just surf the web.

Usage by App

Just for clarification:

iTerm - terminal app
Forklift - file/ssh/ftp file management
SublimeText 2 - Text Editor

So Chrome wins pretty easy but I do spend a lot of my time in a terminal. I hate pixels and gui's. I wish I could do everything from the command line. The other 21% of my time is spent programming in SublimeText. Even though it is not a large amount 5% of my time sitting at my computer is managing files in Forklift. That is still a lot of time for something like moving a file from one directory to another.

These two graphs show pretty much the same thing. What hours of the day do I must most of my work. 8pm-10pm is when I do most of my work. However I do like the area graph about what apps I am using at what time.

You can see that I have a large spike in the morning with Chrome. This is when I get up and surf the internet looking for news or reading RSS feeds. I have a pretty constant usage of the terminal throughout the entire day but my development really picks up in the later hours with SublimeText and iTerm.

Hours by day

Thursday, Saturday, and Sunday are my big computing days. This mostly because I am at work throughout the week so I do what I want on the weekends.

Finally my usage throughout the past few months. You can read this graph a few different ways. You can look at the usage trends over time and see if I am starting to use a computer more or less. You can also look at how many hours I am on a computer during a single day. On average it's about ~4.5hrs a day. Not too bad at all.

You can study the graphs if you want and see if doing something like this would be advantageous to you. I always enjoy looking over data and seeing trends in my own life and how I use and interact with things. You can use data like this to determine when you are most productive by hour or day, day of week, etc...

Development Infrastructure

2012-09-10T00:00:00+00:00

Having servers that automate much of your development process is cool. Entities living out on the web just waiting to process any task that you throw their way.

I currently have two servers in my cluster that run my network. Everything from my public web sites, to mongodb and Postgresql, to private git and build servers. Why have all of this? Because it makes me happy when I push changes to my git server and within seconds the project is built, unit tests ran, and a report is emailed to me.

Public Site

I run a Python stack for my web server. Python 2.7 with django as the framework running on-top of lighttpd. Lighttpd also acts as the proxy server for the rest of the services that run on the cluster. Postgresql is the datastore for my sites. I still run mongodb on the server for quick, scheme-less storage.

On Node1 is my git repositories via Stash. Stash is not necessary but it provides a nice UI for managing repositories and users. I could do all of this via ssh but for $10 Starter License it is worth it. You can easily browse and see diffs on the web as well.

The only drawl back to Stash is that is runs on-top of Java and hogs RAM. The Stash instance consumes about 700-900MB of RAM for not doing much. The other memory hog is a minecraft server that I run. Minecraft consumes about 2GB+. This server has to be robust because if the minecraft sever goes down, I will have some angry users texting me until it's back online.

Build server

I run Jenkins-Ci as my build server. I have tried Bamboo for a week but the license was too restrictive for my needs, ( not enough jobs on the starter license ). For my python projects Jenkins has many nice plugins.

Even though the build server is running on Debian it can build both Python and .NET projects thanks to mono ( xbuild ). I am having some issues running nunit tests for a non-root user. I have not been able to figure out what the issue is yet so my .NET projects do not have automated tests.

Server Software

Both servers run Debian and are administered via ssh, no passwords just ssh keys. Each node does it's daily backup to the other with ~3day off-site dumps to Amazon S3.

Node1

Debian 6
lighttpd
git 1.7+ ( built from src )
django
mongodb
postgresql
sun-java
stash
Amazon S3 as CDN for content

Domains Hosted * CodeAssistant

Node2

Debian 6
sun-java
jenkins
mono ( built from git repo )
git 1.7+( built from src )

Future Changes Currently my blog crosbymichael.com ( this site ) is hosted on dreamhost. Also HTTPAssistant is a wordpress site hosted on dreamhost as well. My dreamhost service expires Feb. of next year so I will be moving both of these sites to Node1. I will replace wordpress with a static one page landing page. Hopefully I will have HTTPAssistant 2.0 done and replace that ugly thing.

web.py server setup

2012-08-04T00:00:00+00:00

Python is my favorite language to work with. It is so versatile. I can automate tasks, write quick scripts, process data, and even run a web application. It is very portable and I love the syntax. For a web framework I use web.py because it stays out of your way and does not impose itself on your application. With other frameworks you have to do it their way or struggle the entire time. It's awesome but setting up a "production" server is a bitch. Tutorials suck and getting Python to run under Apache is a waste of 6+ hours with all the config bullshit. Leave it and do it my way.

What you will learn:

Install lighttpd on Ubuntu/Debian
Install MySQL
Setup web.py to run under lighttpd
Setup MySQLDB for Python to MySQL

Ubuntu Server

Ubuntu or any Debian distro will work with this tutorial. I am currently running my server on Ubuntu Server 11.04. You need to make sure that Apache is not installed on your server. If it is remove everything Apache. Make sure you have shell plus access to install software on your server.

MySQL

Now we will need to install MySQL on your server before we do anything else. Just skip this part if you don't use MySQL. This is very easy, just run these two commands to install MySQL and the development libraries that Python needs.

apt-get install mysql-server libmysqlclient-dev

Remember that you should be running these as root or sudo. If you didn't know this just give up now. You will also need to setup your MySQL username and permissions for your web application to use. I am not going to go over this process because I am lazy. Google it if you don't know how.

lighttpd

Now we can install the lighttpd server.

apt-get install lighttpd

You can go ahead and browse to the site to see if lighttpd is functioning properly.

Python Build Tools and MySQLDB

With our web server and MySQL database setup on the server it is time to get our Python environment setup. Start by installing the buildtools and other Python development tools.

apt-get install python-setuptools python-dev build-essential

This will install "easy_install" and other tools so we can build the python MySQL driver. After this is complete we need to download and install the MySQLDB driver. This is easy if you have the libmysqlclient-dev and setuptools installed.

wget http://downloads.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz
tar -zxf MySQL-python-1.2.3.tar.gz

This will download and unzip the driver. Now "cd" in to the folder and run these two commands:

python setup.py build
python setup.py install

Web.py

All we have left to do is install web.py, flup, and configure lighttpd for our application. The first two are easy.

easy_install web.py
easy_install flup

Now lets configure lighttpd.

cd /etc/lighttpd/

Replace the contents of lighttpd.conf with the text below:

server.modules = (
    "mod_access",
    "mod_alias",
    "mod_compress",
    "mod_accesslog",
)

server.document-root        = "/var/www"
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"


## Use ipv6 if available
#include_shell "/usr/share/lighttpd/use-ipv6.pl"


compress.cache-dir          = "/var/cache/lighttpd/compress/"
compress.filetype           = ( "application/x-javascript", "text/css", "text/html", "text/plain" )

include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"

server.modules   += ( "mod_fastcgi" )
server.modules   += ( "mod_rewrite" )

 fastcgi.server = ( "/index.py" =>
 ("/" => ( "socket" => "/tmp/fastcgi.socket",
    "bin-path" => "/var/www/index.py",
    "max-procs" => 1,
   "bin-environment" => (
     "REAL_SCRIPT_NAME" => ""
   ),
   "check-local" => "disable"
 ))
 )

 url.rewrite-once = (
   "^/favicon.ico$" => "/static/favicon.ico",
   "^/static/(.*)$" => "/static/$1",
   "^/(.*)$" => "/index.py/$1",
 )

Note the server.username and server.groupname. This user/group will need proper permissions to your application files. Study the config above. Everything is simple and you should be able to change whatever settings that you need.

Make sure you updated the server.document-root path to where the location that you are going to be placing your application in. I just use /var/www for my applications. This path will need to be updated in the fastcgi.server section along with your main python file for web.py. I usually call my main application file "index.py" and the above config is setup to reflect that.

Now that we have lighttpd set up we can go into our /var/www directory and set up a sample web.py application ( if you do not already have a working app ). We can just use this simple template. Make a note of the shebang at the top of the file. You will need to add that to your application file. Just save the file as "index.py".

#!/usr/bin/env python

import web

urls = ('/(.*)', 'Index')

application = web.application(urls, globals())
web.config.debug = True

class Index:

    def GET(self, name=''):
        return 'Hello World'

    def POST(self, name=''):
        return 'Hello World'

if __name__ == '__main__':application.run()

Make the file's owner www-data and make it executable. After that restart lighttpd.

chown www-data:www-data index.py
chmod +x index.py
service lighttpd restart

Go ahead and navigate to your site in your browser and you should see "Hello World". If you do not, cat out the lighttpd error log at:

cat /var/log/lighttpd/error.log

That's it. I hope this helps to save hours of your life, hours that I wasted trying to get web.py to run under Apache bullshit. This is the way to go, lighttpd is fast, easy to config, and perfect for web.py.

Objective - C LINQ

2012-07-06T00:00:00+00:00

I have been working on two iOS applications over the past few days and have been longing for a LINQ like syntax in Objective-C. LINQ is an amazing way to work with collections and it really adds a productivity boost to your coding. Since Objective C does not have LINQ I thought I would code it myself. Right now it only supports non-destructive operations but here it is.

@interface NSArray (IQuery)

- (id) first;
- (id) last;
- (id) firstOrDefault: (int(^)(id)) predicate;
- (id) firstOrDefault;

- (BOOL) any;
- (BOOL) any:(int (^)(id)) predicate;
- (BOOL) all:(int (^)(id)) predicate;
- (NSArray *) ofClass: (Class) class;
- (NSArray *) where: (int (^)(id)) predicate;
- (NSArray *) select: (id (^)(id)) predicate;
- (NSArray *) skip: (id (^)(id)) predicate;
- (void) forEach: (void(^)(id)) action;

@end

This is the current interface. Most methods return BOOL or an NSArray so that the query operations can be chained together.

Code

What we want to do in this example is to take an array with strings and ints, filter it by class, then by strings starting with 'h'. Here is how you do it:

NSArray * array = [[NSArray alloc] initWithObjects:
                          @"hot dog", 
                          @"hamburger", 
                          @"pizza", 
                          @"taco", 
                          @"pop", 
                          0, 
                          5, 
                          3, 
                          nil];

NSArray * filter = [[array 
                           ofClass:[NSString class]]
                           where:^(NSString * item) {
                            return [item characterAtIndex:0] == 'm';
                        }];

[filter forEach:^(NSString * i) { NSLog(i); }];

So that is it. The first IQuery method is "ofClass". This takes a class and returns an NSArray of all the items in the array that are of that type(class). It is the same idea as C#'s OfType.

Next is "where". This takes a predicate block that will return any item that satisfies the condition. The syntax is not as clean as C# implementation but it's a start.

Now after we have this filtered list with the chain of calls to "ofClass" and "where", we can now use the "forEach" method to enumerate the array, passing a block as the action to perform on each item.

Select

One last thing, if you want to select a certain property from an item in the array, you can use the "select" method to do so. Here we select the price for each of the items.

NSArray * array = [[NSArray alloc] initWithObjects:
                         [[Items alloc] initWithNameAndPrice:@"hamburger" :[NSNumber numberWithInt:3]], 
                         [[Items alloc] initWithNameAndPrice:@"hot dog" :[NSNumber numberWithInt:5]],
                         [[Items alloc] initWithNameAndPrice:@"pop" :[NSNumber numberWithInt:2]], 
                       nil];

NSArray * prices = [array select:(id)^(Items * item) {
    return item.price;
}];

This will return an array of NSNumber items. Nothing more.

That's all for now. If you are interested in the project I will have the binary to download soon. Just email me with any comments or questions.

Objective C Design Patterns - Factory

2012-05-21T00:00:00+00:00

Design patterns can help you solve complex coding problems with proven solutions. This will be a mini series about different design pattern implementations in Objective - C. I will be using a simple console app and updating the code on github with each new pattern.

Factory Pattern

This pattern provides a consistent way for object creation. A factory provides one point of creation for one or a group of objects. It also helps to not code to a concrete class.

When to use it? Many objects need created that share an interface Creation of complex objects Hide concrete classes and implementation from client code Provide one point of entry for the creation of a class or group of classes

Static Factory

Most of the time I find myself using a static factory instead of an abstract factory. A static factory is what we are going to focus on now. In the console app I have an interface for logging text to the console. There are two implementations for this interface. One uses NSLog and the other uses fprintf().

#import <Foundation/Foundation.h>

@protocol CMConsoleOutput <NSObject>

- (void) outputText: (NSString *) content;

@end


//Class one
#import "CMPrintLog.h"

@implementation CMPrintLog

- (void) outputText:(NSString *)content {

    fprintf(stdout, "%s\n", [content UTF8String]);
}

@end


//Class two
#import "CMLogOutput.h"

@implementation CMLogOutput

- (void) outputText:(NSString *)content {
    NSLog(content);
}

@end

Now we need a factory to choose what implementation the client should use without knowing the exact types of objects that are returned. The client does not care what they get as long as it implements the interface. It also helps to cut down the number of imports you have in your header files. You no longer have to import 5 different class headers; you just have to import one factory header. I went ahead an created the factory class file, typedef for identifying what type of output object the client wants, and a static method for creating/returning the object.

typedef enum ouputTypes {
    Timestamp,
    NoTimestamp
} OutputTypes;


@interface CMOutputFactory : NSObject

+ (id<CMConsoleOutput>) create: (OutputTypes) type;

@end

Now the only thing that we need to remember is to autorelease the objects that we create if you are not using ARC.

@implementation CMOutputFactory

+(id<CMConsoleOutput>) create:(OutputTypes)type {
    switch (type) {
        case Timestamp:
            return [[CMLogOutput alloc] init];
        case NoTimestamp:
            return  [[CMPrintLog alloc] init];
    }
}

@end

Finally we just need to use our new factory.

#import <Foundation/Foundation.h>
#import "CMOutputFactory.h"

int main(int argc, const char * argv[])
{

    @autoreleasepool {

        id withTimestamp = [CMOutputFactory create:Timestamp];
        id noTimestamp = [CMOutputFactory create:NoTimestamp];

        [withTimestamp outputText:@"Output without a time stamp"];

        [noTimestamp outputText:@"Without a time stamp"];
    }
    return 0;
}

Very easy, we only have one import and a consistent way to create objects. It is very straight forward for people consuming an API's classes when you provide a factory for object creation. They do not need to know the inner workings of your API, they just want a certain type of functionality and your factory provides that. If you have any questions just ask.

CodeAssistant

2012-05-14T00:00:00+00:00

My second application was published this week.

CodeAssistant is a simple tool to test code snippets in virtually any language on Windows. It has been in development for four months and has been a great learning experience for domain driven development and .NET's WPF.

The application is build on top of WPF using the MVVM design pattern. I created a MVVM framework called MVVM Async that you can find it on github. WPF and MVVM are still in their infancy and there are some rough edges and verbose syntax that needs fixed ( command delegates anyone? ). I like WPF and MVVM but I wish they would have just used the proved MVC tooling form ASP.NET MVC in WPF. There are a lot of developers familiar with MVC, partials, and the such that would have been a good fit for WPF. But no, they had to build another framework and you have to relearn everything all over again.

This has been the first project where I have built it from the ground up with domain driven design and solid patterns. Each of the layers in the application are completely decoupled from the others and are in their own projects. The view/application layer only has a reference to the ViewModel project and the ViewModel project only references the Domain layer.

Future Projects

I don't have any plans for a future project. I have been wanting to work on a game. Desktop or iOS, I have not decided. I may play with the XNA development toolkit before making a decision. A friend of mine said that Corona is a good platform to build games on. I have a few game ideas but don't know yet.

If anyone has a game that they want to work on, it does not matter what platform, I can code in any language, let me know.

Setup Ruby on Rails with Nginx and Unicorn

2012-02-27T00:00:00+00:00

I lost an entire day of coding trying to get his to work. We are all lucky that I am a developer and not a systems admin.

However around 9pm I finally have it running. I burned through a lot of screwed up VM's trying to get this to work but finally made it. I tried Apache, Nginx, Passenger, and finally Nginx and Unicorn worked like it should. I don't know what the issues were but the actual rails app was not being ran.

I'll try to go through every single step I make, from installing Debian and the settings that I use to the installing and compiling gems, rails, and Nginx. I will also try to make this into a bash script to automate much of the install incase you f something up.

Requirements

Email
Rails
MySQL
Nginx

Installing Debian 6

I am going to use Debain 6 ( net install ) because it has the bare minimum. Only what we install is what it is going to have. You can also use Ubuntu Server 10.04 or 11.x with these commands. They should work the same. I have tested this on Ubuntu Server 10.04 so I know it works.

Ok boot into your linux install and configure your languages. I set the hostname as:

nginx-rails-server

The domain name to my url.

Create your root and user passwords then for the disk partitioning just use the entire hard drive. I also choose to keep all system and user files on one partition.

Pick one of the package servers for debian. It does not matter.

Install Package Selection

Here is my package selection for install. We do not want a webserver or anything else. I usually install the Mail server so that I don't have to mess with it but since we are going to use Nginx and install mySQL manually all we want is the SSH and Mail servers installed right now.

Install GRUB boot load and restart.

Once your booted up, login really quick and run ifconfig to get the ip of the machine. We will want to SSH into it so we can use our desktop Terminal ( copy and paste ) instead of using a keyboard on the actual machine or through the VM.

Once you have the IP login as root via ssh. Lets nano and paste in the setup script from below. Chmod 755 it and let it rip.

Application Install

So now we need to get our base applications installed on the server. If you are using Ubutnu Server you will want to run:

sudo apt-get update
sudo apt-get upgrade

Also from here on out I will not place sudo infront of all the commands that need it. If you don't konw when you need to run the command with sudo or when not to, just go home. Don't waste my time emailing why when you run a command on Ubuntu it says "permission denied."

Here is the script that I will run to get the base apps installed from apt.

#!/bin/bash

apt-get update
apt-get upgrade -y


apt-get install wget build-essential ruby1.8 ruby1.8-dev irb1.8 rdoc1.8 zlib1g-dev libopenssl-ruby1.8 libopenssl-ruby libzlib-ruby libssl-dev libpcre3-dev -y

apt-get install vim -y
apt-get install git-core -y
apt-get install screen -y
apt-get install ctags -y
apt-get install curl -y
apt-get install libsqlite3-dev -y
apt-get install libcurl4-openssl-dev -y
apt-get install libssl-dev -y

If you notice I did not install mySQL in here because it will ask for the root passwords and I wanted this base install script to be fully automatic.

Now that that is done lets go ahead and isntall mySQL

apt-get install mysql-server libmysqlclient16 libmysqlclient-dev mysql-client mysql-common -y

When asked, pick a good mySQL root password.

Gems

Now that we have everything from apt installed lets get rubygems installed. I don't want to install gems via apt so we are going to grab the tarbal and install that way.

cd /opt/
wget http://production.cf.rubygems.org/rubygems/rubygems-1.5.1.tgz
tar -zxvf rubygems-1.5.1.tgz
cd /opt/rubygems-1.5.1/
ruby setup.rb

This will install gems in:

/usr/bin/gem1.8

Logout of your bash session and them back in and you should be able to type gem1.8 to execute some commands.

Create another bash setup script for all these gem installs. They will take a while so it's easier to just let it roll. I chose not to install the rdoc and ri because this is the server and we don't need it.

#!/bin/bash
gem1.8 install fastthread --no-rdoc --no-ri 
gem1.8 install rake rack --no-rdoc --no-ri 
gem1.8 install rails --no-rdoc --no-ri 
gem1.8 install mysql --no-rdoc --no-ri 
gem1.8 install unicorn --no-rdoc --no-ri 
gem1.8 install sqlite3-ruby --no-rdoc --no-ri 
gem1.8 install execjs --no-rdoc --no-ri

Javascript engine

Rails needs a javascript engine to run. Pick your favorite one and install it. I will isntall node.js for this. I will not use apt or gems to install node.js but will grab the latest stable release from github and compile it.

Lets grab the latest stable release from github and download it in our /opt/ dir.

wget -O nodejs.zip https://github.com/joyent/node/zipball/v0.6.11

One thing that I forgot to install is zip so that we can unzip this file. So quickly install

apt-get install zip

then unzip node.

unzip nodejs.zip

Lets cd into joyent-node-* folder and start building it

./configure
make
make install

Also make sure that you install coffee-rails and jquery-rails. I got this error when trying to make the test app below. It may be a good idea to make a test app as root so bundler to handle these for us.

gem1.8 install coffee-rails --no-rdoc --no-ri
gem1.8 install jquery-rails --no-rdoc --no-ri

Testing rails

Now that we have everything setup lets login to our user account ( not root ) and create a sample rails app and run it with the builtin WEBrick server to make sure that rails works on our server.

In your home dir:

rails new testRails
cd testRails/
rails server

Your server should start up fine so navigate to it's ip and make sure that rails is running and the environment links displays information about your setup.

Hopefully your rails test app looks like this in the browser. If not then don't even try to go forward. Get the WEBrick server working first and then move on.

Nginx

We will go ahead and install nginx via the apt repo. We first need to add the key to the resposity so download it then add it.

wget http://nginx.org/keys/nginx_signing.key
apt-key add nginx_signing.key

Now add the repos to your /etc/apt/sources.list

deb http://nginx.org/packages/debian/ squeeze nginx
deb-src http://nginx.org/packages/debian/ squeeze nginx

Then

apt-get update
apt-get install nginx

It's a really quick install. After the install is finished you can test to make sure that it works by going to your machines ip address. You should see the welcome to nginx text.

User group and dir

Now lets setup a new directory that nginx and our rails apps will be in.

cd /var
mkdir www
usermod -a -G www-data nginx
usermod -a -G www-data YOURUSERNAME

chgrp -R www-data www/
chmod -R 775 www/

Now our user and nginx should be part of www-data group and own /var/www

Nginx Config

Now lets grab the config file form the unicorn repo and place it in the nginx config directory overwriting the default config.

cd /etc/nginx/
cp nginx.conf nginx.conf.old
rm nginx.conf
wget https://raw.github.com/defunkt/unicorn/master/examples/nginx.conf

Now we need to setup our paths in the nginx.conf file.

Uncomment this line:

listen 80 default deferred; # for Linux

Change the root paths to there our apps public dir is:

root /var/www/testapp/public;

Save and restart nginx

/etc/init.d/nginx restart

Now lets get unicorn setup with init.d and make a test app in the www dir via our other user.

Create the test app

rails new testapp

Everything should go smooth if you have things setup right. You don't need root or sudo to make the test app in var/www because we added ourself and nginx to the group that owns the folder. Lets cd into the test app's config dir and download the unicorn config file.

cd testapp/config/
wget -O unicorn.rb https://raw.github.com/defunkt/unicorn/master/examples/unicorn.conf.rb

Vim or nano into it and setup the paths to your app. Here is my config:

# Sample verbose configuration file for Unicorn (not Rack)
#
# This configuration file documents many features of Unicorn
# that may not be needed for some applications. See
# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb
# for a much simpler configuration file.
#
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.

# Use at least one worker per core if you're on a dedicated server,
# more will usually help for _short_ waits on databases/caches.
worker_processes 4

# Since Unicorn is never exposed to outside clients, it does not need to
# run on the standard HTTP port (80), there is no reason to start Unicorn
# as root unless it's from system init scripts.
# If running the master process as root and the workers as an unprivileged
# user, do this to switch euid/egid in the workers (also chowns logs):
# user "unprivileged_user", "unprivileged_group"

# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
working_directory "/var/www/testapp" # available in 0.94.0+

# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen "/tmp/.sock", :backlog => 64
listen 8080, :tcp_nopush => true

# nuke workers after 30 seconds instead of 60 seconds (the default)
timeout 30

# feel free to point this anywhere accessible on the filesystem
pid "/var/www/testapp/shared/pids/unicorn.pid"

# By default, the Unicorn logger will write to stderr.
# Additionally, ome applications/frameworks log to stderr or stdout,
# so prevent them from going to /dev/null when daemonized here:
stderr_path "/var/www/testapp/shared/log/unicorn.stderr.log"
stdout_path "/var/www/testapp/shared/log/unicorn.stdout.log"

# combine Ruby 2.0.0dev or REE with "preload_app true" for memory savings
# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!

  # The following is only recommended for memory/DB-constrained
  # installations.  It is not needed if your system can house
  # twice as many worker_processes as you have configured.
  #
  # # This allows a new master process to incrementally
  # # phase out the old master process with SIGTTOU to avoid a
  # # thundering herd (especially in the "preload_app false" case)
  # # when doing a transparent upgrade.  The last worker spawned
  # # will then kill off the old master process with a SIGQUIT.
  # old_pid = "#{server.config[:pid]}.oldbin"
  # if old_pid != server.pid
  #   begin
  #     sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
  #     Process.kill(sig, File.read(old_pid).to_i)
  #   rescue Errno::ENOENT, Errno::ESRCH
  #   end
  # end
  #
  # Throttle the master from forking too quickly by sleeping.  Due
  # to the implementation of standard Unix signal handlers, this
  # helps (but does not completely) prevent identical, repeated signals
  # from being lost when the receiving process is busy.
  # sleep 1
end


after_fork do |server, worker|
  # per-process listener ports for debugging/admin/migrations
  # addr = "127.0.0.1:#{9293 + worker.nr}"
  # server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)

  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection

  # if preload_app is true, then you may also want to check and
  # restart any other shared sockets/descriptors such as Memcached,
  # and Redis.  TokyoCabinet file handles are safe to reuse
  # between any number of forked children (assuming your kernel
  # correctly implements pread()/pwrite() system calls)
end

They are easily found and just point them to your app dir.

![Unicorn Config][8]

Now lets run unicorn and test out our app. Make two new folders in your app before login back in as root and running unicorn.

mkdir /shared
cd shared/
mkdir pids
mkdir log

unicorn_rails -c /var/www/testapp/config/unicorn.rb -D

Now go to your server ip in your browser and you should see the same thing with the ability to click the envirionment link with data displayed. If you do no see any data then unicorn is not running or setup correctly. If you do see it then we are done. It's working and ready to go. You may want to add unicorn to an init.d script so that it starts with your server.

Unicorn init.d

As root:

cd /etc/init.d/
vim unicorn

Here is the init.d script that I am using so just paste it into the file.

#! /bin/sh

# File: /etc/init.d/unicorn

### BEGIN INIT INFO
# Provides:          unicorn
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the unicorn web server
# Description:       starts unicorn
### END INIT INFO

DAEMON=/usr/bin/unicorn
DAEMON_OPTS="-c /var/www/testapp/config/unicorn.rb -D"
NAME=unicorn
DESC="Unicorn app for testapp"
PID=/var/web/testapp/shared/pid/unicorn.pid

case "$1" in
  start)
    echo -n "Starting $DESC: "
    $DAEMON $DAEMON_OPTS
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
        kill -QUIT `cat $PID`
    echo "$NAME."
    ;;
  restart)
    echo -n "Restarting $DESC: "
        kill -QUIT `cat $PID`
    sleep 1
    $DAEMON $DAEMON_OPTS
    echo "$NAME."
    ;;
  reload)
        echo -n "Reloading $DESC configuration: "
        kill -HUP `cat $PID`
        echo "$NAME."
        ;;
  *)
    echo "Usage: $NAME {start|stop|restart|reload}" >&2
    exit 1
    ;;
esac

exit 0

Edit to your correct paths then:

chmod +x unicorn
/usr/sbin/update-rc.d -f unicorn defaults

Reboot and see if nginx and unicorn are successfully servering your app.

Everything should work on the defult page via port 80. If you do not see images or the environment info that means unicorn is not running. Check the log files that we setup in the config ( the paths ) and see what is happening. I hope this tutorial works for everyone. I had a lot of trouble setting this up so hopefully you will not repeat my pain.

Multithreading in C#

2012-01-28T00:00:00+00:00

From my point of view the reason why we create multithreaded applications is so the user experience does not suffer. We have operations that need completed but running these on the main thread will block the UI and piss off our users.

Most of the operations have some similarities. Atleast for myself, the most common use case is the user requests information, we process it, then display the results back. That is the most common use that I use background threads for. When we display these results back, we need to write code that is executed on the Main thread to update the UI.

Working with delegates, callbacks, and IAsyncResults suck. They cause you to write more methods and change your code. There should be an easy way to write code normally and have all the benefits of async programming. That is what I try to accomplish in my WPF MVVM-Async framework. It allows you to write code normally and create multithreading applications with lambdas.

MVVM-Async

So first thing that you need to do to take advantage of the framework is to make your ViewModel inherit from ViewModelBase and pass it a type for the model. By doing this you automatically get access to the Model object.

public class EditorViewModel : ViewModelBase<EditorModel>
...
this.Model.EditorModelProperties...

Async

Now just code and when you want to dispatch a process in a background thread call DispatchAsync() and pass it a lambda.

DispatchAsync(()=> {

    int i = 0;
    bla bla bla

});

The framework will automatically spawn a new thread and execute whatever you placed inside the lambda on that thread. You can even call another method and pass any variables to it. By using lambdas you stay in the scope of the method calling DispatchAsync.

DispatchAsync(() => {

    SomeOtherFunc(name, date, age);

});

Updating the UI

So now we have this lambda executing on a background thread and processing data. We need to take that data and update our UI.

DispatchAsync(() => {

    string fullName = GetFullName(first, last);

    MyViewsTextBoxDataSource = fullName;

});

This is very simple but you get the point. If we were to try to set one of our view's controls from a background thread we would throw an exception because you can only set Controls on the thread that created it. So without using callbacks we can do this easily with the framework.

DispatchAsync(() => {

    string fullName = GetFullName(first, last);

    DispatchMainAsync(() => {

        MyViewsTextBoxDataSource = fullName;

    });

});

How hard was that? If we want to dispatch a lambda to the main thread that updates the UI, we can write it inline, with all our variables in scope, and have that code executed on the main thread. No exceptions.

Easy, simple, and you don't have to have callbacks or other crap to mess with. If you do want a callback when a thread has finished executing the framework does support this.

void DispatchAsyncWithCallback(action, callback);
void DispatchAsyncWithCallback(action, callback, whereTo);

The whereTo tells the framework on what thread do you want the callback executed one. Call me back on a background thread or call me back on the main thread.

DispatchAsyncWithCallback(() => {

    doSomeStuff();

}, () => {

    doSomeStuffOnTheCallback();

}, DispatchOn.Main); //Dispatch the callback on the main thread.

It's all done inline.

Queues

There is also support of Queues. You can load up a DispatchQueue with tasks and invoke the entire queue.

void AddQueue(DispatchQueue queue);
void InvokeQueue(string name);
void ProcessAllQueues();

Timed Execution

You can also tell the framework when to dispatch a thread.

DispatchIn(() => { somethingToDo(); }, "5min");

Or at a specific time.

DateTime someTime;
DispatchAt(() => { somethingToDo(); }, someTime);

Final

I am still working on this and will be releasing the code soon after CodeAssistant is complete. CodeAssistant is using this framework.

tCal - iCal events in terminal

2012-01-14T00:00:00+00:00

I just puked up some old code for a little app that displays your iCal events in the terminal. I have it as one of my startup scripts so when I start a new bash session I see my events for the next 7 days. It's a very simple but cool app. I am not proud of the design but I just cleaned up the code tonight and threw it on tCal on Github.

Usage

It is very simple to use. Just throw it in a folder that is in your PATH and type tCal to see your events for the next 5 days (default). if you want more or less days then just:

tCal -d 20

There is no config to setup with your calendars, it just works.

Here is a little taste for the code. Head over to Github for it all.

int main(int argc, const char * argv[])
{

    @autoreleasepool {
        CMCalendar * calendar = [[CMCalendar alloc] init];
        CMArgumentParser * parser = [[CMArgumentParser alloc] init];
        NSInteger days;

        @try {
            days = [(NSString *)[parser argumentForName:@"-d"] integerValue];
        }
        @catch (NSException *exception) {
            days = DEFAULTDAYS;
        }
        NSArray * events = [calendar getEventsForDays:days];
        for (id <ITerminalOutput> output in events) {
            [output display];
        }

    }
    return 0;
}

If you have any ideas just send them to me.

Download Binary

Bash Informational Startup

2011-11-11T00:00:00+00:00

If you use Terminal on your Mac like I do. It is the first application that is run at starup and the last to close. When you start Terminal in a new window or tab that is a great time to review your computer stats, iCal events, and disk sizes. I saw a lifehacker post abut a menu bar app that shows all the disks that are connected to your Mac and their size and free space. I also use todotxt for my todo lists and because it does not support dates for your tasks I keep my dates for events in iCal. I wanted to see all of this whenever I startup a new bash session so here is how I did it.

So this is what I see when I start a new bash session. All my information that I want is there plain text and easy to read.

There is my computers uptime at the very top with load averages.

Next is all the disk space information of my internal hard drive and external USB hard drives. This is a straight forward easy command.

Next up is my iCal events for today and tommorrow. You can easily set how many days forward you want. This is done by a simple Objective - C command line app that I wrote.

Last is todotxt. All my tasks.

So here is the last part of my bash_profile that makes this all happen. You can download all my app that displays your iCal events from my github of dotfiles (it's in the bin dir). You will also find all kinds of awesome stuff in it.

End of my .bash_profile

echo "-----------------------------------------------"
#Display uptime
uptime
echo "-----------------------------------------------"
#Display disk space usage
df -hl
echo "-----------------------------------------------"
echo "Up coming events:"
echo "-----------------"
#Call to my cmd line app to show iCal events
CommandLineAssistant
echo "-----------------------------------------------"
echo "Todo Tasks:"
echo "-----------"
#List my todos 
todo list
echo ""

Compile Vim for Python

2011-10-13T00:00:00+00:00

Update

I just added the newest MacVim compiled for python using the 10.7 Lion SDK. You can downloading it below.

If you want to use the Python auto complete in vim then you have to compile vim for python on your system. If you are running OSX I have just compiled MacVim for Snow Leopard and I am working on a compiled Lion version but the SL one works fine. Then you can setup an alias for vim so that it points to the binary in the MacVim app so you can still use vim from the command line and as the MacVim app.

It works really well and is fairly easy this way.

MacVim with Python SL MacVim with Python Lion

First download the already compiled MacVim.app from the link above and extract it to your Applications folder.

Then download:

pythoncomplete.vim

vim files and put it in your ~/.vim/autoload/ folder. It should be named:

pythoncomplete.vim

If you need, you can see my .dotfiles on github that already has my vimrc and vim dir in it.

My dotfiles

Then just add:

filetype plugin on

Last thing if you want to run this version form terminal just add:

alias vim=/Applications/MacVim.app/Contents/MacOS/Vim

to your ~/.bash_profile

You can see all these settings and more in my dotfiles.

All done. To use the auto complete just hit control+x then still holding down the control key it o

Python talking to Arduino

2011-10-08T00:00:00+00:00

I just setup my Arduino with an LCD display. You can follow the tutorial here to get the LCD and board wired up. We will also be using the sample code from that tutorial.

The Problem

The issue that I ran into was displaying messages from the Arduino IDE's SerialMonitor worked fine but when I was trying to display a message that I sent via python, it was not showing on the screen. After some help I finally have it working so this is how you fix it.

Python Code

You will need to install pyserial so that you can talk to your board via USB. It's simple if you have build tools installed so just:

sudo easy_install pyserial

Now if we look at the code that does not work first you can see what you need to change. We need to import serial and then setup the connection to the board.

import serial 
port = serial.Serial('pathtoyourboard', 9600)

You will need to look in your Arduino IDE for the path to your board and the 9600 is the rate that is definded in the Arduino code that we are loading onto it. Now we need to write some data do the Arduino and close the connection.

port.write('sometext')
port.close()

If we load the code from the tutorial that I linked at the beginning of this post onto your board and then ran this python code, we would just see a blank LCD. The probelm is that whenever the serial connection is first made, it takes the Arduino two seconds to get that port open and setup. So for our python code to work we need to import the time module and add a sleep for atleast 2 seconds right after our port = serial.Serial statement. Here is the what the final python code will look like with it working and the message changed.

import serial
import time

port = serial.Serial('/dev/tty.usbmodemfd141', 9600)
time.sleep(2)

message = ['My', 'name', 'is', 'michael']

for word in message:
    port.write(word)
    time.sleep(1)

port.close()

So this code makes the connection and waits 2 seconds for the Arduino to get setup, then starts writing data to it. Everything works find and it's good to go. So in the future when you are working with Python and Arduino via USB, wait 2 seconds before sending data to it or it will not get it.

How to backup gmail the right way

2011-10-04T00:00:00+00:00

So like many people my life is stored in my gmail. I trust google to keep my data safe but since so many things, from contacts to many different online accounts, are all registered under your gmail address it is important to have that data whenever you need it. There are some services online that will backup your gmail but it is still stored on their servers. I feel a lot better if it was stored locally on my computers so that I have a copy of my emails whenever I need them. So this is how you do it.

Got Your Back

Got Your Back is a python app that will connect to your gmail via oAuth and suck down your messages in simple txt files. They have an *.eml extension but you can open all these files with a text editor and see the contents of your emails. It is fast and also does incremental backups. So after the initial backup of your account, Got Your Back will only download the messages that it doe not have instead of downloading everything each time you do a backup. I use a simple cron job to get all my new messages everyday, automatically without having to think about it.

Installing

You can get to the download page of got your back here. They have downloads for Unix and Windows systems. I have an server setup running debian that handles all my account backups so this tutorial will focus on running gyb (got your back) on Unix systems. So download the source package or use subversion to download the source onto your computer:

svn checkout http://got-your-back.googlecode.com/svn/trunk/ got-your-back-read-only

If you download the zip extract it to whatever folder you want and cd into the directory. Now because gyb uses oAuth to login to your account you will need to initially run this script on a computer with a desktop. Because I use a server over ssh to download my backups I need to first get the oAuth token on a computer with a web browser first.

Getting oAuth Tokens

So now that we are cd into the got-your-back directory run this command to authenticate and get an estimate of the size and number of messages that need to be backed up.

python gyb --email youremail@gmail.com --estimate

So make sure that you input your correct email address. If you are already logged into your email account on your computer and you are trying to backup a different account, you will need to logout first. So gyb will tell you that it will launch a browser, so just hit Enter and you will see a gmail page asking if you want to authenticate gyb to your gmail. Hit yes and then go back to the command line. No hit Enter again if it is still waiting and wait for it to make an estimate of your gmail's size and number of messages.

So now it should give you an estimate. I had 28k messages to download at 800mb. It does not download attachments. If you do an ls now you will see a new file. It will be a file with your email address and an extension of .cfg. This file now holds your oAuth token so if you are going to run the backups on a server over ssh, you will have to copy this file over to your server for it to work. So either zip, tar, or 7z the entire gyb directory with your email's config file in it and transfer it to your server. If you are just going to run it on the same computer then you don't have to do anything yet.

Inital backup

So now that you have the config file and gyb on whatever computer that you want to setup the automated backups on lets run the initial backup. First lets make the gyb.py file runnable.

chmod 755 gyb.py

So run this command to perform the initial backup.

python gyb.py --email youremail@gmail.com --backup

This will perform the initial backup so depending on how many messages your estimate was, this make take awhile.

Cron job for everyday backups

So now that we have an initial backup we will need to do an incremental backup everyday for all of the new messages that we receive. So lets add a cron job with crontab that will run gyb.py everyday. So remember where that file is on your computer because we will need it.

crontab -e

Then input this into your crontab file with the proper changes.

0 2 * * * python /home/michael/got-your-back-read-only/gyb.py --email youremail@gmail.com --backup

So what this does is make a backup at 3am everyday running gyb.py with your specified email. Easy and done. It will now do incremental backups of only your new messages and you will always be able to get that data because it is now backed up locally on your computer or server.

How to manage your website with git

2011-09-28T00:00:00+00:00

So this is much easier if you have a static site that is generated using pelican or jerkyll. Both of these apps are great for generating blogs out of markdown or other simple text markup language. I use pelican because it is built with python and that is what I develop in.

Local

So on your local machine you will need your basic folder structure setup and if you are not already using git to manage your files, now is the time to start. In terminal cd into your site dir and:

git init

This is create the local git repo that you need. Go ahead and commit the files that you already have done and now we can work on the server.

Server

Make sure that you have ssh access to your web server. The first thing you need to do is cd into your users main directory, not your www or directory that hosts the files that are displayed to your users and we will make a new directory:

mkdir cmsite.git
cd cmsite.git
git init --bare

This will create a bare git repo that will accept the files when you do a push to it. Now we will need to setup a hook that will checkout the latest push into the actual directory that hosts our content. So now we need to add the hook the the hooks directory in cmsite.git in my case.

vim hooks/post-receive
#!/bin/sh
GIT_WORK_TREE=/home/crosbymichael/crosbymichael.com git checkout -f

Make sure that this file is executable so do a quick:

chmod 755 hooks/post-receive

Final Steps

Now we just need to setup our new remote repo on our local machine. So cd into your local git repo and type:

git remote add website ssh://username@yoursite.com/~/cmsite.git

Now all your have to do is push to production when you need to update your site and git does all the rest.

git push website master

Because you can have more than one remote in git, you could have a test site and your production site, and another offsite repo for backups. You can run tests on your test site and then when everything is good to go you just push to production.

AVAudioPlayer Memory Leak

2011-09-11T00:00:00+00:00

This is a little bug in the AVFoundation framework. When using the AVAudioPlayer in you will get memory leaks when running in the iOS Simulator but not on devices.

It is a pain whenever you do most of your testing on the desktop during development and then start testing on device after you have a working app. So just remember if you are using AVAudioPlayer on the iOS Simulator and you are getting memory leaks that look like this:

Look at the type of leaks. note the responsible library is AudioToolbox not even AVFoundation, go ahead and profile it on a device and see if the leaks go away. If they don't, then it's something that you need to fix in your code, if they do then you are good to go.

SVN to GIT and Extracting Individual Projects

2011-08-15T00:00:00+00:00

This post is still not formatting. I switched from worpress and this is a long post and have not had time to update it.

So here is the issue. You first started to use version control and because you were coding solo and the low learning curve, you started to use Subversion (SVN). It worked great for you because you had your own repository and you added all your projects to it and it was easy. Then you started to work with other people on projects and the very first commit that you made to your svn repo ended in conflicts. Not a good way to start. So you started to use more powerful version controle systems like GIT and now you wonder why you ever used svn in the first place.

However, you have a years worth of revision history sitting in your svn repository that you want out and placed into your shinny new git repo. You have already been working in your git repo for a few months, you just checked out the latest copy from your svn repo and added it to a new git repo, and now you have a branches and commits of all your recent work. It's time to convert your svn to git and unite your past revision history with the present. Lets get to work.

Video is at the bottom if this is confusing.

Problems:

SVN repo has many different projects in the one large repository

SVN has a year worth of revision history with commits from many different projects

You checked out the latest copy from svn and made git repositories for each of your projects

These individual project git repos now have many commits of their own and branches of your recent work

Goal:

Extract individual projects with only their complete revision history from svn and merge that with your current work and revision history in your individual git repos.

Lets look at our starting point. SVN revision history and GIT revision history.

Last svn file change and git's last change. You can see that the files are different. SVN on left and GIT on right.

First we need to suck our svn repo down to a machine where we have shell access. I had a large 4GB svn repo with 25+ projects in it. It had 280 commits from various projects. ie: commit 278 from Encoder Assistant changes, commit 279 from HTTP Assistant changes, and commit 280 from another project. So the svn repo was tracking overall revision history for the entire repo and not individual projects. This was a mistake on my part, I was new to svn and my hosting provider offered free private svn repos to use. So it was a free way for me to have revision history and also offsite backup of my code.

To grab your svn repo from various locations you need to first make a new folder and cd into it. Yes, you will have to use the terminal.

git svn clone url-to-repo

(not the working copy, the actual svn repo)

Local file:///Users/michael/svnrepo

Http http://yoursite.com/svnrepo

SSH ssh://username@yoursite.com/svnrepo

So to cone I will do this:

[highlight]git svn clone file:///Users/michael/Development/Video/WIP/SVNTOGIT/[/highlight]

If you get an error with the wrong url to the svn repo then you will have to delete any folders in your current folder before you can try again.

We can see by the terminal output and then doing an ls that it received 3 revisions(commits) and created a new folder of the svn repo name. Now cd into it and you can see that there are two projects exactly in the same as the svn repo. If we want to check to see if git cloned the full revision history we can type:

[highlight]git log[/highlight]

See it's all there.

If you only had one project in your svn repo then you can skip the next step.

Now we need to extract one project out of this new git repo. We only want one project per repo in our new setup. To do this we need to do a filter.

Lets extract the contents of ProjectOne to the root dir of this new git repo. (If you have more than one project you will want to clone this git repo so that you can keep do this over and over again for each project.)

[highlight]git filter-branch --subdirectory-filter ProjectOne/ -- --all[/highlight]

Now you can see that it rewrote the branch and by doing an ls you can see the contents of ProjectOne are now in the root of your repo. There is only one code.txt file from this repo.

Lets do a git log again and see that only the ProjectOne revision history is extracted with the ProjectOne files.

You can see only two commits, the init and Added more code. These are only the commits that affected ProjectOne. Nothing for ProjectTwo.

Lets make a new dir to keep the same folder structure as your CURRENT git repo that we have been working in.

[highlight]mkdir ProjectOne[/highlight]

Now we need to move the files back into this project one folder.

[highlight]mv * ProjectOne [/highlight]

This will give you an error about moving ProjectOne into it's self but it works and moves every other file and folder into our new ProjectOne folder. You can cd into it and check for your self.

Now we need to commit the changes that we have just done to the repo.

[highlight]git add .[/highlight]

This adds and removed all changes that we have made.

[highlight]git commit -a[/highlight]

This will commit all the changes we made. Give it a message for the commit.

Now we need to merge the revision history, branches, etc from our CURRENT repo with the new git repo's svn revisions. We are still in the same folder of the new git repo that we have created.

Lets create a new branch to add our current work into. We will now refer to our git repo with the svn history as our OLD repo and the git repo with the CURRENT changes as the CURRENT repo. This will get confusing to pay attention.

To create and new branch and check it out at the same time we do:

[highlight]git checkout -b current-repo[/highlight]

Now we need to pull in the changes from the CURRENT repo into the current-repo branch of the OLD repo (the one with the svn revision history and extracted ProjectOne)

git pull pathtorepo

So in my example

[highlight]git pull ~/Development/Video/WIP/GitCopy/svn-repo/[/highlight]

So if you notice, it pulled the CURRENT repo into the OLD repo and when it tried to merge the changes into the branch it received a conflict. Don't worry, resolving merging conflict's in git is easy. If you are on OSX like me or have a merge tool setup, ie: FileMerge all you have to do when you receive a conflict on a merge it type:

[highlight]git mergetool[/highlight]

It will tell you what file you are going to be working with and when you hit enter, it will launch the tool so you can work on the conflict in the file.

Now we can see the code on the left is from the last commit from our svn repo and the code on the right is from our CURRENT git repo with all the changes. Now you just go down through the differences and select the left or right to keep. After you are finished, I am just going to keep all the right side changes, save the file and close your merge tool.

Now if we type:

[highlight]git status [/highlight]

We can see that we have the modified file to commit. If you have untracked files or if you have files with a .orig extension, conflicted files, add or remove those before committing.

[highlight]rm ProjectOne/code.txt.orig[/highlight]

Now commit

[highlight]git commit -a[/highlight]

You can see that git has already added a comment for you stating that this was a merge from another repo and also adds the repo path or url. Write and quit your comment tool and it's done. If we type in:

[highlight]gitk[/highlight]

This will bring up an gui to show our git repos history.

Now you can see that we not only have the Init and Added more code commit's that were originally from the svn repo but also all the commits from the git project added to our new branch. If you want to do another merge to merge the changes from the current-repo branch back into the master branch for the project you can do that and now we can finally call this repository our CURRENT, full revision history, repository of awesomeness.

If you were able to follow along, your welcome.

Setup your own Git Server over SSH

2011-08-08T00:00:00+00:00

The thing about dvsc that makes me nervous is that all the code and revision history is on your local computer. This makes it very fast when committing, branching, and merging but what if you have a drive failure or worse? All your code is gone if you don't have backups. You have many different ways to backup your git and mercurial repositories. Especially this open-source project that I am working when it is done: DVCS-Backup

If you want to work with others on a project and need a git server setup to be an intermediary, then this is how you can do it as well as having an offsite backup copy of your repos.

First find a server that you are going to use. Setup keyless ssh access to it following my tutorial here

After that ssh into your server and make a new user that will manage the repos if sharing or just user your own account if it's just for you.

Install git on your server. I run debian so it's a simple:

apt-get install git-core

To get git installed.

Now create a new folder for the repo that you want to push to. I have many repos on my development computer that I work from so we are going to push an already populated repo to the server.

mkdir somereponame.git

You add the .git to the end of the folder name to that it's easy to distinguish between other folders. .git means this is a git repo.

Now cd into it and create a bare repo.

git init --bare

Now that server part is done. Easy.

Now go to your local dev computer and cd into the repo that you want to push to the server. Make sure that you don't have a remote origin already setup in this repo. To clear it out just:

git remote rm origin

Now we need to set the origin to your remote server.

git remote add origin ssh://sshusername@yourserver.com/~/myrepo.git

Now your done. Time to push. If you have more than one branch, you can either push one or all the branches to the server. I just push all of them because I want the server to have a full backup or my local repo.

git push origin --all

But for inividual branches:

git push origin branchname

That's it. Now when you want to get data from the server just do a pull.

git pull origin branchname

If you want to clone the repo to a new folder:

git clone ssh://sshusernane@yourserver.com/~/myrepo.git

Or you can make a new dir, cd into it and create a new local repo, set the origin to your server and pull specific branches.

mkdir newrepo
cd newrepo
git init
git remote add origin ssh://sshusername@yourserver.com/~/myrepo.git
git pull origin branchname

Flex 4.5 vs Pure AS3

2011-08-07T00:00:00+00:00

I have built a few applications for Android using the Flex framework. It is a great framework and I can build, test, and publish apps very quickly. It takes productivity to the max.

However, I have noticed that some of my apps take a noticeable time to launch on my device for the first time. Even simple apps like QuickIP where all it does it check for external and wifi IP address. So I ran a few comparison tests between pure AS3 projects and Flex projects to see three things. Startup time, file (swf) size, and memory usage. All important on a mobile device. Here are the results.

Mobile Pure AS3

Startup Time: 99ms
SWF Size: 1.5kb
Memory Usage*: 6.328mb

Mobile Flex mxml

Startup Time: 653ms
SWF Size: 2.839mb
Memory usage*: 21.397mb

Memory usage is a sample taken after 5 seconds of the app being launched. All the apps are just code to test startup time and memory usage after 5 seconds. The Flex mobile project was a blank project. Each project launched a view that was run on my desktop, not on a device.

So how does this compare?

AS3 over Flex

AS3 launches in 15.16% of the time it takes a Flex project
AS3's file size is .05% of the Flex projects size
AS3 uses 29.58% of the memory that a Flex project uses

As for the desktop, the numbers are closer to the same. Remember that I launched the mobile project on my desktop rather than on a device so the tests have a consistent test environment.

Desktop Pure AS3

Startup Time: 96ms
SWF Size: 1.5kb
Memory Usage*: 6.184mb

Desktop Flex mxml

Startup Time: 556ms
SWF Size: 2.114mb
Memory usage*: 19.021mb

As you can see the desktop Flex project is a little lighter on the resources than a comparable Mobile Flex app. I even did some tests on the desktop such as only using spark and no mx. I have even changed the WindowedApplication to just Application (no visible window as launch) and was able to get the startup time down to 450ms and only using ~15mb of ram. I tried to do tests with a Flex project's initial application file as a .as instead of a .mxml file but they were unsuccessful in launching a visible window. However just launching an initial .as instead of .mxml did make the Flex project have the same ~99ms startup time, ~1.5kb swf size, and ~6mb memory footprint.

So the question is... Is the speed of productivity in building an app on the Flex framework worth the loss of speed running the app? I guess it depends on what type of app you are building and what your user base is. The big difference between the Flex framework and pure As3 is components. You have to build all the components your self in pure AS3 projects. But how will productivity change for pure As3 projects after you already have a library of components built? What if you make all your assets and skin the Flex components anyway with bitmap or fxg assets? Do the components matter then?

These are the questions that I am asking my self currently. I am going to rebuild my QuickIP app with pure AS3 since it has all custom bitmap assets then compare the performance on my Android device and see if the time of development makes a noticeable jump in performance.

Setup Password-less Login over SSH

2011-08-07T00:00:00+00:00

Everyone needs a fast instance in a datacenter to work with. Amazon's EC2 or any other VPS provider will give you access. Sometimes you need to transfer files around fast or do some data processing. Having a fast instance will help you get your job done faster.

I have a tutorial that I needed to upload to an FTP server and do an FXP copy. The file is resting on another FTP server form a basic hosting provider. I do not have FXP access. So I launched an instance on EC2. Logged in via ssh. wget the file from my hosting server. Then connected to the marketplace FTP and uploaded the file. Easy, fast, and all the transferring was done on someone else's bandwidth.

To make this easier here is how to setup ssh to use a ssh key file instead of a password. This has more advantages than just making it quicker to ssh into my instances. You can use this to setup automated backups using rsync.

How To:

On your local machine you need to jump into terminal for this. Go to your home dir and make a new .ssh folder if it does not exist. To see if it does exist just type:

ls -a

If not exist then:

mkdir .ssh

Now cd into it:

cd .ssh

Now we need to generate an rsa key file:

ssh-keygen -t rsa

Do not enter a passphrase, just leave it blank.

Now you need to ssh into your instance or remote computer and make an .ssh folder in your user home folder. Same as above.

Now cd into and make a new file called:

authorized_keys

In this file, paste the contents of your local computers newly generated id_rsa.pub file into it.

Now, still on your remote computer or instance, you need to change the permission of the authorized_keys file to 644:

chmod 644 authorized_keys

All done. Exit your ssh session and login again and you will notice that you don't have to enter a password now.

I suggest changing your password to something really long and secure since you won't be using it anymore, unless logging into to it from a different computer.