0

Go interfaces make test stubbing easy

Go's "object-orientation" approach is through interfaces. Interfaces provide a way of specifying the behavior expected of an object, but rather than saying what an object itself can do, they specify what's expected of an object. If any object meets the interface specification it can be used anywhere that interface is expected.

I was working on a new, small piece of software that does image compression for CloudFlare and found a nice use for interfaces when stubbing out a complex piece of code in the unit test suite. Central to this code is a collection of goroutines that run jobs. Jobs are provided from a priority queue and performed in priority order.

The jobs ask for images to be compressed in myriad ways and the actual package that does the work contained complex code for compressing JPEGs, GIFs and PNGs. It had its own unit tests that checked that the compression worked as expected.

But I wanted a way to test the part of the code that runs the jobs (and, itself, doesn't actually know what the jobs do). Because I only want to test if the jobs got run correctly (and not the compression) I don't want to have to create (and configure) the complex job type that gets used when the code really runs.

What I wanted was a DummyJob.

The Worker package actually runs jobs in a goroutine like this:

func (w *Worker) do(id int, ready chan int) {
    for {
        ready <- id

        j, ok := <-w.In
        if !ok {
            return
        }

        if err := j.Do(); err != nil {
            logger.Printf("Error performing job %v: %s", j, err)
        }
    }
}

do gets started as a goroutine passed a unique ID (the id parameter) and a channel called ready. Whenever do is able to perform work it sends a message containing its id down ready and then waits for a job on the worker w.In channel. Many such workers run concurrently and a separate goroutine pulls the IDs of workers that are ready for work from the ready channel and sends them work.

If you look at do above you'll see that the job (stored in j) is only required to offer a single method:

func (j *CompressionJob) Do() error

The worker's do just calls the job's Do function and checks for an error return. But the code originally had w.In defined like this:

w := &Worker{In: make(chan *job.CompressionJob)}

which would have required that the test suite for Worker know how to create a CompressionJob and make it runnable. Instead I defined a new interface like this:

type Job interface {
    Priority() int
    Do() error
}

The Priority method is used by the queueing mechanism to figure out the order in which jobs should be run. Then all I needed to do was change the creation of the Worker to

w := &Worker{In: make(chan job.Job)}

The w.In channel is no longer a channel of CompressionJobs, but of interfaces of type Job. This shows a really powerful aspect of Go: anything that meets the Job interface can be sent down that channel and only a tiny amount of code had to be changed to use an interface instead of the more 'concrete' type CompressionJob.

Then in the unit test suite for Worker I was able to create a DummyJob like this:

var Done bool

type DummyJob struct {
}

func (j DummyJob) Priority() int {
    return 1
}

func (j DummyJob) Do() error {
   Done = true
   return nil
}

It sets a Done flag when the Worker's do function actually runs the DummyJob. Since DummyJob meets the Job interface it can be sent down the w.In channel to a Worker for processing.

Creating that Job interface totally isolated the interface that the Worker needs to be able to run jobs and hides any of the other details greatly simplifying the unit test suite. Most interesting of all, no changes at all were needed to CompressionJob to achieve this.

0

CloudFlare hiring Go programmers in London and San Francisco

Are you familiar with the Go programming language and looking for a job in San Francisco or London? Then think about applying to CloudFlare. We're looking for people with experience writing Go in both locations.

CC BY-SA 2.0 by Yuko Honda (cropped, resized)

CloudFlare uses Go extensively to build our service and we need to people to build and maintain those systems. We've written a complete DNS server in Go, our Railgun service is all Go and we're moving more and more systems to Go programs.

We've recently written about our open source Red October Go project for securing secrets, and open-sourced our CFSSL Go-based PKI package. Go is now making its way into our data pipeline and be used for processing huge amounts of data.

We even have a Go-specific section on our GitHub.

If you're interested in working in Go on a high-performance global network like CloudFlare, send us an email.

Not into Go? We're hiring for all sorts of other positions and technologies.

0

Experimenting with mozjpeg 2.0

One of the services that CloudFlare provides to paying customers is called Polish. Polish automatically recompresses images cached by CloudFlare to ensure that they are as small as possible and can be delivered to web browsers as quickly as possible.

We've recently rolled out a new version of Polish that uses updated techniques (and was completely rewritten from a collection of programs into a single executable written in Go). As part of that rewrite we looked at the performance of the recently released mozjpeg 2.0 project for JPEG compression.

To get a sense of its performance (both in terms of compression and in terms of CPU usage) when compared to libjpeg-turbo I randomly selected 10,000 JPEG images (totaling 2,564,135,285 bytes for an average image size of about 256KB) cached by CloudFlare and recompressed them using the jpegtran program provided by libjpeg-turbo 1.3.1 and mozjpeg 2.0. The exact command used in both cases was:

jpegtran -outfile out.jpg -optimise -copy none in.jpg

Of the 10,000 images in cache, mozjpeg 2.0 failed to make 691 of them any smaller compared with 3,471 for libjpeg-turbo. So mozjpeg 2.0 was significantly better at recompressing images.

On average Continue reading

0

Courage to change things

This was an internal email that I sent to the CloudFlare team about how we are not afraid to throw away old code. We thought it was worth sharing with a wider audience.

Date: Thu, 10 Jul 2014 10:24:21 +0100
Subject: Courage to change things
From: John Graham-Cumming
To: Everyone

Folks,

At the Q3 planning meeting I started by making some remarks about how much
code we are changing at CloudFlare. I understand that there were audio
problems and people may not have heard these clearly, so I'm just going to
reiterate them in writing.

One of the things that CloudFlare is being brave about is looking at old code
and deciding to rewrite it. Lots of companies live with legacy code and build
on it and it eventuallybecomes a maintenance nightmare and slows the company
down.

Over the last year we've made major strides in rewriting parts of our code
base so that they are faster, more maintainable, and easier to enhance. There
are many parts of the Q3 roadmap that include replacing old parts of our
stack. This is incredibly important as it enables us to be more agile and 
more stable in future.

We should feel good  Continue reading