Writing HTTP Middleware in Go · Justinas Stankevičius (original) (raw)

In the context of web development, "middleware" usually stands for "a part of an application that wraps the original application, adding additional functionality". It's a concept that usually seems to be somewhat underappreciated, but I think middleware is great.

For one, a good middleware has a single responsibility, is pluggable and self-contained. That means you can plug it in your app at the interface level and have it just work. It doesn't affect your coding style, it isn't a framework, but merely another layer in your request handling cycle. There's no need to rewrite your code: if you decide that you want the middleware, you add it into the equation, if you change your mind, you remove it. That's it.

Looking at Go, HTTP middleware is quite prevalent, even in the standard library. Although it might not be obvious at first, functions in the net/http package, like StripPrefixor TimeoutHandlerare exactly what we defined middleware to be: they wrap your handler and take additional steps when dealing with requests or responses.

My recent Go package nosurfis middleware too. I intentionally designed it as one from the very beginning. In most cases you don't need to be aware of things happening at the application layer to do a CSRF check: nosurf, like any proper middleware, stands completely on its own and works with any tools that use the standard net/http interface.

You can also use middleware for:

Writing a simple middleware

For the first example, we'll write middleware that only allows users visit our website through a single domain (specified by HTTP in the Host header). A middleware like that could serve to protect the web application from host spoofing attacks.

Constructing the type

For starters, let's define a type for the middleware. We'll call it SingleHost.

type SingleHost struct { handler http.Handler allowedHost string }

It consists of only two fields:

As we made the field names lowercase, making them private to our package, we should also make a constructor for our type.

func NewSingleHost(handler http.Handler, allowedHost string) *SingleHost { return &SingleHost{handler: handler, allowedHost: allowedHost} }

Request handling

Now, for the actual logic. To implement http.Handler, our type only needs to have one method:

type Handler interface { ServeHTTP(ResponseWriter, *Request) }

And here it is:

func (s *SingleHost) ServeHTTP(w http.ResponseWriter, r *http.Request) { host := r.Host if host == s.allowedHost { s.handler.ServeHTTP(w, r) } else { w.WriteHeader(403) } }

ServeHTTP method simply checks the Host header on the request:

The original handler's ServeHTTP is never called in the latter case, so not only does it not get a say in this, it won't even know such a request arrived at all.

Now that we're done coding our middleware, we just need to plug it in. Instead of passing our original Handlerdirectly into the net/http server, we wrap it in the middleware.

singleHosted = NewSingleHost(myHandler, "example.com") http.ListenAndServe(":8080", singleHosted)

An alternative approach

The middleware we just wrote is really simple: it literally consists of 15 lines of code. For writing such middleware, there exists a method with less boilerplate. Thanks to Go's support of first class functions and closures, and having the neat http.HandlerFunc wrapper, we'll be able to implement this as a simple function, rather than a separate struct type. Here is the function-based version of our middleware in its entirety.

func SingleHost(handler http.Handler, allowedHost string) http.Handler { ourFunc := func(w http.ResponseWriter, r *http.Request) { host := r.Host if host == allowedHost { handler.ServeHTTP(w, r) } else { w.WriteHeader(403) } } return http.HandlerFunc(ourFunc) }

Here we declare a simple function called SingleHostthat takes in a Handler to wrap and the allowed hostname. Inside it, we construct a function analogous to ServeHTTPfrom the previous version of our middleware. Our inner function is actually a closure, so it can access the variables from the outer function. Finally, HandlerFunclets us use this function as a http.Handler.

Deciding whether to use a HandlerFuncor to roll out your own http.Handler type is ultimately up to you. While for basic cases a function might be enough, if you find your middleware growing, you might want to consider making your own struct type and separate the logic into several methods.

Meanwhile, the standard library actually uses both ways of building middleware.StripPrefixis a function that returns a HandlerFunc, while TimeoutHandler, although a function too, returns a custom struct type that handles the requests.

A more complex case

Our SingleHost middleware was trivial: we checked one attribute of the request and either passed the request to the original handler, not caring about it anymore, or returned a response ourselves and didn't let the original handler touch it at all. Nevertheless, there are cases where, rather than acting based on what the request is, our middleware has to post-process the response after the original handler has written it, modifying it in some way.

Appending data is easy

If we just want to append some data after the body written by the wrapped handler, all we have to do is call Write() after it finishes:

`type AppendMiddleware struct { handler http.Handler }

func (a *AppendMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) { a.handler.ServeHTTP(w, r) w.Write([]byte("Middleware says hello.")) } `

The response body will now consist of whatever the original handler outputted, followed by Middleware says hello..

The problem

Doing other types of response manipulations is a bit harder though. Say, we'd like to prepend data to the response instead of appending it.

If we call Write() before the original handler does, it will lose control over the status code and headers, since the first Write() writes them out immediately.

Modifying the original output in any other way (say, replacing strings in it), changing certain response headers or setting a different status code won't work because of a similar reason: when the wrapped handler returns, those will have already be sent to the client.

To counter this we need a particular kind of ResponseWriterthat would work as a buffer, gathering the response and storing it for later use (and modifications). We would then pass this buffering ResponseWriterto the original handler instead of giving it the real RW, thus preventing it from actually sending the response to the user just yet.

Luckily, there's a tool just like that in the Go standard library.ResponseRecorderin the net/http/httptest package does all we need: it saves the response status code, a map of response headers and accumulates the body into a buffer of bytes. Although (like the package name implies) it's intended to be used in tests, it fits our use case as well.

Let's look at an example of middleware that uses ResponseRecorder and modifies everything in a response, just for the sake of completeness.

`type ModifierMiddleware struct { handler http.Handler }

func (m *ModifierMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) { rec := httptest.NewRecorder() // passing a ResponseRecorder instead of the original RW m.handler.ServeHTTP(rec, r) // after this finishes, we have the response recorded // and can modify it before copying it to the original RW

// we copy the original headers first
for k, v := range rec.Header() {
    w.Header()[k] = v
}
// and set an additional one
w.Header().Set("X-We-Modified-This", "Yup")
// only then the status code, as this call writes out the headers 
w.WriteHeader(418)

// The body hasn't been written (to the real RW) yet,
// so we can prepend some data.
data := []byte("Middleware says hello again. ")

// But the Content-Length might have been set already,
// we should modify it by adding the length
// of our own data.
// Ignoring the error is fine here:
// if Content-Length is empty or otherwise invalid,
// Atoi() will return zero,
// which is just what we'd want in that case.
clen, _ := strconv.Atoi(r.Header.Get("Content-Length"))
clen += len(data)
r.Header.Set("Content-Length", strconv.Itoa(clen))

// finally, write out our data
w.Write(data)
// then write out the original body
w.Write(rec.Body.Bytes())

} `

And here's the response we get by wrapping a handler that would otherwise simply return "Success!" with our middleware.

`HTTP/1.1 418 I'm a teapot X-We-Modified-This: Yup Content-Type: text/plain; charset=utf-8 Content-Length: 37 Date: Tue, 03 Sep 2013 18:41:39 GMT

Middleware says hello again. Success! `

This opens up a whole lot of new possibilities. The wrapped handler is now completely in or control: even after it handling the request, we can manipulate the response in any way we want.

Sharing data with other handlers

In various cases, your middleware might need to expose certain information to other middleware or your app itself. For example, nosurf needs to give the user a way to access the CSRF token and the reason of failure (if any).

A nice pattern for this is to use a map, usually an unexported one, that maps http.Request pointers to pieces of the data needed, and then expose package (or handler) level functions to access the data.

I used this pattern for nosurf too. Here, I created a global context map. Note that a mutex is also needed, since Go's maps aren't safe for concurrent accessby default.

`type csrfContext struct { token string reason error }

var ( contextMap = make(map[*http.Request]*csrfContext) cmMutex = new(sync.RWMutex) ) `

The data is set by the handler and exposed via exported functions like Token().

`func Token(req *http.Request) string { cmMutex.RLock() defer cmMutex.RUnlock()

ctx, ok := contextMap[req]
if !ok {
        return ""
}

return ctx.token

} `

You can find the whole implementation in the context.gofile in nosurf's repository.

While I chose to implement this on my own for nosurf, there exists a handy gorilla/contextpackage that implements a generic map for saving request information. In most cases, it should suffice and protect you from pitfalls of implementing a shared storage on your own. It even has middleware of its ownthat clears the request data after it's been served.

All in all

The intention of this article was both to draw fellow gophers' attention to middleware as a concept and to demonstrate some of the basic building blocks for writing middleware in Go. Despite being a relatively young language, Go has an amazing standard HTTP interface. It's one of the factors that make coding middleware for Go a painless and even fun process.

Nevertheless, there is still a lack of quality HTTP tools for Go. Most, if not all, of the middleware ideas for GoI mentioned earlier are yet to come to life. Now that you know how to build middleware for Go, why not do it yourself? ;)

P.S. You can find samples for all the middleware written for this post in a GitHub gist.