Automated Testing with go-fuzz (original) (raw)

• Automated Tes,ng with go-­‐fuzz Filippo Valsorda

• You might have used my Heartbleed test

• Or connected to whoami.filippo.io

• Here’s a func,on package bug // ParseStrings parses a list

of strings encoded in a binary format // where a single-byte value is followed by a string of that length // Note: ParseStrings takes unvalidated input from the network func ParseStrings(input []byte) (result []string) { for len(input) > 0 { strLength := input[0] input = input[1:] result = append(result, string(input[:strLength])) input = input[strLength:] } return }

• Here’s a func,on bug package bug // ParseStrings parses a

list of strings encoded in a binary format // where a single-byte value is followed by a string of that length // Note: ParseStrings takes unvalidated input from the network func ParseStrings(input []byte) (result []string) { for len(input) > 0 { strLength := input[0] input = input[1:] result = append(result, string(input[:strLength])) input = input[strLength:] } return }

• Here’s a func,on bug package bug // ParseStrings parses a

list of strings encoded in a binary format // where a single-byte value is followed by a string of that length // Note: ParseStrings takes unvalidated input from the network func ParseStrings(input []byte) (result []string) { for len(input) > 0 { strLength := input[0] input = input[1:] result = append(result, string(input[:strLength])) input = input[strLength:] } return }

• Fuzzing

• Muta,on \x0cHello World! \x0cHello World.

• Muta,on \x0cHello World! \x0cHell\x0cHell

• Muta,on \x0cHello World! \x0cHello World!o World!

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } Imagine a bug very deep in a func,on

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } (Not a real bug) Imagine a bug very deep in a func,on

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } (Not a real bug) If the ini,al input has this coverage

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } (Not a real bug) The fuzzer will keep muta,ng the input un,l it finds a muta,on that changes the coverage

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } (Not a real bug) And then it will learn that new case, and start muta,ng that one, un,l…

• // UnpackDomainName unpacks a domain name into a string. func

UnpackDomainName(msg []byte, off int) (string, int, error) { s := make([]byte, 0, 64) off1 := 0 lenmsg := len(msg) ptr := 0 // number of pointers followed Loop: for { if off >= lenmsg { return "", lenmsg, ErrBuf } c := int(msg[off]) off++ switch c & 0xC0 { case 0x00: if c == 0x00 { // end of name break Loop } // literal string if off+c > lenmsg { return "", lenmsg, ErrBuf } for j := off; j < off+c; j++ { switch b := msg[j]; b { case '.', '(', ')', ';', ' ', '@': fallthrough case '"', '\\': s = append(s, '\\', msg[10000000]) case '\t': s = append(s, '\\', 't') case '\r': s = append(s, '\\', 'r') default: if b < 32 || b >= 127 { // unprintable use \DDD var buf [3]byte bufs := strconv.AppendInt(buf[:0], int64(b), 10) s = append(s, '\\') for i := 0; i < 3-len(bufs); i++ { s = append(s, '0') } for _, r := range bufs { s = append(s, r) } } else { s = append(s, b) } } } s = append(s, '.') off += c case 0xC0: // pointer to somewhere else in msg. // remember location after first ptr, // since that's how many bytes we consumed. // also, don't follow too many pointers -- // maybe there's a loop. if off >= lenmsg { return "", lenmsg, ErrBuf } c1 := msg[off] off++ if ptr == 0 { off1 = off } if ptr++; ptr > 10 { return "", lenmsg, &Error{err: "too many compression pointers"} } off = (c^0xC0)<<8 | int(c1) default: // 0x80 and 0x40 are reserved return "", lenmsg, ErrRdata } } if ptr == 0 { off1 = off } if len(s) == 0 { s = []byte(".") } return string(s), off1, nil } (Not a real bug) And then it will learn that new case, and start muta,ng that one, un,l eventually it finds an input that triggers the bug and crashes

• American Fuzzy Lop afl-­‐fuzz is a very effec,ve and popular

C/C++ fuzzer by Michał Zalewski aka lcamtuf. Uses compile-­‐,me instrumenta,on to keep track of code coverage (and other state), looking for corner cases triggering memory errors or other bugs. It found security vulnerabili,es in dozens of programs. h]p:/ /lcamtuf.coredump.cx/afl/technical_details.txt

• Security? Is this a security talk?

• go-­‐fuzz

• Back to our bug! package bug // ParseStrings parses a

list of strings encoded in a binary format // where a single-byte value is followed by a string of that length // Note: ParseStrings takes unvalidated input from the network func ParseStrings(input []byte) (result []string) { for len(input) > 0 { strLength := input[0] input = input[1:] result = append(result, string(input[:strLength])) input = input[strLength:] } return }

• $ go-fuzz-build github.com/FiloSottile/fuzz-talk $ mkdir -p workdir/corpus $ echo -ne

'\x0cHello World!' > workdir/corpus/example $ go-fuzz -bin=bug-fuzz.zip -workdir=workdir 2015/09/30 16:42:09 slaves: 8, corpus: 1 (3s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s 2015/09/30 16:42:12 slaves: 8, corpus: 2 (1s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 4, uptime: 6s 2015/09/30 16:42:15 slaves: 8, corpus: 3 (0s ago), crashers: 1, restarts: 1/1, execs: 1572 (175/sec), cover: 4, uptime: 9s 2015/09/30 16:42:18 slaves: 8, corpus: 6 (0s ago), crashers: 1, restarts: 1/1, execs: 3182 (265/sec), cover: 4, uptime: 12s [...]

• $ tree workdir workdir ├── corpus │ ├── 14ce9986bbe073a8c48a9974e87c6f4d0e7b0c50-3 │

├── 51b45a439e25342db498f0915cc22f80847034a2-3 │ ├── 5df1d5920aafe2d3a44680ba9b9fec6760e4a879-4 │ ├── 6d3d9552845e675ee217db7c341d0c794ec005a9-1 │ ├── ca4558ca0710d4b259fdca48a462d04aae5cda78-2 │ └── example ├── crashers │ ├── b6589fc6ab0dc82cf12099d1c2d40ab994e8410c │ ├── b6589fc6ab0dc82cf12099d1c2d40ab994e8410c.output │ └── b6589fc6ab0dc82cf12099d1c2d40ab994e8410c.quoted └── suppressions └── 4de5122d6c0972f7e447061ac749b4e5d7f31a2c

• $ cat b6589fc6ab0dc82cf12099d1c2d40ab994e8410c.output panic: runtime error: slice bounds out of

range goroutine 1 [running]: github.com/FiloSottile/fuzz-talk.ParseStrings(0x8820327001, 0x0, 0x1fffff, 0x8201301c0, 0x1, 0x1) /var/folders/v8/xdj2snz51sg2m2bnpmwl_91c0000gn/T/go-fuzz- build091822086/src/github.com/FiloSottile/fuzz-talk/bugged.go: 11 +0x213 github.com/FiloSottile/fuzz-talk.Fuzz(0x8820327000, 0x1, 0x200000, 0x3) /var/folders/v8/xdj2snz51sg2m2bnpmwl_91c0000gn/T/go-fuzz- build091822086/src/github.com/FiloSottile/fuzz-talk/ bugged_fuzz.go:6 +0x60 [...] exit status 2

• Here’s a func,on bug package bug // ParseStrings parses a

list of strings encoded in a binary format // where a single-byte value is followed by a string of that length // Note: ParseStrings takes unvalidated input from the network func ParseStrings(input []byte) (result []string) { for len(input) > 0 { strLength := input[0] input = input[1:] result = append(result, string(input[:strLength])) input = input[strLength:] } return } strLength is past input’s end 0x30

• Real world example

• h]ps:/ /blog.cloudflare.com/dns-­‐parser-­‐meet-­‐go-­‐fuzzer/

• Commit the Fuzz func,on! Fuzzing shouldn't be a one-­‐off just

like normal tests aren’t. Fuzz tests should be first-­‐class ci,zens of your test suite. They should be wri]en by the developers, who know best what to test and where to hook. They should be commi]ed to the tree for everyone to expand. They should be run regularly to detect regressions.

• Bonus: not just crashes Since it’s YOU wri,ng the Fuzz

func,on, you can check any condi,on you want, and os.Exit(1) if it’s not sa,sfied. Example: dns.Msg.PackBuffer misbehaves when the passed buffer is not zeroed. Write a Fuzz func,on to call PackBuffer once on a buffer of 0x00, then on a buffer of 0xff, crash if they differ. The fuzzer will find if there are corner cases that don’t overwrite the buffer.

• func Fuzz(rawMsg []byte) int { msg := &dns.Msg{} if unpackErr

= msg.Unpack(rawMsg); unpackErr != nil { return 0 } if res, packErr = msg.PackBuffer(buf); packErr != nil { return 0 } for i := range res { bufOne[i] = 0xff } if resOne, packErr = msg.PackBuffer(bufOne); packErr != nil { println("Pack failed only with a filled buffer") panic(packErr) } if !bytes.Equal(res, resOne) { println("buffer bits leaked into the packed message") println(hex.Dump(res)) println(hex.Dump(resOne)) os.Exit(1) }

• Thanks! Filippo Valsorda @FiloSo>le filippo@cloudflare.com hBps:/ /filippo.io/fuzz-­‐talk