Property Based Testing

Unlike unit testing, which tests specific test cases, property based testing tests the higher level properties of piece of code. It involves generating random inputs and verifying that the outputs match certain properties or invariants. Instead of thinking about specific test cases it makes you write code that handles all cases. QuickCheck is a library originally written in Haskell, but has been ported to a number of languages including go’s testing/quick, which implements utilities for writing property based tests. I try to use property based testing as much as possible since I find that property based tests reflect requirements better than unit tests. In particular, property based testing is helpful when testing features that handle user (either end user or other services) input since these tests will help find specific cases that your program doesn’t handle properly that might be overlooked with unit tests.

Another example where property based tests are an obvious choice is for testing methods that are inverses of each other like writing custom marshaling and unmarshaling methods. So let’s pretend we have to write methods to implement go’s encoding.BinaryMarshaler and encoding.BinaryUnmarshaler interfaces for a custom cat type. Here’s what the quick check test could look like.

The test fails immediately since the methods don’t do anything yet and the output shows the values generated as arguments for the name and age in the check function. Let’s start implementing the methods and see if we can get the test to pass. An incorrect implementation might marshal by converting the age to a string then returning the name and age separated by a comma.

Note that this isn’t guaranteed to fail since the inputs are generated randomly, so I picked a random seed that will fail. The output

main.go:51: #36: failed on input ",\U000fc131𨚽\U000d40c3", 5554241157310712471

makes it clear why this implementation doesn’t work. The name that was randomly generated had a comma as the first character. In a real program there might be additional validation done to ensure that the name field only contains certain characters, but for this example lets continue without any additional validation. A couple of details to note is that the random string values generated by go are always valid UTF-8 and custom types can also be used as arguments to a check function if they implement the quick.Generator interface.

To fix the issue with the name potentially containing a comma, we can use some functions from the encoding/binary package to encode the age as a variable length byte sequence. Since the binary.PutVarint and binary.Varint functions return the number of bytes written and read we can just convert the name to a byte slice and append it to the variable length encoded age.

That gets the tests to pass and gives pretty good confidence that this code satisfies the property that our cat type can be unmarshaled and marshaled back into the same value. Of course property based testing can’t prove a complete lack of bugs and program correctness, but it is another testing method that helps find edge cases that aren’t handled correctly. A quote from Edsger Dijkstra’s The Humble Programmer.

program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.