Putting Your Haskell to the Test!

How many times have you encountered a regression bug in your production code? This can be one of the most demoralizing experiences for a software engineer. You shipped code and were confident it worked. And now it turns out it broke something else. The best way to avoid these bugs is to have tests that check your code for these cases. So how does testing work in Haskell?

In Haskell, we have a mantra that if your code compiles it ought to work. This is might be more true in Haskell than in other languages. But it’s still a tongue-in-cheek comment that doesn't quite pass muster. There are often different ways to accomplish the same goal in Haskell. But we should strive to write in ways that make it more likely the compiler will catch our errors. For instance, we could use newtypes as an alternative to type synonyms to limit errors.

At a certain point though, you have to start writing tests if you want to be confident about your code. Luckily, as a pure functional language, Haskell has some advantages in testing. In certain ways, it is far easier and more natural to test than, say, object oriented languages. Its functional features allow us to take more confidence from testing Haskell. Let’s examine why.

Functional Testing Advantages

Testing works best when we are testing specific functions. We pass input, we get output, and we expect the output to match our expectations. In Haskell, this is a approach is a natural fit. Functions are first class citizens. And our programs are largely defined by the composition of functions. Thus our code is by default broken down into our testable units.

Compare this to an object oriented language, like Java. We can test the static methods of a class easily enough. These often aren't so different from pure functions. But now consider calling a method on an object, especially a void method. Since the method has no return value, its effects are all internal. And often, we will have no way of checking the internal effects, since the fields could be private.

We'll also likely want to try checking certain edge cases. But this might involve constructing objects with arbitrary state. Again, we'll run into difficulties with private fields.

In Haskell, all our functions have return values, rather than depending on effects. This makes it easy for us to check their true results. Pure functions also give us another big win. Our functions generally have no side effects and do not depend on global state. Thus we don't have to worry about as many pathological cases that could impact our system.

Test Driven Development

So now that we know why we’re somewhat confident about our testing, let’s explore the process of writing tests. The first step is to defined the public API for a particular module. To do this, we define a particular function we’re going to expose, and the types that it will take as input as output. Then we can stub it out as undefined, as suggested in this article on Compile Driven Learning. This makes it so that our code that calls it will still compile.

Now the great temptation for much all developers is to jump in and write the function. After all, it’s a new function, and you should be excited about it!

But you’ll be much better off in the long run if you first take the time to define your test cases. You should first define specific sets of inputs to your function. Then you should match those with the expected output of those parameters. We’ll go over the details of this in the next section. Then you’ll write your tests in the test suite, and you should be able to compile and run the tests. Since your function is still undefined, they'll all fail. But now you can implement the function incrementally.

Your next goal is to get the function to run to completion. Whenever you find a value you aren't sure how to fill in, try to come up with a base value. Once it runs to completion, the tests will tell you about incorrect values, instead of errors. Then you can gradually get more and more things right. Perhaps some of your tests will check out, but you missed a particular corner case. The tests will let you know about it.

HUnit

One way of going about testing your code is to use QuickCheck. This approach is more focused on abstract properties than on concrete examples. We go through a few examples with this library in this article on Monad Laws. In this article we’ll test some code using the HUnit library combined with the Tasty testing framework.

Suppose to start out, we’re writing a function that will take three inputs. It should multiply the first two, and subtract the third. We’ll start out by making it undefined:

simpleMathFunction :: Int -> Int -> Int -> Int
simpleMathFunction a b c = undefined

We’ll take out combinations of input and output and make them into test cases like this:

simpleMathTests :: TestTree
simpleMathTests = testGroup "Simple Math Tests"
  [ testCase "Small Numbers" .
      simpleMathFunction 3 4 5 @?= 7
  , testCase "Bigger Numbers" .
      simpleMathFunction 22 12 64 @?= 20
  ]

We start by defining a group of tests with a broader description. Then we make individual test cases that each have their own name for themselves. Then in each of these we use the @?= operator to check that the actual value is equal to the expected value. Make sure you get the order right, putting the actual value first. Otherwise you'll see confusing output. Then we can run this within a test suite and we’ll get the following information:

Simple Math Tests
  Small Numbers:  FAIL
    Exception: Prelude.undefined
  Bigger Numbers: FAIL
    Exception: Prelude.undefined

So as expected, our test cases fail, so we know how we can go about improving our code. So let’s implement this function:

simpleMathFunction :: Int -> Int -> Int -> Int
simpleMathFunction a b c = a * b - c

And now everything succeeds!

Simple Math Tests
  Small Numbers:  OK
  Bigger Numbers: OK

All 2 tests passed (0.00s)

Behavior Driven Development

As you work on bigger projects, you’ll find you aren’t just interacting with other engineers on your team. There are often less technical stakeholders like project managers and QA testers. These folks are less interested in the inner working of the code, but still concerned with the broader behavior of the code. In these cases, you may want to adopt “behavior driven development.” This is like test driven development, but with a different flavor. In this framework, you describe your code and its expected effects via a set of behaviors. Ideally, these are abstract enough that less technical people can understand them.

You as the engineer then want to be able to translate these behaviors into code. Luckily, Haskell is an immensely expressive language. You can often define your functions in such a way that they can almost read like English.

Hspec

In Haskell, you can implement behavior driven development with the Hspec library. With this library, you describe your functions in a particularly expressive way. All your test specifications will belong to a Spec monad.

In this monad, you can use composable functions to describe the test cases. You will generally begin a description of a test case with the “describe” function. This takes a string describing the general overview of the test case.

simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
  ...

You can then modify it by adding a different “context” for each individual case. The context function also takes a string. However, the idiomatic usage of context is that your string should begin with the words “when” or “with”.

simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
  context "when the numbers are small" $
    ...
  context "when the numbers are big" $
    ...

Now you’ll describe each the actual test cases. You’ll use the function “it”, and then a comparison. The combinators in the Hspec framework are functions with descriptive names like shouldBe. So your case will start with a sentence-like description and context of the case. The the case finishes “it should have a certain result": x “should be” y. Here’s what it looks like in practice:

main :: IO ()
main = hspec simpleMathSpec


simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
  context "when the numbers are small" $
    it "Should match the our expected value" $
      simpleMathFunction 3 4 5 `shouldBe` 7
  context "when the numbers are big" $
    it "Should match the our expected value" $
      simpleMathFunction 22 12 64 `shouldBe` 200

It’s also possible to omit the context completely:

simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
  it "Should match the our expected value" $
    simpleMathFunction 3 4 5 `shouldBe` 7
  it "Should match the our expected value" $
    simpleMathFunction 22 12 64 `shouldBe` 200

At the end, you’ll get neatly formatted output with descriptions of the different test cases. By writing expressive function names and adding your own combinators, you can make your test code even more self documenting.

Tests of our simple math function
  when the numbers are small
    Should match the our expected value
  when the numbers are big
    Should match the our expected value

Finished in 0.0002 seconds
2 examples, 0 failures

Conclusion

So this concludes our overview of testing in Haskell. We went through a brief description of the general practices of test-driven development. We saw why it’s even more powerful in a functional, typed language like Haskell. We went over some of the basic testing mechanisms you’ll find in the HUnit library. We then described the process of "behavior driven development", and how it differs from normal TDD. We concluded by showing how the HSpec library brings BDD to life in Haskell.

If you want to see TDD in action and learn about a cool functional paradigm along the way, you should check out our Recursion Workbook. It has 10 practice problems complete with tests, so you can walk through the process of incrementally improving your code and finally seeing the tests pass!

If you’ve never programmed in Haskell before and want to see what all the rage is about, you should download our Getting Started Checklist. It’ll walk you through some of the basics of downloading and installing Haskell and show you a few other tools to help you along your way!