Testing Part 1: TDD and Basic Libraries
How many times have you encountered a regression bug in your production code? This can be one of the most demoralizing experiences for a software engineer. You shipped code and were confident it worked. And now it turns out it broke something else. Even worse can be when you discover too late that while your code gives the correct output, it is not nearly performant enough. So your system starts breaking just as your starting to scale, leaving a bad impression for many new users.
The best way to avoid these issues is to have automated code that verifies test conditions and the performance of your program. In this series on Testing with Haskell, we'll see what libraries we can use to test and profile our code. This first part goes over the general ideas behind test driven development (TDD) and some of the basic libraries we can use to make it work in Haskell. We'll also quickly examine why Haskell is a good fit for TDD.
If you're already familiar with libraries like HUnit and HSpec, you can move onto part 2 of this series, where we discuss how to identify performance issues using profiling.
To use testing properly, you'll need to have some understanding of how we organize projects in Haskell. I recommend you learn how to use Stack to organize your Haskell code. Learn how by taking our free Stack mini-course!
Functional Testing Advantages
Testing works best when we are testing specific functions. We pass input, we get output, and we expect the output to match our expectations. In Haskell, this is a approach is a natural fit. Functions are first class citizens. And our programs are largely defined by the composition of functions. Thus our code is by default broken down into our testable units.
Compare this to an object oriented language, like Java. We can test the static methods of a class easily enough. These often aren't so different from pure functions. But now consider calling a method on an object, especially a void method. Since the method has no return value, its effects are all internal. And often, we will have no way of checking the internal effects, since the fields could be private.
We'll also likely want to try checking certain edge cases. But this might involve constructing objects with arbitrary state. Again, we'll run into difficulties with private fields.
In Haskell, all our functions have return values, rather than depending on effects. This makes it easy for us to check their true results. Pure functions also give us another big win. Our functions generally have no side effects and do not depend on global state. Thus we don't have to worry about as many pathological cases that could impact our system.
Test Driven Development
So now that we know why we’re somewhat confident about our testing, let’s explore the process of writing tests. The first step is to define the public API for a particular module. To do this, we define a particular function we’re going to expose, and the types that it will take as input as output. Then we can stub it out as undefined, as suggested in this article on Compile Driven Learning. This makes it so that our code that calls it will still compile.
Now the great temptation for much all developers is to jump in and write the function. After all, it’s a new function, and you should be excited about it!
But you’ll be much better off in the long run if you first take the time to define your test cases. You should first define specific sets of inputs to your function. Then you should match those with the expected output of those parameters. We’ll go over the details of this in the next section. Then you’ll write your tests in the test suite, and you should be able to compile and run the tests. Since your function is still undefined, they'll all fail. But now you can implement the function incrementally.
Your next goal is to get the function to run to completion. Whenever you find a value you aren't sure how to fill in, try to come up with a base value. Once it runs to completion, the tests will tell you about incorrect values, instead of errors. Then you can gradually get more and more things right. Perhaps some of your tests will check out, but you missed a particular corner case. The tests will let you know about it.
One way of going about testing your code is to use
QuickCheck. This approach is more focused on abstract properties than on concrete examples. We go through a few examples with this library in this article on Monad Laws. In this article we’ll test some code using the
HUnit library combined with the
Tasty testing framework.
Suppose to start out, we’re writing a function that will take three inputs. It should multiply the first two, and subtract the third. We’ll start out by making it undefined:
simpleMathFunction :: Int -> Int -> Int -> Int simpleMathFunction a b c = undefined
We’ll take out combinations of input and output and make them into test cases like this:
simpleMathTests :: TestTree simpleMathTests = testGroup "Simple Math Tests" [ testCase "Small Numbers" . simpleMathFunction 3 4 5 @?= 7 , testCase "Bigger Numbers" . simpleMathFunction 22 12 64 @?= 20 ]
We start by defining a group of tests with a broader description. Then we make individual test cases that each have their own name for themselves. Then in each of these we use the
@?= operator to check that the actual value is equal to the expected value. Make sure you get the order right, putting the actual value first. Otherwise you'll see confusing output. Then we can run this within a test suite and we’ll get the following information:
Simple Math Tests Small Numbers: FAIL Exception: Prelude.undefined Bigger Numbers: FAIL Exception: Prelude.undefined
So as expected, our test cases fail, so we know how we can go about improving our code. So let’s implement this function:
simpleMathFunction :: Int -> Int -> Int -> Int simpleMathFunction a b c = a * b - c
And now everything succeeds!
Simple Math Tests Small Numbers: OK Bigger Numbers: OK All 2 tests passed (0.00s)
Behavior Driven Development
As you work on bigger projects, you’ll find you aren’t just interacting with other engineers on your team. There are often less technical stakeholders like project managers and QA testers. These folks are less interested in the inner working of the code, but still concerned with the broader behavior of the code. In these cases, you may want to adopt “behavior driven development.” This is like test driven development, but with a different flavor. In this framework, you describe your code and its expected effects via a set of behaviors. Ideally, these are abstract enough that less technical people can understand them.
You as the engineer then want to be able to translate these behaviors into code. Luckily, Haskell is an immensely expressive language. You can often define your functions in such a way that they can almost read like English.
In Haskell, you can implement behavior driven development with the
Hspec library. With this library, you describe your functions in a particularly expressive way. All your test specifications will belong to a
In this monad, you can use composable functions to describe the test cases. You will generally begin a description of a test case with the “describe” function. This takes a string describing the general overview of the test case.
simpleMathSpec :: Spec simpleMathSpec = describe "Tests of our simple math function" $ do ...
You can then modify it by adding a different “context” for each individual case. The context function also takes a string. However, the idiomatic usage of
context is that your string should begin with the words “when” or “with”.
simpleMathSpec :: Spec simpleMathSpec = describe "Tests of our simple math function" $ do context "when the numbers are small" $ ... context "when the numbers are big" $ ...
Now you’ll describe each the actual test cases. You’ll use the function “it”, and then a comparison. The combinators in the
Hspec framework are functions with descriptive names like
shouldBe. So your case will start with a sentence-like description and context of the case. The the case finishes “it should have a certain result":
x “should be”
y. Here’s what it looks like in practice:
main :: IO () main = hspec simpleMathSpec simpleMathSpec :: Spec simpleMathSpec = describe "Tests of our simple math function" $ do context "when the numbers are small" $ it "Should match the our expected value" $ simpleMathFunction 3 4 5 `shouldBe` 7 context "when the numbers are big" $ it "Should match the our expected value" $ simpleMathFunction 22 12 64 `shouldBe` 200
It’s also possible to omit the context completely:
simpleMathSpec :: Spec simpleMathSpec = describe "Tests of our simple math function" $ do it "Should match the our expected value" $ simpleMathFunction 3 4 5 `shouldBe` 7 it "Should match the our expected value" $ simpleMathFunction 22 12 64 `shouldBe` 200
At the end, you’ll get neatly formatted output with descriptions of the different test cases. By writing expressive function names and adding your own combinators, you can make your test code even more self documenting.
Tests of our simple math function when the numbers are small Should match the our expected value when the numbers are big Should match the our expected value Finished in 0.0002 seconds 2 examples, 0 failures
This concludes our introduction to testing in Haskell. We went through a brief description of the general practices of test-driven development. We saw why it’s even more powerful in a functional, typed language like Haskell. We went over some of the basic testing mechanisms you’ll find in the HUnit library. We then described the process of "behavior driven development", and how it differs from normal TDD. We concluded by showing how the
HSpec library brings BDD to life in Haskell.
But testing correctness is only half the story! We also need to be sure that our code is performant enough. In part 2 of this series, we'll discuss how we can use the Criterion library to identify performance issues in our system.
If you want to see TDD in action and learn about a cool functional paradigm along the way, you should check out our Recursion Workbook. It has 10 practice problems complete with tests, so you can walk through the process of incrementally improving your code and finally seeing the tests pass!
If you want to learn the basics of writing your own test suites, you need to understand how Haskell code is organized! Take our quick and free Stack mini-course to learn how to use the Stack tool for this!