Flexible Data with Aeson
At a certain point, our Haskell programs have to be compatible with other programs running on the web. This is especially useful given the growing usage of micro-services as an architecture. Regardless, it's very common to be transferring data between applications on different stacks. You’ll need some kind of format that allows you to transfer your data and read it on both ends.
There are many different ways of doing this. But much of the current eco-system of web programming depends on the JSON format. JSON stands for “JavaScript Object Notation”. It is a way of encoding data that is compatible with Javascript programs. But it is also a useful serialization system that any language can parse. In this article, we’ll explore how to use this format for our data in Haskell.
JSON 101
JSON encodes data in a few different types. There are four basic types: strings, numbers, booleans, and null values. These all functions in a predictable way. Each line here is a valid JSON value.
“Hello”
4.5
3
true
false
nullJSON then offers two different ways of combining objects. The first is an array. Arrays contain multiple values, and represent them with a bracket delimited list. Unlike Haskell lists, you can put multiple types of objects in a single array. You can even put arrays in your arrays.
[1, 2, 3]
[“Hello”, false, 5.5]
[[1,2], [3,4]]Coming from Haskell, this kind of structure seems a little sketchy. After all, the type of whatever is in your array is not clear. But you can think of multi-type arrays as tuples, rather than lists, and it makes a bit more sense. The final and most important way to make an object in JSON is through the object format. Objects are delimited by braces. They are like arrays in that they can contain any number of values. However, each value is also assigned to a string name as a key-value pair.
{
“Name” : “Name”,
“Age” : 23,
“Grades” : [“A”, “B”, “C”]
}These objects are infinitely nest-able, so you can have arrays of objects of arrays and so on.
Encoding a Haskell Type
So that’s all nice, but how do we actually use this format to transfer our Haskell data? We’ll let’s start with a dummy example. We’ll make a Haskell data type and show a couple possible JSON interpretations of that data. First, our type and some sample values:
data Person = Person
{ name :: String
, age :: Int
, occupation :: Occupation
} deriving (Show)
data Occupation = Occupation
{ title :: String
, tenure :: Int
, salary :: Int
} deriving (Show)
person1 :: Person
person1 = Person
{ name = “John Doe”
, age = 26
, occupation = Occupation
{ title = “teacher”
, tenure = 5
, salary = 60000
}
}
person2 :: Person
person2 = Person
{ name = “Jane Doe”
, age = 25
, occupation = Occupation
{ title = “engineer”
, tenure = 4
, salary = 90000
}
}Now there are many different ways we can choose to encode these values. The most basic way might look something like this:
{
“name” : “John Doe”,
“age” : 26,
“occupation” : {
“title” : “teacher”,
“tenure” : 5,
“salary” : 60000
}
}
{
“name” : “Jane Doe”,
“age” : 25,
“occupation” : {
“title” : “engineer”,
“tenure” : 4,
“salary” : 90000
}
}Now, we might want to help our friends using dynamically typed languages. To do this, we could provide some more context around the type contained in the object. This format might look something more like this:
{
“type” : “person”,
“contents” : {
“name” : “John Doe”,
“age” : 26,
“occupation” = {
“type” : “Occupation”,
“contents” : {
“title” : “teacher”,
“tenure” : 5,
“salary” : 60000
}
}
}
}Either way, we ultimately have to decide on a format with whoever we’re trying to interoperate with. It’s likely though that you’ll be working with an external API that’s already set their expectations. You have to make sure you match them.
Data.Aeson
Now that we know what our target values are, we need a Haskell representation for them. This comes from the Data.Aeson library (named for the father of the mythological character "Jason"). This library contains the type Value. It encapsulates all the basic JSON types in its constructors. (I’ve substituted a couple type synonyms for clarity):
data Value =
Object (HashMap Text Value) |
Array (Vector Value) |
String Text |
Number Scientific |
Bool Bool |
NullSo we see all six elements represented. So when we want to represent our items, we’ll do so using these constructors. And like with many useful Haskell ideas, we’ll use typeclasses here to provide some structure. We'll use two typeclasses: ToJSON and FromJSON. We’ll first go over converting our Haskell types into JSON. We have to define the toJSON function on our type, which will turn it into a Value.
In general, we’ll want to stick our data type into an object, and this object will have a series of “Pairs”. A Pair is the Data.Aeson representation of a key-value pair, and it consists of a Text and another value. Then we’ll combine these pairs together into a JSON Object by using the object function. We’ll start with the Occupation type, since this is somewhat simpler.
{-# LANGUAGE OverloadedString #-}
import Data.Aeson (ToJSON(..), Value(..), object, (.=))
...
instance ToJSON Occupation where
toJSON :: Occupation -> Value
toJSON occupation = object
[ “title” .= toJSON (title occupation)
, “tenure” .= toJSON (tenure occupation)
, “salary” .= toJSON (salary occupation)
]The .= operator creates a Pair for us. All our fields are simple types, which already have their own ToJSON instances. This means we can use toJSON on them instead of the Value constructors. Note also we use the overloaded strings extension (as introduced here) since we use string literals in place of Text objects. Once we’ve defined our instance for the Occupation type, we can call toJSON on an occupation object. This makes it easy to define an instance for the Person type.
instance ToJSON Person where
toJSON person = object
[ “name” .= toJSON (name person)
, “age” .= toJSON (age person)
, “occupation” .= toJSON (occupation person)
]And now we can create JSON values from our data type! In general, we’ll also want to be able to parse JSON values and turn those into our data types. We'll use monadic notation to encapsulate the possibility of failure. If the keys we are looking for don’t show up, we want to throw an error:
import Data.Aeson (ToJSON(..), Value(..), object, (.=), (.:), FromJSON(..), withObject)
...
instance FromJSON Occupation where
parseJSON = withObject “Occupation” $ \o -> do
title_ <- o .: “title”
tenure_ <- o .: “tenure”
salary_ <- o .: “salary”
return $ Occupation title_ tenure_ salary_
instance FromJSON Person where
parseJSON = withObject “Person” $ \o -> do
name_ <- o .: “name”
age_ <- o .: “age”
occupation_ <- o .: “occupation”
return $ Person name_ age_ occupation_A few notes here. The parseJSON functions are defined through eta reduction. This is why there seems to be no parameter. The .: operator can grab any data type that conforms to FromJSON itself. Just as we could use toJSON with simple types like String and Int, we can also parse them right out of the box. Plus we described how to parse an Occupation, which is why we can use the operator on the occupation field. Also, the first parameter to the withObject function is an error message we’ll get if our parsing fails. One last note is that our FromJSON and ToJSON instances are inverses of each other. This is definitely a good property for you to enforce within your own API definitions.
Now that we have these instances, we can see the JSON bytestrings for our different objects:
>> Data.Aeson.encode person1
"{\"age\":26,\"name\":\"John Doe\",\"occupation\":{\"salary\":60000,\"tenure\":5,\"title\":\"teacher\"}}"
>> Data.Aeson.encode person2
"{\"age\":25,\"name\":\"Jane Doe\",\"occupation\":{\"salary\":90000,\"tenure\":4,\"title\":\"engineer\"}}"Deriving Instances
You might look at the instances we’ve derived and think that the code looks boilerplate-y. In truth, these aren’t particularly interesting instances. Keep in mind though, an external API might have some weird requirements. So it’s good to know how to create these instances by hand. Regardless, you might be wondering if it’s possible to derive these instances in the same way we can derive Eq or Ord. We can, but it’s a little complicated. There are actually two ways to do it, and they both involve compiler extensions. The first way looks more familiar as a deriving route. We’ll end up putting deriving (ToJSON, FromJSON) in our types. Before we do that though, we have to get them to derive the Generic typeclass.
Generic is a class that allows GHC to represent your types at a level of generic constructors. To use it, you first need to turn on the compiler extension for DeriveGeneric. This lets you derive the generic typeclass for your data. You then need to turn on the DeriveAnyClass extension as well. Once you have done this, you can then derive the Generic, ToJSON and FromJSON instances for your types.
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DeriveAnyClass #-}
…
data Person = Person
{ name :: String
, age :: Int
, occupation :: Occupation
} deriving (Show, Generic, ToJSON, FromJSON)
data Occupation = Occupation
{ title :: String
, tenure :: Int
, salary :: Int
} deriving (Show, Generic, ToJSON, FromJSON)With these definitions in place, we’ll get the same encoding output we got with our manual instances:
>> Data.Aeson.encode person1
"{\"age\":26,\"name\":\"John Doe\",\"occupation\":{\"salary\":60000,\"tenure\":5,\"title\":\"teacher\"}}"
>> Data.Aeson.encode person2
"{\"age\":25,\"name\":\"Jane Doe\",\"occupation\":{\"salary\":90000,\"tenure\":4,\"title\":\"engineer\"}}"As you can tell, this is a super cool idea and it has widespread uses all over the Haskell ecosystem. There are many useful typeclasses with a similar pattern to ToJSON and FromJSON. You need the instance to satisfy a library constraint. But the instance you'll write is very boilerplate-y. You can get a lot of these instances by using the Generic typeclass together with DeriveAnyClass.
The second route involves writing Template Haskell. Template Haskell is another compiler extension allowing GHC to generate code for you. There are many libraries that have specific Template Haskell functions. These allow you to avoid a fair amount of boilerplate code that would otherwise be very tedious. Data.Aeson is one of these libraries.
First you need to enable the Template Haskell extension. Then import Data.Aeson.TH and you can use the simple function deriveJSON on your type. This will give you some spiffy new ToJSON and FromJSON instances.
{-# LANGUAGE TemplateHaskell #-}
import Data.Aeson.TH (deriveJSON, defaultOptions)
data Person = Person
{ name :: String
, age :: Int
, occupation :: Occupation
} deriving (Show)
data Occupation = Occupation
{ title :: String
, tenure :: Int
, salary :: Int
} deriving (Show)
-- The two apostrophes before a type name is template haskell syntax
deriveJSON defaultOptions ''Occupation
deriveJSON defaultOptions ''PersonWe’ll once again get similar output:
>> Data.Aeson.encode person1
"{\"name\":\"John Doe\",\"age\":26,\"occupation\":{\"title\":\"teacher\",\"tenure\":5,\"salary\":60000}}"
>> Data.Aeson.encode person2
"{\"name\":\"Jane Doe\",\"age\":25,\"occupation\":{\"title\":\"engineer\",\"tenure\":4,\"salary\":90000}}"Unlike deriving these types wholecloth, you actually have options here. Notice we passed defaultOptions initially. We can change this and instead pass some modified options. For instance, we can prepend our field names with the type if we want:
deriveJSON (defaultOptions { fieldLabelModifier = ("occupation_" ++)}) ''Occupation
deriveJSON (defaultOptions { fieldLabelModifier = ("person_" ++)}) ''PersonResulting in the output:
>> Data.Aeson.encode person1
"{\"person_name\":\"John Doe\",\"person_age\":26,\"person_occupation\":{\"occupation_title\":\"teacher\",\"occupation_tenure\":5,\"occupation_salary\":60000}}"Template Haskell can be convenient. It reduces the amount of boilerplate code you have to write. But it will also make your code take longer to compile. There’s also a price in accessibility when you use template Haskell. Most Haskell newbies will recognize deriving syntax. But if you use deriveJSON, they might be scratching their heads wondering where exactly you've defined the JSON instances.
Encoding, Decoding and Sending over Network
Once we’ve defined our different instances, you might be wondering how we actually use them. The answer depends on the library you’re using. For instance, the Servant library makes it easy. You don’t need to do any serialization work beyond defining your instances! Servant endpoints define their return types as well as their response content types. Once you've defined these, the serialization happens behind the scenes. Servant is an awesome library if you need to make an API. You should check out the talk I gave at BayHac 2017 if you want a basic introduction to the library. Also take a look at the code samples from that talk for a particular example.
Now, other libraries will require you to deal with JSON bytestrings. Luckily, this is also pretty easy once you’ve defined the FromJSON and ToJSON instances. As I’ve been showing in examples through the article, Data.Aeson has the encode function. This will take your JSON enabled type and turn it into a ByteString that you can send over the network. Very simple.
>> Data.Aeson.encode person1
"{\"name\":\"John Doe\",\"age\":26,\"occupation\":{\"title\":\"teacher\",\"tenure\":5,\"salary\":60000}}"As always, decoding is a little bit trickier. You have to account for the possibility that the data is not formatted correctly. You can call the simple decode function. This gives you a Maybe value, so that if parsing is unsuccessful, you’ll get Nothing. In the interpreter, you should also make sure to specify the result type you want from decode, or else you’ll get that it tries to parse it as (), which will fail.
>> let b = Data.Aeson.encode person1
>> Data.Aeson.decode b :: Maybe Person
Just (Person {name = "John Doe", age = 26, occupation = Occupation {title = "teacher", tenure = 5, salary = 60000}})To better handle error cases though, you should prefer eitherDecode. This will give you a an error message if it fails.
>> Data.Aeson.eitherDecode b :: Either String Person
Right (Person {name = "John Doe", age = 26, occupation = Occupation {title = "teacher", tenure = 5, salary = 60000}})
>> let badString = "{\"name\":\"John Doe\",\"occupation\":{\"salary\":60000,\"tenure\":5,\"title\":\"teacher\"}}"
>> Data.Aeson.decode badString :: Maybe Person
Nothing
>> Data.Aeson.eitherDecode badString :: Either String Person
Left "Error in $: key \"age\" not present"Summary
So now we know all the most important points about serializing our Haskell data into JSON. The first step is to define ToJSON and FromJSON instances for the types you want to serialize. These are straightforward to write out most of the time. But there are also a couple different mechanisms for deriving them. Once you’ve done that, certain libraries like Servant can use these instances right out of the box. But handling manual ByteString values is also easy. You can simply use the encode function and the various flavors of decode.
Perhaps you’ve always thought to yourself, “Haskell can’t possibly be useful for web programming.” I hope this has opened your eyes to some of the possibilities. You should try Haskell out! Download our Getting Started Checklist for some pointers to some useful tools!
Maybe you’ve done some Haskell already, but you're worried that you’re getting way ahead of yourself by trying to do web programming. You should download our Recursion Workbook. It’ll help you solidify your functional programming skills!
Smart Data with Conduits
If you’re a programmer now, there’s one reality you’d best be getting used to. People expect you to know how to deal with big data. The kind of data that will take a while to process. The kind that will crash your program if you try to bring it all into memory at the same time. But you also want to avoid making individual SQL requests to a database to access every single row. That would be awfully slow. So how do you solve this problem? More specifically, how do you solve this problem in Haskell?
This is exactly the type of problem the Data.Conduit library exists for. This article will go through a simple example. We’ll stream some integers using a conduit "source" and add them all together. Luckily, we only ever need to see one number at a time from the source, so we have no need of bringing a ton a data into memory. Let’s first take a look at how we create this source.
Getting Our Data
Suppose we are just trying to find the sum of the numbers from 1 up to 10 million. The naive way to do this would be to have our “source” of numbers be a raw list, and then we would take the sum of this list:
myIntegerSourceBad :: [Integer]
myIntegerSourceBad = [1..10000000]
sumFromSource :: [Integer] -> Integer
sumFromSource lst = sum lstThis method necessitates having the entire list in memory at the same time. Imagine if we were querying a database with tens of millions of entries. That would be problematic. Our first stroke to solve this will be to use Data.Conduit to create a “Source” of integers from this list. We can do this with the sourceList function. Note that Identity is the simple Identity monad. That is, the monad base monad that doesn’t actually do anything:
import Control.Monad.Identity (Identity)
import Data.Conduit (Source)
import Data.Conduit.List (sourceList)
myIntegerSource :: Source Identity Integer
myIntegerSource = sourceList [1..10000000]So now instead of returning a lump list of Int values, we’re actually getting a stream of values we call a Source. Let’s describe the meaning of some of the conduit types so we’ll have a better idea of what’s going on here.
Conduit Types
The conduit types are all built on ConduitM, which is the fundamental monad for conduits. This type has four different parameters. We’ll write out its definition like so:
type ConduitM i o m rYou should think of each conduit as a data processing funnel. The i type is the input that you can bring into the funnel. The o type stands for output. These are the values you can push out of the funnel. The m type is the underlying monad. We'll see examples with both the Identity monad and the IO monad.
We can think of the last r type as the “result”. But it’s different from the output. This is what the function itself will return as its value once it’s done. We'll see a return type when we write our sink later. But otherwise, we'll generally just use (). As a result, we can rewrite some types using the Conduit type synonym. This synonym is helpful for ignoring the return type. But it is also annoying because it flips the placement of the m and o types. Don’t let this trip you up.
type Conduit i m o = ConduitM i o m ()Now we can define our source and sink types. A Source is a conduit that takes no input (so i is the unit type) and produces some output. Conversely, a Sink is a conduit that takes some input but produces no output (so the Void type):
type Source m o = Conduit () m o
type Sink i m r = ConduitM i Void m rSo in the above example, our function is a “source”. It creates a series of integer values without taking any inputs. We’ll see how we can match up different conduits that have different matching types.
Primitive Functions
There are three basic functions in the conduits library: yield, await, and leftover. Yield is how we pass a value downstream to another conduit. In other words, it is a “production” function, or a source of values. We can only yield types that fit with the output type of our conduit function.
So then how do we receive values from upstream? The answer is the await function. This operates as a sink and allows a function to remain inert while it waits to receive a value from upstream. Naturally, this value must be of the type of the input of the conduit. Now, the actual resulting type of await is a Maybe value. We could receive a value of Nothing. This indicates that our upstream source has terminated and we won't receive any more values. This is usually the circumstance in which we’ll let our function return.
The final function is the leftover function. This allows you to take a value that you have already pulled from upstream and put it back upstream. This way, we can consume the value again from a different sink, or a different iteration of this sink. One caveat the docs add is that you should not use leftover with values that you have have created yourself. You should ONLY use values you got from upstream.
Writing Our Conduits
So above we’ve already written a source conduit for our toy program. We return a list of integers so we can stream them into our program 1-by-1. Now we’ll write a sink that will take these values and add them together. This will take the form of a recursive function with an accumulator argument. Sinks often function recursively to receive their next input.
So we’ll start off by awaiting some value from the upstream conduit.
myIntegerSink :: Integer -> Sink Integer Identity Integer
myIntegerSink accum = do
maybeFirstVal <- await
...If that value is Nothing, we’ll know there are no more values coming. We can then proceed to return whatever accumulated value we have.
myIntegerSink :: Integer -> Sink Integer Identity Integer
myIntegerSink accum = do
maybeFirstVal <- await
case maybeFirstVal of
Nothing -> return accum
...If that value contains another Int, we’ll first calculate the new sum by adding them together. Then we’ll make a recursive call to the sink function with the new accumulator argument:
myIntegerSink :: Integer -> Sink Integer Identity Integer
myIntegerSink accum = do
maybeFirstVal <- await
case maybeFirstVal of
Nothing -> return accum
Just val -> do
let newSum = val + accum
myIntegerSink newSumAnd that’s it really! Our example is simple so the code ends up being simple as well.
Combining Our Conduits
Now we’ll want to combine our conduits. We do this with the “fuse” operator, written out as =$=. Haskell libraries can be notorious for having strange operators. This library isn’t necessarily an exception. But think of conduits as a tunnel, and this operator looks like it's connecting two parts of a tunnel.
With this operator, the output type of the first conduit needs to match the input type of the second conduit. With how we’ve set up our conduits, this is the case. So now to polish things off, we use runConduitPure with our combined conduit:
fullConduit :: Integer
fullConduit = runConduitPure $
myIntegerSource =$= myIntegerSink 0It’s generally quite easy using fuse to add more conduits. For instance, suppose we wanted even numbers to count double for our purposes. We could accomplish this in our source or sink, but we could also add another conduit. It will take an Int as input and yield an Int as output. It will want for its input integer, check its value, and then double the value if it is even. Then we will yield the resulting value. Once again, if we haven’t seen the end of the conduit, we need to recursively jump back into the conduit.
myDoublingConduit :: Conduit Integer Identity Integer
myDoublingConduit = do
maybeVal <- await
case maybeVal of
Nothing -> return ()
Just val -> do
let newVal = if val `mod` 2 == 0
then val * 2
else val
yield newVal
myDoublingConduitThen we can stick this conduit between our other conduits with the fuse operator!
fullConduit :: Integer
fullConduit = runConduitPure $
myIntegerSource =$= myDoublingConduit =$= myIntegerSink 0Vectorizing
There are also times when you’ll want to batch up certain transactions. A great example of this is when you want to insert all results from your sink into a database. You don’t want to keep sending individual insert queries down the line. You can instead wait and group a bunch of inputs together and send batch inserts.
For our toy example, we’ll gather our ints in groups of 100000 so that we can give log progress updates along the way. This will require changing our conduits to live on top of the IO monad. But once we’ve made this change, we can use the conduitVector function like so:
fullConduitIO :: IO Integer
fullConduitIO = runConduit $
myIntegerSourceIO =$= conduitVector 100000 =$= myIntegerVectorSink 0
myIntegerSourceIO :: Source IO Integer
myIntegerSourceIO = sourceList [1..100000000]
myIntegerVectorSink :: Integer -> Sink (Vector Integer) IO Integer
myIntegerVectorSink accum = do
maybeFirstVal <- await
case maybeFirstVal of
Nothing -> return accum
Just vals -> do
let newSum = (Vec.sum vals) + accum
lift $ print newSum
myIntegerVectorSink newSumBy vectorizing the conduit, we need to change the sink so that it takes Vector Int as its input type instead of Int. Then we get a vector from await, so we have to sum those values as well.
Summary
The Data.Conduit library allows you to deal with large amounts of data in sustainable ways. You can use it to stream data from a database or some other source. This is more efficient than bringing a large chunk of information into memory all at once. It also allows you to pass information through “tunnels”, called conduits. You can make these perform many complicated operations. You mainly compose conduit functions from the yield, await and leftover functions. You merge conduits together into a larger conduit with the “fuse” operator =$=. You can also use the conduitVector function to batch certain operators.
This was more advanced of a topic. If you’ve never written Haskell before, it’s not as scary as this article makes it out to be! Check out our Getting Started Checklist to take the first steps on your Haskell journey!
If you’ve done a little bit of Haskell before but need some more practice on the fundamentals, you should download our Recursion Workbook. It has some great material on recursion as well as 10 practice problems!
Next week we’ll keep up with some more advanced topics. We’ll look into the Data.Aeson library and how it allows us to serialize Haskell objects into JSON format! So stay tuned to Monday Morning Haskell!
Appendix
I mentioned earlier that operators can be a major pain point in Haskell. Unfortunately, so can documentation. This can be especially true when tracking down the right docs for the conduit concepts. So for reference, here are all the imports I used:
import Conduit (conduitVector, lift)
import Control.Monad.Identity (Identity)
import Data.Conduit (Source, Sink, await, runConduitPure, (=$=), Conduit, yield, runConduit)
import Data.Conduit.List (sourceList)
import Data.Vector hiding (sum)
import qualified Data.Vector as VecNote that Data.Conduit and Data.Conduit.List come from the conduit library on hackage. However, the Conduit module actually comes from conduit-combinators. This is very deceptive.
Numbers of Every Shape and Size
Last week we explored the many different string types in Haskell. But this isn't the only situation where we seem to have an abundance of similar types. We can also see this in Haskell's numeric types. Again, we have the issue that we often want to represent numbers in slightly different ways. But Haskell's type system forces us to have different types for each different case. Interoperating between these types can be very painful.
This is another one of those things with Haskell that can definitely throw beginners for a loop. It can even make them say, “ugh, this language sucks, I’m going back to Javascript" (where numbers are easier). But there are a few simple rules you have to get under your belt, and then things become much better. Let's start by summarizing the different numeric types we can use.
Int Types
The most familiar integral type is the simple Int type. This represents your standard machine-varying, bounded, signed integer. According to the documentation, it is guaranteed to have a range of at least -2^29 to 2^29. So if you’re on a 32-bit machine, your Int might have a different set of bounds than on a 64-bit machine. Luckily, Int is a member of the Bounded typeclass. So in your code, you can always query the max and min bounds to check for overflow.
class Bounded a where
minBound :: a
maxBound :: aNow, suppose you don’t want to be at the whims of the machine for how large a value you can store. Sometimes, you want to know exactly how much memory your int will take. There are several different Int types that allow you to do exactly this. We have Int8, Int16, Int32, and Int64. They allow you to have more definite bounds on your number, while giving you the same basic functionality as Int. Obviously Int8 is 8 bits, Int16 is 16 bits, and so on.
Now, there are also circumstances where you want your integer to be unbounded. In this case, you’ll need to use the Integer type. This type establishes no limit on how big your number can be. It does not implement the Bounded typeclass. Naturally, this comes at a performance penalty. If your number is too large to fit in a single register in memory, Haskell will use a byte array to represent the number. Operations on this number will be slower. Hardware is designed to make mathematical operations on smaller values extremely fast. You won’t be able to get these speedups on larger numbers. But if you need the higher bounds then you don't have a choice.
Word Types
Once you understand the different Int classes, next up are the Word types. Each Int size has a corresponding Word type. These types work the same way, except they are unsigned. So whereas an Int8 goes from -128 to 127, a Word8 can contain values from 0 to 255. In fact, Data.ByteString has a function to give you a list of Word8 values as a representation of the byte string.
So what lessons then can we take from these different integral types? How do we choose what type to use? For most cases, Int is probably fine. Depending on the domain, you may be able to limit the number of bytes that you need to use. This is where the various sizes of integers come into play. If you know your value will be positive and not negative, definitely use Word types instead of Int types. This will give you more wiggle room against those bounds. If you're dealing with values that might exceed 64-bit bounds, you have no choice but to use Integer. Be aware that if the values are this large, you'll face a performance penalty.
The Integral Typeclass
You'll often come across a situation where you want to write a more general function on numbers. You don't know exactly what type you'll be dealing with. But you know it'll be one of these integer types. Like a good Haskell developer, you want your code to be as polymorphic as possible. This is what the Integral typeclass is for. It encapsulates a few different pieces of functionality.
First, it facilitates the conversion between the different integral types. It supplies a toInteger function with the type:
toInteger :: Integral a => a -> IntegerThis allows you to use the unbounded Integer type as a go-between amongst your integral types. It allows you to write reasonably polymorphic code. There can also be drawbacks though. By using the lowest common denominator, you might make your program's performance worse. So for performance critical code, you might need to reason more methodically about which type you should use.
The other functions within the Integral typeclass deal with division on integral types. For instance, there are quotient and remainder functions. These allow you to perform integral division when you don't know the exact type.
class Integral a where
toInteger :: a -> Integer
quot :: a -> a -> a
-- ^^ Integer division, e.g. 9 `quot` 2 = 4
rem :: a -> a -> a
-- ^^ Remainder, e.g. 9 `rem` 2 = 1
div :: a -> a -> a
mod :: a -> a -> aAs a note, the div and mod functions are like quot and rem, but round towards negative infinity instead of 0. It can be a little constricting to have to jump through these hoops. This is especially the case when your javascript friends can just get away with always writing 5 / 2. But that’s the price of a strong typed system.
Floating Point Numbers
Of course, we also have a couple different types to represent floating point numbers. Haskell has two main floating point types: Float and Double. The Float type is a single-precision floating point number. And of course, Double is double precision. They represent the same basic concept, but Double allows a larger range of values with more precision. At the same time, it takes up twice as much memory as a Float. Converting between these two types is easy using the following functions:
float2Double :: Float -> Double
double2Float :: Double -> FloatThere is a typeclass that encapsulates these types (as well as a couple more obscure versions). This is the Floating typeclass. It allows a host of different operations to be performed on these types. A lot of these are mathematical functions.
For instance, one of these functions is just pi. So if your function takes any type in the Floating typeclass, you can still get a reliable value for pi. Floating also incorporates other math concepts as well, like square roots, exponents, trigonometric functions, and so on.
Other Numeric Typeclasses
There are a few other numeric typeclasses. They encapsulate behavior both for floating numbers and integers. For instance, we have the Real typeclass which allows us to convert anything to a Rational number. There is also the Fractional typeclass. It allows us to perform certain operations that are natural on fractions. These include calculating reciprocals and performing true (non-integer) division. We can then mash these two classes together to get RealFrac. This typeclass allows us to express a number as a true fraction (so a tuple of two integral numbers). It has several other useful functions like ceiling, round, truncate, and so on.
Conversion Mania
We’ve gone over some of the conversions between similar types. But it’s difficult to keep track of all the different ways to convert between values. For a more exhaustive list, check out Gentle Introduction to Haskell. Besides what we've covered, two types of transitions stand out the most.
First, there is fromIntegral. This allows you to convert from any integral type to any numeric type. It should be your go-to when you need a floating point number but have some integer type. Second, there is the process of going from a floating point number to an integer type. You’ll generally use one of round, floor and ceiling, depending on your desired coercion. Finally, remember the toInteger function. It is generally the answer when going between integral types.
fromIntegral :: (Integral a, Num b) => a -> b
round :: (Fractional a, Integral b) => a -> b
floor :: (Fractional a, Integral b) => a -> b
ceiling :: (Fractional a, Integral b) => a -> b
toInteger :: (Integral a) => a -> IntegerScientific
If you do any level of web programming, you’ll likely be encoding things in JSON using the Data.Aeson library. We’ll go more in depth on this library in a later article (it’s super useful). But it uses the Scientific type, which represents numbers in “scientific” notation. In this format, you have a base multiplied by 10 to a certain power. You won’t encounter this type too often, but it’s useful to know how to construct it. In particular you’ll often want to take simple integer or floating point values and convert them back and forth. Here are some code examples how:
-- Creates 70000
createScientificNum :: Scientific
createScientificNum = scientific 7 4
-- Conversion to float (i.e. 70000.0)
convertToFloat :: Double
convertToFloat = toBoundedRealFloat createScientificNum
-- Convert to integer, might fail if the scientific is not an integer
convertToInt :: Maybe Integer
convertToInt = toBoundedInteger createScientificNumNum Typeclass
So on top of the different floating and integral typeclasses, we also have the Num typeclass. This brings them all together under one roof. The required elements of this typeclass are your basic math operators.
class Num a where
(+), (*), (-) :: a -> a -> a
negate :: a -> a
abs :: a -> a
signum :: a -> a
fromInteger :: Integer -> aThis class is also somewhat analogous to the IsString typeclass. The IsString class let us use a string literal to represent our different types in code. In the same way, the Num typeclass allows us to represent our value using a numeric literal. We don’t even need a compiler extension for it! This is particularly helpful when you make a "newtype" around a number. It's nice to not have to always write the constructor for it.
newtype MyNumberType = MyNumberType Int
instance Num MyNumberType where
-- Implement math functions using underlying int value
...
myNumWithLiteral :: MyNumberType
myNumWithLiteral = 5Conclusion
Numbers, like string types, can be super confusing in Haskell. It’s hard to keep them straight. It’s especially difficult when you’re trying to interoperate between different types. It feels like since numeric types are so similar it should be easy to use them. And for once the type system seems to get in the way of simplicity here.
Luckily, polymorphism allows us many different avenues to fix this. We can almost always make our functions apply to lots of different numeric types. We’ll often need to convert our types to make things interoperate. But there are generally some nice functions that allow us to do this with ease.
If you’ve never programmed in Haskell before and want to try it out, you should check out our Getting Started Checklist. It will walk you through installing Haskell on your computer. It will also point you towards some tools that will help you in learning the language.
If you’ve experimented a little bit and want some extra practice, you should download our Recursion Workbook. It contains two chapters of materials as well as 10 practice problems!
Untangling Haskell's Strings
Haskell is not without its faults. One of the most universally acknowledged annoyances, even for pros, is keeping track of the different string types. There are, in total, five different types representing strings in Haskell. Remember Haskell is strongly typed. So if we want to represent strings in different ways, we have to have different types for them. This motivates the need for these five types, all with slightly different use cases. It's not so bad when you're using any one of them. But when you constantly have to convert back and forth between them, it can be a major hassle. In this article we'll go over these five different types. We'll examine their different use cases, and observe how to convert between them.
Strings
The String type is the most basic form of representing strings in Haskell. It is a simple type synonym for a list of unicode characters (the Char type). So whenever you see [Char] in your compile errors, know this refers to the basic String type. By default, when you enter in a string literal in your Haskell code, the compiler infers it as a String.
myFirstString :: String
myFirstString = HelloThe list representation of strings gives us some useful behavior. Since a string is actually a list, we can use all kinds of familiar functions from Data.List.
>> let a = "Hello"
>> map Data.Char.toUpper a
"HELLO"
>> 'x' : a
"xHello"
>> a ++ " Person!"
"Hello Person!"
>> Data.List.sort "dbca"
"abcd"The main drawback of the vanilla string type is its inefficiency. This comes as a consequence of immutability. For instance suppose we have:
myFirstString :: String
myFirstString = "Hello"
myModifiedString :: String
myModifiedString = map toLower (sort (myFirstString ++ " Person!"))This will allocate a total of 4 strings. The first is myFirstString. The second is the " Person!" literal. We can then append these without making another string. But then the sorted version will be a third allocation, and the fourth will be the lowercased version. This constant allocation can make our code non-performant and inappropriate for heavy duty operations.
Text
The Text family of string types solves this dilemma. There are two Text types: strict and lazy. Most often, you will use the strict form. However, the lazy form is useful in certain circumstances where you know you won't need the full string.
The main advantage Text has is that its functions are subject to "fusion". This means the compiler can actually prevent the issue of multiple allocations we saw in the last example. For instance, if we look at this:
import qualified Data.Char as C
import qualified Data.Text as T
optimizedTextVersion :: T.Text
optimizedTextVersion = T.cons 'c' (T.map C.toLower (T.append (T.Text "Hello ") (T.Text " Person!")))This will only actually allocate a single Text object at runtime. This will make it it substantially more efficient than the String version. So for industrial use of heavy text processing, you are much better off using the Text type than the String type.
ByteString
The third family of types fall into the ByteString category. As with Text, there are strict and lazy variants of bytestrings. Lazy bytestrings are a bit more common than lazy text values though. Bytestrings are the lowest level representation of the characters. It is the closest you can get to the real machine level interpretation of them. At their core, bytestrings are a list of Word8 objects. A Word8 is simply an 8-bit number representing a unicode character.
Most networking libraries will use bytestrings, as they make the most sense for serialization. When you send information across platforms, you can't be sure about the encoding on the other end. If you store information in a database, you will often want to use bytestrings as well. Like Text types, they are generally far more efficient than strings.
Conversion
So with all these types floating around, the real problem is converting between them. It can be enormously frustrating when you want to write some basic code but you have a different string type. You'll have to look up the conversion if you don't remember, and this can be annoying. Our first example will be with String and Text. This is quite straightforward. The Data.Text package exports these two functions, which do exactly what you want:
pack :: String -> Text
unpack :: Text -> StringThere are equivalents in Data.Text.Lazy. We'll find similar functions for going between ByteStrings and Strings. They exist in the Data.ByteString.Char8 package:
pack :: String -> ByteString
unpack :: ByteString -> StringNote these only work with strict ByteStrings. To convert between strict and lazy, you'll want functions from the .Lazy version of a text type. For instance, Data.Text.Lazy exports:
-- (Lazy to strict)
toStrict :: Data.Text.Lazy.Text -> Data.Text.Text
-- (Strict to lazy)
fromStrict :: Data.Text.Text -> Data.Text.Lazy.TextThere are equivalents in Data.ByteString.Lazy. The final conversion we'll go over is between Text and ByteString. You could use String as an intermediate type with the functions above. But this makes certain assumptions about the encoding and is subject to failure. Going from Text to ByteString is straightforward, assuming you know your data format. The following functions exist in Data.Text.Encoding:
encodeUtf8 :: Text -> ByteString
-- LE = Little Endian format, BE = Big Endian
encodeUtf16LE :: Text -> ByteString
encodeUtf16BE :: Text -> ByteString
encodeUtf32LE :: Text -> ByteString
encodeUtf32BE :: Text -> ByteStringIn general, you'll use UTF8 encoded text and thus encodeUtf8. Decoding is a little more complicated. Simple functions exist in this same library:
decodeUtf8 :: ByteString -> Text
decodeUtf16LE :: ByteString -> Text
decodeUtf16BE :: ByteString -> Text
decodeUtf32LE :: ByteString -> Text
decodeUtf32BE :: ByteString -> TextBut these can throw errors if your bytestring does not match the format. Run-time exceptions are bad, so for UTF8, we have this function:
decodeUtf8' :: ByteString -> Either UnicodeException TextWhich let's us wrap this in an Either and handle possible errors. For the other formats, we have to rely on functions like:
decodeUtf16LEWith :: OnDecodeError -> ByteString -> TextWhere OnDecodeError is a specific type of handler. These functions can be particularly cumbersome and difficult to deal with. Luckily, you'll most often be using UTF8.
OverloadedStrings
So we haven't touched too much on language extensions yet in my articles. But here's our first real example of one. It's intended to show you language extensions aren't particularly scary! As we saw earlier, Haskell will in general interpret string literals in your code as the String type. This means you are unable to have the following code:
-- Fails
myText :: Text
myText = "Hello"
myBytestring :: ByteString
myBytestring = "Hello"The compiler expects both of these values to be of String type, and not the types you gave. So this will normally throw compiler errors. However, with the OverloadedStrings extension, you can fix this! Extensions use tags like {-# LANGUAGE … #-}. They are generally added at the top of your source file.
{-# LANGUAGE OverloadedStrings #-}
-- This works!
myText :: Text
myText = "Hello"
myBytestring :: ByteString
myBytestring = "Hello"In fact, for any type you make, you can create an instance of the IsString typeclass. This will allow you to use string literals to represent it.
{-# LANGUAGE OverloadedStrings #-}
import qualified Data.String (IsString(..))
data MyType = MyType String
instance IsString MyType where
fromString s = MyType s
myTypeAsString :: MyType
myTypeAsString = "Hello"You can also enable this extension within GHCI. You need to use the command :set -XOverloadedStrings.
Summary
Haskell uses 5 different types for representing strings. Two of these are lazy versions. The String type is a type synonym for a list of characters, and is generally inefficient. Text represents strings somewhat differently, and can fuse operations together for efficiency. ByteString is a low level representation most suited to serialization. There are a lot of ways to convert between types, and it's hard to keep them straight. Finally, the OverloadedStrings compiler can make your life easier. It allows you to use a string literal refer to any of your different string types.
If you haven't had a chance to get started with Haskell yet, you should get our Getting Started Checklist. It will guide you through installation and some basics!
Be sure to check back next week! Now that we understand strings, we'll divide into another potentially thorny system of different types. We'll investigate the different numeric types and the simple conversions we can run between them.
4 Steps to a Better Imports List
So your thoughts on seeing this title might be “Holy cow, an article about imports...could there be anymore more boring?” But bear with me. This is actually something that is pretty easy to do correctly. But if you're even a bit lazy (as I often am), you’ll do it wrong. And that can have really bad effects on you and your teammates' productivity.
Imagine someone is trying to make their first contribution to your codebase. They have no idea which functions are defined where. They aren’t necessarily familiar with the libraries you use. So what happens when they come across a function they don't know? They'll search for the definition in the file itself. But if it's not there, they'll have to look to the imports section.
Once you’ve built a Haskell program of even modest size, you'll appreciate the importance of the imports section of any source file. Almost any nontrivial program requires code beyond the base libraries. This means you’ll have to import library code that you got through Stack or Cabal. You’ll also want different parts of your code working together. So naturally your different modules will import each other as well.
When you write your imports list properly, your contributors will love you. They'll know exactly where they need to look to find the right documentation. Write it poorly, and they’ll be stuck and frustrated. They'll lose all kinds of time googling and hoogling function definitions.
Tech stacks like Java and iOS have mature IDEs like IntelliJ or XCode. These make it easy someone to click on a method and find documentation for it. But you can't count on people having these features for their Haskell environment yet. So now imagine the function or expression they’re looking for is not defined in the file they’re looking at. They’ll need to figure out themselves which module imported it. Here are some good practices to make this an easy process.
Only Import the Functions You Need
The first way to make your imports clearer is to specify which functions you import. The biggest temptation is to only write the module name in the import. This will allow you to use any function from that library. But you can also limit the functions you use. You do this with a parenthesized list after the module name.
import Data.List.Split (splitOn)
import Data.Maybe (isJust, isNothing)Now suppose someone sees the splitOn function in your code and doesn’t know what it does, or what its type is. By looking at your imports list, they know they can find out by googling the Data.List.Split library.
Qualifying Imports
The second way to clarify your imports is to use the qualified keyword. This means that you must prefix every function you use from this module by a name assigned to the module. You can either use the full module name, or you can use use the as keyword. This indicates you will refer to the module by a different, generally shorter name.
import qualified Data.Map as M
import qualified Data.List.Split
...
myMap :: M.Map String [String]
myMap = M.fromList [(“first”, Data.List.Split.splitOn “abababababa” “a”)]In this example, a contributor can see exactly where our functions and types came from. The fromList function and the “Map” data structure belong to the Data.Map module, thanks to the “M” prefix. The splitOn function also clearly comes from Data.List.Split.
You can even import the same module in different ways. This allows you to namespace certain functions in different ways. This helps to avoid prefixing type names, so your type signatures remain clean. In this example, we explicitly import the ByteString type from Data.ByteString. Then we also make it a qualified import, allowing us to use other functions like empty.
import qualified Data.ByteString as B
import Data.ByteString (ByteString)
…
myByteString :: ByteString
myByteString = B.emptyOrganizing Your Imports
Next, you should separate the internal imports from the external ones. That is, you should have two lists. The first list consists of built-in packages. The second list has modules that are in the codebase itself. In this example, the “OWA” modules are from within the codebase. The other modules are either Haskell base libraries or downloaded from Hackage:
import qualified Data.List as L
import System.IO (openFile)
import qualified Text.PrettyPrint.Leijen as PPrint
import OWAPrintUtil
import OWASwiftAbSynThis is important because it tells someone where to look for the modules. If it's from the first list, they will immediately know whether they need to look online. For imports on the second list, they can find the file within the codebase.
You can provide more help by name-spacing your module names. For instance, you can attach your project name (or an abbreviation) to the front of all your modules’ names, as above. An even better approach is to separate your modules by folder and have period spacing. This makes it more clear where to look within the file structure of the codebase for it.
Organizing your code cleanly in the first place can also help this process. For instance, it might be a good idea to have one module that contains all the types for a particular section of the codebase. Then it should be obvious to users where the different type names are coming from. This can save you from needing to qualify all your type names, or have a huge list of types imported from a module.
Making the List Easy to Read
On a final note, you want to make it easy to read your import list. If you have a single qualified import in a list, line up all the other imports with it. This means spacing them out as if they also used the word qualified. This makes it so the actual names of the modules all line up.
Next, write your list in alphabetical order. This helps people find the right module in the list. Finally, also try to line up the as statements and specific function imports as best you can. This way, it’s easy for people to see what different module prefixes you're using. This is another feature you can get from Haskell text editor plugins.
import qualified Data.ByteString (ByteString)
import qualified Data.Map as M
import Data.Maybe as MaySummary
Organizing your imports is key to making your code accessible to others developers! Make it clear where functions come from by. You can do this in two ways. You can either qualify your imports or specify which functions you use from a module. Separate library imports from your own code imports. This let's people know if they need to look online or in the codebase for the module. Make the imports list itself easy to read by alphabetizing it and lining all the names up.
Stay tuned for next week! We’ll take a tour through all the different string types in Haskell! This is another topic that ought to be simple but has many pitfalls, especially for beginners! We'll focus on the different ways to convert between them.
If you’ve never a line of Haskell before don’t despair! You can check out our Getting Started Checklist. It will walk you through downloading Haskell and point you in the direction of some good learning resources.
If you’ve done a little bit of Haskell before but want some practice on the fundamentals, you should take a look at our Recursion Workbook. It has two chapters of content followed by 10 practice problems for you to look at!
Putting Your Haskell to the Test!
How many times have you encountered a regression bug in your production code? This can be one of the most demoralizing experiences for a software engineer. You shipped code and were confident it worked. And now it turns out it broke something else. The best way to avoid these bugs is to have tests that check your code for these cases. So how does testing work in Haskell?
In Haskell, we have a mantra that if your code compiles it ought to work. This is might be more true in Haskell than in other languages. But it’s still a tongue-in-cheek comment that doesn't quite pass muster. There are often different ways to accomplish the same goal in Haskell. But we should strive to write in ways that make it more likely the compiler will catch our errors. For instance, we could use newtypes as an alternative to type synonyms to limit errors.
At a certain point though, you have to start writing tests if you want to be confident about your code. Luckily, as a pure functional language, Haskell has some advantages in testing. In certain ways, it is far easier and more natural to test than, say, object oriented languages. Its functional features allow us to take more confidence from testing Haskell. Let’s examine why.
Functional Testing Advantages
Testing works best when we are testing specific functions. We pass input, we get output, and we expect the output to match our expectations. In Haskell, this is a approach is a natural fit. Functions are first class citizens. And our programs are largely defined by the composition of functions. Thus our code is by default broken down into our testable units.
Compare this to an object oriented language, like Java. We can test the static methods of a class easily enough. These often aren't so different from pure functions. But now consider calling a method on an object, especially a void method. Since the method has no return value, its effects are all internal. And often, we will have no way of checking the internal effects, since the fields could be private.
We'll also likely want to try checking certain edge cases. But this might involve constructing objects with arbitrary state. Again, we'll run into difficulties with private fields.
In Haskell, all our functions have return values, rather than depending on effects. This makes it easy for us to check their true results. Pure functions also give us another big win. Our functions generally have no side effects and do not depend on global state. Thus we don't have to worry about as many pathological cases that could impact our system.
Test Driven Development
So now that we know why we’re somewhat confident about our testing, let’s explore the process of writing tests. The first step is to defined the public API for a particular module. To do this, we define a particular function we’re going to expose, and the types that it will take as input as output. Then we can stub it out as undefined, as suggested in this article on Compile Driven Learning. This makes it so that our code that calls it will still compile.
Now the great temptation for much all developers is to jump in and write the function. After all, it’s a new function, and you should be excited about it!
But you’ll be much better off in the long run if you first take the time to define your test cases. You should first define specific sets of inputs to your function. Then you should match those with the expected output of those parameters. We’ll go over the details of this in the next section. Then you’ll write your tests in the test suite, and you should be able to compile and run the tests. Since your function is still undefined, they'll all fail. But now you can implement the function incrementally.
Your next goal is to get the function to run to completion. Whenever you find a value you aren't sure how to fill in, try to come up with a base value. Once it runs to completion, the tests will tell you about incorrect values, instead of errors. Then you can gradually get more and more things right. Perhaps some of your tests will check out, but you missed a particular corner case. The tests will let you know about it.
HUnit
One way of going about testing your code is to use QuickCheck. This approach is more focused on abstract properties than on concrete examples. We go through a few examples with this library in this article on Monad Laws. In this article we’ll test some code using the HUnit library combined with the Tasty testing framework.
Suppose to start out, we’re writing a function that will take three inputs. It should multiply the first two, and subtract the third. We’ll start out by making it undefined:
simpleMathFunction :: Int -> Int -> Int -> Int
simpleMathFunction a b c = undefinedWe’ll take out combinations of input and output and make them into test cases like this:
simpleMathTests :: TestTree
simpleMathTests = testGroup "Simple Math Tests"
[ testCase "Small Numbers" .
simpleMathFunction 3 4 5 @?= 7
, testCase "Bigger Numbers" .
simpleMathFunction 22 12 64 @?= 20
]We start by defining a group of tests with a broader description. Then we make individual test cases that each have their own name for themselves. Then in each of these we use the @?= operator to check that the actual value is equal to the expected value. Make sure you get the order right, putting the actual value first. Otherwise you'll see confusing output. Then we can run this within a test suite and we’ll get the following information:
Simple Math Tests
Small Numbers: FAIL
Exception: Prelude.undefined
Bigger Numbers: FAIL
Exception: Prelude.undefinedSo as expected, our test cases fail, so we know how we can go about improving our code. So let’s implement this function:
simpleMathFunction :: Int -> Int -> Int -> Int
simpleMathFunction a b c = a * b - cAnd now everything succeeds!
Simple Math Tests
Small Numbers: OK
Bigger Numbers: OK
All 2 tests passed (0.00s)Behavior Driven Development
As you work on bigger projects, you’ll find you aren’t just interacting with other engineers on your team. There are often less technical stakeholders like project managers and QA testers. These folks are less interested in the inner working of the code, but still concerned with the broader behavior of the code. In these cases, you may want to adopt “behavior driven development.” This is like test driven development, but with a different flavor. In this framework, you describe your code and its expected effects via a set of behaviors. Ideally, these are abstract enough that less technical people can understand them.
You as the engineer then want to be able to translate these behaviors into code. Luckily, Haskell is an immensely expressive language. You can often define your functions in such a way that they can almost read like English.
Hspec
In Haskell, you can implement behavior driven development with the Hspec library. With this library, you describe your functions in a particularly expressive way. All your test specifications will belong to a Spec monad.
In this monad, you can use composable functions to describe the test cases. You will generally begin a description of a test case with the “describe” function. This takes a string describing the general overview of the test case.
simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
...You can then modify it by adding a different “context” for each individual case. The context function also takes a string. However, the idiomatic usage of context is that your string should begin with the words “when” or “with”.
simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
context "when the numbers are small" $
...
context "when the numbers are big" $
...Now you’ll describe each the actual test cases. You’ll use the function “it”, and then a comparison. The combinators in the Hspec framework are functions with descriptive names like shouldBe. So your case will start with a sentence-like description and context of the case. The the case finishes “it should have a certain result": x “should be” y. Here’s what it looks like in practice:
main :: IO ()
main = hspec simpleMathSpec
simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
context "when the numbers are small" $
it "Should match the our expected value" $
simpleMathFunction 3 4 5 `shouldBe` 7
context "when the numbers are big" $
it "Should match the our expected value" $
simpleMathFunction 22 12 64 `shouldBe` 200It’s also possible to omit the context completely:
simpleMathSpec :: Spec
simpleMathSpec = describe "Tests of our simple math function" $ do
it "Should match the our expected value" $
simpleMathFunction 3 4 5 `shouldBe` 7
it "Should match the our expected value" $
simpleMathFunction 22 12 64 `shouldBe` 200At the end, you’ll get neatly formatted output with descriptions of the different test cases. By writing expressive function names and adding your own combinators, you can make your test code even more self documenting.
Tests of our simple math function
when the numbers are small
Should match the our expected value
when the numbers are big
Should match the our expected value
Finished in 0.0002 seconds
2 examples, 0 failuresConclusion
So this concludes our overview of testing in Haskell. We went through a brief description of the general practices of test-driven development. We saw why it’s even more powerful in a functional, typed language like Haskell. We went over some of the basic testing mechanisms you’ll find in the HUnit library. We then described the process of "behavior driven development", and how it differs from normal TDD. We concluded by showing how the HSpec library brings BDD to life in Haskell.
If you want to see TDD in action and learn about a cool functional paradigm along the way, you should check out our Recursion Workbook. It has 10 practice problems complete with tests, so you can walk through the process of incrementally improving your code and finally seeing the tests pass!
If you’ve never programmed in Haskell before and want to see what all the rage is about, you should download our Getting Started Checklist. It’ll walk you through some of the basics of downloading and installing Haskell and show you a few other tools to help you along your way!
A Project in Haskell: One Week Apps
Lately we've focused on some of the finer points of how to learn Haskell. But at a certain point we want to actually build things. This next series of articles will focus on some of the more practical aspects of Haskell. This week, I'm going to introduce one of my own Haskell projects. We’ll first examine some of the reasons I chose Haskell. Then we’ll look at how I’ve organized the project and some of the pros and cons of those choices.
Introduction to One Week Apps
The program I've created is called One Week Apps (Names have never been my strong suit...I'm open to better ideas). It is designed to help you rapidly create mobile prototypes. It allows you to specify important elements of your app in a simple domain specific language. As of current writing, the supported app features are:
- Colors
- Fonts
- Alert popups
- Programmer errors (think NSError)
- Localized strings
- View layouts
- Simple model objects.
As an example, suppose you wanted to create an app from scratch and add a simple login screen. You’ll start by using the owa new command to generate the XCode project itself. First you'll enter some information about the app though a command line prompt. Then it will generate the project as well as some directories for you to place your code.
>> owa new
Creating new OWA project!
What is the name of your app:
>> Blog Example
What is the prefix of your app (3 letters):
>> BEX
What is the author's name:
>> James
What is your company name (optional):
>> Monday Morning Haskell
Your new app "Blog Example" has been created!We then start specifying the different elements of our app. To represent colors, we'll put the following code in a .colors file in the generated app directory.
Color labelColor
Hex 0x000000
Color fieldColor
Hex 0xAAAAAA
Color buttonColor
Blue 255
Red 0
Green 0We can also specify some different fonts for the elements on your screen:
Font labelFont
FontFamily HelveticaNeue
Size 12.0
Styles Light
Font fieldFont
FontFamily Helvetica
Size 16
Font buttonFont
FontFamily Arial
Size 24
Styles BoldNow we'll add some localization to the strings:
NAME_TITLE = “Name”
“NAME_PLACEHOLDER” = “Enter Your Name”
PASSWORD_TITLE = “Password”
“PASSWORD_PLACEHOLDER” = “Enter Your Password”
SUBMIT_BUTTON_TITLE = “Log In!”Finally, we'll specify the view layout itself. We can use the colors and fonts we wrote above. We can also modify the layout of the different elements.
View loginView
Type BEXLoginView
Elements
Label nameLabel
Text “NAME_TITLE”
TextColor labelColor
Font labelFont
Layout
AlignTop 40
AlignLeft 40
Height 30
Width 100
TextField nameField
PlaceholderText “NAME_PLACEHOLDER”
PlaceholderTextColor fieldColor
PlaceholderFont fieldFont
Layout
Below nameLabel 20
AlignLeft nameLabel
Height 30
Width 300
Label passwordLabel
Text “PASSWORD_TITLE”
TextColor labelColor
Font labelFont
Layout
Below nameField 40
AlignLeft nameLabel
Height 30
Width 100
TextField passwordField
PlaceholderText “PASSWORD_PLACEHOLDER”
PlaceholderTextColor fieldColor
PlaceholderFont fieldFont
Layout
Below passwordLabel 40
AlignLeft nameLabel
Height 30
Width 300
Button submitButton
Text “SUBMIT_BUTTON_TITLE”
TextColor buttonColor
Font buttonFont
Layout
Below passwordField 40
CenterX
Height 45
Width 200Then we'll run the owa gen command to generate the files. We need to add them manually to your XCode project (for now), but they should at least appear in the proper folder. Finally, we'll add a few simple lines of connecting code in the view controller:
class ViewController: UIViewController {
override func loadView() {
view = BEXLoginView()
}
...And we can take a look at our basic view:
A basic login screen
Rationale For Haskell
When I first came up with the idea for this project, I knew Haskell was a good choice. One major feature we should notice is the simple structure of the program. It has limited IO boundaries. We read in a bunch of files at the start, and then write to a bunch of files at the very end. In between, all we have is the complicated business logic. We parse file contents and turn them into algebraic data structures. Then we perform more transformations on those depending on the chosen language (you can currently use Objective C or Swift).
There is little in the way of global state to track (at least now when there's no "compiling" occurring). There are no database connections whatsoever. Many of the things that can make Haskell clunky to deal with (IO stuff and global state) aren’t present in the main body of the program.
On the contrary, the body consists of computations playing to Haskell’s strengths. These include string processing, representing data abstractly, an so on. These were the main criteria for choosing Haskell for this project. Of course, there are libraries to deal with all the “clunky” issues I mentioned earlier. But we don’t even need to learn those for this project.
Architecture
Now I’ll give a basic overview of how I architected this program. I decided to go with a multi-package approach. We can see the different packages here in my stack.yaml file:
packages:
- './owa-core'
- './owa-model'
- './owa-utils'
- './owa-parse'
- './owa-objc'
- './owa-swift'- The
modelsection contains the datatypes for the objects of the mobile app. Other packages rely on this. - The
parsepackage contains all code and tests related to parsing files. - The
objcpackage contains all code creating Objective C files and printing them out. It also has tests for these features. - The
swiftpackage does the same for Swift code. - The
corepackage handles functionality like the CLI, interpreting the commands, searching for files, and so on. - The
utilspackage contains certain extra code needed by multiple packages.
Pro and Cons
Let’s look at some of the advantages and disadvantages of this architecture. As an alternative, we could have used a single-package structure. One advantage of the chosen approach is shorter compile time within the development cycle. There is a tradeoff with total compile time. Having many packages lead to more linking between libraries, which takes a while. However, once you've compiled the project once, each successive re-compile should take much less time. You’ll only be re-compiling the module you happen to be working on in most cases. This leads to a faster development cycle. In this case, the development cycle is more important than the overall compile time. If the program needed to be compiled from scratch on a deployment machine with some regularity, we might make a different choice.
The organization of the code is another important factor. It is now very obvious where you’ll want to add parsing code. If a feature seems broken, you know where to go to find a failing test or add in a new test to reproduce the bug. This system works far better than the haphazard version-based test organization I had earlier.
Another advantage I would list is it’s now a cleaner process to add another language to support. To support a new language, there will be few changes necessary to the core module. You’ll add another package (say owa-android) and add a few more options to Core. This should make it an easy repository to fork for, say, anyone who wanted to make an android version.
Let’s also consider some of the disadvantages. It is unlikely many of these packages will be used in total isolation from one another. The owa-parse module firmly depends on the owa-model package, for instance. The language specific modules will not interact with each other. But they're also not particularly useful unless you're using the parser anyways.
Additionally, the utils module is a real eyesore. It has a few random utilities needed by the testing code for several packages as well as the printing code. It seems to suggest there should only be one package for testing, but I find this to be unsatisfactory. It suggests instead there should be common printing code. It may be a reasonable solution to simply leave this code duplicated in certain places. This is another code-smell. But if different languages evolve separately, then their utility code might also.
Summary
So what did we learn about Haskell from this project? Haskell is great at internal processing. But programs with wide IO bounds can cause some headaches. Luckily, One Week Apps has lots of internal processing, but limited IO bounds. So Haskell was a natural choice. Meanwhile, multi-package organization has some advantages in terms of code organization. But it can also lead to some headaches as far as placing common code.
One Week Apps is now open source, so feel free to check it out on Github! Do you love the idea and think it would supercharge your app-writing? You should contact me and check out our issues page!
Want to contribute but have never touched Haskell? You're in luck, because we've got a couple great resources for getting started! First, you should check out our Getting Started Checklist. It'll walk you through some of the basic steps for installing Haskell. It'll also show you some awesome resources to kickstart your Haskell code writing.
If you have a little experience but want more practice, you should download our Recursion Workbook. It'll teach you about recursion, a fundamental functional paradigm. It also has some practice problems to get you learning!
Haskell and Deliberate Practice
Have you ever been in a situation where you tried learning something, put in the proper time, and got stuck? Chances are you weren't learning in the best way. But how can you know what constitutes good learning in your field? It can be frustrating to try to find advice for how to learn a specific topic on the internet. Most people do not think about the manner in which they learn. They do learn, but they can’t externalize and teach other people what they did, so there aren't guides for this sort of thing.
In our first article on learning, we focused on some higher level learning techniques. The aim of these was to get ourselves more learning time and motivation. But even if your macro-techniques are good, you ultimately need domain specific practice. The second article in this series showed one specific technique for learning Haskell. It's a start, but I still want to share some more general ideas.
This article will go over a couple more key ideas from The Art of Learning by Josh Waitzkin. The first concept we’ll talk about is the idea of deliberate practice. The goal of deliberate practice is to zero in on a specific idea and try to improve a certain skill until it become subconscious. The second concept is the role of mistakes in learning any new skill. We'll use these to propel ourselves forward and prevent ourselves from tripping over the same ideas in the future.
Deliberate Practice
Suppose for a moment you’re learning to play a particular piece of music on the piano (or any instrument). The biggest temptation out there is to “learn” the piece by repeatedly playing it from start to finish. You’ll get a fair amount right, and you’ll get a fair amount wrong. Eventually, you’ll get most of it right. This is a tempting method of practice for a few different reasons:
- It’s the “obvious” choice.
- It let’s us do the parts we already enjoy and are good at, which feels good.
- It is, after all, most like what we’ll actually end up doing in a performance.
However, it’s a suboptimal method from a learning perspective. If you want to improve your ability to play the piece from start to finish, you should focus on your weakest areas. You have to find the specific passages that you are struggling with. Once you’ve determined those, you can break them down even further. You can find specific measures or even notes that you have difficulty with. You should practice these weaknesses over and over again, fixing one small thing at a time. At a certain point, you’ll need to go all the way through, but this should wait until you’re confident on all your weak spots.
The focus on small things is the most important part. You can’t take a tricky passage you know nothing about and play it perfectly from start to finish. You might start with one section that forces you make a quick hand movement. You might practice a dozen times just focusing on getting the last note and then moving your hand. Nothing else is important for these dozen repetitions. Once you’ve made the hand movement subconscious, you can move onto another idea. The next step might be to make sure you hit the first three notes after the hand movement.
This sums up the idea of deliberate practice. We focus on one thing at a time, and practice that one thing deliberately. Mindless practice, or practice for the sake of practice will only give us slow progress. It may even impede our progress if we build up bad habits. We can apply it to any skill out there, including coding. We want to build up tiny habits that will gradually make us better.
Mistakes
So deliberate practice is a system of building up skills that we want. However, there are also plenty of habits we don’t want. We do many things that we realize later are errors. And the worst thing is when we realize we’ve made the same error over and over again!
Waitzkin notes in Art of Learning, “If a student of any disciple could avoid ever repeating the same mistake twice, they would skyrocket to the top of their field.” We could also do this if we avoid mistakes entirely, but this isn’t possible. We’ll always make mistakes the first time we try something.
So first we have to embrace the certainty that mistakes will happen. Once we’ve done this, we can have a plan for dealing with them. We won’t be able to avoid ever repeating mistakes, but we can take steps that will reduce the rate at which we do. And if we’re able to do this, we’ll see major improvement. Our solution will ultimately be to keep a record of the mistakes we make. By writing things, down, we'll dramatically reduce the repetition of errors.
Practicing Haskell
So now we need to step back into the land of coding and ask ourselves how we can apply these ideas to Haskell. What specific areas of our coding practices can we focus on? Let’s start with a couple lessons from Compile Driven Learning.
You could write an application, and focus on nothing but building the following habit: before you write a function, stub it out as undefined and make sure the type signature compiles. It doesn't matter if you do anything else right! After you've gotten into that habit, you could also take another step. You could make sure you always write the function’s invocation (where your other code calls the function) before you implement it.
I’ve chosen these examples because there are two things that slow us the most when writing functions. The first is a lack of clarity because we don’t know exactly how our code will use the function. The second is the repetition of work when we realize we need to re-write the function. This can happen because there was an extra type we didn’t account for or something like that. These habits are designed to get you to plan more up front so that your life is simpler when it comes to implementation.
Here are a couple of other ideas in a similar vein:
- Before writing a function, write a comment describing that function.
- Before you use expressions from a library, add it to your .cabal file. Then, write the import statement to make sure you are using the right dependency.
Another great practice is to know how you will test a piece of functionality before you implement it. In a couple weeks we'll explore test driven development, where we'll write test cases for our functions. But if it’s a very simple feature, like getting a line of input and parsing it in some way, you can get away with simpler ideas. You could commit to running your program on the command line with a couple types of input, for instance. As long as you know your approach before you start coding, it counts. It might be most helpful to write the test plan in some document first.
So in almost all the above cases, the “trigger” for building this habit is writing a new function. The trigger is the most important part of building a new habit. It is the action that tips your brain off that you should be doing something you’re not accustomed to doing. In this case, the trigger could be writing the :: for the type signature. Every time you do this, remind yourself of your current goal.
Here’s an idea with a different trigger. Every time you pick a structure to contain your data (list, sequence, set, map, etc.), brainstorm at least three alternatives. Once you get beyond the basics, you’ll find each structure has its unique strengths. It would be most helpful if you wrote down your reasoning for your final choice. The trigger in this case could be every time you write the data keyword. For a more extreme version, the trigger could be writing down the left bracket to start a list. Each time you do this, ask yourself if you could be using a different structure.
Here's one final possibility. Every time you make a type synonym, ask yourself if you would be better served making a newtype out of it instead. This often leads to better compile time behaviors. You'll likely see clearer error messages, and you'll catch more of your errors at compile time. The trigger here is also simple: any time you type the keyword type.
Here’s the most important thing though. Don’t try more than one of these at a time! You want to pick one thing, practice it until it becomes subconscious, and then move on to other things. This is the hardest part about deliberate practice: maintaining your patience. The biggest temptation is to move on and try new things often before the good habits are solidified. Once you shift your focus onto other items, you might lose whatever it was you were working on! Remember to treat learning like compound interest! You need to make small investments that stack up over a long period time. You can't necessarily hurry the process.
Tracking Mistakes
Let's also consider the various ways we can avoid making the same mistakes in the future. Again, these are different from the “skills” you build up with deliberate practice. They don't occur much, and you don’t want to “practice” them. You just want to remember how you fixed some issue so you can solve it again if it does come up. You should keep a list in a google doc of all the worst mistakes you’ve encountered in your programming. The google doc should record three things for each mistake.
- What was the compiler message or runtime behavior?
- What was the problem with your code?
- How did you fix it?
So for an example, think about a time you were certain your code was correct. You look at the error, then back to your code, then back to the error. And you're still sure your code is right. Of course, the compiler is (almost) always right. You want to document these so they don't trip you up again.
Other good candidates are those runtime errors where you cannot for the life of you track down where the error even occurred in your code. You’ll want to write down what this experience was like so the next time it happens, you’ll be able to fix it quickly. By writing about it, you’ll also motivate yourself to avoid it as well.
Then there are also dumb mistakes that you should record because it’ll teach you the right way faster. Like when you’re starting out you might use the (+) operator to try to append two strings instead of the (++) operator. By writing down errors like this, you'll learn quirky language features much faster.
One final group of things you should track down is awesome solutions. Not just bug fixes, but solutions to your central programming problems. For instance, you found your program was too slow, but you used a better data structure to improve it. Not only does it feel good to write about things that went well, you’ll have a record of what you did. That way, you’ll be able to apply it next time as well. These kinds of items (both the good and the bad) make good fodder for technical interviews. Interviewers are often keen to see what kinds of challenges you’ve overcome to show your growth potential as an engineer.
I have one example that demonstrates the good and bad of recording mistakes. I had a nasty bug when I was trying to build my Haskell project using Cabal. I remember it being a linker error that didn’t point to any particular file. I did a good job in making a mental note that the solution was to add something to the “.cabal” file. But I didn’t write down the full context or full solution. So in the future, I'll see a linker error and know I have to do something in the “.cabal” file, but I won’t be sure exactly what. So I’ll still be more likely to repeat this error than I would if I had written down the full resolution.
Summary
It’s an oft-repeated mantra that practice makes perfect. But as anyone who’s mastered a skill can tell you, only good practice makes perfect. Bad or mindless practice will leave you stuck. Or worse, it will ingrain poor habits that will take more time to undo. Deliberate practice is the process of solidifying knowledge by building up tiny habits. You pick one thing to focus on, and ignore everything else. Then you learn that focus until it has become subconscious. Only then do you move on to learning other things. This approach requires a great deal of patience.
One final thing we have to understand about learning is the need to embrace the possibility that we will make mistakes. Once we have done this, we can make a plan for recording those mistakes. This way, we can learn from them and not repeat them. This will dramatically improve our pace of development.
If you want start out on your journey of deliberate practice, you should download our Recursion Workbook! In addition to some content on recursion, it contains 10 practice problems. The answer start from undefined so you can build up your solutions step-by-step. It's a great way to learn deliberate practice ideas!
If you’ve never written a line of Haskell before, don’t be afraid! You should check out our Getting Started Checklist. It will walk you through installing Haskell and give you some helpful tools for starting on your Haskell journey!
BayHac 2017!
I spent most of this last weekend at BayHac, the Bay Area Haskell Hackathon! It was hosted at Takt headquarters in San Francisco. I had been looking forward to the event for a long time and I was not disappointed. I had a blast and met a ton of smart, interesting people. I also got to speak at an event for the first time, which was a lot of fun.
The event had two primary sections of activity. First, there was a packed schedule of speakers talking about a wide array of topics. The talks ranged from some basic Stack usage all the way to some eye-opening category theory. Second, several open source maintainers coordinated some hacking on their projects.
Now, given the event was a “Hackathon”, I should perhaps have focused more on this second part than I did. That said, the litany of fascinating topics for the talks made this difficult. I managed a little bit in the end, but I'll start by going through my observations on the talks. A few major themes stood out. Many of my articles and posts try to highlight some of these themes. But it helps enormously to see concrete examples of these.
Language Safety
The first theme was Haskell’s safety as a language. The first talk of the event was given by Greg Horn, who discussed his work at Kittyhawk. He has been developing a flight-control system for small aircraft in Haskell. There isn’t a large margin for error when dealing with objects flying high above the ground. One small bug involving any kind of crash could be extraordinarily expensive. This is a large part of his motivation in using Haskell.
As I stress so often about Haskell, he is confident he can write and refactor Haskell code without breaking anything. Further there are even compile time guarantees that OTHER people can’t break things. His code also involved compiling Haskell code to C to run on embedded systems. But he managed to make most of his code pure. So the generated C code was a subset that excluded a lot of unsafe activities.
Needless to say, it was cool to see Haskell used on such a cool application. It’s even cooler to know that it’s not just being used for the sake of using Haskell. There are real language level concepts that distinguish Haskell and make it more suitable for certain tasks.
Expressiveness and Simplicity
Another talk came from Tikhon Jelvis. He discussed his work optimizing Target’s supply chain. His work is notable for tackling some fascinating mathematical problems involving non-deterministic probability. He managed to express complex probabilistic math within a monad. The simplicity of the final code was mind-boggling.
But another more prosaic issue also struck me as a real showcase of the power of Haskell's simpler ideas. Target’s database has a curious identification schema on their items. One object could potentially have three different identifiers. In dynamically typed languages, you'd likely be stuck using strings for them all. It can be a real pain to keep the different identifiers straight. You're likely to use the wrong one at some point. Tracking down bugs can be nigh impossible. Even if you make wrapper objects, you still likely won’t get compile time guarantees. And it’s quite possible a novice developer will be able to slip in incorrect usages.
Haskell avoids this with newtypes. We can simply wrap each type of ID in a different type. This gives us compile time guarantees that we are not comparing different ID types. This example strengthened my firm belief that even Haskell's simple concepts can give you some big advantages.
The Rise of Frontend Haskell
Takt’s trio of Luite Stegeman, Greg Hale, and Doug Beardsley (previously featured on Monday Morning Haskell!) all gave interesting talks on various topics related to front-end web development in Haskell. Luite discussed some of the intricacies of GHCJS. Doug shared some problems he had encountered over the course of using Reflex FRP, and how he’d solved them. I didn’t get a chance to catch much of Greg’s talk (though they’ll all be up online soon). But I gather from the title it involved showing how nice and easy our life can be when we use Haskell full-stack and get to use the same types everywhere!
Full-stack Haskell applications have existed for a while. But I'm very encouraged by the progress I see in the front-end community, especially with FRP. There seem to be a lot of new ideas and solutions out there. And they're constantly getting better and better. My fingers are crossed that Haskell could soon be widely considered a strong competitor as a full-stack web language.
Breaking Down the Complex
Haskell has a lot of complicated ideas. But several speakers did a great job breaking them down and showing concrete examples. There were a couple talks that accomplished this for some category theory concepts. Julie Moronuki (co-author of the Haskell Book) discussed monoids, semi-groups, and their connections to boolean algebra. Rúnar Bjarnason (co-author of this Scala Book) went into some details on adjunctions and showed how common they are.
Meanwhile Jon Wiegley and Sandy Maguire tackled a couple more concepts that are pretty high on my TODO list for learning: lenses and free monads. I’ve definitely tried learning about lenses before. But I still have little competency beyond playing “monkey see, monkey do” and copying from other examples in the codebase. But now it might have finally clicked. We’ll see. I’m a firm believer in the Feynman Technique. So you can probably expect to see a blog post on lenses from me sometime soon.
Baby’s First Talk!
Besides all these crazy smart people, BayHac also let me give a talk! I gave an introduction to the fantastic Servant library. In all the languages I have programmed in, it is the best tool I have come across for building or even calling an API. You start by describing your API as a type. Then you write your server side handlers. Once you've done this, Servant can generate a lot of other code for you. You'll get things like client side functions and documentation almost ready made. I’ll link to the video of the presentation once it’s up. But in the meantime you can check get the slides and code samples I used for the presentation. They should give you a decent taste of the Servant library so you can get started!
Tensor-Flow and Some Hacking
In the midst of all this I sort’ve managed to do something approaching “hacking”. Of the projects listed before hand, the one that stood out to me most was Haskell Tensor Flow. I've been focusing a lot on the Haskell language lately. This means I haven’t gotten much of a chance to focus on some other skills, particularly machine learning and AI. So I was super psyched at getting to learn more about this library, which allows you to use Google’s awesome Tensor Flow API from Haskell. I didn’t manage to hack much of anything together. I mostly spent my time wrapping my head around the Tensor Flow concepts and the Haskell types involved. Judah Jacobson’s presentation helped a lot in this regard. I’m hoping to learn enough about the library to start making contributions in the future.
Conclusion
So to summarize, BayHac was a fantastic event. On the plus side I learned a ton about different areas of Haskell. Meeting more people in the Haskell community was a great experience. I look forward to having more opportunities to share knowledge with others in the future. The only minus is that I now have way too many things I want to learn and not enough time. I can cope with a language that puts me in that position.
The last big takeaway from BayHac was how open and inviting the community is to newcomers. If you’ve never written a line of Haskell before, you should check out our Getting Started Checklist. It’ll walk you through installing the language and point you to some basic tools to help you on your way!
Compile Driven Learning
Imagine this. You've made awesome progress on your pet project. You need to add one more component to pull everything together. You bring in an outside library to help with this component and you...get stuck. You wonder how you're supposed to get started. You take a gander at the documentation for the library. It's not particularly helpful.
While documentation for Haskell libraries isn't always great, there is a saving grace. As we’ve explored before, Haskell is strictly typed. Generally, when it compiles, it works the way we expect it to. At least this is more common in Haskell than other languages. This can be a double-edged sword when it comes to learning new libraries.
On the one hand, if you can cobble together the correct types for functions, you're well on your way to success. However, if you don't know much about the types in the library, it's hard to know where to start. What do you do if you don't know how to construct anything of the right type? You can try to write a lot of code, but then you’ll get a mountain of error messages. Since you aren’t familiar with the types, they’ll be difficult to decipher.
In this article, I'll share my approach to solving this learning problem. I refer to it as "Compile Driven Learning". To learn a new library or system, you should start out by writing as little code as you can to make the code continue to compile. It is intricately related to the ideas of test driven development. We'll go over this idea in more detail in next week's article, but here's the 10000 ft. overview.
Test Driven Development
Test driven development is a paradigm of software development where you write your tests before writing your source code. You consider the effects you want the code to have, and what the exposed functions should be. Then you write tests establishing expectations about the exposed functions. You only write the source code for a feature once you’re satisfied with the scope of your tests.
Once you’ve done this, the test results drive the development. You don’t have to spend too much time figuring out what piece of code you should implement. You find the first failing test case, make it pass, rinse and repeat. You want to write as little code as you can to make the test pass. Obviously, you shouldn't just be hard-coding function definitions to fit the tests. Your test cases should be robust enough that this is impossible.
Now, if you’re trying to write as little code as possible to make the tests pass, you might end up with disorganized code. This would not be good. The main idea in TDD to combat this is the Red-Green-Refactor cycle. First you write tests, which fail (red). Then you make the tests pass (green). Then you refactor your code to make it live up to whatever style standards you are using (refactor). When you've finished this, you move on to the next piece of functionality.
Compile Driven Learning
TDD is great, but we can’t necessarily apply it to learning a new library. If you don’t know the types, you can’t write good tests. So you can use this process instead. In a way, we’re using the type system and the compiler as a test of our understanding of the code. We can use this knowledge to try to keep our code compiling as much as possible to accomplish two goals:
- Drive our development and know exactly what we’re intending to implement next.
- Avoid the discouraging “mountain of errors” effect.
The approach looks like this:
- Define the function you’re implementing, and then stub it out as
undefined. (Your code should still compile.) - Make the smallest progress you can in defining the function so the code still compiles.
- Determine the next piece of code to write, whether it is an
undefinedvalue you need to fill in, or a stubbed portion of a constructor for an object. - Repeat 2-3.
Notice at the end of every step of this process, we should still have compiling code. The undefined value is a wonderful tool here. It is a value in Haskell which can take on any type, so you can stub out any function or value with it. The key is to be able to see the next layer of implementation.
CDL In Practice
Here’s an example of running through this process from "One Week Apps", one of my side projects. First I defined an function I wanted to write:
swiftFileFromView :: OWAAppInfo -> OWAView -> SwiftFile
swiftFileFromView = undefinedThis function says we want to be able to take an “App Info” object about our Swift application, as well as a View object, and generate a Swift file for the view. Now we have to determine the next step. We want our code to compile while still making progress toward solving the problem. The SwiftFile type is a wrapper around a list of FileSection items. So we are able to do this:
swiftFileFromView :: OWAAppInfo -> OWAView -> SwiftFile
swiftFileFromView _ _ = SwiftFile []This still compiles! Admittedly, it is quite incomplete! But we’ve made a tiny step in the right direction.
For the next step, we have to determine what FileSection objects go into the list. In this case we want three different sections. First we have the comments section at the top. Second, we have an “imports” section. Then we have the main implementation section. So we can put expressions for these in the list, and then stub them out below:
swiftFileFromView :: OWAAppInfo -> OWAView -> SwiftFile
swiftFileFromView _ _ = SwiftFile [commentSection, importsSection, classSection]
where
commentSection = undefined
importsSection = undefined
classSection = undefinedThis code still compiles. Now we can fill in the sections one-by-one instead of burdening ourselves with writing all the code at once. Each will have its own component parts, which we’ll break down further.
Using our knowledge of the FileSection type, we can use the BlockCommentSection constructor. This just takes a list of strings. Likewise, we’ll use the ImportsSection constructor for the imports section. It also takes a list. So we can make progress like so:
swiftFileFromView :: OWAAppInfo -> OWAView -> SwiftFile
swiftFileFromView _ _ = SwiftFile [commentSection, importsSection, classSection]
where
commentSection = BlockCommentSection []
importsSection = ImportsSection []
classSection = undefinedSo once again, our code still compiles, and we’ve made small progress. Now we’ll determine what strings we need for the comments section, and add those. Then we can add the Import objects for the imports section. If we screw up, we’ll see a single error message and we’ll know exactly where the issue is. This makes for a much faster development process.
Summary
We talked about this approach for learning new libraries, but it’s great for normal development as well! Avoid the temptation to dive in and write several hundred lines of code! You’ll regret it when dealing with dozens of error messages! Slow and steady truly does win the race here. You’ll get your work done much faster if you break it down piece by piece, and use the compiler to sanity check your work.
If you want to take a stab at implementing Compile Driven Learning, you should check out our free Recusion Workbook! It has 10 practice problems that start out as undefined. You can try implement them yourself step-by-step and see if you can get the tests to pass!
If you’ve never written any Haskell before and want to try it out, you should read our Getting Started Checklist. It’ll tell you everything you need to know about writing your first lines of Haskell!
Finally, stay tuned for next week when we’ll go into more depth about Test Driven Development. We’ll see how we can achieve a similar effect to CDL by using test cases instead of merely seeing if our code compiles.
Learning to Learn Haskell
A month or two ago, we discussed the Intimidation Factor of Haskell. That article focused on why people see Haskell as challenging and why they shouldn't. Today we’ll pick up on some of the threads of that conversation. We'll explore the how of learning Haskell (and other things). We’ll examine some general ideas on learning and discuss how to apply them to programming.
Warren Buffett and Compound Interest
There’s an oft-repeated line on productivity about Warren Buffett. He says he reads at least 500 pages a day, and this is one of the major keys to his success. Knowledge, according to Buffett, is like compound interest. The more you acquire, the more it accumulates and is able to build on itself as you make more connections.
The why of this statement is fantastic. I’ve found it rings true as I explore different topics. I have seen how my knowledge has begun building on itself. And yet, the how is often misunderstood and misinterpreted. This leads people to have a difficult time implementing Buffett's principle.
The simple fact is that the average person does not have time to read 500 pages a day. First, if he reads so much, Warren Buffett is likely an accomplished speed reader, so he needs less time. Second, he is in far more control of his time than most other people due to his wealth. In my current job as a software engineer, I cannot spend a full 80% of my working day on “reading and thinking”. I would have some very unhappy teammates and project managers to deal with.
The average person will see this advice and decide to start reading a ton outside of working hours. They might even succeed in hitting 500 pages a day...for a couple days. Then of course life gets in the way. They won’t have time over a few days to do it, and they’ll drop the habit.
A Better Application
So how do we achieve the compound knowledge effect? The real misinterpretation I find about this advice is this. The key factor in compound interest is time, not average investment. Making small, repeated contributions will have major rewards later on. Of course, big, repeated contributions will have major rewards as well. But if the investment causes us to give up the habit, we'll be worse off over time.
Once we accept this idea, we can apply it to other topics, including Haskell programing. We might be tempted to devote an hour every day to learning some particular Haskell concept. But this is often unsustainable. It is far easier to devote at least 15 minutes a day, or even 10 minutes a day. This will ensure we’re going to continue learning. On any given day, it can be hard to carve out a full hour for something. Your schedule might not allow that contiguous block of time. But you should always be able to find a 15 minute block. This will dramtically lower the barrier of starting each day, so you'll be more likely to succeed.
In keeping with the compound interest principle, progress is momentum based. By committing to 15 minutes a day on a couple different projects, I’ve made a ton of progress. I've accomplished far more than if I had tried to carve out an hour here and there. I was only able to start writing online because I devoted 20 minutes a day to writing. Once I stuck with that for a month, I was in good shape.
Josh Waitzkin and Confronting Difficulties
Another of the most important ideas about learning I’ve encountered comes from The Art of Learning, by Josh Waitzkin. He is a former chess prodigy and grandmaster turned world-champion martial artist. He describes a story that was all-too familiar to me as a fellow childhood chess player. He saw many young players with a lot of potential. They would beat everyone around them at their school or local chess club. But they never brought themselves to face stronger opposition. As a result, they ended up not improving, and ultimately quit chess. They were so invested in the idea of winning every game that losing to anyone was too much of a blow to their pride.
If we focus on our egos too much, we’ll be afraid of appearing weak. The causes us to avoid confronting the areas of our knowledge where we actually are weak. These are exactly the areas we need to strengthen! If we never address these areas, we’ll never improve, and we won't be able to beat big challenges.
Confronting Haskell
So how does this affect learning Haskell, or programming in general? After all, programming is not a competitive game. And yet, there are still ways in which this mentality can hurt us. We might stay away from a particular topic because it seems difficult. We're concerned that we'll try to learn it and fail. And we worry this failure will reveal how peculiarly unfit we are to be Haskell developers. Worse, we're afraid to ask other programmers for help. What if they look down on us for our lack of knowledge?
I have three main responses to this. First, I'll repeat a note from the intimidation article. A topic is infinitely more intimidating when you know nothing about it. Once you know even the most basic definitions, you have a reasonable idea of what you're missing. Get as basic an idea of it as you can and write it down in plain English. You might not know the subject. But it will no longer be an "unknown-unknown".
Second, who cares if you put in effort toward learning something and fail? Try again! It can take several iterations of learning on a single topic before you understand it. It took me at least three tries before I understood monads.
Finally, the very people we are afraid to admit our weaknesses to are the same people who can actually help us overcome these weakness. Better yet, they are often more than happy to do so! This involves getting over our primal fears of appearing inferior and being rejected by others. This is difficult but not impossible.
Conclusion
So remember the key lessons here. Focus a little bit at first. Don’t commit to learning more than 15 minutes a day, and pick a project with clear progress. Keep momentum going by working at something every day. Don't worry if a concept seems difficult. It's OK to take several tries at learning something. And above all, never be afraid to ask for help.
So what are you waiting for? If you've always wanted to learn Haskell but never have, download our Getting Started Checklist to start your journey!
Have you done a little Haskell, but still don't understand some functional programming concepts? Check out our free Recursion Workbook to learn about recursion, higher order functions, and try some practice problems!
Interview with Doug Beardsley!
This week on Monday Morning Haskell, we have an exciting change of pace! I recently met with Doug Beardsley, a Haskell engineer at Takt. He was kind enough to agree to an interview with me. We discussed a number of topics, including learning Haskell, his work at Takt, and his view of the Haskell community as a whole.
You can find out more about him and his projects on his Github page. One of the more interesting things I saw there was his monad challenges repository. If you’ve read through the Monday Morning Haskell monad series and want some more hands on experience, I encourage you to try these challenges!
Perhaps the most striking response I heard from Doug was his answer about how welcoming and helpful the Haskell community is. Haskellers are always eager to answer newcomers’ questions! So if you haven’t written any Haskell yet, I encourage you to look through our Getting Started Checklist and take the first steps on your Haskell journey!
Without further ado, let’s get on to the interview!
Interview
Let’s start with some information about yourself. How long have been a software engineer? How long have you been using Haskell?
I started programming back in 1990ish, and I was pretty young. It wasn’t really software engineering level programming back then obviously, but I just immediately fell in love with it. I then learned C and programmed with it all through high school. And so I knew I wanted to be a computer science major in college. I went through that, graduated, started programming in the early 2000’s in industry, first in the defense industry for five/six years. Then I got a job programming Haskell. I moved to New York for that, and I’ve been doing Haskell pretty much full time ever since.
How long ago was that?
I started programming in Haskell professionally in 2010. I didn’t jump straight into Haskell professionally from zero. I think I started learning Haskell around 2006, maybe 2007.
So what Haskell concept was the hardest for you to learn?
Wow that’s a good question. I feel like with everything, there’s a spectrum. As you evolve, new things become harder. I definitely had a lot of trouble with monads early on. I feel like you don’t learn them at any one time. You just use them enough, and then you gradually build up a working knowledge of the tools and the APIs that are available around Control.Monad. I read through that so many times. If, like me, you don’t have a photographic memory, you just gotta keep going back through refreshing your memory, and eventually the patterns just kind of sink in.
I don’t know if I would say that was the hardest concept though. I feel like I can still use a much better understanding of say, profunctors, or some of the more obscure things that are going on in the lens package. But I don’t know. I kinda don’t like questions like “What is the superlative of something”, because it depends on the day really.
That makes sense. What advice would you have for someone who has tried to learn Haskell and has either given up, or is having a really tough time? Are there any 80/20 ideas or important easy wins when learning Haskell?
The biggest thing for me is you need to have intrinsic motivation. So in my case, I had an app that I wanted to build, and I had been building it in Drupal, with PHP, and that just fell flat on its face very quickly. I really wanted to build the app, for myself, not for a startup or to make money; I wanted it for myself. So I had a very high level of intrinsic motivation.
I wasn’t really a big fan of web programming back then, I was more of a back-end kind of programmer, and I thought “I want to learn a new language, Haskell seems like one of the more challenging ones that would stretch me more, maybe it’ll make web programming interesting”. As someone once said on IRC, “a spoonful of Haskell makes the web programming go down”. Which I think is a very apt description of my case, so I just dove in. I started working on it. It was a way bigger project than I should have been working on as a Haskell novice.
But I think that was good. You need to be working on something you think is maybe a little bit beyond you. And if you also have that intrinsic motivation taken care of, then it almost solves itself. You’re going to bang your head against a wall until you beat it. I’ve heard a friend of mine is really big into natural language learning and says you just have to find some way of getting over this incredible hurdle because it’s so hard. You just have to make yourself do it. For Haskell I think it’s a very similarly challenging endeavor, and you really have to have intrinsic motivation.
So the WHY is almost more important than any particular WHAT?
Yea I think so. I just started out with Project Euler problems, which is something I hear a lot of people do. And OK great, it’s like “Oh, I’ll spend some time working on this problem.” You spend an hour on it, and you don’t learn a whole lot of Haskell. You learn a little bit of the basic syntax, but not a lot that’s useful in terms of real world development. And there’s not a whole lot of intrinsic motivation there. I suppose maybe you could say, “Oh I want to finish all the problems.” But that’s not really a question of learning Haskell. That’s more a question of learning math than any particular programming language.
And so I think you need a project that’s going to push you into a fair number of realistic, real world situations. In my case, a web app was suitable. Web apps do pretty well because they involve a fair number of pieces of technology and teach you the software development process. But yea, it’s not so much the “what”, for sure. It’s just, you gotta have a reason that lights a fire under you.
I guess my own experience is a little bit like that. I had this product idea and I had done simple versions of a couple of the pieces of that idea in college in a class where I first learned Haskell. And I thought, well I could just use Haskell for this whole project. So then I put the whole project together and that’s what started getting me towards learning the more practical side of things like Stack. It also got me to learn more about monads and that sort of stuff.
So having worked on Haskell in industry, what is your take on the idea widely held in many parts of the programming community that Haskell isn’t suitable for production use? Obviously you’re very opinionated here.
Yea it just couldn’t be farther from the truth. There are plenty of people in the world who are doing production stuff in Haskell and I think that proves it. For instance, JanRain, I think, has used Snap [a Haskell library] for years. I assume they still are, but I used to be able to say “hey, when you go to ladygaga.com, you’re hitting a snap server”. And that’s real stuff. That’s solving real problems in the real world.
We don’t have yet the Google or the Facebook or something of that scale. Though Facebook actually is using Haskell (on the side), just not with their forward user facing stuff. So we don’t have a huge example to point at, but there’s certainly people using Haskell every day in production. So it’s just straight up not true that it can’t be done.
So what is the biggest challenge you’ve had using Haskell on an industry project?
There’s a couple of things that jump out. One was a some kind of machine learning or more numerical algorithm I was working on. It might’ve been a hidden markov model or something with a lot of math going on. And I had a fair amount of trouble getting that right. This was quite a while back, so I was less experienced than I am now.
I think part of the challenge there was not having the right abstractions for multi-dimensional matrix math. And I think Haskell is still not really in a fantastic state there. You can do it, you can accomplish the matrix math with say, HMatrix, and I was able to. But in terms of wrangling dimensions and units and stuff like that, I probably didn’t know to take advantage of some of the things that are available now.
There were definitely some challenges there and it seemed like a very difficult thing to get right. Although I suspect it’s difficult to get right no matter what language you’re using. So it may be the case that Haskell just doesn’t have as much of an advantage there until we get some really phenomenal matrix libraries that could help us solve that problem a lot more nicely.
Another situation where I had frustrations was with an ETL [Extract-Transform-Load == migrating a legacy system] that I was working on and it was so miserable to get right. I was just doing it the naive way bashing through the problem head-on using the IO monad. I was trying to abstract things just using functions, type classes, and whatever other functional techniques I knew about.
Maybe what I should have done is gone back and looked at some kind of a more fundamental approach, perhaps an EDSL [Embedded Domain Specific Language] with say, a GADT [Generalized Algebraic Data Type] and specifying operations representing the structure of what I was trying to do. Then I would have an interpreter that would interpret that in a test type of context where I might be counting different operations that are happening in all manner of checks that you might want to do. Then I would have another interpreter that interprets it in IO and you can kind of test these things in a more structured way.
I think there were a lot of things I could have done better in that project, but maybe as happens in so many projects, there was time pressure. And if that’s the background situation, then you’re less likely to take your time and look for a really good solution to that problem.
Like you said, there’s often a brute-force solution and then a more elegant solution in most software projects. But if you’re operating under major time pressures like so many projects are, it’s hard to find it, especially in Haskell.
I feel like if I had maybe had prior experience with using some of those more elegant solutions, I would have been more likely to jump to them. And this is maybe an obstacle in a more general way. Learning to find a reason to use some of these things, so then you’ll be able to use them in the real world.
Yea that’s a very interesting problem.
To what extent do you think abstract math is necessary to understand Haskell, and to what extent do you think it is helpful? Is there a particular instance where you’ve used higher math to solve a Haskell problem?
I don’t think it is necessary at all to be productive with Haskell. I wouldn’t consider myself super strong in category theory. I took an abstract algebra class in college, so I’m not at the bottom of the barrel, but I don’t think it was super significant to learning Haskell. Things definitely come up. You’ve got all kinds of examples of places where people have leveraged abstract math. But I feel like the knowledge of those structures might actually be part of the answer to the problem we were talking about a second ago.
If you are adequately motivated and you know about this structure somehow, you can use it. The Lens library comes to mind as one. You may not know how profunctors or prisms or whatever other concepts there are, but you know about the lens library, so you can learn about concepts that way, in a library/problem directed way. Maybe knowing the structures would help you know what to use for a more elegant solution to a problem.
But at the same time, when I have studied some category theory concepts, it’s not obvious at all how I’m going to apply them to programming, so we need to do much better as a community at bridging this gap between theory and practice. I think we already do better than any community out there though. Which is good because Haskell does have this reputation of being a very theoretical language. I think we do a great job, but I think we can still improve a lot.
Yea I’m wondering...is there even a theoretical PHP for that community to bridge?
Yea maybe that’s an oxy-moron...
What’s your favorite miscellaneous Haskell library?
Boy, that’s an interesting question. One thing that pops to mind is libraries that represent paths more semantically. So you can concatenate them and manipulate them without having to parse and split on slashes and things like that. I feel like I should have an answer to this question.
Perhaps it’s just far enough back in my brain that I’m not thinking of it right now. But yea, paths are something I encountered a while back. When you really need it, they’re a really cool thing to have...being able to target absolute paths and relative paths. There’s perhaps a whole algebra of paths there that some libraries can give you which is interesting.
What is your least favorite thing about Haskell?
I would say it’s the fact that sometimes we DON’T have referential transparency. The thing that comes to mind that I’ve encountered just recently was that I wanted to just make something generic over a particular prism but I couldn’t, because each of the prisms I would have ended up using returned a different value. So it needed to be more monomorphic than I would have ideally wanted.
So instead of having the absolute right level of abstraction, I had to add a level of repetition to get an Int64 or whatever I was prism-ing out to. We do have referential transparency but occasionally there’s some of these real-world type system restrictions that force us to be more monomorphic than we would like.
So let’s talk a little bit about Takt, where you work. I guess first, do you want to give a quick elevator pitch about Takt? What do you do, what problems do you solve, both the company as a whole, and you yourself?
The company as a whole has lots of very interesting problems. We’re a fairly large-scale data processing/data-science company automating for clients and making intelligent decisions to the point of trying to create a more personalized experience for our clients’ customers. There’s tons of interesting problems to solve there. The one that I’ve been working on most recently is just building a web front-end. And so we’re using Haskell and GHCJS and the Reflex FRP library to build UIs that are robust and easy to refactor and be confident about. So that’s my area of focus.
The company is just a fantastic place to work. It’s really refreshing to have a group of people who are unified in the pursuit of Haskell as the right solution to many of our problems. There are some cases where maybe it’s not the right solution. Maybe you just need some quick and dirty scripts on a server. Especially if you don’t have GHC on that server. So there’s plenty of real world situations where you end up reaching for something else. But as often as we can we like to reach for Haskell, and that’s a really fun environment to be involved in.
In what ways do you think Haskell is uniquely suited for solving Takt’s problems?
I feel that Haskell is uniquely suited for solving ALL the world’s software problems! I truly believe that Haskell or at least the ideas that Haskell is based on are the future of software. The future of more reliable, robust software. And so I feel like it’s going to the future for all of Takt’s software problems.
What is one piece of advice for someone you might have for someone who might want to apply to Takt?
Well if you’re not a Haskell programmer, you should learn some Haskell for sure. I even wrote a blog post awhile back about how to get a Haskell job. You can’t really expect people to hire you for a language that is as different from the mainstream as Haskell is without some prior experience. You can’t expect a company to hire you as any kind of programmer without having some prior experience programming. So I think that’s a pretty natural thing. It’s maybe frustrating for some people who have trouble finding the time to invest in learning Haskell, but it’s kind of a reality.
For Haskell programmers out there who are interested in working at Takt, talk to us! We’d love to hear from you. If you don’t want to just put your resume on a faceless website, you can contact me and open a conversation that way. It’s really nice when candidates have public work that we can look at and get an idea of what kind of Haskell skills you have. We don’t require that because there are plenty of candidates out there that don’t have that for whatever reason, but it can’t hurt.
We’re in the business of trying to reduce risk, and the more we can know about you, the better. Open source contributions are really helpful there. And the more visible that particular project is, the more it’s going to be significant in making us take a look at you and make us excited to talk to you.
Awesome. So let’s finish up with a couple questions about the broader community. What’s your take on the Haskell community at large? What are its strengths, weaknesses, etc..
I think we’re super strong in the area of just being willing to help newcomers. Super friendly. I can’t possibly count how many hours I’ve spent getting help online from people who were just giving me their time just to help me learn some abstract concept where I was taking forever to catch on. It’s just a fantastically friendly and talented community.
I’ve had the opportunity to look at jobs where we were advertised in both a Haskell forum and a non-Haskell forum and the Haskell candidates were just an order of magnitude better and more promising. Now I can’t say that proves anything, because maybe there were other differences about the way those two postings were done that would bias it somehow. But it seems to suggest that it’s a phenomenal filter. If you’re involved in the Haskell community, then you have a lot of intrinsic motivation like we talked about earlier.
You just can’t get to a working knowledge of Haskell by sitting around, being lazy, and watching TV. You gotta actually work for it. Anything in life that requires people to really work for it is going to be a strong predictor of success or maybe the ability to be successful, or a lot of attributes that are aligned and I think are valuable in potential employees.
Very interesting. What changes do you see happening in the Haskell community or the language itself in the next year?
Hmmm that’s a good question. Well I hope to be able to say in the next year or two that Takt is contributing some new libraries and new points in the ecosystem that are missing. Hopefully we can get some more companies using Haskell for adoption. I think there’s some more work recently to make Haskell more tractable for newcomers, and I think that’s fantastic. That’s an area where we’ve struggled a little bit because we are so different from the mainstream. And I really look forward to seeing things grow.
Awesome! That’s all the questions I have. Thanks so much for doing this interview.
It’s a pleasure!
Wrap Up
Hopefully our interview helped give you a better idea of what it’s like working with Haskell in industry. And as Doug mentioned, the Haskell community is very welcoming to newcomers! If you’ve never tried Haskell before, check out our Getting Started Checklist. It’ll show you what tools you need to start writing your first lines of Haskell.
If you’ve done a little Haskell already but want some more practice, you should try our free Recursion Workbook. It has lots of material on recursion and also includes 10 practice problems!
Be sure to check out the Monday Morning Haskell Blog next week. We’ll have a new piece on how we can use Haskell compiled nature to help drive our development and improve our productivity!
Obey the (Type) Laws!
We should now have a decent grasp on functors, applicative functors, and monads. Be sure to check these articles out if you need a refresher! Now we understand the concepts, so it’s time to learn the laws around them.
Remember Haskell represents each of these mathematical classes by a type class. Each of these type classes has one or two main functions. So, as long as we implement those functions and it type checks, we have a new functor/applicative/monad, right?
Well not quite. Yes, your program will compile and you’ll be able to use the instances. But this doesn't mean your instances follow the mathematical constructs. If they don't, your instances won't fulfill other programmers’ expectations. Each type class has its own “laws”. For instance, let's think back to the GovDirectory type we created in the functor article. Suppose we made a different functor instance than we had there:
data GovDirectory a = GovDirectory {
mayor :: a,
interimMayor :: Maybe a,
cabinet :: Map String a,
councilMembers :: [a]
}
instance Functor GovDirectory where
fmap f oldDirectory = GovDirectory {
mayor = f (mayor oldDirectory),
interimMayor = Nothing,
cabinet = f <$> cabinet oldDirectory,
councilMembers = f <$> councilMembers oldDirectory
}As we’ll see, this would violate one of the functor laws. In this case it would not be a true functor. Its behavior would confuse any other programmer trying to use it. We should take care to make sure that our instances make sense. Once you get a feel for these type classes, the likelihood is that the instances you’ll create follow the laws. So don’t sweat it if a few of these are confusing. This article will be very math-y, and we won’t dwell too much on the concepts. You can understand and use these classes without knowing these laws by heart. So without further ado, let’s dive into the laws!
Functor Laws
There are two functor laws. The first is an identity law. We’ll see some variation of this idea for each of the type classes. Remember how fmap "maps" a function over our container. If we map the identity function over a container, the result should be the same container object:
fmap id = idIn other words, our functor should not be applying any extra changes or side effects. It should only apply the function. The second law is a composition law. It states our functor implementation should not break the composition of functions:
fmap (g . f) = fmap g . fmap f
-- For reference, remember the type of the composition operator:
(.) :: (b -> c) -> (a -> b) -> (a -> c)On one side we compose two functions, and map the resulting function over our container. On the other side, we map the first function, get the result, and map the second function over it. The functor composition law states these outcomes should be identical. This sounds complex. But you don't need to worry about it much. If you break the composition law in Haskell, you'll also likely break the identity law.
Those are the only two laws for functors! So let's move on to applicative functors.
Applicative Laws
Applicative functors are a little more complicated. They have four different laws. The first is easy though. It's another simple identity law. It says:
pure id <*> v = vOn the left side, we wrap the identity function. Then we apply it over our container. The applicative identity law states this should result in an identical object. Simple enough.
The second law is the homomorphism law. Suppose we wrap a function and an object in pure. We can then apply the wrapped function over the wrapped object. Of course, we could also apply the normal function over the normal object, and THEN wrap it in pure. The homomorphism law states these results should be the same.
pure f <*> pure x = pure (f x)We should see a distinct pattern here. The overriding theme of almost all these laws is that our type classes are containers. The type class function should not have any side effects. All they should do is facilitate the wrapping, unwrapping, and transformation of data.
The third law is the interchange law. It’s a little more complicated, so don’t sweat it too much. It states that the order that we wrap things shouldn’t matter. One on side, we apply any applicative over a pure wrapped object. On the other side, first we wrap a function applying the object as an argument. Then we apply this to the first applicative. These should be the same.
u <*> pure y = pure ($ y) <*> uThe final applicative law mimics the second functor law. It is a composition law. It states that function composition holds across applications within the functor:
pure (.) <*> u <*> v <*> w = u <*> (v <*> w)The sheer number of laws here can be a little overwhelming. Remember, the instances you make will probably follow the laws! Let’s move on to our final example: monads.
Monad Laws
Monads have three laws. The first two are simple identity laws, like our other classes have had:
return a >>= f = f
m >>= return = mThese are the left and right identities. They state effectively that the only thing the return function is allowed to do is to wrap the object (sound familiar?). It cannot manipulate the data in any way. Our main takeaway from these is that the following code samples are equivalent:
func1 :: IO String
func1 = do
str <- getLine
return str
func2 :: IO String
func2 = getLineThe third law is a bit more interesting. It tells us that associativity holds within monads:
(m >>= f) >>= g = m >>= (\x -> f x >>= g)But we see this third law has a parallel structure to the other composition laws. In the first case, we apply two functions in two steps. In the second case, we compose the functions first, and THEN apply the result. These should be the same.
So in summary, there are two main ideas from all the laws. First, identity should be preserve over wrapper functions, like pure and return. Second, function composition should hold across our structures.
Checking the Laws
As I stated before, most of the instances that you come up with will naturally follow these rules. As you get more experience with the different type classes, this will be even more true. Yet, it also pays to be sure. Haskell has an excellent tool for verifying your instances pass a certain law.
This utility is QuickCheck. It can take any a certain rule, generate many different test cases on the rule, and verify the rule holds. In this section, we’ll run a few tests on our GovDirectory functor instance. We'll see how QuickCheck proves its initial failure, and ultimate success. First we need to implement the Arbitrary type class over our type. We can do this a long as the inner type is also Arbitrary, such as a built-in string type. Then we’ll use all the other Arbitrary instances that exist over our inner types:
instance Arbitrary a => Arbitrary (GovDirectory a) where
arbitrary = do
m <- arbitrary
im <- arbitrary
cab <- arbitrary
cm <- arbitrary
return $ GovDirectory
{ mayor = m
, interimMayor = im
, cabinet = cab
, councilMembers = cm }Once you have done that, you can write a test case over a particular rule. In this case, we check the identity function for functors:
main :: IO ()
main = quickCheck govDirectoryFunctorCheck
govDirectoryFunctorCheck :: GovDirectory String -> Bool
govDirectoryFunctorCheck gd = fmap id gd == gdNow let’s test this on the faulty instance we used above. We can see that a particular test will fail:
*** Failed! Falsifiable (after 2 tests):
GovDirectory {mayor = "", interimMayor = Just "\156", cabinet = fromList [("","")], councilMembers = []}It specifies for us an arbitrary instance that failed the test. Now suppose we correct the instance:
interimMayor = f <$> (interimMayor oldDirectory),We’ll see the tests pass!
+++ OK, passed 100 tests.Summary
We've discussed three major type classes: functors, applicative functors, and monads. They all have particular laws their instances should follow. Other programmers who use your code will expect any instances you make to follow these laws. Once you are familiar with the types, you will likely create instances that follow the laws. But if you are unsure, you can use the QuickCheck utility to verify them.
This concludes our series on monads! You should now have all the tools you need to start using them in practice. Remember that they are a difficult concept, and you'll likely have to review them a couple times. But eventually, you'll understand them!
If you're now inspired to get started with Haskell, make sure to check out our free Getting Started Checklist! It'll help kickstart your Haskell experience by helping you through the download process and making your first project with Stack!
If you're up for a bigger challenge, you should get our Recursion Workbook. It's also free! It has a couple chapters of material on recursion and higher order functions. It also has 10 practice problems for you to try out!
Making Sense of Multiple Monads
We’ve recently how the maybe monad has helped us avoid triangle of doom code patterns. Without it, we had to check each function call for success. However, the examples we looked at were all pure code examples. Consider this:
main :: IO
main = do
maybeUserName <- readUserName
case maybeUserName of
Nothing -> print “Invalid user name!”
Just (uName) -> do
maybeEmail <- readEmail
case maybeEmail of
Nothing -> print “Invalid email!”
Just (email) -> do
maybePassword <- readPassword
Case maybePassword of
Nothing -> print “Invalid Password”
Just password -> login uName email password
readUserName :: IO (Maybe String)
readUserName = do
str <- getLIne
if length str > 5
then return $ Just str
else return Nothing
readEmail :: IO (Maybe String)
...
readPassword :: IO (Maybe String)
...
login :: String -> String -> String -> IO ()
...In this example, all our potentially problematic code takes place within the IO monad. How can we use the Maybe monad when we’re already in another monad?
Monad Transformers
Luckily, we can get the desired behavior by using monad transformers to combine monads. In this example, we’ll wrap the IO actions within a transformer called MaybeT.
A monad transformer is fundamentally a wrapper type. It is generally parameterized by another monadic type. You can then run actions from the inner monad, while adding your own customized behavior for combining actions in this new monad. The common transformers add T to the end of an existing monad. Here’s the definition of MaybeT:
newtype MaybeT m a = MaybeT { runMaybeT :: m (Maybe a) }
instance (Monad m) => Monad (MaybeT m) where
return = lift . return
x >>= f = MaybeT $ do
v <- runMaybeT x
case v of
Nothing -> return Nothing
Just y -> runMaybeT (f y)So MaybeT itself is simply a newtype. It in turn contains a wrapper around a Maybe value. If the type m is a monad, we can also make a monad out of MaybeT.
Let’s consider our example. We want to use MaybeT to wrap the IO monad, so we can run IO actions. This means our new monad is MaybeT IO. Our three helper functions all return strings, so they each get the type MaybeT IO String.
To convert the old IO code into the MaybeT monad, all we need to do is wrap the IO action in the MaybeT constructor.
readUserName :: MaybeT IO String
readUserName = MaybeT $ do
str <- getLIne
if length str > 5
then return $ Just str
else return Nothing
readEmail :: MaybeT IO String
...
readPassword :: MaybeT IO String
...Now we can wrap all three of these calls into a single monadic action, and do a single pattern match to get the results. We’ll use the runMaybeT function to unwrap the Maybe value from the MaybeT:
main :: IO ()
main = do
maybeCreds <- runMaybeT $ do
usr <- readUserName
email <- readEmail
pass <- readPassword
return (usr, email, pass)
case maybeCreds of
Nothing -> print "Couldn't login!"
Just (u, e, p) -> login u e pAnd this new code will have the proper short-circuiting behavior of the Maybe monad! If any of the read functions fail, our code will immediately return Nothing.
Adding More Layers
Here’s the best part about monad transformers. Since our newly created type is a monad itself, we can wrap it inside another transformer! Pretty much all common monads have transformer types in the same way the MaybeT is a transformer for the ordinary Maybe monad.
For a quick example, suppose we had an Env type containing some user information. We could wrap this environment in a Reader. However, we want to still have access to IO functionality, so we’ll use the ReaderT transformer. Then we can wrap the result in MaybeT transformer.
type Env = (Maybe String, Maybe String, Maybe String)
readUserName :: MaybeT (ReaderT Env IO) String
readUserName = MaybeT $ do
(maybeOldUser, _, _) <- ask
case maybeOldUser of
Just str -> return str
Nothing -> do
-- lift allows normal IO functions from inside ReaderT Env IO!
input <- lift getLine
if length input > 5
then return (Just input)
else return NothingNotice we had to use lift to run the IO function getLine. In a monad transformer, the lift function allows you to run actions in the underlying monad. So using lift in the ReaderT Env IO action allows IO functions. Within a MaybeT (ReaderT Env IO) function, calling lift would allow you to run a Reader function. We don’t need this above since the bulk of the code lies in Reader actions wrapped by the MaybeT constructor.
To understand the concept of lifting, think of your monad layer as a stack. When you have a ReaderT Env IO action, imagine a Reader Env monad on top of the IO monad. An IO action exists on the bottom layer. So to run it from the upper layer, you need to lift it up. If your stack is more than two layers, you can lift multiple times. Calling lift twice from the MaybeT (ReaderT Env IO) monad will allow you to call IO functions.
It’s inconvenient to have to know how many times to call lift to get to a particular level of the chain. Thus helper functions are frequently used for this. Additionally, since monad transformers can run several layers deep, the types can get complicated, so it is typical to use type synonyms liberally.
type TripleMonad a = MaybeT (ReaderT Env IO) a
performReader :: ReaderT Env IO a -> TripleMonad a
performReader = lift
performIO :: IO a -> TripleMonad a
performIO = lift . liftTypeclasses
As a similar idea, there are some typeclasses which allow you to make certain assumptions about the monad stack below. For instance, you often don’t care what the exact stack is, but you just need IO to exist somewhere on the stack. This is the purpose of the MonadIO typeclass:
class (Monad m) => MonadIO m where
liftIO :: IO a -> m aWe can use this behavior to get a function to print even when we don’t know its exact monad:
debugFunc :: (MonadIO m) => String -> m a
debugFunc input = do
liftIO $ print “Interpreting Input: “ ++ input
…One final note: You cannot, in general, wrap another monad with the IO monad using a transformer. You can, however, make the other monadic value the return type of an IO action.
func :: IO (Maybe String)
-- This type makes sense
func2 :: IO_T (ReaderT Env (Maybe)) string
-- This does not existSummary
Monad Transformers allow us to wrap monads within other monads. All of the basic built-in monads have transformer types. We name these types by adding T to the end of the name, like MaybeT. Monad transformers let us get useful behavior from all the different monads on our stack. The lift function allows us to run functions within monads further down the stack.
Monad transformers are extremely important when trying to write meaningful Haskell code. If you want to get started with Haskell, be sure to check out our free checklist for Haskell tools.
Want to practice some Haskell skills, but aren’t ready for monads? You can also take a look at our recursion workbook (it’s also free!). It has two chapters of content on recursion and higher order functions, as well as 10 practice problems.
Stay tuned, because next week we will complete our discussion of our abstract wrapper types (functors, applicatives, monads) by exploring the laws governing their behavior.
The Monadic State of Mind
In this article, we’re going to describe one final monad. The State monad is among the most common and most useful. We’ve seen the Reader monad, which allows you to have a common global value among several functions. We also examined the Writer monad, which allows you to keep adding to a particular monoid value. The State monad combines these ideas to give us a full read-write state. We’ll see how this allows our code to have many of the qualities of other programming languages that Haskell seems to lack.
Motivating Example: Monopoly
For this article, we’ll use a simple model for a Monopoly-like game. The main object is the GameState data type containing several important pieces of information.
data GameState = GameState
{ players :: [Player]
, chanceDeck :: [GameCard]
, properties :: Map Property PropertyState
, piecePositions :: Map Player Property
. generator :: StdGen }
data PropertyState = Unowned | Owned Player
data ChanceCard = …
data Player = …
data BoardPostion = …
data GameAction = ...Let’s think at a high level about how some of our game functions would work. We could, for instance, have a function for rolling the dice. This would output a number and alter our game’s number generator. We would then make a move based on the dice output and the current player. This would change the piece positions in the board state as well as leaving us with an output action to resolve (like drawing a card, or taking action on a property).
Buying a property also changes the board’s state. Drawing a chance card would update the state of the deck while returning us a GameCard to resolve. We see a common pattern here among the different actions. Almost all of them will update GameState in some way, and some of them will have an additional piece of output we’ll want to use.
The State Monad
This is exactly the situation the State monad deals with. The State monad wraps computations in the context of reading and modifying a global state object. This context chains two operations together by determining what the state should be after the first operation, and then resolving the second operation with the new state.
It is parameterized by a single type parameter s, the state type in use. So just like the Reader has a single type we read from, the State has a single type we can both read from and write to.
The two main functions we’ll use within the State monad with are get and put. They do exactly what you expect they’d do. The get function works much like the ask function of the reader monad, retrieving our state value. Meanwhile, put works similarly to tell in the Writer monad, where we’ll pass an updated state. Finally we observe there will still be a final return type on each expression in State, just as there is in any other monad. Thus our different function types will look like this for a return type of a:
State GameState a
Our Monopoly Functions
Now we can examine some of the different functions mentioned above and determine their types. We have for instance, rolling the dice:
rollDice :: State GameState Int
rollDice = do
currentState <- get
let gen = generator currentState
let (d1, gen') = randomR (1,6) gen
let (d2, gen'') = randomR (1,6) gen'
put (currentState { generator = gen'' } )
return (d1 + d2)This outputs an Int to us, and modifies the random number generator stored in our state! Now we also have the function making a move:
movePiece :: Player -> Int -> State GameState Property
movePiece player roll = do
currentState <- get
let currentPositions = piecePositions currentState
let currentPos = fromJust (M.lookup player currentPositions)
let next = nextProperty currentPos roll
let newMap = M.insert player next currentPositions
put (currentState { piecePositions = newMap } )
return next
nextProperty :: Property -> Int -> Property
...This will give us the output of the new property we landed on, while also modifying the board with our new position of our piece. Based on the resulting position, we might take different actions, like drawing a chance card:
drawChance :: State GameState ChanceCard
drawChance = do
currentState <- get
let (fstCard : deck) = chanceDeck currentState
put (currentState { chanceDeck = deck } )
return fstCardAs we said above, this will modify the pile of available cards in the chance pile. There are other stateful functions we could describe, such as resolving a property purchase, or paying rent to another player. These would also exist within the state monad.
buyProperty :: Player -> Property -> State GameState ()
…
payRent :: Player -> Property -> State GameState ()
...So finally, we can combine all these functions together with do-syntax, and it actually looks quite clean! We don’t need to worry about the side effects. The different monadic functions handle them. Here’s a sample of what your function might look like to play one turn of the game:
resolveTurn :: State GameState ()
resolveTurn = do
currentState <- get
let playerToMove = currentPlayer currentState
roll <- rollDice
newProperty <- movePiece playerToMove roll
action <- actionForProperty playerToMove newProperty
resolveAction action
switchNextPlayer
return ()Obviously, we haven’t described all these functions, but the general idea should be clear. They would all exist within the state monad.
State, IO, and Other Languages
When thinking about Haskell, it is often seen as a restriction that we can’t have global variables we can modify, like you could with Java class variables. However, we see now this isn’t true. We could have a data type with exactly the same functionality as a Java class, where many functions can modify the global state of the class object using the State monad.
The difference is in Haskell we simply put a label on these types of functions. We don’t allow it to happen for free. We want to know when side effects can potentially happen, because knowing when they can happen makes our code easier to reason about. In a Java class, many of the methods won’t actually need to modify the state. But they could, which makes it harder to debug them. In Haskell we can simply make these pure functions, and our code will be simpler.
IO is the same way. It’s not like we can’t perform IO in Haskell. Instead, we want to label the areas where we can, to increase our certainty about the areas where we don’t need to. When we know part of our code cannot communicate with the outside world, we can be far more certain of its behavior.
Summary
The State monad allows us to have a global readable and writable state. It gives Haskell exactly the kind of flexibility you expect to find in any other programming language. But by separating stateful code from our pure code, our pure code becomes much easier to reason about.
Since we have so many monads under our belts now, the next step is to know how to combine them. Next week we’ll talk about monad transformers, and how those enable us to use multiple monadic functionalities together!
If this has piqued your curiosity for Haskell but you don’t know where to begin, checkout out our checklist to learn more!
If this has inspired you to try out some Haskell code, be sure to try out our free workbook on recursion and higher order function. It includes 10 practice problems so you can hone your skills!
How to Read and Write (with Monads!)
So last week we discussed what a monad is. It isn’t some scary thing only wizards with arcane knowledge of category theory can understand. It’s just a type class with a couple functions describing a particular context. These functions, when used properly, can dramatically expand what we can do while keeping our code purely functional.
We haven’t gone over all the “laws” these functions need to follow. But if we explore enough examples, we’ll have an intuitive grasp of what should happen. We saw some simple examples last time with the Maybe, Either, and IO monads. In this article, we will look at the Reader and Writer monads.
Global Variables (or a lack thereof)
In Haskell, our code is generally “pure”, meaning functions can only interact with the arguments passed to them. This effectively means we cannot have global variables. We can have global expressions, but these are fixed at compile time. If user behavior might change them, we have to wrap them in the IO monad, which means they can’t be used from pure code.
Consider this example where we might want to have an Environment containing different parameters as a global variable. However, we might have to load these from a config file or a command line interface, which requires the IO monad.
main :: IO ()
main = do
env <- loadEnv
let str = func1 env
print str
data Environment = Environment
{ param1 :: String
, param2 :: String
, param3 :: String }
loadEnv :: IO Environment
loadEnv = …
func1 :: Environment -> String
func1 env = “Result: “ ++ (show (func2 env))
func2 :: Environment -> Int
func2 env = 2 + floor (func3 env)
func3 :: Environment -> Float
func3 env = … -- Some calculation based on the environmentThe only function actually using the environment is func3. However func3 is an impure function. This means it cannot directly call loadEnv, an impure function. This means the environment has to be passed through as a variable to the other functions, just so they can ultimately pass it to func3. In a language with global variables, we could save env as a global value in main. Then func3 could access it directly. There would be no need to have it as a parameter to func1 and func2. In larger programs, these “pass-through” variables can cause a lot of headaches.
The Reader Solution
The Reader monad solves this problem. It effectively creates a global read-only value of a specified type. All functions within the monad can “read” the type. Let’s look at how the Reader monad changes the shape of our code. Our functions no longer need the Environment as an explicit parameter, as they can access it through the monad.
main :: IO ()
main = do
env <- loadEnv
let str = runReader func1 env
print str
data Environment = Environment
{ param1 :: String
, param2 :: String
, param3 :: String }
loadEnv :: IO Environment
loadEnv = …
func1 :: Reader Environment String
func1 = do
res <- func2
return (“Result: “ ++ (show res))
func2 :: Reader Environment Int
func2 = do
env <- ask
let res3 = func3 env
return (2 + (floor res3))
func3 :: Environment -> Float
...The ask function unwraps the environment so we can use it. The monad’s bind action allows us to glue different Reader actions together together. In order to call a reader action from pure code, all we need to do is call the runReader function and supply the environment as a parameter. All functions within the action will be able to treat it like a global variable.
It might not seem like we’ve accomplished much, but our code is much more intuitive now. We keep func3 as it was. It makes sense to describe it as a function from an Environment to a value. However, our other two functions no longer take the environment as an explicit parameter. They simply exist in a context where the environment is a global variable.
Accumulating Values
Now, to motivate the Writer monad, let’s talk about the accumulation problem. Suppose we have a few different functions. Each will perform some string operations we’ve assigned an arbitrary “cost” to. We want to keep track of how “expensive” it was to run the full computation. We can do this by using accumulator arguments to keep track of the cost we’ve seen so far. We then keep passing the accumulated value along.
-- Calls func2 if even length, func3 and func4 if odd
func1 :: String -> (Int, String)
func1 input = if length input `mod` 2 == 0
then func2 (0, input)
else (i1 + i2, str1 ++ str2)
where
(i1, str1) = func3 (0, tail input)
(i2, str2) = func4 (0, take 1 input)
-- Calls func4 on truncated version
func2 :: (Int, String) -> (Int, String)
func2 (prev, input) = if (length input) > 10
then func4 (prev + 1, take 9 input)
else (10, input)
-- Calls func2 on expanded version if a multiple of 3
func3 :: (Int, String) -> (Int, String)
func3 (prev, input) = if (length input) `mod` 3 == 0
then (prev + f2resI + 3, f2resStr)
else (prev + 1, tail input)
where
(f2resI, f2resStr) = func2 (prev, input ++ "ab")
func4 :: (Int, String) -> (Int, String)
func4 (prev, input) = if (length input) < 10
then (prev + length input, input ++ input)
else (prev + 5, take 5 input)However, an Int isn’t the only type of value we could accumulate. We could instead be accumulating a list of strings to print as log messages so we know what computations were run. There is a generalization of this behavior: the Monoid typeclass.
The Monoid Typeclass
In this example, Int is a simple example of a Monoid. Let’s look at the monoid typeclass definition:
class Monoid a where
mempty :: a
mappend :: a -> a -> aThis is effectively an accumulation class. It defines two functions. The mempty function is an initial value for our monoid. Then with mappend, we can combine two values of this type into a result. It is quite easy to how we can make a monoid instance for Int:
instance Monoid Int where
memty = 0
mappend a b = a + bOur accumulator starts at 0, and we combine values by adding them.
Using Writer to Track the Accumulator
The Writer monad is parameterized by some monoidal type. Its main job is to keep track of an accumulated value of this type. So it’s operations live in the context of having a global value that they can modify in this particular way. We can change our code examples above to use the Writer monad as follows:
func1 :: String -> (String, Int)
func1 input = if length input `mod` 2 == 0
then runWriter (func2 input)
else runWriter $ do
str1 <- func3 input
str2 <- func4 (take 1 input)
return (str1 ++ str2)
func2 :: String -> Writer Int String
func2 input = if (length input) > 10
then do
tell 1
func4 (take 9 input)
else do
tell 10
return input
func3 :: String -> Writer Int String
func3 input = if (length input) `mod` 3 == 0
then do
tell 3
func2 (input ++ “ab”)
else do
tell 1
return $ tail input
func4 :: String -> Writer Int String
func4 input = if (length input) < 10
then do
tell (length input)
return (input ++ input)
else do
tell 5
return (take 5 input)Notice we no longer need to actually explicitly keep track of the accumulator. It is now wrapped by the Writer monad. We can increase it in any of our functions by calling “tell”. Now our code is much simpler and our types are cleaner.
Conclusion
The Reader and Writer monads both offer pure functional ways to deal with common side effects. The Reader monad allows you to keep track of a shared global state. It allows you to avoid passing that state as an explicit parameter to functions that don’t really use it. The Writer monad allows you to keep track of a global accumulated value using a monoid. Next week we’ll learn how we can wrap these ideas into one with the State monad!
Hopefully this article has helped convinced you that monads (and Haskell for that matter) aren’t all that scary! If this has inspired you to pick up Haskell and start writing some code, check out our free checklist for getting stated!
Not quite ready for monads but want to try some different Haskell skills? Check out our recursion workbook. It includes 2 chapters of material on recursion and higher order functions, as well as 10 practice problems with a test harness.
(Finally) Understanding Monads (Part 1)
We should now have a decent grasp on functors and applicative functors (check out the links if you aren’t!). Now it’s time to take the next step up. We’re going to tackle the dreaded concept of Monads. There are dozens of monad tutorials and descriptions on the internet. This makes sense. Monads are vital to writing any kind of meaningful program in Haskell. They aren’t the hardest concept in functional programming, but they are the biggest roadblock because of their importance. In this series of articles, we’re going to try tackling the concept in small, manageable chunks.
So without further ado, here’s my crack at a definition: A Monad wraps a value or a computation with a particular context. A monad must define both a means of wrapping normal values in the context, and a way of combining computations within the context.
This definition is quite broad. So let’s look at a more practical level to try to make sense of this.
The Monad Typeclass
Just like with functors and applicative functors, Haskell represents monads with a type class. It has two functions:
class Monad m where
return :: a -> m a
(>>=) :: m a -> a -> m b -> m bThese two functions correspond to the two ideas from above. The return function specifies a how to wrap values in the monad’s context. The >>= operator, which we call the “bind” function, specifies how to combine two operations within the context. Let’s clarify this further by exploring a few specific monad instances.
The Maybe Monad
Just as Maybe is a functor and an applicative functor, it is also a monad. To motivate the Maybe monad, let’s consider this code.
maybeFunc1 :: String -> Maybe Int
maybeFunc1 “” = Nothing
maybeFunc1 str = Just $ length str
maybeFunc2 :: Int -> Maybe Float
maybeFunc2 i = if i `mod` 2 == 0
then Nothing
Else Just ((fromIntegral i) * 3.14159)
maybeFunc3 :: Float -> Maybe [Int]
maybeFunc3 f = if f > 15.0
then Nothing
else $ Just [floor f, ceil f]
runMaybeFuncs :: String -> Maybe [Int]
runMaybeFuncs input = case maybeFunc1 input of
Nothing -> Nothing
Just i -> case maybeFunc2 i of
Nothing -> Nothing
Just f -> maybeFunc3 fWe can see we’re starting to develop a hideous triangle pattern as we continue pattern matching on the results of successive function calls. If we were to add more Maybe functions onto this, it would keep getting worse. When we consider Maybe as a monad, we can make the code much cleaner. Let’s take a look at how Haskell implements Maybe as a monad to see how.
instance Monad Maybe where
return = Just
Nothing >>= _ = Nothing
Just a >>= f = f aThe context the Maybe monad describes is simple. Computations in Maybe can either fail or succeed with a value. We can take any value and wrap it in this context by calling the value a “success”. We do this with the Just constructor. We represent failure by Nothing.
We combine computations in this context by examining the result of the first computation. If it succeeded, we takes its value, and pass it to the second computation. If it failed, then we have no value to pass to the next step. So the total computation is a failure. Let’s look at how we can use the bind operator to combine our operations:
runMaybeFuncs :: String -> Maybe [Int]
runMaybeFuncs input = maybeFunc1 input >>= maybeFunc2 >>= maybeFunc3This looks much cleaner! Let’s see why the types work out. The result of maybeFunc1 input is simply Maybe Int. Then the bind operator allows us to take this Maybe Int value and combine it with maybeFunc2, whose type is Int -> Maybe Float. The bind operator resolves these to a Maybe Float. Then we pass this similarly through the bind operator to maybeFunc3, resulting in our final type, Maybe [Int].
Your functions will not always combine so cleanly though. This is where do notation comes into play. We can rewrite the above as:
runMaybeFuncs :: String -> Maybe [Int]
runMaybeFuncs input = do
i <- maybeFunc1 input
f <- maybeFunc2 f
maybeFunc3 fThe <- operator is special. It effectively unwraps the value on the right-hand side from the monad. This means the value i has type Int, even though the result of maybeFunc1 is Maybe Int. The bind operation happens under the hood, and if the function returns Nothing, then the entire runMaybeFuncs function will return Nothing.
At first glance, this looks more complicated than the bind example. However, it gives us a lot more flexibility. Consider if we wanted to add 2 to the integer before calling maybeFunc2. This is easy to deal with in do notation, but more difficult when simply using binds:
runMaybeFuncs :: String -> Maybe [Int]
runMaybeFuncs input = do
i <- maybeFunc1 input
f <- maybeFunc2 (i + 2)
maybeFunc3 f
-- Not so nice
runMaybeFuncsBind :: String -> Maybe [Int]
runMaybeFuncsBind input = maybeFunc1 input
>>= (\i -> maybeFunc2 (i + 2))
>>= maybeFunc3The gains are even more obvious if we want to use multiple previous results in a function call. Using binds, we would have to continually accumulate arguments in anonymous functions. One note about do notation: we never use <- to unwrap the final operation in a do-block. Our call to maybeFunc3 has the type Maybe [Int]. This is our final type (not [Int]) so we do not unwrap.
The Either Monad
Now, let’s examine the Either monad, which is quite similar to the Maybe monad. Here’s the definition:
instance Monad (Either a) where
return r = Right r
(Left l) >>= _ = Left l
(Right r) >>= f = f rWhereas the Maybe either succeeds with a value or fails, the Either monad attaches information to failures. Just like Maybe, it wraps values in its context by calling them successful. The monadic behavior also combines operations by short-circuiting on the first failure. Let’s see how we can use this to make our code from above more clear.
maybeFunc1 :: String -> Either String Int
maybeFunc1 “” = Left “String cannot be empty!”
maybeFunc1 str = Right $ length str
maybeFunc2 :: Int -> Either String Float
maybeFunc2 i = if i `mod` 2 == 0
then Left “Length cannot be even!”
else Right ((fromIntegral i) * 3.14159)
maybeFunc3 :: Float -> Either String [Int]
maybeFunc3 f = if f > 15.0
then Left “Float is too large!”
else $ Right [floor f, ceil f]
runMaybeFuncs :: String -> Either String [Int]
runMaybeFuncs input = do
i <- maybeFunc1 input
f <- maybeFunc2 i
maybeFunc3 fBefore, every failure just gave us a Nothing value:
>> runMaybeFuncs ""
Nothing
>> runMaybeFuncs "Hi"
Nothing
>> runMaybeFuncs "Hithere"
Nothing
>> runMaybeFuncs "Hit"
Just [9,10]Now when we run our code, we can look at the resulting error string, and this will tell us which function actually failed.
>> runMaybeFuncs ""
Left "String cannot be empty!"
>> runMaybeFuncs "Hi"
Left "Length cannot be even!"
>> runMaybeFuncs "Hithere"
Left "Float is too large!"
>> runMaybeFuncs "Hit"
Right [9,10]Notice we parameterize the Either monad by the error type. If we have:
maybeFunc2 :: Either CustomError Float
…This function is in a different monad now. It won’t be quite as simple to combine this with our other functions. If you’re curious how we might do this, check out this answer on quora.
The IO Monad
The IO Monad is perhaps the most important monad in Haskell. It is also one of the hardest monads to understand starting out. Its actual implementation is a bit too intricate to discuss when first learning monads. So we’ll learn by example.
The IO monad wraps computations in the following context: “This computation can read information from or write information to the terminal, file system, operating system, and/or network”. If you want to get user input, print a message to the user, read information from a file, or make a network call, you’ll need to do so within the IO Monad. These are “side effects”. We cannot perform them from “pure” Haskell code.
The most important job of pretty much any computer program is to interact with the outside world in some way. For this reason, the root of all executable Haskell code is a function called main, with the type IO (). So every program starts in the IO monad. From here you can get any input you need, call into relatively “pure” code with the inputs, and then output the result in some way. The reverse does not work. You cannot call into IO code from pure code, the same way you can call into a Maybe function from pure code.
Let’s look at a simple program showing a few of the basic IO functions. We’ll use do-notation to illustrate the similarity to the other monads we’ve discussed. We list the types of each IO function for clarity.
main :: IO ()
main = do
-- getLine :: IO String
input <- getLIne
let uppercased = map Data.Char.toUpper input
-- print :: String -> IO ()
print uppercasedSo we see once again each line of our program has type IO a. (A let statement can occur in any monad). Just as we could unwrap i in the maybe example to get an Int instead of a Maybe Int, we can use <- to unwrap the result of getLine as a String. We can then manipulate this value using string functions, and pass the result to the print function.
This is a simple echo program. It reads a line from the terminal, and then prints the line back out in all caps. Hopefully it gives you a basic understanding of how IO works. We’ll get into more details in the next couple articles.
Summary
A monad wraps computations in a particular context. It defines functions for wrapping values in its context, and combining operations in the context. Maybe is a monad. We describe its context by saying its computations can succeed or fail. Either is similar to Maybe, except it can add error information to failures. The IO monad is hugely important, encapsulating the context of operations reading from and writing to the terminal, network, and file system. The easiest way to learn monadic code is to use do notation. In this notation, every line has a right-side value of the monad. You can then unwrap the value on the left side using the <- operator.
Stay tuned next week as we continue our exploration of monads. We’ll examine the Reader and Writer monads, and demonstrate how they encapsulate different kinds of side effects then we might get from the IO monad.
Hopefully this article has started you off on (finally) understanding monads. If you haven’t written any Haskell code yet and want to get started so you can test your knowledge of monads, be sure to check out our free checklist for getting started with Haskell!
Not quite ready for monads but want to try some different Haskell skills? Check out our recursion workbook. It includes 2 chapters of material on recursion and higher order functions, as well as 10 practice problems with a test harness.
Applicatives: One Step Further
So last week, we discussed the Functor typeclass. We found it allows us to run transformations on data regardless of how the data is wrapped. No matter if our data were in a List, a Maybe, an Either, or even a custom type, we could simply call fmap. However, what happens when we try to combine wrapped data? For instance, if we try to have GHCI interpret these calculations, we’ll get type errors:
>> (Just 4) * (Just 5)
>> Nothing * (Just 2)Functors Falling Short
Can functors help us here? We can use fmap to wrap multiplication by the particular wrapped Maybe value:
>> let f = (*) <$> (Just 4)
>> :t f
f :: Num a => Maybe (a -> a)
>> (*) <$> Nothing
NothingThis gives us a partial function wrapped in a Maybe. But we still cannot unwrap this and apply it to (Just 5) in a generic fashion. So we have to resort to code specific to the Maybe type:
funcMaybe :: Maybe (a -> b) -> Maybe a -> Maybe b
funcMaybe Nothing _ = Nothing
funcMaybe (Just f) val = f <$> valThis obviously won’t work with other functors types.
Applicatives to the Rescue
This is exactly what the Applicative typeclass is for. It has two main functions:
pure :: a -> f a
(<*>) :: f (a -> b) -> f a -> f bThe pure function takes some value and wraps it in a minimal context. The <*> function, called sequential application, takes two parameters. First, it takes a function wrapped in the context. Next, a wrapped value. Its output is the result of applying the function to the value, rewrapped in the context. An instance is called an applicative functor because it allows us to apply a wrapped function. Since sequential application takes a wrapped function, we typically begin our use of applicatives by wrapping something with either pure or fmap. This will become more clear with some examples.
Let’s first consider multiplying Maybe values. If we are multiply by a constant value we can use the functor approach. But we can also use the applicative approach by wrapping the constant function in pure and then using sequential application:
>> (4 *) <$> (Just 5)
Just 20
>> (4 *) <$> Nothing
Nothing
>> pure (4 *) <*> (Just 5)
Just 20
>> pure (4 *) <*> Nothing
NothingNow if we want to multiply 2 maybe values, we start by wrapping the bare multiplication function in pure. Then we sequentially apply both Maybe values:
>> pure (*) <*> (Just 4) <*> (Just 5)
Just 20
>> pure (*) <*> Nothing <*> (Just 5)
Nothing
>> pure (*) <*> (Just 4) <*> Nothing
NothingImplementing Applicatives
From these examples, we can tell the Applicative instance for Maybe is implemented exactly how we would expect. The pure function simply wraps a value with Just. Then to chain things together, if either the function or the value is Nothing, we output Nothing. Otherwise apply the function to the value and re-wrap with Just.
instance Applicative Maybe where
pure = Just
(<*>) Nothing _ = Nothing
(<*>) _ Nothing = Nothing
(<*>) (Just f) (Just x) = Just (f x)The Applicative instance for Lists is a little more interesting. It doesn’t exactly give the behavior we might first expect.
instance Applicative [] where
pure a = [a]
fs <*> xs = [f x | f <- fs, x <- xs]The pure function is what we expect. We take a value and wrap it as a singleton in a list. When we chain operations, we now take a LIST of functions. We might expect to apply each function to the value in the corresponding position. However, what actually happens is we apply every function in the first list to every value in the second list. When we have only one function, this results in familiar mapping behavior. But when we have multiple functions, we see the difference:
>> pure (4 *) <*> [1,2,3]
[4,8,12]
>> [(1+), (5*), (10*)] <*> [1,2,3]
[2,3,4,5,10,15,10,20,30]This makes it easy to do certain operations, like finding every pairwise product of two lists:
>> pure (*) <*> [1,2,3] <*> [10,20,30]
[10,20,30,20,40,60,30,60,90]You might be wondering how we might do parallel application of functions. For instance, we might want to use the second list example above, but have the result be [2,10,30]. There is a construct for this, called ZipList! It is a newtype around list, whose Applicative instance is designed to use this behavior.
>> ZipList [(1+), (5*), (10*)] <*> [5,10,15]
ZipList {getZipList = [6,50,150]}Summary
- Applicative functors take the idea of normal functors one step further.
- They allow function application to occur within the wrapper context.
- In some circumstances, this allows us to reuse generic code.
- In other cases, this gives us clean idioms to express simple concepts and solve common problems.
The Easiest Haskell Idiom
Once you master the basics of Haskell, the next important step is to understand the patterns that make it easier to write good, idiomatic Haskell code. The next few posts will focus on some of the most important patterns to learn. The simplest of these is functors.
A Simple Example
Here’s a simple example to start us on our way. This code converts an input string like “John Doe 24” into a tuple. We want to consider all inputs though, so the resulting type is a Maybe.
tupleFromInputString :: String -> Maybe (String, String, Int)
tupleFromInputString input = if length stringComponents /= 3
then Nothing
else Just (stringComponents !! 0, stringComponents !! 1, age)
where
stringComponents = words input
age = (read (stringComponents !! 2) :: Int)This simple function simply takes a string and converts it into parameters for first name, last name, and age. Suppose we have another part of our program using a data type to represent a person instead of a tuple. We might write a conversion function between these two different types. We want to account for the possibility of failure. So we’ll have another function handling that case.
data Person = Person {
firstName :: String,
lastName :: String,
age :: Int
}
personFromTuple :: (String, String, Int) -> Person
personFromTuple (fName, lName, age) = Person fName lName age
convertTuple :: Maybe (String, String, Int) -> Maybe Person
convertTuple Nothing = Nothing
convertTuple (Just t) = Just (personFromTuple t)A Change of Format
But imagine our original program changes to read in a whole list of names:
listFromInputString :: String -> [(String, String, Int)]
listFromInputString contents = mapMaybe tupleFromInputString (lines contents)
tupleFromInputString :: String -> Maybe (String, String, Int)
...Now if we passed this result to the code using Person, we would have to change the type of the convertTuple function. It would have a parallel structure though. Maybe and List can both act as containers for other values. Sometimes, we don’t care how values are wrapped. We just want to transform whatever underlying value exists, and then return the new value in the same wrapper.
Introduction to Functors
With this idea in mind, we can start understanding functors. First and foremost, Functor is a typeclass in Haskell. In order for a data type to be an instance of the Functor typeclass, it must implement a single function: fmap.
fmap :: (a -> b) -> f a -> f bThe fmap function takes two inputs. First, it demands a function between two data types. The second parameter is some container of the first type. The output then is a container of the second type. Now let’s look at a few different Functor instances for some familiar types. For lists, fmap is simply defined as the basic map function:
instance Functor [] where
fmap = mapIn fact, fmap is a generalization of mapping. For example, the Map data type is also a functor. It uses its own map function for fmap. Functors simply take this idea of transforming all underlying values and apply it to other types. With this in mind, let’s observe how Maybe is a functor:
instance Functor Maybe where
fmap _ Nothing = Nothing
fmap f (Just a) = Just (f a)This looks a lot like our original convertTuple function! If we have no value in the first place, then the result is Nothing. If we do have a value, simply apply the function to the value, and rewrap it in Just. The Either data type can be seen as a Maybe type with more information about how it failed. It has similar behavior:
instance Functor (Either a) where
fmap _ (Left x) = Left x
fmap f (Right y) = Right (f y)Note the first type parameter of this instance is fixed. Only the second parameter of an Either value is changed by fmap. Based on these examples, we can see how to rewrite convertTuple to be more generic:
convertTuple :: Functor f => f (String, String, Int) -> f Person
convertTuple = fmap personFromTupleMaking Our Own Functors
We can also take our own data type and define an instance of Functor. Suppose we have the following data type representing a directory of local government officials. It is parameterized by the type a. This means we allow different directories using different representations of a person:
data GovDirectory a = GovDirectory {
mayor :: a,
interimMayor :: Maybe a,
cabinet :: Map String a,
councilMembers :: [a]
}One part of our application might represent people with tuples. Its type would be GovDirectory (String, String, Int). However, another part could use the type GovDirectory Person. We can define the following Functor instance for GovDirectory by defining fmap. Since our underlying types are mostly functors themselves, this mostly involves calling fmap on the individual fields!
instance Functor GovDirectory where
fmap f oldDirectory = GovDirectory {
mayor = f (mayor oldDirectory),
interimMayor = f <$> interimMayor oldDirectory,
cabinet = f <$> cabinet oldDirectory,
councilMembers = f <$> councilMembers oldDirectory
}Note <$> is simply a synonym for fmap. Now we have our own functor instance, sp transforming the underlying data type of our directory class is easy! We can just use:
convertTuple <$> oldDirectorySummary
- Functors, in general, wrap some kind of data
- In Haskell,
Functoris a typeclass, with a single functionfmap - The
fmapfunction allows us to transform the underlying data without caring how the data is contained. - This allows us to write much more flexible code in certain circumstances.
Stay tuned for next week, when we’ll discuss applicative functors! If you’re starting to get a grasp for Haskell and want to try new skills, be sure to check out our free workbook on Recursion, which comes with 10 practice problems!
Easing Haskell's Intimidating Glare
All of my previous articles have discussed basic language features and concepts in Haskell. My hope has been to provide new Haskellers with suitable starting material to help them get started. Even just as a language, Haskell is complex. There are many technical challenges and paradigm shifts one has to overcome to learn it. This is especially true for those coming from imperative languages.
However, this article will focus on something equally important. It’s no secret that most people consider Haskell not just a difficult language to learn technically, but an intimidating language. It has undoubted psychological hurdles. People seem to give up on Haskell at a higher rate than most other languages. By naming these problems, we can overcome them.
An Academic Language
People have long considered Haskell primarily as a research language. It builds on the lambda calculus, possibly the simplest, purest programming language. This gives it a host of connections to cool concepts in abstract mathematics, primarily the province of professors and PhD. students. The connection is so elegant many mathematical ideas can be well represented in Haskell.
But this connection has a price in accessibility. Important Haskell concepts include functors, monads, categories, etc. These are cool, but few without a math degree have any intuition for what the terms mean. Compare these to terms from other languages: class, iterator, loop, template. These terms are a lot more intuitive, so the languages using them have an automatic advantage in accessibility.
Aside from this terminology point though, the great academic interest is a good thing, not a bad thing. However, on the production side of things, the tooling has not been sufficient. It was simply too difficult to maintain a large-scale Haskell project. As a result, companies had little incentive to use it. This meant there was little to no balance of the academic influence.
Knowledge Distribution
The net result of Haskell’s academic primacy is a skewed knowledge base. The nature of academia is relatively few people spend a large amount of time on a relatively small set of problems. Consider another academic field, like virology. You have some experts who understand viruses at an extremely high level, and the rest of us know quite little. There are no hobbyist virologists. Unfortunately, this kind of distribution is not conducive to introducing new people to a topic.
Naturally, people have to learn from those who know more than they do. But the truth is they don’t want their teachers to be too much better. It helps tremendously to learn from someone who was in your shoes not too long ago. They’ll more likely remember the pitfalls and frustrations they encountered early on, so they’ll be able to help you avoid them. But when the distribution skews towards the extreme, there is no middle class. There are fewer people who can optimally teach new learners. In addition to not remembering old mistakes, experts tend to use overly complicated terminology. New folks may feel intimidated by this and despair.
Turning the Tide of Production
The lack of production work mentioned above contributes substantially to this divide. Many other languages, like C++, have strong academic followings. But since so many companies use C++ in production, it does not face the knowledge distribution problem Haskell does. Companies using C++ have no choice but to train people to use the language. Many of these people stick with the language long enough to train the next generation. This creates a more normal looking knowledge distribution curve.
The good news for Haskell is there have been major tooling improvements in the last few years. This has brought about a renaissance for the language. More companies are starting to use it in production. More meetups are happening; more people are writing libraries for the most crucial tasks. If this trend continues, Haskell will hopefully reach a tipping point, normalizing the knowledge distribution curve.
The Key Insight
If you are someone who is interested in learning Haskell, or who has tried learning Haskell in the past, there is one key thing to know. While the abstract mathematics is cool, understanding it is mostly unnecessary. Monads are essential, there is no denying. But category theory is overkill for most day-to-day problems you’ll solve. The dozens of language extensions might seem intimidating, but you can pick them up one-by-one as you go.
At the last Haskell eXchange, Don Stewart from Standard Chartered gave a talk about the company’s use of Haskell. He explained they rarely use anything outside of vanilla Haskell constructs*. They just don’t need to. Anything you can do with, say, lenses, you can accomplish without them.
Haskell is different from most other programming languages. It constrains you in ways those languages do not. But the constraints are not nearly as binding as they seem. You can’t use for loops. So use recursion. You can’t re-assign variables. So create new names for expressions. You just have take it one step at a time.
Taking Action
If this has inspired you to get started with Haskell, check out this checklist to learn the tools you need to get started.
If you’re already familiar with the basics and want to take the next step up, you should take a look at our workbook. You’ll learn about recursion and get to take a shot at 10 practice problems to test your skills!
Note
*At least with respect to their normal Haskell code. Some of their code is in a proprietary language of theirs called Mu, which is built on Haskell but obviously different.
