2. Property-Based Testing Tools

Common forms of automated testing such as the one seen in Test-Driven Development (TDD) with unit tests tend to require very little support from tooling. Simple test harnesses for execution and assertions may be all that is provided by xUnit-style of frameworks. Property-Based testing, by comparison requires heavier machinery to be provided to the developer in order to be of any use, and various frameworks exist with varying levels of completeness.

This chapter gives a tour of the overall landscape for property-based testing, introduces the framework chosen for this book, and explains how to get everything in place to be able to follow along with the text.

2.1. In General

Even though property-based testing may feel familiar to fuzzing and other testing techniques based on randomness, it is a distinct entity with a distinct set of tools. Property-based testing sits in between model-checking (a formal method) and fuzzing (fairly random). It does so by having input generation that is based in randomness, but with hints to guide its exploration of the search space. The forms this takes will be explained in Custom Generators. The generated input is then passed to 'properties', which are specific tests or models of the system under test. Because the search space is neither deterministic nor exhaustive, property-based testing cannot be considered to be a formal method.

Property-based testing in its more popular form is derived from the initial Quickcheck tool, written in Haskell and published for the first time in 1999. It was designed by Koen Claessen and John Hughes, and quickly touted as one of the most interesting things in the Haskell community.

The original authors of QuickCheck ended up forming Quviq, a company offering a commercial version of the tool that was more advanced. Since the market was a bit small, they eventually made a version of Quickcheck in Erlang (which boasted some larger industrial actors than Haskell). Quickcheck for Erlang was rapidly embraced by the community and rapidly found interested industrial parties. The Erlang community then made two other spin-off versions: PropEr, which is licensed GPLv3 (with a promise not to cause any "problem" with Open Source projects), and Triq, which uses a more permissive Apache v2 license.

The quickcheck framework was also copied into many other languages, including C, C++, Scheme, Clojure, Common Lisp, D, Elm, F#, Go, Java, JavaScript, Julia, Objective-C, OCaml, Perl, Prolog, PHP, Python, R, Ruby, Rust, Scala, Smalltalk, SML, and Swift.

None of them can truly compare to the full commercial Quviq Erlang Quickcheck version in the completeness of their implementation, although some come closer than others. Erlang, Haskell and Scala, for example, all have fairly complete implementations and better support within their communities than Javascript, Ruby, and Go, which tend to only have frameworks implementing a subset of the features and very limited visibility within their respective communities.

An interesting language for Property-based testing is Python, which has the Hypothesis[1] framework. The ideas behind it can appear similar, but it integrates them more fully into existing python semantics and uses rather unique internal mechanisms to work. The feature set it provides does not match the full set provided by Quviq Erlang Quickcheck, but the opposite is also true; the feature sets of either tools overlap but are distinct. Hypothesis has taken its own path when it comes to property-based testing, and comparing both families of frameworks is a bit tricky.[2]

2.2. In This Book

This book uses Erlang as the host language for property-based testing, with PropEr as its main tool. The reason for this is that the most powerful property-based test frameworks now live in Erlang, in no small part thanks to the influence of Quviq. Their implementation is, bar none, the best one available across any language. PropEr is its most powerful free counterpart in the community, and Triq is principally used by people who want something more powerful than the free (as in beer) version of QuickCheck Mini made available by Quviq, but do not want the risk of writing tests in PropEr which could have an interesting impact on their testing code with the GPL.

Fortunately, the vast majority of property-based testing libraries out there use similar terminology and test formats as the Erlang versions; most of them would feel familiar and would require limited adjustment after having read this book. As mentioned earlier, some languages only have a partial set of Quickcheck’s features implemented, so not all examples may be available or may make sense in all languages.

The core principles remain the same however, and learning from this book should make it possible to use similar techniques in any other language, as long as their frameworks support equivalent features.

2.3. Prerequisites and Setup

The reader should be familiar with Erlang concepts such as:

  • basic data types, including numbers, atoms, tuples, lists, strings, and binaries

  • the syntax for Dialyzer and Erlang type specifications

  • language constructs such as modules, functions, macros, module attributes, variables, and function calls

  • concepts like pattern matching, recursion, higher order functions

The reader should also have an installed version of Erlang with a working copy of rebar3, Erlang’s build tool. Once the tool is installed, a project can be set up from the command line:

→ rebar3 new lib pbt
===> Writing pbt/src/pbt.erl
===> Writing pbt/src/pbt.app.src
===> Writing pbt/rebar.config
===> Writing pbt/.gitignore
===> Writing pbt/LICENSE
===> Writing pbt/README.md
→ cd pbt

The rebar.config file should be edited to contain the following text, which will ensure PropEr is available to the project and ready to be used:

%% the plugin itself
{plugins, [rebar3_proper]}.

%% The PropEr dependency is still required to compile the test cases
{profiles,
    [{test, [
        {deps, [{proper, "1.2.0"}, recon]}
    ]}
]}.

Rebar3, for this project, can now run PropEr tests. Options for the plugin can be displayed by calling:

→ rebar3 help proper
[...]

Everything should then be in place to move forward.

1. https://hypothesis.works
2. Add to this that the concerns and priorities Python developers and Erlang developers tend to have when testing may be fairly different, and divergences can soon be expected to appear.