PropEr Testing

By default, the data types provided by PropEr include very generic Erlang types. Very generic data types imply a very large search space, and it may therefore be difficult to find examples of data that are varied enough to uncover edge cases.

The last exercise of Stateless Properties' questions asked to generate a property that validates the system’s ability to count words in strings. A candidate property for that test could be:

This property has to operate on generic strings that are fairly random-looking. The test itself focuses on word counting, which is a fairly specific activity based on a fairly specific subset of strings. It should not be surprising that the fairly random strings might therefore be a suboptimal fit for a word-counting test. The property can pass dozens of iterations without problem, and it should be reliable, but the test would be more trustworthy if it had a more specialized generator.

In fact, if this word-counting candidate property were to be run over more tests than the default number of iterations (100), a failure would be detected whenever two or more space characters (' ') are in a sequence in a generated string, or when a string with just spaces is submitted. This error should logically happen really early on in a good property test that is related to breaking up text on space characters, but since the default generators are so general, the odds for this specific case to show up are relatively low.

Custom generators can fortunately be written, allowing the data generated to be retargeted in arbitrary ways. In the case of a word counter, the test input would be far more relevant if the following generators were available:

In fact, the test itself could be simpler. By creating alternating sequences of words and splitting characters, while maintaining a count of the words put in it, a custom generator could return strings with their expected word count. The property test would only need to make sure that the program returns the expected count:

In order to write a wordy_string() generator, the property-based testing framework must offer some primitives. Before optimizing the generator, though, it would be useful to know how to measure how good their distribution is, and then see the primitives themselves.

To turn weak generators into better one first requires to demonstrate that the generator is weak. Analysis of the data set for each execution is required. The simplest way is to call proper_gen:pick() on a generator and look at the results. This tends to be cumbersome and time consuming.

Generators can nearly all be parametrized by an implicit size parameter handled by the framework. The size parameter is initially small, but as the number of test runs increase, the value is grown. The end objective is to start with simpler generated data and progressively add to their complexity.

Controlling the size manually can therefore be useful if highly complex data is what appears to trigger property failures. In such cases, the function resize(Size, Generator) can be called over an existing generator to using a given size value. The Size value must be a positive integer.

The prop_collect2() function introduced in Generator Distribution has a collector that would report the byte size of generated binaries, with the following result set:

By resizing the generator to some arbitrary value, the data size can be increased:

The value 150 was chosen by trial and error. Running the test again reports new statistics:

The data sizes are still varied; they are not fixed to exactly 150, but where 80% of the results were 20 bytes of smaller before, the vast majority of them now sits well above 40 bytes in size.

One caveat of using the resize function with static values is that some of the variability of result sets is possibly lost. It is possible that both smaller or larger sizes would prompt interesting results. Another caveat is that relative sizing of various elements becomes cumbersome. For example, the following property test’s generator requires shorter names than biographies:

Because size requirements vary between elements of the generator, multiple resize/2 function calls must be made and kept synchronized. Instead, PropEr and Quickcheck introduce a macro of the form ?SIZED(VarName, Expression), which introduces the variable VarName into the scope of Expression, bound to the 'size' value for the current execution.

In this property, the bio string is desired to be 35 times larger than the currently chosen size. Doing this allows to scale both the name and the bio relative to each other without imposing an anchor in terms of absolute size. This, in turn, provides some additional flexibility for the framework to do what it should:

This shows both names and bios tending to test over a much wider range of sizes, and in a less brittle manner.

All the generators shown until now were simple sequences of function calls, returning abstract pieces of data to be evaluated later. Since the generators return data, they can be wrapped into arbitrary functions. This allows to turn properties like this:

Into this one:

Any generator can be defined as such a function.

In the course of writing generators, one of the common patterns is the need to generate data types that cannot simply be described in basic Erlang terms. For example, one may want to generate a queue of key/value pairs, using the queue module. Just using tuples and lists won’t be enough to enforce the internal constraint of the data structure.

One simple implementation may be to first generate a list of key/value pairs, and then turn them into queues inside the property expression:

This makes reuse a bit annoying, since every property requiring a queue would then need to coordinate between its generator and priming them partly before the actual test. A better approach would be to use the ?LET(InstanceOfType, TypeGenerator, Transform) macro to apply a transformation to the generated dataset itself, without preventing the generator from being composable with others:

Since the macro returns a generator itself (rather than an actualized data type), this allows for much nicer usage overall, since composability is preserved.

A common trait to all default generators is that they’re pretty broad in data they generate, and from time to time, there’s specific counterexamples that need to be excluded. Using the non_empty() generator to remove empty lists or binaries from the generated data set is one example of this. Such a filter generator can be implemented with the ?SUCHTHAT(InstanceOfType, TypeGenerator, BooleanExp):

If any term L is an empty list or binary, the generated data will be discarded and generation will try a gain.

Similarly, generating a list of even or uneven numbers could be done by using ?SUCHTHAT() macros:

However, in this specific case it may be faster to use a transform to get there:

Since these transforms can generate correct data on the first try every time, they will be more efficient than using ?SUCHTHAT. One should be on the lookout for possibilities of reworking a restriction into a transformation.

Another thing to note is that not just guard expressions are accepted in SUCHTHAT macros. For example, the following generator looks for ISO Latin-1^[1] strings by using the io_lib:printable_latin1_list/1 function:^[2], which will let us restrict down the range of string().

The problem in this case is that for such generators as string(), the search space is large enough to be fairly ineffective. Generation can become long or fail too often. It then becomes interesting to look at building custom generators probabilistically.

Most of the default generators introduced in Default Generators tend to strictly return specific data types and are fairly straightforward. A few of them can be parametrized to influence how they work, and one of them specifically sees enough usage to warrant its own section.

The frequency([{N, Types}]) generator is special in that it allows probabilistic distributions of various types and sizes, and ends up being used in a lot of custom generators to allow such control.

The frequency/1 generator will prove especially useful when designing generators for complex data types where a uniform distribution is not desirable. Looking back at strings, the frequency of some characters could be used to make them look more 'realistic'.

When we were playing with the various string representations, we mentioned that it was often tricky to get values that were representative of real text: just using string() tended to yield a lot of control characters, codepoints that were very "out there", and very little in terms of latin1 or ASCII characters regular people are used to. This can be remediated with frequency/1:

This generator produces something a lot closer to realistic text than what is obtained through string(). By tweaking the frequency numbers, specific characters can see their probability raised or lowered as required to properly exercise a property. A CSV^[3] parser would likely require a higher frequency of quotation marks, commas, and line breaks than what would usually be obtained, to properly exercise expected corner cases, for example.

Many generators require step-wise decision making, and others require a repetitive structure in order to work. Recursion is well-indicated for those, but recursive generators have a few tricks to consider.

The word count problem was specifically well-tailored for a recursive generator. The skeleton property looked like this:

Where wordy_string() generator returns fully formed bits of text with a count of the number of words in it. To generate the text itself, two generators are required:

With an implementation such as:

In order to be able to produce a word count, the step-by-step approach will require generating a word, then a splitter, increment the count, and then repeat again, and so on for the entire string. This form of iteration is a perfect fit for a recursive generator. All recursive functions have the same fundamental elements:

In the case of a wordy string, this could look like:

This would, through a probabilistic and recursive generator, produce strings of around 50 words. The base case returns the generated fragments and count with a probability of 1 and the recursive case adds to both with a probability of 50. Changing the weight of either clauses would impact string length. The generator is not quite right though, because it will generate a list of string fragments, not the flat list required for a true string. The LET macro can be used for this:

Running this generator would unfortunately never terminate. This is because the wordy_string/2 generator calls itself, and Erlang is an eager language. Before frequency/1 has the time to make a selection, the recursive call to wordy_string/2 has already taken place. This will repeat indefinitely (or at least until memory runs out).

For those specific case, QuickCheck and PropEr have a ?LAZY() macro, which allows to defer the evaluation of an expression until it is actually required by the generator. This yields the final form:

Note that a wrapper has also been added so that the generator can be called without arguments. With this and the laziness, the generator can then take a time proportional to the length of the generated string to run, rather than infinity:

This is a funny definition of 'words', but then again, those are what the specifications are. Property-based testing is a great way to reveal how woefully underspecified a software project is. With the generator plugged into the previous prop_word_count definition, the property can now run for an arbitrary number of tests:

The distribution of words to spaces should now be much higher than before. The same kind of logic can be applied to any recursive data structure.

The one caveat with this approach is that probabilities for recursion can mean two things:

In some cases, the potential variation for very large or very small structures will be desirable, but not always. Recursive functions can be written the way they usually are in Erlang, using a marker such as a counter that decrements to know when to stop:

Because the structure is flat, the ?LAZY macro can be avoided. All that is needed is a wrapper to seed the size parameter. It turns out that the ?SIZED macro is perfect for this:

The variability of the data structure size will then be managed by the framework. Someone wanting smaller or larger lists could still use resize/2 to impact the total size.

In the case where the generator is used to build a given data structure through side-effects, such as sequences of writes to a file, filling an ETS table, or just sending messages to a process, a counter-example built from regular or recursive generators will look like on opaque structure. In the case of a file or socket, it may be a 'Port' data type (#Port<0.123>—Erlang’s file descriptors, more or less); in the case of ETS, an integer table identifier; for a process, a Pid (<0.123.0>). Another difficult to read type of generator is for data structures that are too abstract in representation to reasonably make sense out of them from the printout. In such cases, digging for the bad state would be cumbersome, possibly requiring temporary debugging code to be added to the property since the opaque data type shows no relevant information. Symbolic calls can help.

A symbolic call is just a special notation for function calls that can be used in generators. Rather than executing the operation straight away, the calls are built up as a data structure. Once the generator materializes, then they get executed at once. The notation is {call, Module, Function, ArgumentList}:

By using this format, calls can be made simpler when shrinking:

In properties, they look like:

The first one is the standard generator, giving a standard dictionary. The second one uses symbolic calls, which require calling eval(Gen) ^[6] to execute the symbolic calls into the final dictionary. The automated symbolic calls are automatically evaluated when being returned.

The three properties are set up to fail, to show the difference on shrinking:

With the help of symbolic calls, generated terms that are too obscure (whether through their complex size or because side-effects hide the way the state was obtained) can be made simpler to debug.