PropEr Testing

The simplest way to shrink is through the ?SHRINK(Generator, [AltGenerators]) macro. This macro allows the specification of the actual generator as usual, but also allows the specification of alternative generators to be used when trying to reduce the size of the data structure.

This is useful when there are multiple variations in data that may or may not be the source of failures, and there are simple ways to encode safer patterns through the alternative generators.

For example, when dealing with time, January 1st 1970 is a more common and meaningful zero value than the year 0 itself (never mind that year zero does not necessarily exist in the first place). When dealing with a date and time piece of data, re-centering the 'zero' towards that epoch may be practical.

Let’s take a look at the following set of generators used to create strings of the form "1997-08-04T12:02:18-05:00", in accordance with RFC 8601. This set of generators will be centering its shrinking efforts towards January 1st January 1970. Also note that the RFC is somewhat lax and allows (or rather, does not forbid) the notation of a timezone that is +99:76, even if that is nonsensical from a human perspective.

Various forms of shrinking are used: years will be generated from year 0 to 9999, but will start shrinking on errors with ranges between 1970 and 2000, and if that fails, 1900 to 2100. Multiple alternative generators can be used.

Similarly, timezones will look for 00-99 for their ranges, but will try to settle on hours between 0 an 14 (expected human ranges, not just formatting oddities) and minutes that match currently existing timezones.

You can see a sample of how shrinking would work by calling the sampleshrink function:

In practice, shrinking will be done less linearly than this. A given attempt at shrinking that fails to create another failing case won’t be used. If a test failed on every month of July, the shrinking would end up looking like "1970-07-01T00:00:00+00:00".

A more complex way to shrink is through the ?LETSHRINK([Pattern, …​], [Generator, …​], Expression) macro. The macro always requires patterns and generators to be in lists. One example usage may look like:

In this expression, the generators A, B, and C are merged together into one with A ++ B ++ C, much in the same way we would do it in a regular ?LET macro. The difference is that if the property-based testing framework has to generate a simpler term when shrinking, it will instead pick just one of A, B, or C without applying the transformation and return that directly.

?LETSHRINK is particularly appropriate for recursive structures, data made through branching, and just all kinds of pieces of data that are generated by smashing others together and applying transformations, since taking a part of it is a legitimate way to get a simpler version.

The most common form of ?LETSHRINK is the one to build binary trees. A generator for binary trees whose size is directed by its argument N would look like:

Running sampleshrink/1 on it reveals that shrinking always takes place on the contents of the tree, but not its structure (since the size is bound):

Each of the five trees above contain exactly 4 entries, and all the integers tend to 0, but the tree structure itself is fixed in size. The only way to get the tree to shrink is through parametrizing it with the Size variable obtained from the ?SIZED(Var, Exp) macro. But if the failure require internal tree elements to be large while the tree structure itself is small, then there’s few chances that shrinking by size only will work well.

Here’s the shrink-friendly version:

By building individual branches through the ?LETSHRINK macro and assembling them in there, the framework is able to prune parts of the trees it would detect as irrelevant to a test case failure. This allows to generate much bigger trees to initially find bugs, without losing clarity when getting a good counterexample:

Rather than a constant tree size, each shrink subsequently makes the tree smaller (if the property still fails). This helps in figuring out precisely why a test failed with more accuracy: if the tree size is at play, then it will remain large. If the tree contents are at play, then a smaller tree with a better focus on the bad elements will be generated.

In most cases, for a framework like PropEr or Quickcheck, adding shrinking instructions is not something that needs to be done as part of writing the generator the first time around. Rather, it’s something that will be worth doing once a confusing counter-example is found and the minimal counter-example given by the framework is not understandable on its own.

As with other property-based testing debugging practices, the work will be iterative and a bit explorative. The generator, its shrinking, the test cases, and our understanding of the program itself will all get to improve as we discover the hidden properties embedded in the code that was written.