PropEr Testing

The exercise asks to implement a checkout counter that can give the price of items scanned, with the caveat that some items are cheaper per unit if they are bought in bulk. The following sample table is given:

The checkout function should accept items in any order, so that the sequence ["B","A","B"] would give the 2 for 45 rebate for the Bs being bought.

The only call that needs to be exposed is checkout:total(ListOfItemsBought, UnitPrices, SpecialPrices) → Price.

The requirements are vague, but the following can be extracted:

The rest is seemingly undefined or underspecified.

Some properties that can be extracted from the requirements include:

The first test will require to count the total for the checkout without specials. To avoid a test definition such as sum(ItemList, PriceList) =:= checkout:total(ItemList, PriceList, []), which has the test doing roughly the same thing as the implementation, a generator will be written to return the expected price so that the property is not too circular:

The generator for the property will need to generate the three expected arguments: a list of items bought by the customer (ItemList), the expected price of those items, and then the list of items with their prices as expected by the register itself (PriceList). Since the price list is required to generate the item list and expected prices, the generator will need to come in layers with ?LET macros:

The price list itself is a list of tuples; the list is actualized through the ?LET macro so that the item_list/1 generator can use it as the actual Erlang data structure rather than the abstract intermediary format PropEr uses:

The item_list/1 generator just needs to pick one item from the list at a time, and add their prices together:

Running the tests will show them to fail. A fix for it is a basic implementation:

Which passes successfully.

To validate generators, using the collect/2 function to gather metrics is a good way to start:

Revealing the metrics:

This appears reasonable. Alternatively running the call with collect(lists:usort(ItemList), <Property>) shows good results as well.

The TODO list looks like this at this point:

The special prices property is the last one left to check, but it is a more complex one. The way the existing properties have been written means they should not require changes for this phase of the project, and a new one can be added instead:

The property is similar to the one before, but expects a fourth generated term, which is a list of special prices (SpecialList). Generating the list should be done dynamically, but it will need to rely on the price list to work. That may just take one more ?LET nested into the structure to work. The challenge will be to properly generate the expected price while accounting for specials and while ensuring non-special items keep working in a mixed mode.

A generator that does not pay attention to that mixed mode would create problems. For example, if the checkout counter sells 3 donuts for the price of 2, but the generator created a list of 4 donuts without expecting a special, a correct implementation would account for them and the test would fail. Not a problem since tests can be corrected, but better realize it sooner than later.

An interesting work-around can be discovered by looking at how lists are being generated. As long as a given type of item is generated in a quantity smaller than required for the special price, the special won’t trigger. Similarly, as long as the items are generated in an even multiple of the quantity required for a special, the special will trigger. It follows that both types of lists will never clash even if they are generated independently, and yet they will still merge fine. If a list of 2 donuts at a non-special price is generated, and then a list of 6 donuts (twice the special quantity) is generated for the specials, then a list containing 8 donuts and the sum of both expected prices remains correct.

This should work fine. The generator should therefore have this kind of flow:

Giving the following generator:

The last two steps can be done within the same ?LET expression:

The last two steps can be combined in a single ?LET. There are also three generators used within this one that are new (the item list is being reused from the previous section). The first one is special_list:

The chosen format for the list of specials contains the name of the item, how many are required for the special to start counting, and the price for the whole group. The specials are generated from randomly selected (and deduplicated) entries from the item list so that there is always a matching non-special item to any special one. Note that it is possible for the special price to be higher than the non-special price with this generator.

The two other generators are those having to do with generating lists composed strictly of items that are not on special, and then strictly of items that are on special. The first one follows:

The specials generator is a bit simpler, although it is based on the same model:

The major difference is that it only needs to care about the list of specials, and can return any number of matching items as long as it’s a multiple of the specials count. Running the property unsurprisingly yields a failure:

The tests fail whenever a list of specials is passed to the function. Implementing the specials accounting should help. A simple way to do it can be to count how many of each items are in the list. Account for the specials first, reducing the count every time the specials apply, and then run the list over the items that are not on sale:

The three helper functions are as follows:

The last function simply sums up the cost of leftover items, without specials. The property passes, and the full check list is covered:

Aside from looking at the distribution of results with collect/2 and aggregate/2, a few more tools can be interesting as a way to ensure the tests are exercising code properly. The first of them is code coverage.

Code coverage is simply a measure of how many lines of code have been executed during the test run, in order to get an idea of which areas of the code base might be under-tested. For property-based testing with PropEr, metrics can be obtained by calling:

100% coverage always looks good, but isn’t necessarily so. Lines of code can be executed in ways where they always run in the same manner even though alternative execution flows are possible on each line. Again, coverage should not be seen as a way to figure out whether everything is tested, but rather as a way to identify whether some parts of the code base are definitely untested.

A fairly efficient and simple way to gauge the quality of properties on top of the other methods seen is to willingly introduce a bug in the code base, and then seeing whether the tests can find about it. There will be a surprising number of times where replacing a + with a - or a < with a > will seemingly have no impact on the outcome of tests, which can be worrisome. Figuring out if the code is dead or the tests woefully inadequate is a worthwhile exercise in such cases.

An interesting aspect of the code and tests so far is that the list of items being bought is generated through the list of supported items. It means that by the virtue of having designed the test case from the price list, there is a lack of tests looking for unexpected usage. Such tests try to exercise specifically underspecified paths rather than only well-defined ones.

Rather than generalizing the current properties and making them more complex, it is a good idea to add specific wide-reaching properties that validate different aspects of the code. This also avoids searching only for expectable problems the way it is usually done with traditional tests: limitations to this approach are those of the developer writing the tests, which is underwhelming. Instead, a good idea is to take a fuzz-like approach and to throw a lot of reasonable-looking garbage at the system to see what happens.

Broad properties are not too useful on their own, but as a supplement they’re great. The basic property looks like:

The only property it enforces is "the function returns an integer and does not crash in unexpected ways". A good way to start is through permissive generators and explore from there:

Running the property returns an early failure:

The PropEr representation of strings is lacking, but if a single item is used (with name ""), then the lookups fail. The price of an unknown item cannot be used in the calculation, and the function bails out. This is fine, but could give a clearer exception. A programmer error could be raised instead, alerting to the true nature of the problem:

With the attached patch:

The bug is fixed, and running the property again reveals no more bugs. Before claiming victory, validating the distribution of generators is required. Looking at the distribution of results with collect/2 will shine some light. Gathering metrics for the error case at hand can be done with collect(lists:all(fun(X) → proplists:get_value(X, PriceList) == undefined end, ItemList), Property), and it reveals that 92% of all test runs contain no overlap between the item list and the price list, meaning they all test the case recently fixed.

With the current properties, the view represented is:

Right now, the positive properties give a view of a, and the negative one shows b, but not necessarily much on the gradient in-between. A trick for such cases is to make a kind of 'hybrid' generator that has some constraints, but also is allowed to err. This can be done by mixing expected cases:

Using this generator ensures that some known items and some unknown items will be there, but not always. Running it fortunately reveals a bug:

Whenever a special requires 0 items, a division by zero error happens. It’s surprising that it does not happen in other properties, but at least it is caught in one. Fixing it will require validating the specials. First the property, and then the code:

Finally, the bugs stop cropping up. There are still potential problems though. The space between a and b is still wide to unknown extents. An easy sanity check is to relax the constraints made in the requirements planning (in bold) to figure out what happens when this breaks:

Those are three constraints to relax. The first one can be easy to verify by relaxing the existing generators:

This modification ends up killing one of the properties real fast. Duplicates are not supported, but the generator implementation hid that fact. These assumptions are baked into the model, but not necessarily visible to the user. In fact it would be nice to let them know if the list is valid without needing to buy anything. A quick property can be added to check it:

And an even simpler function can be written to check it:

If a list that sorts and removes duplicate keys has a different length from the original one, the list is invalid since it had to contain dupes. The validator can be wired up inside the main function to avoid accidentally miscalculating prices (better safe than sorry):

And all of a sudden, the tests fail!

The test can be fixed by adding a handler for the same kind of error, and the same anti-duplicate test can also be added for the special list:

And the matching code:

Everything now passes, and users can validate their inputs before running the code.

Even then, more digging could be done. That was just one of three assumptions being tested. Relaxing the other properties could also reveal interesting things:

The spec was very vague though, so those discoveries would have to be discussed with stakeholders to figure out what is acceptable or not before the code hits production. A strict interpretation of the specification will mean the implementation is sufficient. A lax one will find it exploding for multiple cases that may or may not be preposterous.

Of course, in a statically typed language (or in the case of Erlang, with Dialyzer’s type analysis), the strict interpretation of the spec is the only one accepted. The three potential bugs above do not even register as a possibility, and the current state of affairs is very likely acceptable. If strict assumptions are made, and that they are being enforced by manual checks, the compiler, and/or static code analysis, then the program should not get into unexpected states. It may be inflexible but it will not be wrong. At least not on these bugs.

This chapter went through three types of tests in order: regular positive tests, tests that aim to look at broad cases to find unexpected bugs, and negative tests, designed ot check the reliability of the suite. In practice there should be no need for such a clear cut between phases; doing all of them iteratively and continuously tends to work very well. Waiting late for negative testing could mean too many tests are not actually useful. Testing and validating hypothesis early and as frequently as possible is never a bad idea. Looking for broad cases early can help identify unexpected problems that prove a design wrong; finding mistakes early is cheaper than finding them late since less code has been written in the wrong direction.

Property-based testing is a very useful tool to do exploration of what code actually does. It is a worthwhile exercise to start poking at things broadly and to then narrow down the results of generators. It is also worthwhile to start narrow and then widen the generators. The order does not matter, but exploring does.