Testing using Property Based Tests

Quis custodiet ipsos custodes? is a Latin phrase found in the work of the Roman poet Juvenal in his Satires (Satire VI, lines 347–348). It translates to “but who will guard the guardians?”—or, in developer terms, “who tests the tests?”

This post is about property-based testing, an interesting way of testing your code by letting the computer invent troublesome inputs on your behalf. It was introduced by the QuickCheck framework in Haskell and suggests another way to test software. It is not a magic bullet—software engineering has been searching for one of those for decades—but it is a useful companion to traditional testing. It covers the same broad territory as example-based testing, from unit tests to integration tests.

In example-based unit tests you:

define some example inputs
define the expected results
you run your code and check that they match!

Usually, we test the inputs we expect. If we are feeling adventurous—and the coffee is working—we add a few edge cases that might break the function under test. But we have other work to do, and our collection of cases is rarely exhaustive. How can we be sure we have done enough? Who is guarding the guardians?

Enter Property-Based Testing (PBT), wearing a small cape. With PBT:

you describe the properties of the input
you describe the properties of the output
you have the computer try lots of generated examples and check that they do not fail
if one fails, the framework automatically shrinks it to the smallest input that still causes the failure—because debugging one tiny disaster is nicer than debugging a vast one

In Python, Hypothesis is a great property-testing library that works alongside pytest. It is essentially a tireless colleague whose favorite hobby is finding inputs you forgot existed.

We are going to use this library with a small example. We will specify properties from some very simple requirements—the inputs are floats—and Hypothesis will generate examples that try to falsify them. If it finds an error, it will shrink the offending values until it discovers the smallest failure case. Then it will present the evidence with the quiet satisfaction of a cat dropping something unpleasant on the carpet.

Simple app

But enough talking. Let us test a simple calculator app in Python. Behold: arithmetic, in all its glory.

Simple tests

Let us create a few tests for the app using pytest. Nothing exotic yet; everyone may remain calm.

We run the tests created

pytest -v

This results in:

Great! Everything passes. We are brilliant. The software is perfect. Let us absolutely not investigate any further.

But is this really the best we can do?

Of course we are going to investigate further. Let us try property-based testing with Hypothesis.

After a few changes, the test code becomes:

We run again the tests created using

pytest -v

Looking a little more closely at the pytest output, we see this:

What on earth is this trickery? Our simple calculator has somehow discovered existential doubt.

Well it just says:

any float can appear, including NaN, and in that case… BOOM 💥
the code needs to handle NaN (Not a Number), a value whose name is already trying very hard to warn us

So, all in all, it is not rocket science. It is a way of automatically trying to break your functions so you can discover new cases and protect against them. Think of it as hiring a tiny chaos engineer who works at processor speed and never asks to attend the retrospective.

You do not need to throw away your existing tests, either. Hypothesis-based tests can run alongside your current pytest suite. Your carefully handcrafted examples may stay; they are simply getting some unpredictable 🤪 new colleagues .

But I am a data scientist/engineer ! I deal with dataframes and databases every day. Little toy examples will not do!

Real data arrives in dataframes, database rows, nested records, questionable schemas, and columns whose documented type is “integer” but whose emotional type is “it depends.” Testing all that with three carefully selected examples can feel slightly optimistic.

Fortunately, Hypothesis is not limited to generating individual numbers. Its strategies can create lists, dictionaries, structured records, NumPy arrays, and entire pandas dataframes. You can control column names, data types, indexes, sizes, and missing values, then let Hypothesis explore combinations that nobody would deliberately type into a fixture.

For database-facing code, the same idea applies: generate rows or domain objects, pass them through your validation and persistence logic, and check the properties that must always hold. IDs should remain unique. Round trips should preserve values. Invalid records should be rejected. Dates should not quietly become 1970 unless that was genuinely the plan.

You describe what valid—or interestingly invalid—data looks like. Hypothesis supplies the examples. Production no longer has to be the first environment to discover that an empty dataframe, a duplicate key, or one strategically placed NaN changes the meaning of everything.

So, is PBT only a Python/Haskell thing?

Property-based testing has been implemented in many languages. Below are some examples other than the Python library used above. Apparently, the desire to make computers discover our mistakes automatically is universal.