Flaky Pytest Tests

Especially when you're dealing with machine learning algorithms, you may end up writing unit tests that have some element of randomness in them. Many algorithms have a random element in them, so you cannot always guarantee what happens.

Unfortunately, this can lead to some flaky tests. So let's talk about some tricks that you might be able to apply here.

Example

The code below simulates a test that fails randomly.

import random

def test_random_situation():
    assert random.random() > 0.05

Let's suppose that you've seen it fail in Github Actions, but it only fails sometimes. What might you do to investigate?

One thing you could do locally is to run this with pytest.mark.paramtrize.

import pytest
import random

@pytest.mark.parametrize("i", range(100))
def test_random_situation(i):
    assert random.random() > 0.95:

This way, you'll run this function 100 times, which will give you an impression of how often it might fail.