 pytest tricks: xfail

# Xfail in Pytest

Let's now consider a `normalize` function one more time. We're about to test it.

``````import pytest
import numpy as np

def normalize(X):
X = X.astype(np.float)
return (X - X.min())/(X.max() - X.min())

@pytest.fixture(params=[(1,1), (2,2), (3,3), (4,4)], ids=lambda d: f"rows: {d} cols: {d}")
def random_numpy_array(request):
return np.random.normal(request.param)

def test_min_max_normalise(random_numpy_array):
X_norm = normalize(random_numpy_array)
assert X_norm.min() == 0.0
assert X_norm.max() == 1.0
``````

The interesting thing here is that this test will fail, depending on your version of numpy! As of version 1.20 the `np.float` and `np.int` aliases have been deprecated.

## Conditional Skip

One way of dealing with this is that you might consider skipping this test via `@pytest.mark.skipif` depending on the version of Numpy.

``````import pytest
import numpy as np

def normalize(X):
X = X.astype(np.float)
return (X - X.min())/(X.max() - X.min())

@pytest.fixture(params=[(1,1), (2,2), (3,3), (4,4)], ids=lambda d: f"rows: {d} cols: {d}")
def random_numpy_array(request):
return np.random.normal(request.param)

@pytest.mark.skipif(np.__version__ > "1.20.0", reason="new numpy version breaks this API")
def test_min_max_normalise(random_numpy_array):
X_norm = normalize(random_numpy_array)
assert X_norm.min() == 0.0
assert X_norm.max() == 1.0
``````

But in this case, this may not be that relevant because you may not be testing against multiple Numpy versions.

## Xfail

Instead of skipping, we can also declare this test to be something that we expect to fail via `@pytest.mark.xfail`.

``````import pytest
import numpy as np

def normalize(X):
X = X.astype(np.float)
return (X - X.min())/(X.max() - X.min())

@pytest.fixture(params=[(1,1), (2,2), (3,3), (4,4)], ids=lambda d: f"rows: {d} cols: {d}")
def random_numpy_array(request):
return np.random.normal(request.param)

@pytest.mark.xfail(reason="this needs to be fixed once we support Numpy > 1.20")
def test_min_max_normalise(random_numpy_array):
X_norm = normalize(random_numpy_array)
assert X_norm.min() == 0.0
assert X_norm.max() == 1.0
``````

This way, you can declare a failing test that is failing now but that you may want to fix later. In this example the change would be so simple that it wouldn't be worth adding an `xfail`. Sometimes though, especially in a larger project with many dependencies, you might be dealing with a known failure that should not be skipped, but should get adressed. Note that you can also add an extra setting.

• `pytest.mark.xfail(strict=True)` will complain when the test actually passes
• `pytest.mark.xfail(strict=False)` will allow the test to pass, even when the test does not fail

## xfail vs. skip

You might wonder about the difference between `xfail` and `skip` at this point, because they certainly do similar things. In general:

• `skip` is meant for situations where you only want the test to run when certain conditions are met. You can choose to skip a test alltogether, but most typically it's meant to skip tests depending on operating systems, like having some Windows-only tests.
• `xfail` means that you can really expect the test to fail. So if a feature is not yet implemented, or if a bug can't be fixed right now, that's when an `xfail` would be a more appropriate convention.

Note that more information on skipping and xfailing can be found on the pytest documentation.