Sean Gillies (Posts about pytest)

Use pytest's tmp_path fixture

Sean Gillies — Tue, 05 Nov 2024 23:56:14 GMT

If you're not already using pytest's tmp_path fixture, you really should. The fixture provides a temporary directory for testing use. A directory that you can't reference from other test runs or from other test functions in the same run, but that isn't immediately deleted when your tests finish. The directory is created in your account temporary location and is eventually cleaned up by your computer's operating system. Until then, you can open the directories and their files in other applications.

I've been making assertions on datasets in Rasterio's tests and also dragging them into QGIS for a closer look after the tests finish.

pytest.raises excinfo subtlety

Sean Gillies — Tue, 31 Mar 2020 01:45:36 GMT

In the past few weeks I've seen multiple stumbles related to a subtlety of pytest. I'm going to explain how to recognize the issue and what to do about it.

Exceptions are an aspect of a Python package's API, just like the names of the functions, their parameters, and their return types. They also require testing. Pytest provides a handy context manager for this: pytest.raises.

def accept_numbers_lt_3(num):
    if not num < 3:
        raise ValueError("Number is not less than 3")


def test_unaccepted_error():
    with pytest.raises(ValueError):
        accept_numbers_lt_3(3)

That test will pass. The function does raise a ValueError when called with the argument 3.

To make assertions about details of the exception you might try the following.

def test_unaccepted_error_msg():
    with pytest.raises(ValueError) as excinfo:
        accept_numbers_lt_3(3)
        assert str(excinfo.value) == "Number is not less than three"

This test passes too. But wait, we mistyped the expected string. We're asserting that the message ends with "three" and that can't be true, can it? How did this test pass?

Here's the important thing: pytest "magically" changes the interpretation of assert statements, but it doesn't change the behavior of Python's "with" statements. That final assert statement in test_unaccepted_error_msg is never reached. Execution exits from the block when the ValueError (or any other exception) is raised.

The excinfo object has recorded the captured exception so that we can inspect it. We only need to move that assert statement to after the with block.

def test_unaccepted_error_msg():
    with pytest.raises(ValueError) as excinfo:
        accept_numbers_lt_3(3)

    assert str(excinfo.value) == "Number is not less than three"

Now this test will fail, properly. And we can change "three" to "3" and have a passing test of an exception message.

This issue is documented in a note in the pytest.raises docs, but is easy to overlook.

Debugging temporary files using pytest autouse fixtures

Sean Gillies — Fri, 13 Sep 2019 23:10:48 GMT

This week I discovered that Rasterio doesn't always close the temporary in-memory datasets that are used within some of its methods. In testing Rasterio's WarpedVRT class I used a GDAL function to dump descriptions of all open datasets and found a bunch that looked unrelated to WarpedVRT. They were GDAL "MEM" type datasets with UUIDs for names, which didn't tell me much. What were their origins?

They have UUIDs for names because Rasterio imports uuid in its _io module and calls uuid.uuid4() to make temporary dataset names. If only the dataset name included the name of the test in which it was created, then I'd have an entry point into debugging. One way to do this is with a pytest auto-used fixture.

I changed the rasterio._io module's import statement from import uuid to from uuid import uuid4 to make it slightly easier to monkey patch and then I added 5 lines of code to Rasterio's conftest.py file:

@pytest.fixture(autouse=True)
def set_mem_name(request, monkeypatch):
    def youyoueyedeefour():
        return "{}-{}".format(request.node.name, uuid.uuid4())
    monkeypatch.setattr(rasterio._io, "uuid4", youyoueyedeefour)

This set_mem_name fixture uses two standard pytest fixtures: request and monkeypatch. The value of request.node.name is the name of the test and this set_mem_name fixture uses monkeypatch to replace uuid4 in rasterio._io with a custom function that prepends the name of the test to the UUID. The autouse=True argument tells pytest to add this fixture to every test it finds. I didn't need to touch the code of any of Rasterio's tests, not a one.

This quickly revealed to me that the unclosed temporary datasets were coming from tests that asserted certain exceptions were being raised by Rasterio's reprojection code. This code used temporary datasets and didn't close them before raising the exception to the caller. Once I changed the code to do the following, Rasterio no longer leaked datasets from those tests, or in our programs.

try:
    ...
    if condition:
        raise CRSError("error")
    ...
except:
    temp.close()
    raise

If Rasterio used only Python's unittest module, and not pytest, it would be possible to do the same thing. Import rasterio._io in the test case's setUp(), monkey patch it, and then restore it in tearDown(). If all the tests derived from one base class, it would only be necessary to extend that class. The unittest.mock module easily allows every test to be patched with a single decorator statement. It seems like it could be two fewer lines of code, but I don't immediately see how to get the name of the test and use it with only the mock.patch decorator. It looks like one would have to use a patcher's start and stop, which is back to somewhat more boilerplate than with pytest.