pytest.raises excinfo subtlety

In the past few weeks I've seen multiple stumbles related to a subtlety of pytest. I'm going to explain how to recognize the issue and what to do about it.

Exceptions are an aspect of a Python package's API, just like the names of the functions, their parameters, and their return types. They also require testing. Pytest provides a handy context manager for this: pytest.raises.

def accept_numbers_lt_3(num):
    if not num < 3:
        raise ValueError("Number is not less than 3")


def test_unaccepted_error():
    with pytest.raises(ValueError):
        accept_numbers_lt_3(3)

That test will pass. The function does raise a ValueError when called with the argument 3.

To make assertions about details of the exception you might try the following.

def test_unaccepted_error_msg():
    with pytest.raises(ValueError) as excinfo:
        accept_numbers_lt_3(3)
        assert str(excinfo.value) == "Number is not less than three"

This test passes too. But wait, we mistyped the expected string. We're asserting that the message ends with "three" and that can't be true, can it? How did this test pass?

Here's the important thing: pytest "magically" changes the interpretation of assert statements, but it doesn't change the behavior of Python's "with" statements. That final assert statement in test_unaccepted_error_msg is never reached. Execution exits from the block when the ValueError (or any other exception) is raised.

The excinfo object has recorded the captured exception so that we can inspect it. We only need to move that assert statement to after the with block.

def test_unaccepted_error_msg():
    with pytest.raises(ValueError) as excinfo:
        accept_numbers_lt_3(3)

    assert str(excinfo.value) == "Number is not less than three"

Now this test will fail, properly. And we can change "three" to "3" and have a passing test of an exception message.

This issue is documented in a note in the pytest.raises docs, but is easy to overlook.