Sean Gillies (Posts about testing)https://sgillies.net/tags/testing.atom2023-12-31T01:26:21ZSean GilliesNikolapytest.raises excinfo subtletyhttps://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html2020-03-30T19:45:36-06:002020-03-30T19:45:36-06:00Sean Gillies<p>In the past few weeks I've seen multiple stumbles related to a subtlety of
pytest. I'm going to explain how to recognize the issue and what to do about
it.</p>
<p>Exceptions are an aspect of a Python package's API, just like the names of the
functions, their parameters, and their return types. They also require testing.
Pytest provides a handy context manager for this: <a class="reference external" href="https://docs.pytest.org/en/latest/reference.html#pytest.raises">pytest.raises</a>.</p>
<div class="code"><pre class="code python"><a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-1" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-1" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-1"></a><span class="k">def</span> <span class="nf">accept_numbers_lt_3</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-2" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-2" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-2"></a> <span class="k">if</span> <span class="ow">not</span> <span class="n">num</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-3" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-3" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-3"></a> <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Number is not less than 3"</span><span class="p">)</span>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-4" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-4" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-4"></a>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-5" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-5" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-5"></a>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-6" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-6" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-6"></a><span class="k">def</span> <span class="nf">test_unaccepted_error</span><span class="p">():</span>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-7" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-7" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-7"></a> <span class="k">with</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">):</span>
<a id="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-8" name="rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-8" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_4f7f8a4d4c5a485e976c6bd7528a41f0-8"></a> <span class="n">accept_numbers_lt_3</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
<p>That test will pass. The function does raise a ValueError when called with the
argument 3.</p>
<p>To make assertions about details of the exception you might try the following.</p>
<div class="code"><pre class="code python"><a id="rest_code_2730e636745f41068a8321064f6ed451-1" name="rest_code_2730e636745f41068a8321064f6ed451-1" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_2730e636745f41068a8321064f6ed451-1"></a><span class="k">def</span> <span class="nf">test_unaccepted_error_msg</span><span class="p">():</span>
<a id="rest_code_2730e636745f41068a8321064f6ed451-2" name="rest_code_2730e636745f41068a8321064f6ed451-2" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_2730e636745f41068a8321064f6ed451-2"></a> <span class="k">with</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">)</span> <span class="k">as</span> <span class="n">excinfo</span><span class="p">:</span>
<a id="rest_code_2730e636745f41068a8321064f6ed451-3" name="rest_code_2730e636745f41068a8321064f6ed451-3" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_2730e636745f41068a8321064f6ed451-3"></a> <span class="n">accept_numbers_lt_3</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<a id="rest_code_2730e636745f41068a8321064f6ed451-4" name="rest_code_2730e636745f41068a8321064f6ed451-4" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_2730e636745f41068a8321064f6ed451-4"></a> <span class="k">assert</span> <span class="nb">str</span><span class="p">(</span><span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"Number is not less than three"</span>
</pre></div>
<p>This test passes too. But wait, we mistyped the expected string. We're
asserting that the message ends with "three" and that can't be true, can it?
How did this test pass?</p>
<p>Here's the important thing: pytest "magically" changes the interpretation of
assert statements, but it doesn't change the behavior of Python's "with"
statements. That final assert statement in test_unaccepted_error_msg is never
reached. Execution exits from the block when the ValueError (or any other
exception) is raised.</p>
<p>The excinfo object has recorded the captured exception so that we can
inspect it. We only need to move that assert statement to after the with block.</p>
<div class="code"><pre class="code python"><a id="rest_code_367d42a5be2c4c278060eebd0ea51dfc-1" name="rest_code_367d42a5be2c4c278060eebd0ea51dfc-1" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_367d42a5be2c4c278060eebd0ea51dfc-1"></a><span class="k">def</span> <span class="nf">test_unaccepted_error_msg</span><span class="p">():</span>
<a id="rest_code_367d42a5be2c4c278060eebd0ea51dfc-2" name="rest_code_367d42a5be2c4c278060eebd0ea51dfc-2" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_367d42a5be2c4c278060eebd0ea51dfc-2"></a> <span class="k">with</span> <span class="n">pytest</span><span class="o">.</span><span class="n">raises</span><span class="p">(</span><span class="ne">ValueError</span><span class="p">)</span> <span class="k">as</span> <span class="n">excinfo</span><span class="p">:</span>
<a id="rest_code_367d42a5be2c4c278060eebd0ea51dfc-3" name="rest_code_367d42a5be2c4c278060eebd0ea51dfc-3" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_367d42a5be2c4c278060eebd0ea51dfc-3"></a> <span class="n">accept_numbers_lt_3</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<a id="rest_code_367d42a5be2c4c278060eebd0ea51dfc-4" name="rest_code_367d42a5be2c4c278060eebd0ea51dfc-4" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_367d42a5be2c4c278060eebd0ea51dfc-4"></a>
<a id="rest_code_367d42a5be2c4c278060eebd0ea51dfc-5" name="rest_code_367d42a5be2c4c278060eebd0ea51dfc-5" href="https://sgillies.net/2020/03/30/pytest-raises-excinfo-subtlety.html#rest_code_367d42a5be2c4c278060eebd0ea51dfc-5"></a> <span class="k">assert</span> <span class="nb">str</span><span class="p">(</span><span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"Number is not less than three"</span>
</pre></div>
<p>Now this test will fail, properly. And we can change "three" to "3" and have
a passing test of an exception message.</p>
<p>This issue is documented in a note in the pytest.raises docs, but is easy to
overlook.</p>Testing PySpark applicationshttps://sgillies.net/2020/03/29/testing-pyspark-applications.html2020-03-29T14:16:05-06:002020-03-29T14:16:05-06:00Sean Gillies<p>My colleague Kuan Butts has written a <a class="reference external" href="http://kuanbutts.com/2020/03/28/pyspark-unit-testing/">blog post about unit testing patterns
for PySpark</a>. You
should subscribe to his blog if you're interested in transit networks, graphs, and
Python.</p>Debugging temporary files using pytest autouse fixtureshttps://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html2019-09-13T17:10:48-06:002019-09-13T17:10:48-06:00Sean Gillies<p>This week I discovered that Rasterio doesn't always close the temporary
in-memory datasets that are used within some of its methods. In testing
Rasterio's WarpedVRT class I used a GDAL function to dump descriptions of all
open datasets and found a bunch that looked unrelated to WarpedVRT. They were
GDAL "MEM" type datasets with UUIDs for names, which didn't tell me much. What
were their origins?</p>
<p>They have UUIDs for names because Rasterio imports uuid in its _io module and
calls <code class="docutils literal">uuid.uuid4()</code> to make temporary dataset names. If only the dataset
name included the name of the test in which it was created, then I'd have an
entry point into debugging. One way to do this is with a <a class="reference external" href="https://docs.pytest.org/en/latest/fixture.html#autouse-fixtures-xunit-setup-on-steroids">pytest auto-used
fixture</a>.</p>
<p>I changed the rasterio._io module's import statement from <code class="docutils literal">import uuid</code> to
<code class="docutils literal">from uuid import uuid4</code> to make it slightly easier to monkey patch and then
I added 5 lines of code to Rasterio's <code class="docutils literal">conftest.py</code> file:</p>
<div class="code"><pre class="code python"><a id="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-1" name="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-1" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_3dbb9bcf72ca45ec80304496ab4709ef-1"></a><span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span><span class="p">(</span><span class="n">autouse</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<a id="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-2" name="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-2" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_3dbb9bcf72ca45ec80304496ab4709ef-2"></a><span class="k">def</span> <span class="nf">set_mem_name</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">monkeypatch</span><span class="p">):</span>
<a id="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-3" name="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-3" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_3dbb9bcf72ca45ec80304496ab4709ef-3"></a> <span class="k">def</span> <span class="nf">youyoueyedeefour</span><span class="p">():</span>
<a id="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-4" name="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-4" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_3dbb9bcf72ca45ec80304496ab4709ef-4"></a> <span class="k">return</span> <span class="s2">"</span><span class="si">{}</span><span class="s2">-</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">node</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">())</span>
<a id="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-5" name="rest_code_3dbb9bcf72ca45ec80304496ab4709ef-5" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_3dbb9bcf72ca45ec80304496ab4709ef-5"></a> <span class="n">monkeypatch</span><span class="o">.</span><span class="n">setattr</span><span class="p">(</span><span class="n">rasterio</span><span class="o">.</span><span class="n">_io</span><span class="p">,</span> <span class="s2">"uuid4"</span><span class="p">,</span> <span class="n">youyoueyedeefour</span><span class="p">)</span>
</pre></div>
<p>This <code class="docutils literal">set_mem_name</code> fixture uses two standard pytest fixtures: <code class="docutils literal">request</code>
and <code class="docutils literal">monkeypatch</code>. The value of <code class="docutils literal">request.node.name</code> is the name of the test
and this <code class="docutils literal">set_mem_name</code> fixture uses <code class="docutils literal">monkeypatch</code> to replace <code class="docutils literal">uuid4</code> in
<code class="docutils literal">rasterio._io</code> with a custom function that prepends the name of the test to
the UUID. The <code class="docutils literal">autouse=True</code> argument tells pytest to add this fixture to every
test it finds. I didn't need to touch the code of any of Rasterio's tests, not a one.</p>
<p>This quickly revealed to me that the unclosed temporary datasets were coming
from tests that asserted certain exceptions were being raised by Rasterio's
reprojection code. This code used temporary datasets and didn't close them
before raising the exception to the caller. Once I changed the code to do the
following, Rasterio no longer leaked datasets from those tests, or in our
programs.</p>
<div class="code"><pre class="code python"><a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-1" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-1" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-1"></a><span class="k">try</span><span class="p">:</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-2" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-2" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-2"></a> <span class="o">...</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-3" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-3" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-3"></a> <span class="k">if</span> <span class="n">condition</span><span class="p">:</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-4" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-4" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-4"></a> <span class="k">raise</span> <span class="n">CRSError</span><span class="p">(</span><span class="s2">"error"</span><span class="p">)</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-5" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-5" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-5"></a> <span class="o">...</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-6" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-6" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-6"></a><span class="k">except</span><span class="p">:</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-7" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-7" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-7"></a> <span class="n">temp</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<a id="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-8" name="rest_code_51645f5d81434e3ab3c0ef0af18f5e89-8" href="https://sgillies.net/2019/09/13/debugging-temp-files-using-autouse-fixtures.html#rest_code_51645f5d81434e3ab3c0ef0af18f5e89-8"></a> <span class="k">raise</span>
</pre></div>
<p>If Rasterio used only Python's unittest module, and not pytest, it would be
possible to do the same thing. Import rasterio._io in the test case's
<code class="docutils literal">setUp()</code>, monkey patch it, and then restore it in <code class="docutils literal">tearDown()</code>. If all the
tests derived from one base class, it would only be necessary to extend that
class. The unittest.mock module easily <a class="reference external" href="https://docs.python.org/3/library/unittest.mock-examples.html#applying-the-same-patch-to-every-test-method">allows every test to be patched with
a single decorator statement</a>.
It seems like it could be two fewer lines of code, but I don't immediately see
how to get the name of the test and use it with only the mock.patch decorator.
It looks like one would have to use a patcher's <a class="reference external" href="https://docs.python.org/3/library/unittest.mock.html#patch-methods-start-and-stop">start and stop</a>,
which is back to somewhat more boilerplate than with pytest.</p>Mock is Magichttps://sgillies.net/2017/10/19/mock-is-magic.html2017-10-19T19:46:31-06:002017-10-19T19:46:31-06:00Sean Gillies<p>I'm sprinting with my teammates with occasionally spotty internet. We're
developing a module that takes some directory names, archives the directories,
uploads the archive to S3, and then cleans up temporary files. Testing this by
actually posting data to S3 is slow, leaves debris, and is almost pointless:
we're using boto3 and boto3 is, for our purposes, solid. We've only ever found
one new boto bug at Mapbox, and that involved very large streaming uploads. My
teammates and I only need to test that we're making a proper archive, using the
boto3 API properly, and cleaning up afterwards. Whether or not data lands on S3
isn't important for these tests. Python's mock module is one of many Python
tools for faking components during testing. If you're not already using it to
create test doubles for boto components (and AWS services), this post will help
get you started down the right path.</p>
<p>Here's the function to test, with the code I actually want to be testing
glossed over. This post is about boxing out the stuff we don't want to test and
so we'll be looking only at making and using mock boto3 objects.</p>
<div class="code"><pre class="code python"><a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-1" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-1"></a><span class="sd">"""mymodule.py"""</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-2" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-2"></a>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-3" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-3"></a><span class="kn">import</span> <span class="nn">boto3</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-4" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-4"></a>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-5" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-5"></a>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-6" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-6"></a><span class="k">def</span> <span class="nf">archive_and_upload</span><span class="p">(</span><span class="nb">dir</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-7" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-7" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-7"></a> <span class="sd">"""Archive data and upload to S3"""</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-8" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-8" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-8"></a> <span class="c1"># A bunch of code makes a zip file in a ``tmp`` directory.</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-9" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-9" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-9"></a>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-10" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-10" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-10"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span><span class="s2">"s3"</span><span class="p">)</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">bucket</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp</span><span class="p">,</span> <span class="n">zip_file</span><span class="p">))</span>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-11" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-11" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-11"></a>
<a id="rest_code_9a8302f7f30e42aaa5803de0688907f7-12" name="rest_code_9a8302f7f30e42aaa5803de0688907f7-12" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_9a8302f7f30e42aaa5803de0688907f7-12"></a> <span class="c1"># A bunch of code now cleans up temporary resources.</span>
</pre></div>
<p>Now, in the test function that we're discovering and running with pytest, we
create a fake boto3 API using <code class="docutils literal">mock.patch</code>.</p>
<div class="code"><pre class="code python"><a id="rest_code_238077c220914927a61f9d961a7a999c-1" name="rest_code_238077c220914927a61f9d961a7a999c-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-1"></a><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<a id="rest_code_238077c220914927a61f9d961a7a999c-2" name="rest_code_238077c220914927a61f9d961a7a999c-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-2"></a>
<a id="rest_code_238077c220914927a61f9d961a7a999c-3" name="rest_code_238077c220914927a61f9d961a7a999c-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-3"></a><span class="kn">from</span> <span class="nn">mymodule</span> <span class="kn">import</span> <span class="n">archive_and_upload</span>
<a id="rest_code_238077c220914927a61f9d961a7a999c-4" name="rest_code_238077c220914927a61f9d961a7a999c-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-4"></a>
<a id="rest_code_238077c220914927a61f9d961a7a999c-5" name="rest_code_238077c220914927a61f9d961a7a999c-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-5"></a>
<a id="rest_code_238077c220914927a61f9d961a7a999c-6" name="rest_code_238077c220914927a61f9d961a7a999c-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-6"></a><span class="nd">@patch</span><span class="p">(</span><span class="s2">"mymodule.boto3"</span><span class="p">)</span>
<a id="rest_code_238077c220914927a61f9d961a7a999c-7" name="rest_code_238077c220914927a61f9d961a7a999c-7" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-7"></a><span class="k">def</span> <span class="nf">test_archive_and_upload</span><span class="p">(</span><span class="n">boto3</span><span class="p">):</span>
<a id="rest_code_238077c220914927a61f9d961a7a999c-8" name="rest_code_238077c220914927a61f9d961a7a999c-8" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-8"></a> <span class="sd">"""Data is archived, uploaded, and the floor is swept"""</span>
<a id="rest_code_238077c220914927a61f9d961a7a999c-9" name="rest_code_238077c220914927a61f9d961a7a999c-9" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_238077c220914927a61f9d961a7a999c-9"></a> <span class="n">archive_and_upload</span><span class="p">(</span><span class="s2">"test"</span><span class="p">,</span> <span class="s2">"bucket"</span><span class="p">,</span> <span class="s2">"key"</span><span class="p">)</span>
</pre></div>
<p>While the test runs, <code class="docutils literal">boto3</code> in the module is replaced by an instance of
<code class="docutils literal">unittest.mock.MagicMock</code>. We're also able to bind the same mock object to
<code class="docutils literal">boto3</code> for inspection within the test function by passing that as an
argument. These mock objects have almost incredible properties. Substituting one
for the boto3 module gives us a fairly complete API in the sense that all
the methods and properties seem to be there.</p>
<div class="code"><pre class="code pycon"><a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-1" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-1"></a><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">MagicMock</span>
<a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-2" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-2"></a><span class="gp">>>> </span><span class="n">boto3</span> <span class="o">=</span> <span class="n">MagicMock</span><span class="p">()</span>
<a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-3" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-3"></a><span class="gp">>>> </span><span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-4" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-4"></a><span class="go"><MagicMock name='mock.resource()' id='4327834232'></span>
<a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-5" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-5"></a><span class="gp">>>> </span><span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="s1">'mybucket'</span><span class="p">,</span> <span class="s1">'mykey'</span><span class="p">)</span>
<a id="rest_code_4415b7e90ee34dc0bdf25432856d2de5-6" name="rest_code_4415b7e90ee34dc0bdf25432856d2de5-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_4415b7e90ee34dc0bdf25432856d2de5-6"></a><span class="go"><MagicMock name='mock.resource().Object()' id='4327879960'></span>
</pre></div>
<p>It does almost nothing, of course, but that's fine for these tests. One thing
that the mock objects do to help with testing is record how they are accessed
or called. We can assert that certain calls were made with certain arguments.</p>
<div class="code"><pre class="code python"><a id="rest_code_f747fbcbc830440084a2a660564a594a-1" name="rest_code_f747fbcbc830440084a2a660564a594a-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-1"></a><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-2" name="rest_code_f747fbcbc830440084a2a660564a594a-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-2"></a>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-3" name="rest_code_f747fbcbc830440084a2a660564a594a-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-3"></a>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-4" name="rest_code_f747fbcbc830440084a2a660564a594a-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-4"></a><span class="nd">@patch</span><span class="p">(</span><span class="s2">"mymodule.boto3"</span><span class="p">)</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-5" name="rest_code_f747fbcbc830440084a2a660564a594a-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-5"></a><span class="k">def</span> <span class="nf">test_archive_and_upload</span><span class="p">(</span><span class="n">boto3</span><span class="p">):</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-6" name="rest_code_f747fbcbc830440084a2a660564a594a-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-6"></a> <span class="sd">"""Data is archived, uploaded, and the floor is swept"""</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-7" name="rest_code_f747fbcbc830440084a2a660564a594a-7" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-7"></a> <span class="n">archive_and_upload</span><span class="p">(</span><span class="s2">"test"</span><span class="p">,</span> <span class="s2">"bucket"</span><span class="p">,</span> <span class="s2">"key"</span><span class="p">)</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-8" name="rest_code_f747fbcbc830440084a2a660564a594a-8" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-8"></a>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-9" name="rest_code_f747fbcbc830440084a2a660564a594a-9" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-9"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s2">"s3"</span><span class="p">)</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-10" name="rest_code_f747fbcbc830440084a2a660564a594a-10" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-10"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">()</span><span class="o">.</span><span class="n">Object</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s2">"bucket"</span><span class="p">,</span> <span class="s2">"key"</span><span class="p">)</span>
<a id="rest_code_f747fbcbc830440084a2a660564a594a-11" name="rest_code_f747fbcbc830440084a2a660564a594a-11" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_f747fbcbc830440084a2a660564a594a-11"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">()</span><span class="o">.</span><span class="n">Object</span><span class="p">()</span><span class="o">.</span><span class="n">upload_file</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s2">"/tmp/test.zip"</span><span class="p">)</span>
</pre></div>
<p>Asserting that the mock file uploader was called with the correct argument is,
in this case, preferable to posting data to S3. It's fast and leaves no
artifacts to remove. If we wanted to test that <code class="docutils literal">archive_and_upload</code> does the
right thing when AWS and boto3 signal an error, we can set a side effect for
the mock <code class="docutils literal">upload_file</code> method.</p>
<div class="code"><pre class="code python"><a id="rest_code_3667a5cb2c004b1e8165d96518914f07-1" name="rest_code_3667a5cb2c004b1e8165d96518914f07-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-1"></a><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-2" name="rest_code_3667a5cb2c004b1e8165d96518914f07-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-2"></a>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-3" name="rest_code_3667a5cb2c004b1e8165d96518914f07-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-3"></a>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-4" name="rest_code_3667a5cb2c004b1e8165d96518914f07-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-4"></a><span class="nd">@patch</span><span class="p">(</span><span class="s2">"mymodule.boto3"</span><span class="p">)</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-5" name="rest_code_3667a5cb2c004b1e8165d96518914f07-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-5"></a><span class="k">def</span> <span class="nf">test_archive_and_upload_authorized</span><span class="p">(</span><span class="n">boto3</span><span class="p">):</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-6" name="rest_code_3667a5cb2c004b1e8165d96518914f07-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-6"></a> <span class="sd">"""Unauthorized errors are handled"""</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-7" name="rest_code_3667a5cb2c004b1e8165d96518914f07-7" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-7"></a>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-8" name="rest_code_3667a5cb2c004b1e8165d96518914f07-8" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-8"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="o">.</span><span class="n">return_value</span><span class="o">.</span><span class="n">Object</span><span class="o">.</span><span class="n">return_value</span><span class="o">.</span><span class="n">upload_file</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="n">botocore</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">ClientError</span><span class="p">(</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-9" name="rest_code_3667a5cb2c004b1e8165d96518914f07-9" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-9"></a> <span class="p">{</span><span class="s2">"Error"</span><span class="p">:</span> <span class="p">{</span><span class="s2">"Code"</span><span class="p">:</span> <span class="s2">"403"</span><span class="p">,</span> <span class="s2">"Message"</span><span class="p">:</span> <span class="s2">"Unauthorized"</span><span class="p">}},</span> <span class="s2">"PutObject"</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-10" name="rest_code_3667a5cb2c004b1e8165d96518914f07-10" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-10"></a> <span class="p">)</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-11" name="rest_code_3667a5cb2c004b1e8165d96518914f07-11" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-11"></a>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-12" name="rest_code_3667a5cb2c004b1e8165d96518914f07-12" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-12"></a> <span class="n">archive_and_upload</span><span class="p">(</span><span class="s2">"test"</span><span class="p">,</span> <span class="s2">"bucket"</span><span class="p">,</span> <span class="s2">"key"</span><span class="p">)</span>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-13" name="rest_code_3667a5cb2c004b1e8165d96518914f07-13" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-13"></a>
<a id="rest_code_3667a5cb2c004b1e8165d96518914f07-14" name="rest_code_3667a5cb2c004b1e8165d96518914f07-14" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_3667a5cb2c004b1e8165d96518914f07-14"></a> <span class="c1"># assert that exception has been handled.</span>
</pre></div>
<p>A <code class="docutils literal">botocore.exceptions.ClientError</code> will be raised in <code class="docutils literal">archive_and_upload</code>
from the <code class="docutils literal">upload_file</code> call. We could test against a bucket for which we have
no access, but I think the mock is preferable for a unit test. It doesn't
require an internet connection and doesn't require any AWS ACL configuration.</p>
<p>Mock's magic can, however, lead to subtle bugs like the one in the test below. Can you find it?</p>
<div class="code"><pre class="code python"><a id="rest_code_af36731aabe2485aa280d57cae184a36-1" name="rest_code_af36731aabe2485aa280d57cae184a36-1" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-1"></a><span class="kn">from</span> <span class="nn">unittest.mock</span> <span class="kn">import</span> <span class="n">patch</span>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-2" name="rest_code_af36731aabe2485aa280d57cae184a36-2" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-2"></a>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-3" name="rest_code_af36731aabe2485aa280d57cae184a36-3" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-3"></a>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-4" name="rest_code_af36731aabe2485aa280d57cae184a36-4" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-4"></a><span class="nd">@patch</span><span class="p">(</span><span class="s2">"mymodule.boto3"</span><span class="p">)</span>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-5" name="rest_code_af36731aabe2485aa280d57cae184a36-5" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-5"></a><span class="k">def</span> <span class="nf">test_archive_and_upload_wtf</span><span class="p">(</span><span class="n">boto3</span><span class="p">):</span>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-6" name="rest_code_af36731aabe2485aa280d57cae184a36-6" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-6"></a> <span class="sd">"""Why does this keeping failing?"""</span>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-7" name="rest_code_af36731aabe2485aa280d57cae184a36-7" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-7"></a> <span class="n">archive_and_upload</span><span class="p">(</span><span class="s2">"test"</span><span class="p">,</span> <span class="s2">"bucket"</span><span class="p">,</span> <span class="s2">"key"</span><span class="p">)</span>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-8" name="rest_code_af36731aabe2485aa280d57cae184a36-8" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-8"></a>
<a id="rest_code_af36731aabe2485aa280d57cae184a36-9" name="rest_code_af36731aabe2485aa280d57cae184a36-9" href="https://sgillies.net/2017/10/19/mock-is-magic.html#rest_code_af36731aabe2485aa280d57cae184a36-9"></a> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">()</span><span class="o">.</span><span class="n">Object</span><span class="p">()</span><span class="o">.</span><span class="n">upload_file</span><span class="p">()</span><span class="o">.</span><span class="n">assert_called_with</span><span class="p">(</span><span class="s2">"/tmp/test.zip"</span><span class="p">)</span>
</pre></div>
<p>Because all mock methods and properties yield more mocks, it can be hard to
figure out why <code class="docutils literal"><span class="pre">boto3.resource().Object().upload_file()</span></code> is never called with
the expected arguments, even when we're certain the arguments are right.
Unintended parentheses after <code class="docutils literal">upload_file</code> cost me 15 minutes of head
scratching earlier this morning.</p>