Ruth is traveling this week and the second half of next week. Between these trips I've got two weekends of birthday parties and other stuff to cover as a single dad. I'm hosting Beatrice's 11th birthday on the December 14 at a neighborhood pool. Ruth did most of the groundwork, but I'll be the one supervising kids and coordinating with parents. And eating leftover cupcakes.

Both of my kids are changing schools next year. Arabelle is headed to high school and Bea to middle school. This week, all the local schools are having evening open houses and promo nights for prospective students and I'm attending these, too. Not for every school, but one for each of my kids, and that's adding to the busyness. Tonight we went to Poudre High School and got a tour from one of Poudre's senior school ambassadors. Poudre is the local International Baccalaureate high school and also educates kids from the most rural parts of Fort Collins. It's an interesting mix, probably more like my own Intermountain West high school experience than what most Fort Collins kids get these days. We saw classrooms and labs, met teachers, saw the indoor archery range, learned about key hallway landmarks. "Meet me by Yoda after 2nd period" is a thing. Bea has this ambassador role at her elementary school and really enjoyed tagging along. I ran into parents and kids that I'd met when coaching Arabelle's U9 soccer team, which was fun.

Tomorrow, Bea and I are headed to a middle school open house. I'm curious to hear from the principal and meet the teachers. Immediately before the open house is the winter choir concert at Bea's elementary school. Did I mention that I'm busy this week?

A year ago, I blogged about finishing my first week of training for the 2019 Quad Rock 50. I did one such post for each of the 24 weeks in my season and thoroughly enjoyed it. They're all linked at https://sgillies.net/tags/running.html if you're interested. I may do this again in 2020, but for a different race in July, so won't start the trail blogging until February.

GDAL 3.0.0 was released in May. It has been adopted more quickly than I anticipated and lack of support for it in the Rasterio and Fiona packages was becoming a point of pain for users. Rasterio 1.0.25 added support for GDAL 3.0 in August and now Fiona 1.8.9 has similar basic support. We can build the extension modules in these packages with GDAL 3.0 and they will run almost exactly as they did with GDAL 2.4. No new spatial referencing features of GDAL 3.0 and PROJ 6 are exposed by Fiona 1.8.9.

What changes were required to the code? Everywhere an extension module calls one of the OSRImport* functions we now need to call a new OSRSetAxisMappingStrategy function to preserve the "traditional" open source GIS order of coordinates: longitude first, latitude second in common cases. This change was made in core modules and in the three existing shim modules. We had to add a new shim module to go with GDAL 3.0 to make a facade for a function that was removed from GDAL in 3.0. We try to avoid branching on the version of GDAL in Fiona and instead use the shims at build time. This binds installations of the packages to a specific version of GDAL.

Why did it take so long to make the changes? The changes required us to add builds of GDAL 3 and PROJ 6 to our CI server, which takes some iterations and time. Also the cultural knowledge of the project required to change the shim modules is scarce. There are only a few of us who understanding what's going on in there. This is certainly a problem to be overcome in the future.

The shims make the source distribution of the package fragile. There was a bug in 1.8.9 that prevented the fiona/_shim3.c file from being copied to fiona/_shim.c during installation. In 1.8.9.post1 we created a new bug that messed up copy of the fiona/_shim2.c file. These are all fixed and now users can build and install a version of Fiona on Linux or OS X that compiles with and links and existing GDAL 3.0 installation by running GDAL_CONFIG=/gdal3prefix/bin/gdal-config pip install --no-binary fiona -U fiona.

Thanks for your help and patience, everyone.

I'm headed to San Francisco for work almost exactly a year after going on a family trip on which my kids and I made our first walk onto the Golden Gate Bridge. Among the photos we took are these by Arabelle, who was enjoying playing with depth of field out in the unique space of the bridge and bay and sky.

I spent last weekend in Minneapolis, my family flew to Seattle today, I'm flying to San Francisco on Sunday, and I'm planning to cross paths with Ruth, Arabelle, and Beatrice in DEN as they return. I think it goes without saying that it's unsustainable for a family to be flying around all the time. We're trying to reduce our trips and make up for them as much as we can, but this is a period of fairly intense stratospheric presence for my clan.

Arabelle is flying to LA for a week on Catalina Island with a school group in November, her first trip in a plane without me or Ruth. There is at least one kid on the trip who will be flying for their first time at age 13. My kids, who have been flying all their lives, to France multiple times, almost get it, but that first-time plane traveler is a lot like me. When I was a kid, my parents could barely afford to fly us around at all. I boarded my first domestic flight at 15 (Salt Lake City -> Dallas -> Detroit) and didn't make my first international flight (Miami to Caracas, Venezuela) until I was 25.

Air travel is both amazing (the speed, the views out the window!) and terrible (the pollution, the lines). I'm certain that it is going to be more limited in the future. Make the most of it, and give back as much as you can.

I'm back now from Minneapolis and the Loopet Loppet, the 12-hour trail event in Theodore Wirth Park, where I ran 8 laps of the 5 mile loop and 2 laps of the one mile loop. 42 in all. I had a great time and whole-heartedly recommend this event. The trails are challenging and fun. The aid station and facilities are first class. The staff and volunteers are amazing. Thank you, Bitner, for inviting me out and putting me up for the weekend.

Before the Loopet I was certain that it would be my last race of the year, but I'm feeling reinvigorated and not very sore today, and am reconsidering my options for this Saturday. Stay tuned.

After 28 post-1.0 bug fix releases, Rasterio finally has a 1.1.0 release with new features: https://pypi.org/project/rasterio/1.1.0/. The changes are listed here: https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L9-L68. I had fun collaborating with Vincent Sarago on the design and implementation of transform.from_gcps(). I miss working with Vincent, it's been hard getting by without him on my team.

Rasterio has 90 authors now, including new authors Andrew Annex, Caleb Robinson, Christoph Rieke, Darren Weber, Denis Rykov, Guillaume Lostis, Ismail Sunni, Jennifer Reiber Kyle, Koshy Thomas, Leah Wasser, Lukasz, Martin Raspaud, Pablo Sanfilippo, RK Aranas, Raaj Tilak Sarma, Robert Sare, Yuvi Panda, appanacca, derekjanni, firas omrane, jaredairbusaerial, and sshuair.

I'm very pleased about two of the bug fixes in 1.1.0. I blogged recently about fixing some memory leaks; that's one of them. The other bug fix involved GDAL dataset reference counting. I finally got a really good grip on this. In 1.1.0, closing a DatasetReader doesn't automatically destroy the underlying GDAL dataset handle, it decrements the handle's reference count and destroys it only when the count reaches zero. This means that the handle will be kept alive through the lifetime of a VRT object based on the DatasetReader. A fun thing about writing Python extension modules with GDAL is that I get to deal with two reference counting implementations: Python's and GDAL's, as well as two implementations of exceptions: Python's and GDAL's. I started a blog post on these topics a while ago, but it stalled out. I should try to finish it some day.

This post is a follow-up to Friday's and shows how you can dump a listing of open GDAL datasets in your own Python programs using nothing other than Python's standard library and how you can analyze the dumps with Python and an extra module or two.

If you've imported a Python module that links GDAL, such as Rasterio or GDAL's own Python bindings, you can access GDAL's GDALDumpOpenDatasets function using Python's ctypes module. That function takes a FILE pointer as its only argument and you can get a pointer to stdout or stderr from ctypes as well. I will use Rasterio's interactive dataset inspector to demonstrate.

Note that the C level FILE pointer to stdout is ctypes.c_void_p.in_dll(handle, "__stdoutp") on OS X and ctypes.c_void_p.in_dll(handle, "stdout") on Linux. The listing printed by the function bypasses Python and goes to the terminal.

$ rio insp ~/code/rasterio/tests/data/RGB.byte.tif
Rasterio 1.0.28 Interactive Inspector (Python 3.6.4)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> import ctypes
>>> handle = ctypes.CDLL(None)
>>> cstdout = ctypes.c_void_p.in_dll(handle, '__stdoutp')
>>> _ = handle.GDALDumpOpenDatasets(cstdout)
Open GDAL Datasets:
  1 S GTiff  26416000 791x718x3 /Users/seang/code/rasterio/tests/data/RGB.byte.tif

The first field in every record is the reference count of the dataset, the second is whether it is a shared dataset (S) or not (N), the third is the format driver's short name, the third is a thread id, the fourth is the dataset shape, and the fifth is the dataset's identifier.

In the interpreter we find what we expect: one shared open dataset with a reference count of 1.

What if we wanted to process the listing in Python? We would need to capture the low-level file descriptors and expose them in Python. There's a nice blog post about issues and an implementation at https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/. Pytest includes a fixture for this, capfd, and it can be used in a test like the one shown below.

import ctypes


def test_sharing_on(capfd):
    """Datasets are shared"""
    # Open a dataset in not-shared mode.
    ...
    handle = ctypes.CDLL(None)
    cstdout = ctypes.c_void_p.in_dll(handle, "stdout")
    assert 1 == handle.GDALDumpOpenDatasets(cstdout)
    captured = capfd.readouterr()
    assert "1 N GTiff" in captured.out
    assert "1 S GTiff" not in captured.out

There's a package named capturer which is inspired by pytest and does the same kind of thing as a context manager.

$ rio insp ~/code/rasterio/tests/data/RGB.byte.tif
Rasterio 1.0.28 Interactive Inspector (Python 3.6.4)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> import ctypes
>>> handle = ctypes.CDLL(None)
>>> cstdout = ctypes.c_void_p.in_dll(handle, '__stdoutp')
>>> from capturer import CaptureOutput
>>> with CaptureOutput() as capfd:
...     _ = handle.GDALDumpOpenDatasets(cstdout)
...     handle.GDALDumpOpenDatasets(cstdout)
...     captured = capfd.get_text()
...
Open GDAL Datasets:
  1 S GTiff  26416000 791x718x3 /Users/seang/code/rasterio/tests/data/RGB.byte.tif
>>> print(captured)
Open GDAL Datasets:
  1 S GTiff  26416000 791x718x3 /Users/seang/code/rasterio/tests/data/RGB.byte.tif

The sound and smell of charring chiles is a big part of the Colorado farmers market experience in September. The roar of propane burners and rattle of crisped peppers. The sweet, smokey, aroma of burned chile skins and roasted fruit. It's an end-of-summer ritual.

I've bought many a bag of charred chiles at the market, but these days I prefer to buy them fresh and take them home to give them the personal touch over a hot charcoal fire. In my experience, a gently charred chile that is not tumbled is much easier to peel. The blistered skins come off in big pieces and there's no need to rinse them or peel them under water, which would dilute their sublime flavor.

Yesterday I brought home a bag of green poblano (when red ripe and dried, this is known as ancho) and Big Jim chiles, 10 of each. The Big Jim is a New Mexico chile hybrid developed at New Mexico State University. It's the largest of this kind of chile and has a thick skin. Poblanos have a much thinner skin. It would not be a good idea to roast these together in a cage using propane burners: the tender poblanos would be destroyed. It's easy to do them together on a grill.

A week ago, I gave a bunch of Pueblo chiles the same treatment. The Pueblo chile looks a lot like a New Mexico chile, but is quite different. It's a large Mirasol chile that develops pointing up towards the sun. It seems to be a relatively recent import from Mexico. The New Mexico chile points down towards the earth and is derived from chiles grown by Pueblo people of Southwest North America. The Hatch chile is a New Mexico chile grown in the Hatch Valley along the Rio Grande River upstream of Las Cruces, New Mexico. Hatch is a distinguished appellation like Champagne or Napa are for grapes and wine and Palisade is for peaches, and is way ahead of the game in comparison to Colorado's Pueblo chile growers. New Mexico also has an official State Question: "Red or green?" In other words: do you want red chile or green chile on that? Colorado's climate is cooler and red chiles are less often found at market. I love all of these chiles, green or red.

It doesn't take long over a hot fire for big blisters to form on the Big Jims and smaller ones to form on the poblanos. The technique is basic: turn them until the blistering is fairly uniform and then put them, whole, in a dry, heavy pot with a lid to steam for a few minutes. This will let the skins continue to loosen as the chiles cool.

Here are the skinned chiles. The poblanos took about twice as long to peel because the thinner skins come off in smaller pieces. I don't sweat the little scraps of charred skin and never rinse the chiles. You wouldn't rinse a steak or a sausage, would you?

The final steps are trimming the stems, scraping the seeds from the inside, and putting them in freezer containers with the juices on the plates and in the pot. I will use the Big Jims in fall and winter stews and use the poblanos as a condiment with tacos or eggs or grilled steak.

This week I discovered that Rasterio doesn't always close the temporary in-memory datasets that are used within some of its methods. In testing Rasterio's WarpedVRT class I used a GDAL function to dump descriptions of all open datasets and found a bunch that looked unrelated to WarpedVRT. They were GDAL "MEM" type datasets with UUIDs for names, which didn't tell me much. What were their origins?

They have UUIDs for names because Rasterio imports uuid in its _io module and calls uuid.uuid4() to make temporary dataset names. If only the dataset name included the name of the test in which it was created, then I'd have an entry point into debugging. One way to do this is with a pytest auto-used fixture.

I changed the rasterio._io module's import statement from import uuid to from uuid import uuid4 to make it slightly easier to monkey patch and then I added 5 lines of code to Rasterio's conftest.py file:

@pytest.fixture(autouse=True)
def set_mem_name(request, monkeypatch):
    def youyoueyedeefour():
        return "{}-{}".format(request.node.name, uuid.uuid4())
    monkeypatch.setattr(rasterio._io, "uuid4", youyoueyedeefour)

This set_mem_name fixture uses two standard pytest fixtures: request and monkeypatch. The value of request.node.name is the name of the test and this set_mem_name fixture uses monkeypatch to replace uuid4 in rasterio._io with a custom function that prepends the name of the test to the UUID. The autouse=True argument tells pytest to add this fixture to every test it finds. I didn't need to touch the code of any of Rasterio's tests, not a one.

This quickly revealed to me that the unclosed temporary datasets were coming from tests that asserted certain exceptions were being raised by Rasterio's reprojection code. This code used temporary datasets and didn't close them before raising the exception to the caller. Once I changed the code to do the following, Rasterio no longer leaked datasets from those tests, or in our programs.

try:
    ...
    if condition:
        raise CRSError("error")
    ...
except:
    temp.close()
    raise

If Rasterio used only Python's unittest module, and not pytest, it would be possible to do the same thing. Import rasterio._io in the test case's setUp(), monkey patch it, and then restore it in tearDown(). If all the tests derived from one base class, it would only be necessary to extend that class. The unittest.mock module easily allows every test to be patched with a single decorator statement. It seems like it could be two fewer lines of code, but I don't immediately see how to get the name of the test and use it with only the mock.patch decorator. It looks like one would have to use a patcher's start and stop, which is back to somewhat more boilerplate than with pytest.