I knew that a second Trump presidency was very possible, but I'm still surprised that most American voters chose this sociopath. It's super disturbing to see the veneer of decency entirely stripped away from this country. Apparently, this really is who we are.

If you're not already using pytest's tmp_path fixture, you really should. The fixture provides a temporary directory for testing use. A directory that you can't reference from other test runs or from other test functions in the same run, but that isn't immediately deleted when your tests finish. The directory is created in your account temporary location and is eventually cleaned up by your computer's operating system. Until then, you can open the directories and their files in other applications.

I've been making assertions on datasets in Rasterio's tests and also dragging them into QGIS for a closer look after the tests finish.

Just when my running is getting regular again, I mess up my back with yoga and yardwork. I've got searing pain in my left hip that suggests a herniated disc, and have spent most of yesterday and today in bed. I slept okay last night and hope to again. This is another argument for voting by mail: anybody could find themselves unexpectedly disabled right before election day.

I'm very grateful that I got in a solid mountain run before my mishap. With luck I'll be back out there in a few weeks.

There was this trend a few weeks ago on Mastodon where once a day, for 20 days, you posted the cover of an album that was, for whatever reason, a big deal in your life. I really got into it, thanks to Ed Summers.

20 albums are threaded in the link below. They are in an order that is kind of chronological, enough that you can see me evolve from teenage headbanger to aging hipster.

Post by @sgillies@mastodon.social

View on Mastodon

After I gave up on The Bear 100 and Black Squirrel in August, I stopped running for 6 weeks to let my Achilles tendonitis subside. I quit using the stair stepper and elliptical machines at the gym, too. My only activities were bike commuting, yoga, weight lifting, and a weekly "HIIT the water" class. I've joked about aquasizing, but have really been getting into this class. The instructor is gung ho, the regulars are friendly, and it's a good, low impact, workout. I'm going to keep doing it this fall and winter, for sure.

On the 20th of September I went for a flat 2 mile run at Pineridge Open Space. It felt great to run outdoors on dirt, and I didn't feel any worse afterwards. I did another easy, flat trail run 4 days later. This week I did two 3.5 mile runs on trails and another session on an elliptical trainer. 2 hours in all. I'm going to try to increase to 3 hours a week by the end of the year.

I'm grateful to be able to run again. With some luck, 2025 could be a good year.

A month ago I wrote a long-ish post about Fiona 1.10.0. I'll try to keep this one shorter. Rasterio 1.4.0 has three main changes: Python openers, detailed error chaining, and a new statistics API.

Python openers can connect filesystems implemented in Python, like fsspec or tiledb.vfs, to GDAL's own virtual filesystem machinery. In most cases, you should reply on GDAL's built-in virtual filesystem handlers. On the other hand, if you have unique or proprietary data access protocols, then Rasterio's new openers may be useful.

Often enough, I want more visibility into the errors that occur during GDAL I/O functions. I'd like to see all the errors, not just the last one. So, I've implemented Python-like chaining of GDAL errors. It's not perfectly analogous, because we don't have frames for GDAL code like we do for Python, but look at the kind of details you can get now:

>>> src.read()
rasterio._err.CPLE_AppDefinedError: TIFFFillTile:Read error at row 512, col 0, tile 3; got 38232 bytes, expected 47086

The above exception was the direct cause of the following exception:

rasterio._err.CPLE_AppDefinedError: TIFFReadEncodedTile() failed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "rasterio/_io.pyx", line 968, in rasterio._io.DatasetReaderBase._read
    io_multi_band(self._hds, 0, xoff, yoff, width, height, out, indexes_arr, resampling=resampling)
  File "rasterio/_io.pyx", line 207, in rasterio._io.io_multi_band
    with stack_errors() as checker:
  File "rasterio/_io.pyx", line 213, in rasterio._io.io_multi_band
    return checker.exc_wrap_int(retval)
  File "rasterio/_err.pyx", line 307, in rasterio._err.StackChecker.exc_wrap_int
    raise last
rasterio._err.CPLE_AppDefinedError: /app/tests/data/corrupt.tif, band 1: IReadBlock failed at X offset 1, Y offset 1: TIFFReadEncodedTile() failed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "rasterio/_io.pyx", line 650, in rasterio._io.DatasetReaderBase.read
    out = self._read(indexes, out, window, dtype, resampling=resampling)
  File "rasterio/_io.pyx", line 971, in rasterio._io.DatasetReaderBase._read
    raise RasterioIOError("Read or write failed. See context for details.") from cplerr
rasterio.errors.RasterioIOError: Read or write failed. See context for details.

Lastly, the flawed, multimodal statistics() method of datasets has been deprecated and is replaced by new, simpler stats(), clear_stats(), and update_stats() methods.

Rasterio 1.4.0 remains limited to "classical" rasters, those with a handful of bands or channels all of the same type. For hyperspectral data cubes and the like, you should use h5py, xarray, or other emerging software. I'm quite interested in working on new software in that area, but I'd like to do so without any classical raster legacy.

I made a software release last Tuesday. One with new features, deprecations, major refactoring, and packaging and dependency changes, not just bug fixes. Fiona 1.10.0 has three main changes: Python openers, CLI filter commands, and new data classes.

Python openers can connect filesystems implemented in Python, like fsspec or tiledb.vfs, to GDAL's own virtual filesystem machinery. In most cases, you should reply on GDAL's built-in virtual filesystem handlers. On the other hand, if you have unique or proprietary data access protocols, then Fiona's new openers may be useful. As far as I know, Fiona (and Rasterio) have the only open source implementations of GDAL's virtual filesystem plugin system. David Hoese had the initial idea, Even Rouault helped a lot, and I got it over the finish line. I think this is right up there with MemoryFile for my favorite feature that didn't exist in Python-GIS software before.

Fiona's CLI has three new commands, filter (strictly speaking, a new mode of this command), map, and reduce. These provide some great features for Unix-style data processing pipelines and are designed to work well with jq and programs of that nature. Think of them as the data processing part of ogr2ogr, split into 3 simpler commands, reading and writing to stdin/stdout by default, with no SQL and no need to know about different SQL dialects. The documentation contains a new tutorial about using filter, map, and reduce. This work began in planetlabs/fio-planet and now lives in the Fiona CLI core. Thank you, Tim Schaub, for stewarding the transition.

Lastly, Fiona now longer represents GIS features (and their geometries and properties) as Python dicts, but as Python classes: fiona.model.Feature, fiona.model.Geometry, and fiona.model.Properties. These classes provide dict-like access for backwards compatibility, but raise warnings when mutated. These data classes will be immutable in version 2.0.

A lot of GIS-Python attention has moved on to columnar data and massive amounts of time series, trajectories, telemetry, etc, using Parquet and Arrow. But, there's still a need to reason about persistent spatial things in our world and their relationships to each other. Classic GIS features, in other words. Watersheds, counties, neighborhoods. That's what Fiona remains concerned about.

Black Squirrel is the local trail half-marathon that I've finished five times since 2016. I registered to run it again this year, but had to scratch because of my Achilles tendonitis.

Instead, I'm going to volunteer at the event. I'll be among the first at Lory State Park, helping arriving runners park their cars and find the starting line. After that, I'm going to a trail intersection to keep first-time runners on course as they pour down the Howard Trail. It's going to be a beautiful late summer morning and I'm looking forward to being part of the event.

Six years ago my family and I spent some time in France, passing through Lyon and Paris on the way back to Colorado. While Ruth stayed behind at a conference in Montpellier, Arabelle, Bea, and I did Paris in a day. We saw many bones, ate much gelato, hiked many stairs, rode many trains and buses, and enjoyed being together, seeing new places, and using our second language in one of the world's capitals. I cherish my memories of this day. I felt like I was killing the fun Francophile dad role and thoroughly enjoyed the company of my kids. I had a memorable time. We confirmed in June that Arabelle and Bea have strong, positive memories, as well.

I went to Instagram for the first time in years to dig up a photo taken on that day that I haven't published on my blog before.

We didn't go back to the Trocadero in June, assuming that it was closed for Olympics construction, but we did go back to Montmartre and Sacre Coeur. Both Arabelle and Bea have been cultivating Spanish as a third language since 2018 and had a chance to use it on our recent visit. We encountered a Spanish kid of Bea's age who had been separated from her parents and kept her company until we could reach her mom on the phone. She had no phone of her own, but had memorized key numbers.

I just emailed the Bear 100 race director, Cody Draper, and said that I'm injured and won't recover in time to start in 6 weeks. I'm also too injured to do Black Squirrel in 3 weeks and will be emailing Nick and Brad next.

What's the injury? When I came home from vacation and started to ramp up my running, my right Achilles tendon became sore and very inflamed. On July 17 I went out for a 30 mile run and had to bail at mile 20. I took some time off from trail running and have been chugging on an elliptical trainer at my gym. Yesterday I tried some running on the bike path behind my house and couldn't even go a mile without excruciating pain. I'm calling it, my season is over. The last time I had serious Achilles tendonitis like this was at the end of my ultimate frisbee career 20 years ago. It took 4 months before I could run without pain. I'm going to keep going to the gym, ice and rest, see some specialists, and get ready for 2025.