Benjamin Gerard Gillies (1976-2016)
My brother Ben died on August 25 after being seriously ill for most of the summer. I got the news that life supporting measures were no longer helping while I was in Germany and was able to get back to Denver to be with him, his partner, and family at the end. He was just 40 years old.
Ben was a bookseller, he worked at the Hungry Minds (later, Ruminator) and Subtext bookstores in St. Paul, Minnesota and in 2015 opened a store, City Stacks, on Denver’s Wazee Street. City Stacks had a memorial on September 2nd. If you’ve come to this page via a search for news about Ben, I recommend going to that Facebook page. I was touched by the kind words of his friends and customers.
Ben was in the bookselling business to be close to the things he loved most: books and readers. I don’t think there is any aspect of the trade, from printing to distribution to customer service, that he didn’t appreciate and enjoy. I’m sorry he didn’t get to keep selling, trading, and talking about books any longer.
Bye, Ben. I love you. Rest in peace.
Instants and intervals for event-like GeoJSON features
Now that RFC 7946 has been published, I’m returning to my work on describing event-like GeoJSON features at https://github.com/sgillies/geojson-events. My goal is to come up with a representation for instants and intervals of time that does for “when” what GeoJSON has already done for “where.” Applications like the USGS earthquake feeds are what I have in mind for this GeoJSON extension; geojson-events is not suited for GPS tracks or telemetry of moving objects, these kind of measurements require a different model.
I would be grateful for your comments on geojson-events. Please check out the project repository and let me know if this GeoJSON extension is or isn’t useful to you.
RFC 7946, the GeoJSON format
Hello, RFC 7946. Congratulations to everybody involved! I’m very satisfied with it. Taking off my editor hat, I’d like to add some personal commentary on the RFC and the process that produced it.
The results
GeoJSON is already well adopted. What did we accomplish by making it an IETF standard?
- We’ve made a standard for encoding geographic data in internet messages that isn’t just for GIS or web mapping developers and is more likely to be used in network applications like WebRTC.
- GeoJSON is more precisely defined than ever. Bugs in the spec have been fixed. My job at Mapbox involves services that accept GeoJSON uploads and I’m looking forward to more standardized GeoJSON as time goes on.
- At last we can check off the “standards” box when selling GeoJSON services to customers that are required to use formal standards.
Acknowledgements
RFC 7946 is the product of the IETF GeoJSON Working Group (WG) formed in October 2015. All the original authors of the GeoJSON spec participated in the WG: Howard Butler, Martin Daly, Allan Doyle, Sean Gillies (me), Tim Schaub, and Christopher Schmidt.
I scraped the WG email list and the geojson/draft-geojson GitHub repo for the names of other WG participants: Vladimir Agafonkin, Richard Barnes, Ben Campbell, Jose Manuel Cantera Fonseca, Micah Cochran, Simon Cox, Sergey Fedoseev, Atle Sveen Frenvik, Jérôme Gasperi, Pedro Goncalves, Blake Grotewold, Max Hartmann Friedrich, Chris Hills, Tõnis Kärdi, Éric Lemoine, Chris Little, Alexey Melnikov, Calvin Metcalf, Volker Mische, Matthias Müller, David Neufeld, George Percivall, Alexandre Petrescu, Paul Ramsey, Carl Reed, Maik Riechert, Peter Rushforth, James Seppi, Andrew Sheppard, Jerry Sievert, Scott Simmons, Raj Singh, Sergey Tolstov, Dirk-Willem van Gulik, Peter Vretanos, Christine Waigl, James Winterbottom, Jeff Yutzler, and Mohammed Zia. Two participants are identified only by their GitHub user names: breynolds, and roomthily. All these folks helped shape the RFC.
GeoJSON was also shaped by people playing specific IETF roles. The WG chairs, Martin Thomson and Erik Wilde, did a fine job of keeping the draft on track. Our Area Director, Alissa Cooper, was the first IETF member I talked to about chartering a working group and was a super guide throughout the entire process. IESG reviewers Ben Campbell, Stephen Farrell, Suresh Krishnan, Alexey Melnikov, and Kathleen Moriarty asked questions and made suggestions that greatly improved the final draft. I’m grateful to Mary Barnes and Cullen Jennings, the Dispatch WG chairs, for explaining how dispatch works at the IETF and inviting me to present remotely at IETF 92.
I’m grateful to Tom MacWright for all the GeoJSON explainers on his blog and for his rhetorical support. Tom regularly reminded the web – and me – why a GeoJSON I-D was worth the effort.
Stefan Drees was the one who really got this ball rolling with the first commits creating the technical framework for authoring the draft. Thanks, Stefan.
I did much of the draft editing at work and Mapbox paid for my attendance at IETF 93, where the proposal was dispatched and the working group charter was launched. I’m proud to work for a company that’s committed to making the internet work better.
Changes
The changes from the Pre-IETF GeoJSON Format Specification published in 2008 at http://geojson.org are listed at https://tools.ietf.org/html/rfc7946#appendix-B.
To summarize:
- Coordinate reference systems are no longer in the core of the specification; use CRS84 longitude and latitude with GeoJSON from now on.
- Don’t extend coordinate arrays with linear referencing measure, timestamps, or other variables.
- Follow the math and physics right-hand rule (not the surveyor’s right-hand rule) when forming polygon rings.
- Construct bounding boxes so they are not ambiguous at the Earth’s poles and antimeridian.
- Do extend GeoJSON, but don’t change its semantics.
- Don’t write longitude and latitude values with more than 7 decimal places of precision.
- Avoid using GeometryCollection when possible.
- Reference RFC 7946 from now on instead of http://geojson.org.
One change that we glossed over in that appendix is that the set of GeoJSON types is closed. Extensions can’t, for example, add a “Circle” type. That doesn’t mean that there will never be GeoJSON circles, ellipses, and curves, but that they must be added to the core by a future RFC.
Here’s an important point that may be obscured by formalities: RFC 7946 does not define a “GeoJSON 2.0.” This is “GeoJSON.” If you need to be more specific, e.g., in describing support for deprecated features like “crs”, you should refer to “2008 geojson.org GeoJSON” vs. “RFC 7946 GeoJSON” or “application/geo+json GeoJSON.” There is no “GeoJSON 2.0.”
IETF process and people
I’m pleased to report that the IETF process works pretty much as described at https://www.ietf.org/about/standards-process.html. Whenever I got baffled, I found helpful and generous people to point me in the right direction. I’m going to stay involved in the IETF in one way or another.
Bonjour, Montpellier
I woke up yesterday morning in Montpellier, France, my home for the next 12 months. Here is the view of our neighborhood from the roof.
I’m happy to be here, with the last week of house cleaning and travel preparation far behind me. The trip itself, except for delays, was easy. My kids and I (Ruth and our dog went 4 days earlier to save money) wrangled our bags through many halls, stood in many lines, sat in many seats, and emerged into sunshine just as our patience was about to end.
I haven’t left the house except to run with my dog across the river to Castelnau-le-Lez, the village next door, and to walk around the neighborhood with my family after dark to enjoy the silence of vacation season, see the stars, and spot geckos hunting for insects under the street lights. Friends have come by to welcome us back and I’ve unpacked and stashed our empty luggage. In a few minutes I’m going to log in and begin to catch up on projects and look forward to going out to visit friends this evening. There are a bunch of tasks ahead of us this week: getting new phone service and bank accounts, school registration, meeting with a French tutor, and shopping for home and office supplies, but I’m going to put these off for a few more hours and enjoy a break from lines and service counters.
Crazy Legs 10K 2016
Yesterday morning I got out of bed at 5:00 a.m. to eat a bowl of cereal and a banana, drink a cup of coffee and a quart of water, and drive down Taft Hill Rd to Devil’s Backbone Open Space for this year’s edition of the Crazy Legs 10K trail run. Like last year, the weather was beautiful, the trail dry, the greenery very green. During the drive, I saw our Moon appear to go down directly behind Longs Peak. It is, of course, the other way around: the Moon, the horizon, and Longs Peak each move eastward relative to the stars, but the mountain’s apparent velocity is 27 times faster.
I ran the steep and rocky course in 1:04 (unofficially), one minute faster than last year, and finished in 17th place (up 7 spots). Best of all, I did not fall on the trail’s notorious rocks the one time I stumbled.
In my race packet, I found this.
A handwritten thank you note from the organizer to every entrant. A classy detail of a thoroughly classy event.
Rasterio sprint at PyCon
I’m helping Mapbox bring mapping to the PyCon community sprints, June 2-4 in Portland, OR.
Dear people of Portland: whether you’re registered for the conference or not, come join me and other Rasterio contributors for some Python and GDAL hacking. See https://github.com/mapbox/rasterio/wiki/2016-PyCon-Sprint for more details.
Colorado Marathon 2016
Sunday, the 1st of May, I started and finished my first Marathon.
The only thing I don’t like about this race is the godawful early start. I know that the first weekend in May can be uncomfortably warm here and understand that an early start makes restricting traffic on CO-14 more feasible, but in order to eat and begin to digest an adequately sized blob of oatmeal before the 6:30am start I had to set my alarm for 3:30. At 4:30 Ruth and I were headed to a parking garage downtown and at 5:00 we were on buses to the starting line.
It was cloudy and dry downtown, but a little more wet in the canyon and about 14 miles up we reached the snow line. At the Stevens Gulch trailhead I saw about 4 inches of wet snow on unpaved surfaces. It was 34° F (1° C).
At the last minute before leaving the house, I had switched to a long-sleeved shirt under my ultralight wind/rain jacket and was glad I had when we arrived. I’d already committed to wearing tights, hat, and light gloves and found this to be just about right for the conditions. A more breathable jacket would have been better: I ended up slightly wet from the inside due to condensation and fairly cold between finishing my gel packs at mile 16 and reaching “Bagel Hill” at mile 19.
I had a hard time finishing. My hamstrings threatened to cramp up as I sped up on the last steeper downhill stretch around mile 12, but I survived that and was feeling pretty good until mile 18 when the quadricepds muscles in each of my legs began to cramp. I battled quad cramps of increasing frequency and duration the rest of the way in. The 3:45 flag that I’d been closing in on for an hour got away from me and the 4:00 flag passed me at about 20 miles. The final 6 miles were brutal: miles 20-23 were the worst, but once I crossed Taft Hill onto sections of the Poudre Trail I’ve run for almost 20 years, I regained enough confidence and energy and a sense of too-close-to-fail to put in a very modest kick at the end. My time was 4:24, which was just in the second half of all finishers.
I don’t understand the cause of the cramps, but can think of a few factors which might be implicated. The course, while never steep, descended much more than any of my training runs, about 300 meters. It wouldn’t surprise me if I wasn’t quite in the stride that I’d been training for. Also, while I’d run farther than 18 miles in training, I hadn’t done so at race pace. I was confident that I’d be able to go a bit harder/longer than I had in training without breaking down, but found I could not. I’m going to train a little more for the next one. Lastly, the week before the race was super hectic. I was fighting fires all day long and sleeping poorly at night, and wasn’t as rested as I’d hoped to be.
Friends have been reminding me that it’s super cool to finish my first marathon and I really appreciate this perspective: it was harder than I can expected and I’m satisfied to have been able to fight through a bit of adversity and finish.
Ruth’s race went better. She had just returned from a speaking tour on April 30 and briefly considered bailing on the half-marathon to catch up on sleep and nurse our ailing dog, but ended up running very well with a personal best time of 2:00:12.
Montpellier Sabbatical II
Between July 2009 and July 2010, I and two little kids rode Ruth’s sabbatical coattails to Montpellier in the south of France. This was my first blog post on arrival: https://sgillies.net/2009/07/06/nous-sommes-arrivs-montpellier.html. Seven years later, we’re headed back to the Hérault (34)! I’m super excited and a bit anxious, the next few months are going to be a bit crazy.
Moving to France for a year involves a lot of smaller tasks. Some of these we have already ticked off:
- Ruth is going to be hosted by CGBP (Centre de Biologie pour la Gestion des Populations – Population Biology) again. Great people, great facility.
- I’ve got the +1 and support from my team at Mapbox.
- We’ve found and made a deposit on a house in Montpellier. The proprietaires are university profs and very helpful.
- We have a date at the French Consulate General in Los Angeles to get long stay visas. This is a new twist, last time we took care of this in France.
- We’ve tuned up our French language skills. I expect our kids are going to impress their teachers.
Ruth found a rental house in the country that was just 5 mins from her lab via VTT (mountain bike) singletrack (!!) but we decided to go with a house in a more central Montpellier neighborhood.
There is yet much to do:
- Find renters for our house in Fort Collins.
- Pack and ship essentials across the ocean (our 7 and 10 year-olds have much greater requirements than last time), and store other stuff locally.
- Learn how to transport a dog to France.
- Find some late summer or fall trail running races in Europe to replace the Black Squirrel race I’d otherwise run here in September.
- Plan our upcoming vacations. the French do this well in advance and you run the risk of finding everything is booked if you wait till the last minute.
Fort Collins and Montpellier have a lot in common. Both are fast-growing university and government centers with sunny climates and good access to outdoor recreation. But there are things Montpellier has that Fort Collins does not:
- Year-round farmers markets. Local strawberries in April (now!) with cherries and other fruit not far behind.
- Excellent local wine regions like La Clape and Pic Saint-Loup.
- The Lucques variety of olive.
- Bakeries. The Crêperie here in the Fort is good, but there are many places in Montpellier making baguettes just as well. And the sourdough (levain) baking in Montpellier is much better.
- Merguez sausage.
- Fresh fish and oysters.
- A short drive to the beach.
- TGV connections to Paris and Barcelona. Just 3 hours to each.
- Historical public places like Place de la Comédie and the Promenade du Peyrou.
It’s not all milk and honey, however. Montpellier also has hellish traffic (I don’t expect it’s any better seven years later), high unemployment, ghettos of poorly integrated immigrants, and a large homeless population. It’s grittier than Fort Collins, more troubled, and in that sense more real.
I’ll write more about the move when summer arrives.
Rasterio 0.34
Last fall Even Rouault announced that GDAL 2.1 would have a new Amazon S3 virtual file system. Extending GDAL’s capability to make HTTP byte range requests to AWS’s HTTPS + XML S3 API, Even has made it possible to efficiently access partial content of S3 objects using certain formats like GeoTIFF. In other words, metadata of a GeoTIFF on S3 or overviews stored as sub-images can be accessed without retrieving the bulk of its image data. Génial!
With help from Even, Rob Emanuele, and Matt Perry, Rasterio 0.34 has a handy abstraction for this feature. Rasterio uses s3:// URIs instead of GDAL’s /vsis3/ paths because URIs are how we identify resources on the web and because this is the URI scheme – if unregistered – used by the AWS Command Line Interface. The same URIs you use with the AWS CLI
$ aws s3 ls s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF
2015-03-14 17:20:01 51099231 LC81390452014295LGN00_B1.TIF
2015-03-14 17:20:30 6626356 LC81390452014295LGN00_B1.TIF.ovr
can also be used with Rasterio:
$ rio info s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF --indent 2
{
"nodata": null,
"dtype": "uint16",
"crs": "EPSG:32645",
"bounds": [
381885.0,
2279085.0,
610515.0,
2512815.0
],
"count": 1,
"blockxsize": 512,
"driver": "GTiff",
"transform": [
30.0,
0.0,
381885.0,
0.0,
-30.0,
2512815.0
],
"blockysize": 512,
"tiled": true,
"lnglat": [
86.96327090815723,
21.666821827007748
],
"shape": [
7791,
7621
],
"compress": "deflate",
"res": [
30.0,
30.0
],
"width": 7621,
"height": 7791,
"interleave": "band"
}
Rasterio gets its credentials in the same manner as the AWS CLI (see Configuring the AWS Command Line Interface). If you’re already using the AWS CLI no extra configuration is needed to start using Rasterio on S3 raster datasets.
A close read of the GDAL debug logs shows that only 16384 bytes of this 50MB TIFF are fetched in order to get the metadata printed above. That’s an efficiency of 3000:1.
The S3 virtual filesystem is only available in Rasterio if you have a GDAL library version >= 2.1.0dev. The macosx wheels for Rasterio 0.34 on PyPI contain GDAL version 2.1.0dev and are probably the easiest way to try this new feature.
Share and enjoy!
Moab from Landsat 8
I’ve still got the Colorado Plateau on my mind and spent some time yesterday dressing up the image below for a Mapbox Instagram post.
2016-02-28 was a clear day in Southeastern Utah
I used Development Seed’s excellent landsat-util utility to find and download the source imagery, rio stack (from Rasterio) to combine individual bands into an RGB TIFF, rio clip to extract a smaller region, rio convert to scale and reformat the 16-bits per band TIFF into a 8-bit per band PNG, and Imagemagick for sigmoidal constrast enhancement and alpha adjustment.
All of the above was done using Python 3.5.1, which is now an option for landsat-util 0.13.0.