Instants and intervals for event-like GeoJSON features

Now that RFC 7946 has been published, I’m returning to my work on describing event-like GeoJSON features at https://github.com/sgillies/geojson-events. My goal is to come up with a representation for instants and intervals of time that does for “when” what GeoJSON has already done for “where.” Applications like the USGS earthquake feeds are what I have in mind for this GeoJSON extension; geojson-events is not suited for GPS tracks or telemetry of moving objects, these kind of measurements require a different model.

I would be grateful for your comments on geojson-events. Please check out the project repository and let me know if this GeoJSON extension is or isn’t useful to you.

RFC 7946, the GeoJSON format

Hello, RFC 7946. Congratulations to everybody involved! I’m very satisfied with it. Taking off my editor hat, I’d like to add some personal commentary on the RFC and the process that produced it.

The results

GeoJSON is already well adopted. What did we accomplish by making it an IETF standard?

  • We’ve made a standard for encoding geographic data in internet messages that isn’t just for GIS or web mapping developers and is more likely to be used in network applications like WebRTC.
  • GeoJSON is more precisely defined than ever. Bugs in the spec have been fixed. My job at Mapbox involves services that accept GeoJSON uploads and I’m looking forward to more standardized GeoJSON as time goes on.
  • At last we can check off the “standards” box when selling GeoJSON services to customers that are required to use formal standards.

Acknowledgements

RFC 7946 is the product of the IETF GeoJSON Working Group (WG) formed in October 2015. All the original authors of the GeoJSON spec participated in the WG: Howard Butler, Martin Daly, Allan Doyle, Sean Gillies (me), Tim Schaub, and Christopher Schmidt.

I scraped the WG email list and the geojson/draft-geojson GitHub repo for the names of other WG participants: Vladimir Agafonkin, Richard Barnes, Ben Campbell, Jose Manuel Cantera Fonseca, Micah Cochran, Simon Cox, Sergey Fedoseev, Atle Sveen Frenvik, Jérôme Gasperi, Pedro Goncalves, Blake Grotewold, Max Hartmann Friedrich, Chris Hills, Tõnis Kärdi, Éric Lemoine, Chris Little, Alexey Melnikov, Calvin Metcalf, Volker Mische, Matthias Müller, David Neufeld, George Percivall, Alexandre Petrescu, Paul Ramsey, Carl Reed, Maik Riechert, Peter Rushforth, James Seppi, Andrew Sheppard, Jerry Sievert, Scott Simmons, Raj Singh, Sergey Tolstov, Dirk-Willem van Gulik, Peter Vretanos, Christine Waigl, James Winterbottom, Jeff Yutzler, and Mohammed Zia. Two participants are identified only by their GitHub user names: breynolds, and roomthily. All these folks helped shape the RFC.

GeoJSON was also shaped by people playing specific IETF roles. The WG chairs, Martin Thomson and Erik Wilde, did a fine job of keeping the draft on track. Our Area Director, Alissa Cooper, was the first IETF member I talked to about chartering a working group and was a super guide throughout the entire process. IESG reviewers Ben Campbell, Stephen Farrell, Suresh Krishnan, Alexey Melnikov, and Kathleen Moriarty asked questions and made suggestions that greatly improved the final draft. I’m grateful to Mary Barnes and Cullen Jennings, the Dispatch WG chairs, for explaining how dispatch works at the IETF and inviting me to present remotely at IETF 92.

I’m grateful to Tom MacWright for all the GeoJSON explainers on his blog and for his rhetorical support. Tom regularly reminded the web – and me – why a GeoJSON I-D was worth the effort.

Stefan Drees was the one who really got this ball rolling with the first commits creating the technical framework for authoring the draft. Thanks, Stefan.

I did much of the draft editing at work and Mapbox paid for my attendance at IETF 93, where the proposal was dispatched and the working group charter was launched. I’m proud to work for a company that’s committed to making the internet work better.

Changes

The changes from the Pre-IETF GeoJSON Format Specification published in 2008 at http://geojson.org are listed at https://tools.ietf.org/html/rfc7946#appendix-B.

To summarize:

  • Coordinate reference systems are no longer in the core of the specification; use CRS84 longitude and latitude with GeoJSON from now on.
  • Don’t extend coordinate arrays with linear referencing measure, timestamps, or other variables.
  • Follow the math and physics right-hand rule (not the surveyor’s right-hand rule) when forming polygon rings.
  • Construct bounding boxes so they are not ambiguous at the Earth’s poles and antimeridian.
  • Do extend GeoJSON, but don’t change its semantics.
  • Don’t write longitude and latitude values with more than 7 decimal places of precision.
  • Avoid using GeometryCollection when possible.
  • Reference RFC 7946 from now on instead of http://geojson.org.

One change that we glossed over in that appendix is that the set of GeoJSON types is closed. Extensions can’t, for example, add a “Circle” type. That doesn’t mean that there will never be GeoJSON circles, ellipses, and curves, but that they must be added to the core by a future RFC.

Here’s an important point that may be obscured by formalities: RFC 7946 does not define a “GeoJSON 2.0.” This is “GeoJSON.” If you need to be more specific, e.g., in describing support for deprecated features like “crs”, you should refer to “2008 geojson.org GeoJSON” vs. “RFC 7946 GeoJSON” or “application/geo+json GeoJSON.” There is no “GeoJSON 2.0.”

IETF process and people

I’m pleased to report that the IETF process works pretty much as described at https://www.ietf.org/about/standards-process.html. Whenever I got baffled, I found helpful and generous people to point me in the right direction. I’m going to stay involved in the IETF in one way or another.

Bonjour, Montpellier

I woke up yesterday morning in Montpellier, France, my home for the next 12 months. Here is the view of our neighborhood from the roof.

https://c1.staticflickr.com/9/8664/28212652704_6479847ff1_b.jpg

I’m happy to be here, with the last week of house cleaning and travel preparation far behind me. The trip itself, except for delays, was easy. My kids and I (Ruth and our dog went 4 days earlier to save money) wrangled our bags through many halls, stood in many lines, sat in many seats, and emerged into sunshine just as our patience was about to end.

https://c1.staticflickr.com/9/8678/28227771533_5b2edca713_b.jpg

I haven’t left the house except to run with my dog across the river to Castelnau-le-Lez, the village next door, and to walk around the neighborhood with my family after dark to enjoy the silence of vacation season, see the stars, and spot geckos hunting for insects under the street lights. Friends have come by to welcome us back and I’ve unpacked and stashed our empty luggage. In a few minutes I’m going to log in and begin to catch up on projects and look forward to going out to visit friends this evening. There are a bunch of tasks ahead of us this week: getting new phone service and bank accounts, school registration, meeting with a French tutor, and shopping for home and office supplies, but I’m going to put these off for a few more hours and enjoy a break from lines and service counters.

Crazy Legs 10K 2016

Yesterday morning I got out of bed at 5:00 a.m. to eat a bowl of cereal and a banana, drink a cup of coffee and a quart of water, and drive down Taft Hill Rd to Devil’s Backbone Open Space for this year’s edition of the Crazy Legs 10K trail run. Like last year, the weather was beautiful, the trail dry, the greenery very green. During the drive, I saw our Moon appear to go down directly behind Longs Peak. It is, of course, the other way around: the Moon, the horizon, and Longs Peak each move eastward relative to the stars, but the mountain’s apparent velocity is 27 times faster.

I ran the steep and rocky course in 1:04 (unofficially), one minute faster than last year, and finished in 17th place (up 7 spots). Best of all, I did not fall on the trail’s notorious rocks the one time I stumbled.

In my race packet, I found this.

https://c2.staticflickr.com/8/7368/27193799245_58d4575480_b_d.jpg

A handwritten thank you note from the organizer to every entrant. A classy detail of a thoroughly classy event.

Rasterio sprint at PyCon

I’m helping Mapbox bring mapping to the PyCon community sprints, June 2-4 in Portland, OR.

https://c2.staticflickr.com/8/7430/26495508713_10a4fda507_b_d.jpg

Dear people of Portland: whether you’re registered for the conference or not, come join me and other Rasterio contributors for some Python and GDAL hacking. See https://github.com/mapbox/rasterio/wiki/2016-PyCon-Sprint for more details.

Colorado Marathon 2016

Sunday, the 1st of May, I started and finished my first Marathon.

The only thing I don’t like about this race is the godawful early start. I know that the first weekend in May can be uncomfortably warm here and understand that an early start makes restricting traffic on CO-14 more feasible, but in order to eat and begin to digest an adequately sized blob of oatmeal before the 6:30am start I had to set my alarm for 3:30. At 4:30 Ruth and I were headed to a parking garage downtown and at 5:00 we were on buses to the starting line.

It was cloudy and dry downtown, but a little more wet in the canyon and about 14 miles up we reached the snow line. At the Stevens Gulch trailhead I saw about 4 inches of wet snow on unpaved surfaces. It was 34° F (1° C).

https://c2.staticflickr.com/8/7188/26357748774_e48c7cb1ab_b_d.jpg

At the last minute before leaving the house, I had switched to a long-sleeved shirt under my ultralight wind/rain jacket and was glad I had when we arrived. I’d already committed to wearing tights, hat, and light gloves and found this to be just about right for the conditions. A more breathable jacket would have been better: I ended up slightly wet from the inside due to condensation and fairly cold between finishing my gel packs at mile 16 and reaching “Bagel Hill” at mile 19.

https://c2.staticflickr.com/8/7261/26929703676_d1f5fea295_b_d.jpg

I had a hard time finishing. My hamstrings threatened to cramp up as I sped up on the last steeper downhill stretch around mile 12, but I survived that and was feeling pretty good until mile 18 when the quadricepds muscles in each of my legs began to cramp. I battled quad cramps of increasing frequency and duration the rest of the way in. The 3:45 flag that I’d been closing in on for an hour got away from me and the 4:00 flag passed me at about 20 miles. The final 6 miles were brutal: miles 20-23 were the worst, but once I crossed Taft Hill onto sections of the Poudre Trail I’ve run for almost 20 years, I regained enough confidence and energy and a sense of too-close-to-fail to put in a very modest kick at the end. My time was 4:24, which was just in the second half of all finishers.

I don’t understand the cause of the cramps, but can think of a few factors which might be implicated. The course, while never steep, descended much more than any of my training runs, about 300 meters. It wouldn’t surprise me if I wasn’t quite in the stride that I’d been training for. Also, while I’d run farther than 18 miles in training, I hadn’t done so at race pace. I was confident that I’d be able to go a bit harder/longer than I had in training without breaking down, but found I could not. I’m going to train a little more for the next one. Lastly, the week before the race was super hectic. I was fighting fires all day long and sleeping poorly at night, and wasn’t as rested as I’d hoped to be.

Friends have been reminding me that it’s super cool to finish my first marathon and I really appreciate this perspective: it was harder than I can expected and I’m satisfied to have been able to fight through a bit of adversity and finish.

Ruth’s race went better. She had just returned from a speaking tour on April 30 and briefly considered bailing on the half-marathon to catch up on sleep and nurse our ailing dog, but ended up running very well with a personal best time of 2:00:12.

Montpellier Sabbatical II

Between July 2009 and July 2010, I and two little kids rode Ruth’s sabbatical coattails to Montpellier in the south of France. This was my first blog post on arrival: https://sgillies.net/2009/07/06/nous-sommes-arrivs-montpellier.html. Seven years later, we’re headed back to the Hérault (34)! I’m super excited and a bit anxious, the next few months are going to be a bit crazy.

https://c1.staticflickr.com/5/4124/4831955788_39edb3e93d_b_d.jpg

Moving to France for a year involves a lot of smaller tasks. Some of these we have already ticked off:

  • Ruth is going to be hosted by CGBP (Centre de Biologie pour la Gestion des Populations – Population Biology) again. Great people, great facility.
  • I’ve got the +1 and support from my team at Mapbox.
  • We’ve found and made a deposit on a house in Montpellier. The proprietaires are university profs and very helpful.
  • We have a date at the French Consulate General in Los Angeles to get long stay visas. This is a new twist, last time we took care of this in France.
  • We’ve tuned up our French language skills. I expect our kids are going to impress their teachers.

Ruth found a rental house in the country that was just 5 mins from her lab via VTT (mountain bike) singletrack (!!) but we decided to go with a house in a more central Montpellier neighborhood.

There is yet much to do:

  • Find renters for our house in Fort Collins.
  • Pack and ship essentials across the ocean (our 7 and 10 year-olds have much greater requirements than last time), and store other stuff locally.
  • Learn how to transport a dog to France.
  • Find some late summer or fall trail running races in Europe to replace the Black Squirrel race I’d otherwise run here in September.
  • Plan our upcoming vacations. the French do this well in advance and you run the risk of finding everything is booked if you wait till the last minute.

Fort Collins and Montpellier have a lot in common. Both are fast-growing university and government centers with sunny climates and good access to outdoor recreation. But there are things Montpellier has that Fort Collins does not:

  • Year-round farmers markets. Local strawberries in April (now!) with cherries and other fruit not far behind.
  • Excellent local wine regions like La Clape and Pic Saint-Loup.
  • The Lucques variety of olive.
  • Bakeries. The Crêperie here in the Fort is good, but there are many places in Montpellier making baguettes just as well. And the sourdough (levain) baking in Montpellier is much better.
  • Merguez sausage.
  • Fresh fish and oysters.
  • A short drive to the beach.
  • TGV connections to Paris and Barcelona. Just 3 hours to each.
  • Historical public places like Place de la Comédie and the Promenade du Peyrou.

It’s not all milk and honey, however. Montpellier also has hellish traffic (I don’t expect it’s any better seven years later), high unemployment, ghettos of poorly integrated immigrants, and a large homeless population. It’s grittier than Fort Collins, more troubled, and in that sense more real.

I’ll write more about the move when summer arrives.

Rasterio 0.34

Last fall Even Rouault announced that GDAL 2.1 would have a new Amazon S3 virtual file system. Extending GDAL’s capability to make HTTP byte range requests to AWS’s HTTPS + XML S3 API, Even has made it possible to efficiently access partial content of S3 objects using certain formats like GeoTIFF. In other words, metadata of a GeoTIFF on S3 or overviews stored as sub-images can be accessed without retrieving the bulk of its image data. Génial!

With help from Even, Rob Emanuele, and Matt Perry, Rasterio 0.34 has a handy abstraction for this feature. Rasterio uses s3:// URIs instead of GDAL’s /vsis3/ paths because URIs are how we identify resources on the web and because this is the URI scheme – if unregistered – used by the AWS Command Line Interface. The same URIs you use with the AWS CLI

$ aws s3 ls s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF
2015-03-14 17:20:01   51099231 LC81390452014295LGN00_B1.TIF
2015-03-14 17:20:30    6626356 LC81390452014295LGN00_B1.TIF.ovr

can also be used with Rasterio:

$ rio info s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF --indent 2
{
  "nodata": null,
  "dtype": "uint16",
  "crs": "EPSG:32645",
  "bounds": [
    381885.0,
    2279085.0,
    610515.0,
    2512815.0
  ],
  "count": 1,
  "blockxsize": 512,
  "driver": "GTiff",
  "transform": [
    30.0,
    0.0,
    381885.0,
    0.0,
    -30.0,
    2512815.0
  ],
  "blockysize": 512,
  "tiled": true,
  "lnglat": [
    86.96327090815723,
    21.666821827007748
  ],
  "shape": [
    7791,
    7621
  ],
  "compress": "deflate",
  "res": [
    30.0,
    30.0
  ],
  "width": 7621,
  "height": 7791,
  "interleave": "band"
}

Rasterio gets its credentials in the same manner as the AWS CLI (see Configuring the AWS Command Line Interface). If you’re already using the AWS CLI no extra configuration is needed to start using Rasterio on S3 raster datasets.

A close read of the GDAL debug logs shows that only 16384 bytes of this 50MB TIFF are fetched in order to get the metadata printed above. That’s an efficiency of 3000:1.

The S3 virtual filesystem is only available in Rasterio if you have a GDAL library version >= 2.1.0dev. The macosx wheels for Rasterio 0.34 on PyPI contain GDAL version 2.1.0dev and are probably the easiest way to try this new feature.

Share and enjoy!

Moab from Landsat 8

I’ve still got the Colorado Plateau on my mind and spent some time yesterday dressing up the image below for a Mapbox Instagram post.

https://c2.staticflickr.com/2/1548/25444509183_637f031af3_b_d.jpg

2016-02-28 was a clear day in Southeastern Utah

I used Development Seed’s excellent landsat-util utility to find and download the source imagery, rio stack (from Rasterio) to combine individual bands into an RGB TIFF, rio clip to extract a smaller region, rio convert to scale and reformat the 16-bits per band TIFF into a 8-bit per band PNG, and Imagemagick for sigmoidal constrast enhancement and alpha adjustment.

All of the above was done using Python 3.5.1, which is now an option for landsat-util 0.13.0.

Spring Break in Arches and Canyonlands

School was out last week, Poudre Schools and Colorado State, and my family and I took a Spring Break trip to hike in Arches and Canyonlands National Parks. We stayed 4 nights in Moab, Utah, the biggest town in Southeastern Utah and 415 miles from Fort Collins. It’s a long, but scenic drive, and once we got west of Vail Pass the road was clear of snow and it was all smooth sailing down I-70 to Crescent Junction and then down US-191 to Moab.

I grew up in Utah, have family roots in Southern Utah, visited Arches and/or Canyonlands annually in the 70s and 80s, but haven’t been to Moab in a long time. 1992 may have been my last visit. The town has changed a lot on the surface and probably a little bit underneath. Moab’s espresso machine population has increased about twentyfold and you can walk out of a craft brewery with a 4-pack of full strength IPA. While grabbing some picnic food in the City Market, I heard Built to Spill’s The Plan as shopping music. It’s weird. Moab panders shamelessly to the forty-something weekend warrior like me.

On our first day, we drove into Arches N.P. As you see in the photo below, the dramatic features of Arches are largely found in the brick red Entrada Sandstone that sits atop the buff colored Navajo Sandstone.

https://upload.wikimedia.org/wikipedia/commons/thumb/a/a3/The_Three_Gossips_at_Arches_National_Park.jpg/1200px-The_Three_Gossips_at_Arches_National_Park.jpg

By Sanjay Acharya - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=8101229

There’s a particular angle when you’re driving back towards the park’s visitors center and very close to these titanic figures where one appears to be whispering in the ear of another. Something more serious than mere gossip is going on.

My kids loved scrambling on the park’s characteristic joints and fins. These are created by collapse of a deeply buried salt deposit, the sandstone fracturing as its foundation warps.

https://c2.staticflickr.com/2/1540/25332791273_eefb5471a9_b_d.jpg

On the Devil’s Garden Trail in Arches N.P.

Erosion of the fins can produce free standing rock arches. The park touts over 2000 of them. These arches are ephemeral in geologic time; the iconic Wall Arch (in the same Devil’s Garden area) collapsed on August 4, 2008. Landscape Arch, the second longest natural arch in the world, is ever more delicate. According to Wikipedia (no source that I can find) 43 arches are known to have collapsed since 1977.

https://c2.staticflickr.com/2/1639/25660076720_8419dc99c5_b_d.jpg

The Double O Arch

We weren’t the only family on the trail that day. We weren’t even the only family from Fort Collins, and bumped into people we know from school and soccer. On returning home our oldest learned that her class’s student teacher had been to Moab before us, on the earlier end of Spring Break.

Later that same day I did a 7 mile evening trail run from the BLM’s Hidden Valley trailhead up and over the rim southwest of Moab, stopping to check out a few panels of rock art and admire the problems along the Moab Rim jeep trail.

https://c2.staticflickr.com/2/1460/25660096300_d3915c6676_b_d.jpg

Trail sign between Hidden Valley and Moab Rim Trail

There was a point in this run where I realized that I was poorly prepared for any bad turn of events. I’d brought no water, no food, no map, no jacket, no light, and had no internet. I was all alone on the rim, the sun was going down, and the connection between the Hidden Valley and Moab Rim Trails wasn’t as obvious as I had expected. Visions of descending from the rim in the dark haunted me as I lost the trail briefly on a slickrock dome, but I managed to find my way through to the jeep trail and its companion hiking trail, named “Stairmaster.” Ruth and our kids picked me up at the finish, 30 minutes later than planned, and we went back into town for dinner.

https://c2.staticflickr.com/2/1546/25331941443_4e07a5218e_b_d.jpg

Dome above Hidden Valley

We spent one day hiking in Canyonlands N.P., which was practically empty compared to Arches. The highlights of this day for me were the clear views of the nearby La Sal and more distant Henry Mountain ranges afforded from the Island in the Sky district of the park.

https://c2.staticflickr.com/2/1686/25934818746_dfb76096cd_b_d.jpg

Mesa Arch and La Sal Mountains

On our last full day in the area we went back to Arches and Ruth and I traded turns at long runs in the backcountry, starting from Balanced Rock.

https://c2.staticflickr.com/2/1644/25960708835_61f538f563_b_d.jpg

Arches backcountry signage

I ran out and back, 14 miles in all, this time with ample water, food, jacket, and map. Just in case.

https://c2.staticflickr.com/2/1479/25960727405_490e772e6a_b_d.jpg

Whale Rock and the Henry Mountains in the distance

I saw only two other people during my run, a couple touring in their Jeep Cherokee. After they confirmed that I wasn’t stranded, we headed our separate ways.

https://c2.staticflickr.com/2/1594/25839919082_6da924fdfb_b_d.jpg

La Sal Mountains and the Windows

I’ve been on a mission to turn my kids into desert rats and I think I’ve finally succeeded. Every one of us is looking forward to coming back sooner rather than later.

Our admission to the parks was free thanks to Every Kid in a Park, a new White House initiative. Our oldest is in 4th grade and she and her peers can take family and friends to U.S. National Parks through August of 2016.

https://raw.githubusercontent.com/18F/ekip-api/master/ekip/everykid/static/img/nav-logo.png

A cool thing about the initiative’s website is that the source is on GitHub: https://github.com/18F/ekip-api. Good job, 18F!

https://upload.wikimedia.org/wikipedia/commons/1/1d/US-NationalParkService-Logo.svg

National Park Service Logo since 1951