Sean Gillies Blog

The new OSM editor is a pleasure to use and I've been inspired to fix up my neighborhood a bit. It was cool to see these historic homes (designated Fort Collins landmarks, in fact) pop up in MapBox just minutes after I submitted them.

Google Maps, on the other hand, has no historic homes and a rather outdated and (sadly) never to be realized development proposal in the neighborhood. Fixing Google's map for free is not high on my list of priorities.

Next, I'd like to finally get some Pleiades-related data into the OpenHistoricalMap sandbox.

Today was a pretty good day for the GeoJSON format.

Josh Livni announced on Twitter that the Google Map Engine API had been published. The little format that could has a good role in the API.

Ed Summers blogged about wikigeo.js, a library that gives you a GeoJSON interface to Wikipedia API results. Ed is absolutely right about how usable and right for the web mapping software has become since younger web developers and designers have started to displace older earth science programmers like myself.

A good example of which is Tom MacWright's edit geojson app: draw a shape and copy the GeoJSON representation, paste some GeoJSON and render the shape.

The business of geospatial standardization may have hit a rough patch recently, but things aren't all bad. Developers are still finding ways to agree, share, and do good work.

On the 22nd of May, I will be arriving in Minneapolis for the North American edition of the FOSS4G conference and on the next day I will be presenting a talk entitled GeoJSON is Spectacularly Wrong. I had also submitted a Python GIS talk and was surprised and concerned when both were accepted because there's no way I could pull off two. I'd enjoy talking about Python programming a little more, but it looks like the community is slightly more interested in GeoJSON. I hope that I didn't chose the wrong topic. Thursday night I'll be at the fancy shindig and because open source GIS events always need more dudes I will be bringing my brother. Friday I'll be sticking around to hack and catch up with what's going on in the field. It's been a long time since I've been to one of these conferences. Maybe I can share my Python 3 porting experience or offer a "show me your Python script and I'll show you how to make it measurably better/faster, or your money back" service. Yes, like in 2003, I'm going to a Yo la Tengo show one of the nights (here in Colorado, which is why I'm arriving late). No, unlike in 2003 I'm not driving.

The following week I'm going to be at Drew University in Madison, NJ, for the second part of ISAW and Drew's Linked Ancient World Data Institute. Just between you and me, I'm a bit burned out on talking about linked data; it's long past time to just start producing and using linked data to get stuff done. I believe that's what we're going to try to do at Drew this year.

The dumpgj script installed by Fiona 0.14 is among other things a vehicle for changing the way people think about GeoJSON.

There's a growing sense in the GeoJSON community that the crs object is over-engineered. It's rarely used, and those who do use it report that they would be just as happy with OGC URN strings instead of the existing objects. The new version of dumpgj adds an experimental:

"_crs": "urn:ogc:def:crs:OGC:1.3:CRS84"

to a feature collection when the input data is WGS84 longitude and latitude, as with Natural Earth shapefiles, and something like:

"_crs": "urn:ogc:def:crs:EPSG::3857"

for systems in the EPSG database (3857 is Spherical Mercator).

Additionally, dumpgj users can, with --use-ld-context, add more information to their GeoJSON files. The context object added to feature collections states more specifically what is meant by "Feature", "properties", "geometries", etc. Fiona's GeoJSON context looks like this:

{ "@context": {
  "Feature": "http://geovocab.org/spatial#Feature",
  "FeatureCollection": "_:n1",
  "GeometryCollection": "http://geovocab.org/geometry#GeometryCollection",
  "LineString": "http://geovocab.org/geometry#LineString",
  "MultiLineString": "http://geovocab.org/geometry#MultiLineString",
  "MultiPoint": "http://geovocab.org/geometry#MultiPoint",
  "MultiPolygon": "http://geovocab.org/geometry#MultiPolygon",
  "Point": "http://geovocab.org/geometry#Point",
  "Polygon": "http://geovocab.org/geometry#Polygon",
  "_crs": {
    "@id": "_:n2",
    "@type": "@id"
  },
  "bbox": "http://geovocab.org/geometry#bbox",
  "coordinates": "_:n5",
  "features": "_:n3",
  "geometry": "http://geovocab.org/geometry#geometry",
  "id": "@id",
  "properties": "_:n4",
  "type": "@type"
} }

I'm currently mapping GeoJSON's "id" and "type" to JSON-LD keywords, some other items to the GeoVocab spatial and geometry vocabularies, and giving the rest blank node identifiers until the GeoJSON community defines a namespace and permanent identifiers for them.

GeoJSON feature properties are the things that I think stand to benefit the most from a JSON-LD context. If a feature has a "title" property, is that an honorific like "Mr." or "Duchess of York"? Is it "NBA Championship"? Or something else? The dumpgj script has a --add-ld-context-item option that can be used to nail a property down a little more tightly. For example,

$ dumpgj --add-ld-context-item "title=http://purl.org/dc/elements/1.1/title" ...

adds:

"title": "http://purl.org/dc/elements/1.1/title",

to the feature collection's JSON-LD context. This is what "title" usually means in my applications.

Lastly, dumpgj will build the entire GeoJSON object in memory by default, but has a more economical --record-buffered option that only builds feature records in memory and writes them immediately to the output stream.

I've made the jump to developing Fiona primarily for Python 3.3 and Fiona 0.13 is the first release for Python 2.6, 2.7, or 3.3. Thanks to Cython, I'm able to offer Python 2/3 compatibility in a single source distribution.

The fiona.tool script I wrote about earlier is now called dumpgj.

$ dumpgj -h
usage: python -mfiona.tool [-h] [-d] [-n N] [--compact] [--encoding ENC]
                           [--record-buffered] [--ignore-errors]
                           infile [outfile]

Serialize a file's records or description to GeoJSON

positional arguments:
  infile             input file name
  outfile            output file name, defaults to stdout if omitted

optional arguments:
  -h, --help         show this help message and exit
  -d, --description  serialize file's data description (schema) only
  -n N, --indent N   indentation level in N number of chars
  --compact          use compact separators (',', ':')
  --encoding ENC     Specify encoding of the input file
  --record-buffered  buffer writes at record, not collection (default), level
  --ignore-errors    log errors but do not stop serialization

BTW, Comments are off for a while until these Chinese spammers move along.

One of my goals for Fiona is to make it the most boring and predictable package for reading and writing geo data ever. I want it to refute the notion that "spatial is special" and have no gotchas, no surprises. As a benchmark for predictability and dullness consider fiona.tool, a minimal and limited replacement for ogrinfo and ogr2ogr inspired by Python's json.tool. You give it a data file name and write a description to a file or stdout

$ python -mfiona.tool -d docs/data/test_uk.shp

Description of source: <open Collection 'docs/data/test_uk.shp:test_uk', mode
'r' at 0x2038b10>

Coordinate reference system (source.crs):
{'datum': 'WGS84', 'no_defs': True, 'proj': 'longlat'}

Format driver (source.driver):
'ESRI Shapefile'

Data description (source.schema):
{'geometry': 'Polygon',
 'properties': {u'AREA': 'float',
                u'CAT': 'float',
                u'CNTRY_NAME': 'str',
                u'FIPS_CNTRY': 'str',
                u'POP_CNTRY': 'float'}}

or convert the file's data to GeoJSON.

$ python -mfiona.tool --indent 2 docs/data/test_uk.shp /tmp/test_uk.json
$ head /tmp/test_uk.json
{
  "type": "FeatureCollection",
  "features": [
    {
      "geometry": null,
      "id": "0",
      "properties": {
        "POP_CNTRY": 60270708.0,
        "CNTRY_NAME": "United Kingdom",
        "AREA": 244820.0,

Most of the source is devoted to parsing arguments and formatting the descriptions. The core of it is this:

# The Fiona data tool.

if __name__ == '__main__':

    # ...
    args = parser.parse_args()

    with sys.stdout as sink:

        with fiona.open(args.infile, 'r') as source:

            if args.description:
                meta = source.meta.copy()
                meta.update(name=args.infile)
                if args.json:
                    sink.write(json.dumps(meta, indent=args.indent))
                else:
                    sink.write("\nDescription of source: %r" % source)
                    print("\nCoordinate reference system (source.crs):")
                    pprint.pprint(meta['crs'], stream=sink)
                    # ...
            else:
                collection = {'type': 'FeatureCollection'}
                collection['features'] = list(source)
                sink.write(json.dumps(collection, indent=args.indent))

Does that underwhelm you? Are you asking "is that all there is?" Good. It underwhelms me, too. All there really is to Fiona is an open function that returns an iterable object that you can treat as a limited file or io stream and plain old Python dicts. No layers, no features, no shadow classes and reference counting. Just concepts that Python programmers already understand. Files, iterators, and dicts. Nothing special.

By the way, how about that use of sys.stdout as a context manager? I expected it wouldn't be a problem, and it isn't. The Python language keeps getting better and better. I'm developing Fiona for Python 2.7 right now and experimenting with a full switch to 3.3, with backported releases for 2.7 and 2.6.

The snow finally stopped this afternoon. I didn't measure like I feel like I should have, but I read that we received at least 50 cm of snow. That's unconsolidated snow depth. It's fairly dense and has now settled to a foot of firm snow. My youngest daughter lost a boot in the snow this morning and we were unable to find it. It'll show up in a week or so, I suppose.

We desperately needed this water, but I'm still a bit bummed about our salad greens and peas and almost-ready-to-bloom tulips crushed flat under the snow. We're fortunate that trees hadn't begun to leaf out yet and didn't collect enough snow or ice to have their branches broken.

A neat surprise at the end of the afternoon was finding a large mixed flock of Robin and Mountain Bluebird a few houses down the street. I've only seen a couple bluebirds in Fort Collins before, never an entire flock. What a time they picked to head to the hills.

The light was awkward and I'm a rank amateur photographer so I only captured just a fraction of the otherwordly blueness of these beautiful birds.

I've tagged 0.12 and uploaded an sdist to PyPI: https://pypi.python.org/pypi/Fiona/0.12. Changes:

• Fix broken installation of extension modules (#35).
• Log CPL errors at their matching Python log levels.
• Use upper case for encoding names within OGR, lower case in Python.

The 0.11 release was broken and I yanked it, but it had some nice new features and bug fixes, too:

• Cythonize .pyx files (#34).
• Work with or around OGR's internal recoding of record data (#35).
• Fix bug in serialization of int/float PROJ.4 params.

A couple of OS X users who are satisfying Fiona's GDAL requirements with the frameworks from kyngchaos.com have contacted me about this import error:

>>> import fiona
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/__init__.py", line 70, in <module>
    from fiona.collection import Collection, supported_drivers
  File "/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/collection.py", line 7, in <module>
    from fiona.ogrext import Iterator, Session, WritingSession
ImportError: dlopen(/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so, 2): Symbol not found: _CPLSetThreadLocalConfigOption
  Referenced from: /Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so
  Expected in: flat namespace
 in /Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so

This is a sign that you need

DYLD_LIBRARY_PATH=/Library/Frameworks/GDAL.framework/GDAL

in your environment, whether that's a virtualenv or set in your .profile.