2013 (old posts, page 1)

Fiona makes reading and writing data boring

One of my goals for Fiona is to make it the most boring and predictable package for reading and writing geo data ever. I want it to refute the notion that "spatial is special" and have no gotchas, no surprises. As a benchmark for predictability and dullness consider fiona.tool, a minimal and limited replacement for ogrinfo and ogr2ogr inspired by Python's json.tool. You give it a data file name and write a description to a file or stdout

$ python -mfiona.tool -d docs/data/test_uk.shp

Description of source: <open Collection 'docs/data/test_uk.shp:test_uk', mode
'r' at 0x2038b10>

Coordinate reference system (source.crs):
{'datum': 'WGS84', 'no_defs': True, 'proj': 'longlat'}

Format driver (source.driver):
'ESRI Shapefile'

Data description (source.schema):
{'geometry': 'Polygon',
 'properties': {u'AREA': 'float',
                u'CAT': 'float',
                u'CNTRY_NAME': 'str',
                u'FIPS_CNTRY': 'str',
                u'POP_CNTRY': 'float'}}

or convert the file's data to GeoJSON.

$ python -mfiona.tool --indent 2 docs/data/test_uk.shp /tmp/test_uk.json
$ head /tmp/test_uk.json
  "type": "FeatureCollection",
  "features": [
      "geometry": null,
      "id": "0",
      "properties": {
        "POP_CNTRY": 60270708.0,
        "CNTRY_NAME": "United Kingdom",
        "AREA": 244820.0,

Most of the source is devoted to parsing arguments and formatting the descriptions. The core of it is this:

# The Fiona data tool.

if __name__ == '__main__':

    # ...
    args = parser.parse_args()

    with sys.stdout as sink:

        with fiona.open(args.infile, 'r') as source:

            if args.description:
                meta = source.meta.copy()
                if args.json:
                    sink.write(json.dumps(meta, indent=args.indent))
                    sink.write("\nDescription of source: %r" % source)
                    print("\nCoordinate reference system (source.crs):")
                    pprint.pprint(meta['crs'], stream=sink)
                    # ...
                collection = {'type': 'FeatureCollection'}
                collection['features'] = list(source)
                sink.write(json.dumps(collection, indent=args.indent))

Does that underwhelm you? Are you asking "is that all there is?" Good. It underwhelms me, too. All there really is to Fiona is an open function that returns an iterable object that you can treat as a limited file or io stream and plain old Python dicts. No layers, no features, no shadow classes and reference counting. Just concepts that Python programmers already understand. Files, iterators, and dicts. Nothing special.

By the way, how about that use of sys.stdout as a context manager? I expected it wouldn't be a problem, and it isn't. The Python language keeps getting better and better. I'm developing Fiona for Python 2.7 right now and experimenting with a full switch to 3.3, with backported releases for 2.7 and 2.6.


stdout as a context manager

Author: Jonathan Hartley


Lovely post as always, but I'm puzzled by the use of stdout as a context manager, and my google-fu apparently isn't up to it this early in the morning (lots of results for using context managers to redirect stdout)

I can see from here (http://docs.python.org/2/library/stdtypes.html#file.close) that File objects used as context managers get closed when exiting the block, so I expect that you're closing (and hence presumably flushing) stdout when you leave the outermost 'with'. But this is the end of the program, so this would happen anyway.

So why are you doing it? Thanks.

Re: Fiona makes reading and writing data boring

Author: Sean

What I didn't point out is that in the actual code, the sink (output stream) could be either stdout or a Fiona Collection object depending on the parsed script args. That they both work as context managers lets me write less code. The reasons I'm using ``with`` instead of letting the file close at garbage collection time are that it future-proofs my script a bit against code I may add after the features are read and written and that I'm planning to hang some OGR state on the Fiona Collection soon and want to use __enter__ and __exit__ to manage it reliably.

April Snowstorm

The snow finally stopped this afternoon. I didn't measure like I feel like I should have, but I read that we received at least 50 cm of snow. That's unconsolidated snow depth. It's fairly dense and has now settled to a foot of firm snow. My youngest daughter lost a boot in the snow this morning and we were unable to find it. It'll show up in a week or so, I suppose.


We desperately needed this water, but I'm still a bit bummed about our salad greens and peas and almost-ready-to-bloom tulips crushed flat under the snow. We're fortunate that trees hadn't begun to leaf out yet and didn't collect enough snow or ice to have their branches broken.

A neat surprise at the end of the afternoon was finding a large mixed flock of Robin and Mountain Bluebird a few houses down the street. I've only seen a couple bluebirds in Fort Collins before, never an entire flock. What a time they picked to head to the hills.


The light was awkward and I'm a rank amateur photographer so I only captured just a fraction of the otherwordly blueness of these beautiful birds.

Fiona 0.12

I've tagged 0.12 and uploaded an sdist to PyPI: https://pypi.python.org/pypi/Fiona/0.12. Changes:

  • Fix broken installation of extension modules (#35).
  • Log CPL errors at their matching Python log levels.
  • Use upper case for encoding names within OGR, lower case in Python.

The 0.11 release was broken and I yanked it, but it had some nice new features and bug fixes, too:

  • Cythonize .pyx files (#34).
  • Work with or around OGR's internal recoding of record data (#35).
  • Fix bug in serialization of int/float PROJ.4 params.

A couple of OS X users who are satisfying Fiona's GDAL requirements with the frameworks from kyngchaos.com have contacted me about this import error:

>>> import fiona
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/__init__.py", line 70, in <module>
    from fiona.collection import Collection, supported_drivers
  File "/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/collection.py", line 7, in <module>
    from fiona.ogrext import Iterator, Session, WritingSession
ImportError: dlopen(/Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so, 2): Symbol not found: _CPLSetThreadLocalConfigOption
  Referenced from: /Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so
  Expected in: flat namespace
 in /Users/kilroy/Desktop/fiona_env/lib/python2.7/site-packages/fiona/ogrext.so

This is a sign that you need


in your environment, whether that's a virtualenv or set in your .profile.


Re: Fiona 0.12

Author: Martin Laloux

I never had a problem to compile Fiona on Mac OS X with the frameworks from kyngchaos.com

# First, build the extensions (ogrext.so and ogrinit.so):

python setup.py build_ext -I/Library/Frameworks/GDAL.framework/Versions/1.9/Headers -L/Library/Frameworks/GDAL.framework/Versions/1.9/unix/lib -lgdal

# install the module

python setup.py install

Re: Fiona 0.12

Author: Sean

Martin, those people have succeeded in building the extension modules, but are having dlopen failure at run time. Btw, if /Library/Frameworks/GDAL.framework/Programs is on your path, gdal-info will find the headers and library for you. "pip install Fiona" just works in that case.

Re: Fiona 0.12

Author: Frank

Hi Sean,

I just read your tweet about the python + shapefile Fiona to OGR ratio and that made me wonder if I could substitute my ogr2ogr subprocess stuff with Fiona. Thing is I am reading from a Postgis database and converting mostly to some well known text formats like GeoJSON or KML. But I've only found Fiona examples with Shapefiles. Does Fiona support reading from Postgis? What about SQL? With OGR you can pass in a custom SQL query with the -sql switch which is handy when subselecting stuff.

Anyway I think Fiona and Shapely are great projects.


Re: Fiona 0.12

Author: Sean

No, Fiona doesn't support PostGIS, and that's because there are already plenty of Python database adapter packages. I also think ogr2ogr is a killer program and a great way of converting PostGIS tables to other formats.

Re: Fiona 0.12

Author: Frank

Thanks for the clarification. Though I find it a pity that I cannot try Fiona right now I do very well understand the reasons. I started myself once an attempt to write a more pythonic interface to ogr(2ogr) and you quickly hit the wall, especially when it comes to conversion speed.

Wings of Spring

Not only was I out of town during the days the vultures usually return to Fort Collins, I no longer live around the corner from the big roost on Mountain Ave. Last May, my family and I left Old Town for the Sheely Addition and will no longer be able to keep such close tabs on the Harbingers of Spring. I think this may be the last post in this series.

I did see Turkey Vultures during my family trip to Zion National Park last week, but I did not see what I was hoping to see: a California Condor. Rangers told us that Condors have been sighted in the park already this Spring, but I saw not a one despite being outdoors looking up every moment from sunrise to sunset. Until next time I'll have to make do with photos like this one by ChrisWegg:


Previously: 2012, 2011, 2010, 2009, 2008.

Fiona 0.10

I've tagged 0.10 and uploaded an sdist to PyPI: https://pypi.python.org/pypi/Fiona/0.10.


  • Add function to get the width of str type properties.
  • Handle validation and schema representation of 3D geometry types (#29).
  • Return {'geometry': None} in the case of a NULL geometry (#31).

I've also updated the documentation with an explanation in the manual of how "3D" geometries are read and written.

Fiona 0.9

It's tagged and uploaded to PyPI.

Here's a quick example that demonstrates a couple new features: support for variable fixed text field width and mixing of single and multipart one and two-dimensional geometric objects (linestrings and multilinestrings; polygons and multipolygons) in Shapefiles.

>>> c = fiona.open('ne_110m_geography_regions_polys.shp', 'r')

The fiona.open function is the new way to get an open collection, but the old fiona.collection function still works. Keep in mind that the first parameter of fiona.open (the path) is the only positional parameter. All others are keyword parameters, but it's fine to treat the mode, the 2nd parameter in the function definition, as a positional parameter as I did above.

>>> c.name
>>> c.mode
>>> c.closed

A collection has a few file-like properties.

>>> c.crs
{'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj': 'longlat'}
>>> c.driver
'ESRI Shapefile'
>>> import pprint
>>> pprint.pprint(r.schema)
{'geometry': 'Polygon',
 'properties': {u'featurecla': 'str:32',
                u'name': 'str:254',
                u'namealt': 'str:254',
                u'region': 'str:50',
                u'scalerank': 'float',
                u'subregion': 'str:50'}}

The collection describes itself as containing records with polygon type geometries and six properties. Five of those properties are str (text) type, with maximum lengths of values being 32 ('str:32') to 254 ('str:254') characters.

>>> records = list(r)
>>> len([r for r in records if r['geometry']['type'] == 'Polygon'])
>>> len([r for r in records if r['geometry']['type'] == 'MultiPolygon'])

There are 52 records with polygon type geometries in this file and 8 with multipolygons.

All the Fiona examples, including the one only for functional programming geeks, are updated for version 0.9. So is the manual.

I've been remiss in pointing out a couple good blog posts about using Fiona. Tom MacWright wrote GIS with Python, Shapely, and Fiona last fall (widely linked, Tom's reach is huge), and Steve Citron-Pousty wrote Using Open Source GIS tools for spatial data - QGIS, GDAL and Python just the other day. I'm excited to see creative and super productive programmers putting Fiona to work and writing about it.

More Fiona

Last night, I saw that Tom MacWright had posted a script for combining Shapefile fields. I thought I'd try to adapt it to Fiona 0.9 and a slightly more functional (FP, I mean) style, and I'm glad I did because I in the process I discovered that my slightly old version of GDAL and OGR isn't detecting the Windows-1252 encoding of the Natural Earth dataset. Version 0.9.1 of Fiona is now tagged and uploaded to PyPI and it lets a user specify the proper encoding of files if needed.

My version of Tom's script tries to be smart about the width of the new text field and warns if the new values will be truncated. The combining is also done within a function, which allows all output to be written in one statement. Our versions of the script run equally fast.

import fiona
import itertools
import logging
import sys

logging.basicConfig(stream=sys.stderr, level=logging.WARN)

def text_width(val):
    return int((val.split(":")[1:] or ["80"])[0])

with fiona.open(
        encoding='Windows-1252') as inp:

    output_schema = inp.schema.copy()

    # Declare that the output shapefile will have a new text field called
    # a3_sov, with an appropriately sized field width. 254 is the maximum
    # for this format.
    width = min(254,
        text_width(inp.schema['properties']['sov_a3']) +
    output_schema['properties']['a3_sov'] = 'str:%d' % width

    # Define a function that combines properties to produce a value
    # for the 'a3_sov' property. It warns if the value will be truncated
    # at 254 characters.
    def combine_fields(rec):
        val = rec['properties']['sov_a3'] + rec['properties']['sovereignt']
        if len(val) > 254:
            log.warn("Value %s will be truncated to 254 chars", val)
        rec['properties']['a3_sov'] = val
        return rec

    with fiona.open(
            'ne_50m_admin_0_countries_a3_sov.shp', 'w',
            schema=output_schema) as out:

            itertools.imap(combine_fields, inp))

Trying something new

I'm working on submissions for the upcoming FOSS4G-NA conference (aka MapServer User Meeting XI) and am experimenting with opening the abstracts up to comments and forks as I write them at https://gist.github.com/sgillies/4745903. If you like the direction I'm going with the abstracts, feel free to comment. If it looks like I'm overlooking something important, please comment. If you're interested in co-presenting please fork the Gist and we'll talk. The results will go into the FOSS4G-NA submission spreadsheet on the 14th.


Re: Trying something new

Author: Martin Davis

This looks like a couple of good reasons to try and make it to FOSS4GNA-2013!

Shapely 1.2.17

I've tagged 1.2.7 and uploaded an sdist to PyPI. There's an important bug fix for Windows users in this one. Please heed the warning I've attached to the README, dear folks.


Windows users: do not under any circumstances use pip (or easy_install) to uninstall Shapely versions <= 1.2.16. Due to the way Shapely used to install its GEOS DLL and a distribute or setuptools bug, your Python installation may be broken by an uninstall command. Shapely 1.2.17 will uninstall safely.

The highlight of 1.2.17 is the module of affine transformation functions and utilities written by Mike Toews: shapely.affinity. The manual isn't up to date yet, but the module is itself very well documented. Here's a brief example of the rotate function, which calls the module's more general affine_transformation function.

# Default origin is the center of the geometry's bounding box.
# Default unit of angle is the decimal degree, positive is counter-clockwise.
# a) 90°, default origin
from shapely.affinity import rotate
rotated = rotate(line, 90.0)

# b) 90°, origin: (0, 0)
rotated = rotate(line, 90.0, origin=(0.0, 0.0))

In the figures below, the original line is shown in low alpha grey, the rotated line in blue, and the origin of rotation as a point.


I'm pleased to be able to say that 6 different programmers contributed to this release. I think this Shapely package just might catch on after all.

Update (2013-01-28): Windows binaries by Allan Adair: http://pypi.python.org/pypi/Shapely#downloads.

I can has my 15 minutes of Street View fame?

It's been a warm week here in the Fort. The snow is gone and the ground is a bit muddy. Wednesday, I rode my bike to the Pineridge Natural Area trailhead at Cottonwood Glen Park with the intent of running, but found the trail closed. I appreciate that the City is looking out for its trails, but was nonetheless a bit disappointed. Is it open today? I'd probably have to call to find out. If only there was something like a live StreetView so I could check if the gate is open or closed...

Which reminds me that last June I saw a Google StreetView car coming out of the parking lot at the south end of Overland Drive as I came down the Pineridge trail and through the gate. I reported at the time:

The sweaty guy in black with the blurry face running down from Pineridge in a future Street View update? Me.

So I checked and last summer's imagery is now live (I'm not sure when it was updated – my neighborhood's imagery is from August, and probably wasn't the last to be driven, so maybe only quite recently). The sweaty closeups of me are mercifully omitted (Overland dead ends here and there must have been double coverage as the car drove in and back out) but I am clearly visible walking back to my bike as the StreetView car leaves.

Oh, and I have my Twitter archive, too, which is how I tracked down that old post.