Geoprocessing for humans: pygp

I'm not the only one simplifying terrible Python APIs in the geospatial world. Yesterday, I ran across a blog post about software named pygp. Very much about ArcGIS records and fields, it models data differently than Fiona does but similarly eliminates a lot of boilerplate and provides simple access to all coordinates of a record's shape field.

def example_geometry(path):
    Example showing use of Geometry helper class that does the heavy lifting
    on the geometry object and returns something quite similar to WKT/GeoJSON

    Structure is simple, a tuple of tuple of Point objects, very similar to
    Avenue days of geometry and WKT MultiLineString etc.
    (((0, 498266, 6100519, None, None), (0, 499775, 6100281, None, None),
      (0, 500224, 6098694, None, None), (0, 499616, 6097662, None, None),
      (0, 498346, 6096789, None, None)))

    :param path: Workspace Path
    :type path: str
    feature_class = FeatureClass(osjoin(path, POLYGON))
    for srow in
        print srow.get_value(

# End example_geometry function

I don't know whether pygp has eliminated the need to count references to cursors and records or just omitted

del feature_class

from the example. I'd have looked in the code, but I couldn't find a link. I bet a lot of people would love to see it on GitHub.


Re: Geoprocessing for humans: pygp

Author: Pedant

No need to del feature_class - it's a local that will be garbage collected when example_geometry() returns.

Re: Geoprocessing for humans: pygp

Author: Sean

Correct in this case. But what would happen if I created a new FeatureClass instance after the above code in the same function, using the same parameters? Would I encounter locked files? My question isn't hypothetical:

Re: Geoprocessing for humans: pygp

Author: Jason Humber

Backing up a little, in the 9.x world of arcgisscripting there was always an imposed need to delete the row and cursor as a means of closing them and dropping references. This usually looked like del srow, srows statement and was usually placed after the while loop. Being forced to use a while loop and the need to use del always felt odd to us so in the cursor implementation we have in pygp we take care of closing cursors once the loop has exhausted.

So in this case there is no need to del feature_class since the locks on the feature class are dropped when the clean-up/closing is done at the end of the loop.

Re: Geoprocessing for humans: pygp

Author: Jason Humber

Just noticed the last line of your post, drop me an email...

Geoprocessing for humans: date and time

Fiona 0.7 roughly supports OGR date/time fields. Date, time, and datetime field values are turned into strings conforming to RFC 3339 "Date and Time on the Internet: Timestamps". Fiona is ignoring time zones in this version, but then OGR itself doesn't have much support for time zones, and neither do common vector data formats.

There's an example of adding a date type field to a shapefile in

with collection("docs/data/test_uk.shp", "r") as source:
    schema = source.schema.copy()
    schema['geometry'] = 'Point'
    schema['properties']['date'] = 'date'
    with collection(
            "test_write_date.shp", "w", "ESRI Shapefile", schema
            ) as sink:
        for f in source.filter(bbox=(-5.0, 55.0, 0.0, 60.0)):
            f['geometry'] = {
                'type': 'Point',
                'coordinates': f['geometry']['coordinates'][0][0] }
            f['properties']['date'] = "2012-01-29"

A look at the shapefile's feature table in QGIS shows that I'm getting writing of dates right.

Reading that shapefile back in Fiona confirms that dates are read properly.

>>> from fiona import collection
>>> c = collection("test_write_date.shp", "r")
>>> from pprint import pprint
>>> pprint(c.schema)
{'geometry': 'Point',
 'properties': {'AREA': 'float',
                'CAT': 'float',
                'CNTRY_NAME': 'str',
                'FIPS_CNTRY': 'str',
                'POP_CNTRY': 'float',
                'date': 'date'}}
>>> for f in c:
...     print f['properties']['date']

Be careful with this feature. Unless your data is destined for a legacy system, I think you're better off keeping track of time as RFC 3339 strings in a text field. Among other advantages, you'd gain millisecond precision and precise expression of UTC time offset.


Re: Geoprocessing for humans: date and time

Author: Michael Weisman

Good to see more OGR field types supported.

Any reason for using a string representation of dates within Fiona rather than using Python's native datetime objects and converting them to the string representation OGR is expecting at write time?

Re: Geoprocessing for humans: date and time

Author: Sean

One of my goals for Fiona is to go light on classes, stick to built in Python types, and make sure that features can be trivially serialized to JSON. Datetime objects fail that last test. Try json.dumps(datetime.time(...)) and you'll get a TypeError. I did consider (year, month, day, hour, minute, second, millisecond) tuples, but RFC 3339 strings suit my needs better. They're ready to show to humans, sort them lexically and you get cheap temporal sorting, and they're easy to parse. Fiona has fiona.rfc3339.parse_date and friends which take strings and return tuples you could pass to the datetime constructors.


I'm going to be in Santa Clara, CA, Thursday evening through Sunday morning to attend PyCon US 2013. I'm not presenting, but signed up to be the runner for a scientific Python session on Saturday afternoon. Although I'm working in the humanities, (just between you and me) I'm still a science and engineering type at heart and I haven't found much of a humanities computing presence at PyCon. I hope to see you there.

The conference is moving to Montréal next year. Summer, I hope!


Re: PyCon

Author: Martin Davis

If I were you I would wish for Montreal in September. Summer is pretty hot and sticky! Fall is beatiful, though - especially if you happen to hit the 2-week window where the leaves change colour. (I know you get nice fall colours in the Front Range, but nothing beats Northeastern North America in the fall. Speaking as someone from the Wet Coast, where the dominant fall colour is misty green...)

Re: PyCon

Author: Sean

Good point. I'll happily roll with whatever the locals pick, though I'm more likely to be able to stick around for the sprints if it's a family vacation.

Geoprocessing for humans: a pip requirements file

In the geospatial software I'm writing and using these days, concerns are well separated. Fiona reads and writes features. Only. Shapely provides computational geometry algorithms. Only. Pyproj (not my work, but a favorite package) transforms coordinates between spatial reference systems. Only. The separation of concerns helps keep interactions between them predictable and as a user you pay only for what you eat.

A programmer-analyst's daily work has all the above concerns (and more, probably). A pip requirements file makes installing all three packages as easy as installing a single package like osgeo.ogr. I've uploaded one to GitHub: This Gist includes an example of using Fiona, pyproj and Shapely together. Fetching them all, assuming you've got pip and the GDAL/OGR libs and headers already on your system, is just:

$ pip install -r


Re: Geoprocessing for humans: a pip requirements file

Author: Paolo Corti

Hey Sean, really nice post, and just a couple of paragraphs long: well done.

I could read this easily even just after lunch :)

Re: Geoprocessing for humans: a pip requirements file

Author: Sean

That's my mantra: Omit needless words and code.

Notes on learning Clojure

I'm learning Clojure and having fun with it. I never learned a Lisp in school like many programmers my age did. The one variant I did try, about 15 years ago, was Scheme. I did a little Gimp scripting with it but nothing else. I think I had to mature a bit before I could appreciate the Lisp style for what it is.

For a language that's designed to be more simple than easy, it's surprisingly easy to use Java classes in Clojure. This is the first code I've written using JTS classes in a while.

user=> (.buffer
  (.read ( "POINT (0 0)")
#<Polygon POLYGON ((1 0, 0.9807852804032304 -0.1950903220161282, ...))>

I assumed I'd have to write something like a Python C extension module to do this and am thrilled to be wrong.


Re: Notes on learning Clojure

Author: Michael Weisman

One of the great things about alternative JVM languages is how simple it is to use existing java libraries. I managed to get JTS talking to Google Refine through the Jython query interface without having to write any wrappers.

Re: Notes on learning Clojure

Author: Sean

I remember you mentioning this, Mike. So much less complicated than ctypes, Cython, etc.

Shapely 1.2.14

Shapely 1.2.14 is up on PyPI: and the documentation has been updated: Mike Toews has made coordinate and (homogeneous) geometry sequences sliceable. For example, you can now get every other vertex of a line string or linear ring like this:

>>> from shapely.geometry import Point
>>> ring = Point(0.0, 0.0).buffer(1.0).exterior
>>> len(ring.coords)
>>> evens = ring.coords[::2]
>>> len(evens)

and a geometry collecting the odd points of another multi point geometry can be had like:

>>> from shapely.geometry import MultiPoint
>>> multi = MultiPoint(evens)
>>> len(multi)
>>> odds = multi[1::2]
>>> odds.geom_type
>>> len(odds)
>>> list(odds)
[<shapely.geometry.point.Point object at 0x744d30>, ... ]

Heterogeneous geometry collections can't be sliced in 1.2.14. Down the road, maybe.

Linked Ancient World Data Institute

From the ISAW blog:

ISAW will host the Linked Ancient World Data Institute (LAWDI) from May 31st to June 2nd, 2012 in New York City. Applications are due 17 February 2012.

LAWDI, funded by the Office of Digital Humanities of the National Endowment for Humanities, will bring together an international faculty of practitioners working in the field of Linked Data with twenty attendees who are implementing or planning the creation of digital resources.

More information, including a list of faculty, and application instructions are available at the LAWDI page on the Digital Classicist wiki.

I'm excited to get to play a small part in this. I was on the faculty of the UVA Scholars' Lab's Institute for Enabling Geospatial Scholarship in 2009 and got to see first hand how an excellent institute is run. We'll try to live up to the high standards of #geoinst.

Geoprocessing for humans: the Fiona manual

I'm making progress on the Fiona manual: When it's finished, there will be a Fiona 1.0 release. If this doesn't help you understand when and where you'd want to use Fiona over OGR's own Python bindings, let me know. Also let me know if you think I'm misrepresenting those bindings. I'm a big fan of GDAL and OGR, but I don't want to tip toe around its gotchas. I want to banish them.

Home renovation 2012

We finally got our permit and demolition in our new house started on Friday.

There have been a few surprises but nothing terrible yet. It's a ranch style house, built in 1959, and an ordained city landmark. We love the size and layout; all it needs is a new kitchen and a little modernization here and there.

Fiona 0.6.1

Fiona 0.6.1, the version with support for coordinate reference systems (and undefined reference systems thanks to Ariel Núñez) is on PyPI: I think it's ready for wider testing. If you've been able to install OGR's own Python bindings, you'll find Fiona at least as easy to install on Unix or Linux. It might be as simple as

$ pip install Fiona


$ easy_install Fiona

The README has more detailed installation instructions for people like me with non-standard environments.

Update (2012-01-22): See