I've been writing some posts under the rubric of "geoprocessing for humans". These are generally about keeping software simple, predictable, symmetrical, safe, readable, and well-documented. And simple. Most of all: simple.

• 1124. Geoprocessing for humans: close() and with
• 1122. Geoprocessing for humans: pygp
• 1121. Geoprocessing for humans: date and time
• 1119. Geoprocessing for humans: a pip requirements file

A couple of my recent posts are a little different, being about functional programming, partial functions, processing entire files of records without using any for loops, etc – stuff that a GIS programmer/analyst might not recognize as scripting or as even practical for all I know. I think I'll categorize these as "geoprocessing for hipsters". I don't usually begin coding in this way, but often arrive here after a few iterations.

• 1132. Mapping and Reducing
• 1128. Geoprocessing for hipsters: translating features
• 1126. More learning from Haskell
• 1125. Fiona, pyproj, Shapely in a functional style

If you find these rubrics fun and/or pedagogical, jump on in.

Fiona 0.8 has a bunch of important new features: file bounds/extents, sync of bounds and record count with writes, protection against iteration restarts, and more:

0.8 (2012-02-21)
----------------
- Replaced .opened attribute with .closed (product of collection() is always
  opened). Also a __del__() which will close a Collection,  but still not to be
  depended upon.
- Added writerecords method.
- Added a record buffer and better counting of records in a collection.
- Manage one iterator per collection/session.
- Added a read-only bounds property.

I've also expanded the manual (still under construction), documenting among other things what Fiona does to keep users from stumbling over external resources: http://toblerity.github.com/fiona/manual.html#closing-files. Don't del your collections or set them to None and then sacrifice a chicken while crossing your fingers and hoping for the best – just call close() or use with and enjoy some peace of mind.

Update (2012-02-22): "layers" (as in OGR) in the last sentence changed to "collections" (as in Fiona).

Things you find in Pleiades that you don't find in a typical geographic information system include relationships between places that are expressed in the data itself. I blogged about this last summer (accompanying figure reprised below) and talked about it at the 2011 Digital Humanities conference (our poster here).

Until today, connections between places have been a little sparse. Loading 200+ milecastles and turrets with connection to Hadrian's Wall changes the situation at least in Britannia. The representation of Hadrian's Wall in Pleiades doesn't have a published spatial extent of its own, but gets one by virtue of its connections to these other small places. Here is a screenshot from Google Maps.

Here's a closeup near Brampton. The lone placemark at the bottom represents an old quarry that supplied nearby milecastles with rock. For the moment at least, we're asserting that the quarry was connected to the fortifications.

The remains of Hadrian's Wall are a popular hiking itinerary today. The connected places in the maps above don't quite describe the itinerary because they aren't chained to each other, but I can't stop thinking that we should be making it possible to represent ancient itineraries like the Antonine using places from Pleiades.

I'm not the only one simplifying terrible Python APIs in the geospatial world. Yesterday, I ran across a blog post about software named pygp. Very much about ArcGIS records and fields, it models data differently than Fiona does but similarly eliminates a lot of boilerplate and provides simple access to all coordinates of a record's shape field.

def example_geometry(path):
    """
    Example showing use of Geometry helper class that does the heavy lifting
    on the geometry object and returns something quite similar to WKT/GeoJSON

    Structure is simple, a tuple of tuple of Point objects, very similar to
    Avenue days of geometry and WKT MultiLineString etc.
    (((0, 498266, 6100519, None, None), (0, 499775, 6100281, None, None),
      (0, 500224, 6098694, None, None), (0, 499616, 6097662, None, None),
      (0, 498346, 6096789, None, None)))

    :param path: Workspace Path
    :type path: str
    """
    feature_class = FeatureClass(osjoin(path, POLYGON))
    for srow in feature_class.search():
        print srow.get_value(
            feature_class.shape_field_name).as_tuple()

# End example_geometry function

I don't know whether pygp has eliminated the need to count references to cursors and records or just omitted

del feature_class

from the example. I'd have looked in the code, but I couldn't find a link. I bet a lot of people would love to see it on GitHub.

Fiona 0.7 roughly supports OGR date/time fields. Date, time, and datetime field values are turned into strings conforming to RFC 3339 "Date and Time on the Internet: Timestamps". Fiona is ignoring time zones in this version, but then OGR itself doesn't have much support for time zones, and neither do common vector data formats.

There's an example of adding a date type field to a shapefile in test_collection.py.

with collection("docs/data/test_uk.shp", "r") as source:
    schema = source.schema.copy()
    schema['geometry'] = 'Point'
    schema['properties']['date'] = 'date'
    with collection(
            "test_write_date.shp", "w", "ESRI Shapefile", schema
            ) as sink:
        for f in source.filter(bbox=(-5.0, 55.0, 0.0, 60.0)):
            f['geometry'] = {
                'type': 'Point',
                'coordinates': f['geometry']['coordinates'][0][0] }
            f['properties']['date'] = "2012-01-29"
            sink.write(f)

A look at the shapefile's feature table in QGIS shows that I'm getting writing of dates right.

Reading that shapefile back in Fiona confirms that dates are read properly.

>>> from fiona import collection
>>> c = collection("test_write_date.shp", "r")
>>> from pprint import pprint
>>> pprint(c.schema)
{'geometry': 'Point',
 'properties': {'AREA': 'float',
                'CAT': 'float',
                'CNTRY_NAME': 'str',
                'FIPS_CNTRY': 'str',
                'POP_CNTRY': 'float',
                'date': 'date'}}
>>> for f in c:
...     print f['properties']['date']
...
2012-01-29
2012-01-29
2012-01-29
2012-01-29
2012-01-29
2012-01-29
2012-01-29

Be careful with this feature. Unless your data is destined for a legacy system, I think you're better off keeping track of time as RFC 3339 strings in a text field. Among other advantages, you'd gain millisecond precision and precise expression of UTC time offset.

I'm going to be in Santa Clara, CA, Thursday evening through Sunday morning to attend PyCon US 2013. I'm not presenting, but signed up to be the runner for a scientific Python session on Saturday afternoon. Although I'm working in the humanities, (just between you and me) I'm still a science and engineering type at heart and I haven't found much of a humanities computing presence at PyCon. I hope to see you there.

The conference is moving to Montréal next year. Summer, I hope!

In the geospatial software I'm writing and using these days, concerns are well separated. Fiona reads and writes features. Only. Shapely provides computational geometry algorithms. Only. Pyproj (not my work, but a favorite package) transforms coordinates between spatial reference systems. Only. The separation of concerns helps keep interactions between them predictable and as a user you pay only for what you eat.

A programmer-analyst's daily work has all the above concerns (and more, probably). A pip requirements file makes installing all three packages as easy as installing a single package like osgeo.ogr. I've uploaded one to GitHub: https://gist.github.com/1689767. This Gist includes an example of using Fiona, pyproj and Shapely together. Fetching them all, assuming you've got pip and the GDAL/OGR libs and headers already on your system, is just:

$ pip install -r https://raw.github.com/gist/1689767/mersh.txt