Sean Gillies (Posts about software engineering)

Box hugger no more

Sean Gillies — Thu, 25 Jul 2013 06:00:00 GMT

I've been using zc.buildout for years now to build the Pleiades infrastructure in a replicable way, and it's served the project well. The process of setting up a production environment has been this:

ssh in to production
git clone https://github.com/isawnyu/pleiades3-buildout $HOME
cd $HOME
virtualenv
bin/python bootstrap.py
bin/buildout install zope2
bin/buildout
Copy database to $HOME/var
Symlink $HOME to pleiades-production
supervisor restart all

The buildout script yields a database server, multiple app servers, an nginx load balancer, and Varnish caching proxy. Subsequent minor site releases are just a matter of pulling buildout configuration files and re-running step 6. Buildout is repeatable and, with care, idempotent. But the other steps are less repeatable and more manual. That I use screen to keep that shell alive shows exactly how much manual intervention is required. Although I've automated half the process, I've been the kind of devops person Subbu Allamaraju calls a "box hugger". Subbu says:

Two steps to cure box hugging – first, internalize the idea that the box you’ve just finished setting up meticulously is going to burst into flames the very next minute, second treat operations the same way as you would treat software development.

I'm well on the second step: all the Pleiades buildout configuration is version controlled, if not fully tested. But the first step, not so much.

I'm using a new project to reform my ways and be much less of a box hugger. To internalize the ephemeral nature of servers, I'm teaching myself to provision and configure Vagrant VMs with Ansible. My goal is to be able to deploy this project's sites to their production server using only Ansible, never logging in at all. Having no database in this project (it's all based on XML, on GitHub) makes this goal easier to hit. And it turns out that setting up Solr isn't going to be too tough, either. Thanks to this ansible-multi-solr project I've learned to write my own very basic Solr and Tomcat playbook. With just two commands

$ vagrant up
$ ansible-playbook setup.yml -k -i setup_hosts

and a short wait, I get a running Solr instance at http://192.168.35.10:8983/solr/.

I'm late to the party, I know, but Vagrant is killer. I'm also using it to test packaging and installation of Shapely and Fiona. I believe it was Whit Morris who directed me to Vagrant. Thanks, Whit!

I've used neither Chef or Puppet and chose Ansible because it's Python, uses familiar stuff like SSH and JSON, and because the playbook concept is a reasonable leap for me from Buildout. I'm enjoying it very much and hope to be able to contribute something to the project in time.

Thanks, Subbu, for providing the impetus I needed to make the leap from box hugging! I really would rather be developing than deploying and administering, and feel like I'm beginning to get a grip on the tools that will make that possible.

Comments

Re: Box hugger no more

Author: Michael Weisman

Throw jenkins and a few lines of bash in there and you can have production or dev servers auto deploy on git commits to specific branches using the same ansible scripts you use with vagrant for local dev!

Re: Box hugger no more

Author: Sean

Yes, sky's the limit! I didn't realize you were Ansible users at OpenGeo.

Re: Box hugger no more

Author: Kenshi

There is Salt Stack, which is also Python and gaining a lot of traction. http://saltstack.com/community.html

Pleiades software reuse

Sean Gillies — Mon, 26 Mar 2012 06:00:00 GMT

Yesterday I read Melissa Terras On Making, Use and Reuse in Digital Humanities:

That's right. The code we made is now in use by another institution, to do their own transcription project. Hurrah!

It was always our aim in Transcribe Bentham to provide the code to others: it was a key part of our project proposal. But you always have to wonder if that is going to happen. Its the kind of thing that everyone writes in project proposals. And whilst lots of people talk about making things in Digital Humanities, and whether or not you have to make things to be a Digital Humanist, we've shied away - as a community - from the spectre of reuse: who takes our code and reappropriates it once we are done? How can we demonstrate impact through the things we've built being utilised beyond just us and - quite frankly - our mates?

So I'm happy as larry that the code we developed, and the system we have built, is both useful to us, but is now useful to others. I'm not sure how much I want to prod the sleeping monster that is general code reuse in Digital Humanities... dont draw attention to our deficiencies!

But I would be delighted if anyone else could point me to examples where code and systems in Digital Humanities were repurposed beyond their original project, just as we would wish?

I was unable to persuade Recaptcha to let me leave a comment and congratulations on Melissa's blog, so am writing a brief post here. TL;DR Transcribe Bentham: congrats! My own horn: toot! toot!

To my knowledge, no one has set up another instance of Pleiades as a gazetteer. But code written for Pleiades gets reused more and more widely the further down in our stack you look. I designed it to be modular and reusable – a stack of tools, not a single tool – so I'd be disappointed if that wasn't the case. I'll explain how it works for us.

Pleiades uses the Plone (and Zope) web application framework. Our Plone products and packages have taken on a life of their own as the collective.geo project. IW:LEARN is a good example of a collective.geo site. Contributions to collective.geo by almost 20 other people have been making Pleiades better. And this is just the start of our reuse story.

Pleiades and collective.geo Zope and Plone packages are based on Python GIS packages spun off from Pleiades such as Shapely, Rtree, and Geojson. At PyCon a couple of weeks ago, I ran into a lot of Shapely users. I saw it mentioned on slides in talks and on posters in the poster session. Famous web and geography hackers even write about using Shapely from time to time. Shapely feature-wise, Pleiades now gets as much back from others as we give out. This is a fantastic position to be in.

Digging deeper in the stack, I've contributed (as "sgillies") to the development of GEOS as part of my work on Pleiades, so Pleiades has thereby played a tiny role in making thousands of open source GIS programs and web sites more spatially capable. A Python protocol for sharing geospatial data that we invented for Pleiades has been implemented rather widely in GIS software. Anywhere you see __geo_interface__ and shape() or asShape(), or programmers sharing data as GeoJSON-like Python mappings, that's the impact of Pleiades. The GeoJSON format itself has some of its roots in Pleiades.

Even if no one ever sets up another Pleiades site, we're having a significant impact on GIS software and systems, even on big time GIS software being used for Spatial Humanities work. The keys to having a similar impact are, in my mind, 1) modularization and generalization to increase the number of potential users and contributors, and 2) a policy of open sourcing from day zero instead of open sourcing after completion of the project – and after people have lost interest and moved on to other software.

Comments

open sourcing from day zero

Author: Christian Ledermann

Release often, release early. It is important to let the community know what you are up to, though it might feel a little embarrassing to make unfinished buggy alpha code available to the public it has big advantages. E.g. when I released the alpha of collective.geo.index the UI was unfinished and ugly, not one of my priorities. It would probably still be unfinished and ugly if it was not for David who gave the UI some TLC just days after the initial announcement.

Thank you very much for your hard work and your great products :)

Simple and reusable spatial queries

Sean Gillies — Fri, 18 Dec 2009 07:00:00 GMT

Quoting James Fee:

First we’ve got new functionality in the Google Maps Data API. First off you can now perform geospatial and attribution queries on data stored on Google’s MyMap. Now of course this isn’t paleo-type spatial queries, just simple stuff that solve 80% of all queries you’d need to complete. Simple web apps need not fancy complicated APIs and clearly Google is the master of this. So upload your data into Google’s My Maps and then query it to display on a Google Maps application. Simple and sweet.

This is also sweet from an engineering perspective. Yes, simple bounding box or radius search are "only" "80-20 rule" features, but at the same time are also features that power users are going to use almost 100 percent of the time for a lot of tasks. In other words: highly reusable features. Let's say you want to select features of a collection that are contained within another feature. You could build up a result set by iterating over each feature in the collection no matter where it is and evaluate containment with the filtering feature for every single one of them. Better yet would be to first eliminate candidate features that aren't even in the same ballpark as the filtering feature by using an indexed bounding box search. This is the default mode of operation for newer functions in PostGIS, for example, like ST_Contains:

This function call will automatically include a bounding box comparison that will make use of any indexes that are available on the geometries. To avoid index use, use the function _ST_Contains.

(By the way, nice new docs there, PostGIS people!) Simple indexed search is the initial boost to more interesting queries. Indexed operations are best provided by the map data collections themselves. Higher order processing, some of which can be done in parallel, can be left to other resources or services. This is, of course (except for the parallel considerations), the premise of the OGC's Web Processing Service, but taken to an extreme where a WFS provides only bounding box and attribute filtering and the WPS does everything else. A familiar pattern, but I'd be surprised if a future Google spatial data processing platform wasn't as different from OGC WPS as the maps data API is from WFS.

Unofficial Python GIS SIG

Sean Gillies — Wed, 30 Sep 2009 06:00:00 GMT

There was a discussion on the GIS-Python list that made me think that there ought to be a discussion group that helps developers communicate across projects, share good ideas, and generally improve Python's GIS story, much like the Python Web-SIG list does for web software developers. Please help spread word of the Unofficial Python GIS SIG.

Comments

Re: Unofficial Python GIS SIG

Author: Timmie

It's a great idea!

I added this to the OSGEO wiki at

OSGeo Python Library

Will you advertise it on the mailing list of GRASS, QGIS, gvSIG, Mapfish, featureserver/tilecache?

Keep on!

Re: Unofficial Python GIS SIG

Author: Sean

I'm not a subscriber to those particular lists and am counting on others to spread the word. The MapFish developers are on board already.

On web sanity

Sean Gillies — Mon, 07 Sep 2009 06:00:00 GMT

Stefan Tilkov, who has written an excellent REST introduction and an analysis of REST anti-patterns has some criticism for enterprise web apps that applies equally well to "GeoWeb" apps:

In other words: Many web frameworks, especially those within the Java space, try to support the development of desktop applications that are delivered over the Web as opposed to “web sites”, and I believe this idea is deeply flawed: Any Web application should be measured against the criteria that are used to judge web sites.

I think this goes for the "GeoWeb" too. We're delivering too much desktop GIS over the web. It's a point made also by Brian Noyle here, though I see Flex, Silverlight, .NET, and MVC as parts of the problem, not the solution.

Comments

Re: On web sanity

Author: Tim Bowden

Those links are great thanks Sean. I'm developing a desktop app replacement on the web atm, and that material has clarified some of the issues I've been mulling over. I probably won't get it all right this time around, but I'll certainly get closer than I was going to get.

I've previously looked briefly at the idea of RESTful apps, but not in any great detail or real understanding. Now I am, and it's certainly an interesting journey. The app I'm replacing is definitely very much in the traditional desktop space (with a legacy stretching back all the way from its DOS roots; think reverse engineering very old and poorly documented binary data formats). It doesn't *look* like a candidate to be a 'website' type app, as opposed to a 'desktop app on the web'. The requirements *seem* to point that way, that is until you dissect it bit by bit using a RESTful scalpel. I'm not sure how far I'm going to be able to push this model, either because of limitations in my skill and understanding or limitations in the RESTful model, but I'm keen to see where I can go with this.

It's a good thing this is a relatively simple app, as using a RESTful scalpel for the first time isn't so easy. Old mental models of what desktop apps should look like and how they should behave keep getting in the way. I've still not gone very far on this journey (I've not tried implementing any of this yet; next step!), but so far every issue seems to have been answered by sharpening my RESTful scalpel a little more. I think one of the hardest ideas to come to terms with is the idea of “hypermedia as the engine of application state”. All those 'client side' data structures start to melt into very ephemeral constructions. Implementing that is going to be an interesting experience for sure.

Sean, keep pushing the REST barrow. I'm very much a novice, but from my perspective so far, it *seems* to be a winner, despite the mental difficulty of overcoming deeply ingrained ideas from the desktop environment. Hopefully I'll not be too shell shocked when I come out the other side!

Plugins for Shapely

Sean Gillies — Thu, 19 Feb 2009 07:00:00 GMT

I haven't started working on any pure Python (for App Engine) geometry predicates or operators, or any other alternatives to GEOS, but I've begun to make it possible. All GEOS dependency is being moved to its own shapely.geos module, which will be the default plugin provider for Shapely's new plugin framework.

The framework defines a few interfaces (geometry compiling or checking provider, area and length provider, bounds provider, etc) and entry points. When a geometry is asked to compute its bounds, for example, it delegates to a provider that has been wired up to the geometry factory on import. Here's a peek at how the default providers are registered in Shapely's setup.py:

...
setup(name          = 'Shapely',
      version       = '1.1.0',
      ...
      entry_points = """
          [shapely.geometry]
          geometryChecker=shapely.geos.geometry:geometryChecker
          metricProperties=shapely.geos.metrics:metricProperties
          geometryProperties=shapely.geos.topology:geometryProperties
          """,
)

And in shapely.geometry, the providers are activated with a function:

...
use('Shapely>=1.1.0')

from geo import shape, asShape
from point import Point asPoint
from linestring import LineString, asLineString
from polygon import Polygon, asPolygon
from multipoint import MultiPoint, asMultiPoint
from multilinestring import MultiLineString, asMultiLineString
from multipolygon import MultiPolygon, asMultiPolygon
from collection import GeometryCollection

Users can override the default GEOS providers by writing and installing (with setuptools) new packages that provide the same named end points, and "using" those at run time:

from shapely import geometry
geometry.use('MyGeometry')
...

point = geometry.Point(0.0, 0.0)
x = point.buffer(10.0) # calls on provider defined by MyGeometry package

In theory, this makes it possible to a write an application using Shapely that can run on either C Python, Jython, or IronPython using the appropriate backend for the platform, as long as setuptools and pkg_resources work. I've read that they will in the next Jython release. IronPython seems to be a little farther behind. In practice, the plugin framework is helping to improve the testability and quality of Shapely's code.

Comments

Re: Plugins for Shapely

Author: brent

thanks sean. very cool.

for those who didnt know about setuptools entry points, this is a good intro:

http://lucumr.pocoo.org/blogarchive/setuptools-plugins

In order to form a more perfect union

Sean Gillies — Fri, 23 Jan 2009 07:00:00 GMT

Paul Ramsey on GEOS 3.1 and PostGIS 1.4 improvements:

Here's a less contrived result, the 3141 counties in the United States. Using the old ST_Union(), the union takes 42 seconds. Using the new ST_Union() (coming in PostGIS 1.4.0) the union takes 3.7 seconds.

Now that's change we can believe in.

Comments

Re: In order to form a more perfect union

Author: Guillaume

Yes, we can !

Mocking GEOS

Sean Gillies — Tue, 20 Jan 2009 07:00:00 GMT

My use of mocks isn't as sophisticated as Dave's, perhaps, but I stumbled onto a simple testing pattern that might be useful to other Python geospatial/GIS developers who are wrapping C libs using ctypes.

Consider Shapely: it wraps the GEOS library, the quality and accuracy of which we take as a given (though not blindly, because I do contribute fixes and enhancements to GEOS). The predicates and topological functions of GEOS are called from within Python descriptors, classes that perform argument validation and handle GEOS errors. For Shapely, I'm testing these descriptors, the GEOS wrappers, not GEOS itself. What pair of geometries would I have to pass to GEOSDisjoint (for example) in order to get the return value of 2 that signifies an error? Even if known, they might be subject to issues of numerical precision, or be sensitive to changes in GEOS. I'd rather not fuss with this. Instead, I want some function to stand in for GEOSDisjoint and friends, one that takes 2 arguments and has very predictable return values in the range (0, 1, 2). A function like libc's strcmp():

>>> import ctypes
>>> libc = ctypes.CDLL('libc.dylib') # this is OS X
>>> libc.strcmp('\0', '\0')
0
>>> libc.strcmp('\1', '\0')
1
>>> libc.strcmp('\2', '\0')
2

Meaningless, but handy, isomorphism between strcmp() and GEOS binary operations in hand, a generic wrapper for GEOS can be fully tested like this:

import ctypes
import unittest

from shapely import predicates

BN = libc.strcmp

class CompMockGeom(object):
    # Values chosen with libc.strcmp in mind
    vals = {'0': '\0', '1': '\1', '2': '\2'}
    def __init__(self, cat):
        self._geom = ctypes.c_char_p(self.vals[cat])
    comp = predicates.BinaryPredicate(BN)

class BinaryPredicateAttributeTestCase(unittest.TestCase):

    def test_bin_false(self):
        g1 = CompMockGeom('0')
        g2 = CompMockGeom('0')
        self.assertEquals(g1.comp(g2), False)

    def test_bin_true(self):
        g1 = CompMockGeom('1')
        g2 = CompMockGeom('0')
        self.assertEquals(g1.comp(g2), True)

    def test_bin_error(self):
        g1 = CompMockGeom('2')
        g2 = CompMockGeom('0')
        self.assertRaises(predicates.PredicateError, g1.comp, g2)

Comments

Re: Mocking GEOS

Author: ajfowler

Hi, Unrelated to this post, but I came across one of your old blog posts about web-mapping accessibility. Is this topic off of your radar now? aj

Re: Mocking GEOS

Author: Sean

Yes, but it looks like it's on yours. What's up with web map accessibility?

Re: Mocking GEOS

Author: ajfowler

Well I'm looking into creating a text description of a map. There aren't a lot of resources out there, but I'm avidly searching.

I can has Python and GIS environments?

Sean Gillies — Tue, 23 Dec 2008 07:00:00 GMT

I've spent this short week tuning up my new laptop's development environment, and a side effect of this work is a new build system for replicable, isolated Python, GIS, and image/raster processing environments. Ichpage replaces Gdawg on my machine. It supplies:

GDAL (osgeo.gdal, etc)
geojson
geopy
keytree
lxml
Numpy
PIL
PyProj
Rtree
Shapely

and their various library dependencies (libgdal, libgeos_c, libspatialindex, libxml2, libxslt). To get started, clone or get the tarball, cd into the directory, and execute:

$ virtualenv .
$ source ./bin/activate
(ichpage)$ python bootstrap.py
(ichpage)$ buildout
(ichpage)$ . ./setenv
(ichpage)$ labpy
>>> from osgeo import gdal
>>> from shapely.geometry import Point
...

To do tasks include linking the GDAL utilities into the environment's bin directory, adding WorldMill, perhaps adding matplotlib. For now, it's a way for me to manage C libs while I develop Shapely and Rtree, and perhaps useful to other geospatial Python developers.

Comments

Re: I can has Python and GIS environments?

Author: Kurt

keytree?

Re: I can has Python and GIS environments?

Author: Sean

Keytree is a little KML helper for use with Python ElementTree APIs.

Preserving first-generation web/GIS projects

Sean Gillies — Mon, 22 Dec 2008 07:00:00 GMT

Check out this interesting article about the reanimation of an orphaned plant database and its associated ArcIMS instance. The analysis of the issues is sound. I disagree, of course, with their conclusion that ArcIMS is something worth learning and deploying in 2008, and this raises in my mind another issue that the authors did not identify: is not the project's data and its provenance the thing that is most important to preserve? Must the interface cruft around it be preserved in anything other than an archived form, if at all? The ArcIMS user interface and the species database browser are no kind of programmable web APIs; it's unlikely any other application would be broken by a switch to some free web mapping framework or modern search interface.

Now there's a question: switch to what? If this story is just the beginning, and bigger boxes of used, discarded, but potentially useful first-generation web/GIS projects end up in the laps of librarians, a turnkey (and open source, naturally) ArcIMS to MapServer/MapGuide migration tool might be a handy thing. I wouldn't be surprised if such a thing existed. Its authors might want to consider pitching it to GIS librarians in higher education.

Comments

Re: Preserving first-generation web/GIS projects

Author: Jason Birch

That's just freaking bizarre. I was lying in bed last night thinking about whether it would be hard to write an AXL to MapGuide XML transformation.

Re: Preserving first-generation web/GIS projects

Author: James Fee

Wow, deploy ArcIMS in 2008/2009. This is why ESRI can't kill ArcIMS, folks still want to use the darn thing. As much as Jack can get up on stage and basically say the thing is depreciated, people still refuse to listen. Of course part of the problem is ESRI is willing to continue selling licenses to people to deploy it, but I suppose in a higher ed setting site licenses abound and they could have easily gone with the ESRI RESTful API if they wanted to stay ESRI.