2007

Replicable Python Geoprocessing Environments

I've finally followed through on earlier hand waving about combining GDAL with zc.buildout to make replicable Python geoprocessing environments. Grab the Gdawg tarball: Gdawg.tar.gz, extract, and cd into its root. Then build it out like so:

$ python bootstrap.py
$ ./bin/buildout

When done, you'll have a custom Python interpreter with GDAL and (for fun) WorldMill:

sean@lenny:/tmp/Gdawg$ ./bin/gdawgpy

>>> from osgeo import ogr
>>> import mill
>>>

Add any other Python eggs you need by customizing the list of eggs in the gdawgpy section of buildout.cfg:

...
[gdawgpy]
recipe = zc.recipe.egg
interpreter = gdawgpy
eggs =
    GDAL
    WorldMill
    simplejson # etc

All the familiar GDAL/OGR utility programs are found under parts/libgdal/bin.

What do I mean by replicable? You can control versions of every piece of software in this buildout and even put the config file itself under revision control. It's like an FWTools where you control the versions of everything.

Comments

Re: Replicable Python Geoprocessing Environments

Author: Christopher Schmidt

Does it work on Windows?

Re: Replicable Python Geoprocessing Environments

Author: Sean

It might, but not without MinGW or hard work. I'm trying to show people another build tool, not trying to sell a replacement for the FWTools Windows binaries.

Britannia Superior Preview

Also known as Barrington Atlas Map 8. Just got this data back the other day. Roads, settlements, mines -- you can just pick out a few copper mines in what we now call Wales. We'll be rolling it out into the Pleiades site in the new year.

http://sgillies.net/images/map8preview.png

Back

I miss Seattle already. Normal posting from good old Fort Cowpie will resume shortly.

MSWKT

I don't understand how you'd expect a different approach from Microsoft. Being a late entry, the company could benefit from shaking things up a bit.

Update: Or not.

Last REST Post of 2007

Here's the take-away from the past year's discussion about REST and geospatial. My own take, that is. There are differences of opinion among the participants of the Geo-Web-REST group.

REST is an architectural style, not an API. When you hear or read "... REST API ...", that's a smell, a signal that something might not be quite right inside.

REST is the architectural style of the World Wide Web. "Web API" is a more meaningful term, although not all Web APIs are RESTful. Some are GETSful.

The OGC service architecture is not RESTful. The style of the OGC architecture is non-RESTful CGI/RPC. Feature, map, and coverage resources are kept hidden behind service endpoints. We're still partying like it's 1999.

GML is compatible with REST, but XML Schema is still a wretched beast and there are no XLink-traversing GeoWeb browsers of note.

Look to Google Earth and KML for REST inspiration. It's about the links. Also look to Google's AtomPub-ish APIs for YouTube, Picasa, and OpenSocial. There's no transactional WFS involved there, and that's something to think about. Google chose AtomPub for wider interoperability.

REST is not a silver bullet. Distributed applications are hard.

These are the tenets of REST:

  • Give every “thing” an ID
  • Link things together
  • Use standard methods
  • Resources with multiple representations
  • Communicate statelessly

Do read and recommend Stefan Tilkov's Brief Introduction to REST, from which the tenets above are taken. It's excellent.

Disregard the Emerging Technology: Geospatial Web Services and REST article in Directions. I don't care who linked you to it; they probably didn't read it very closely. This sounds mean, but that article will make you dumber about REST.

Comments

Re: Last REST Post of 2007

Author: Norman Barker

Sean, I have just read this http://wisdomofganesh.blogspot.com/2007/12/paying-restafarians-back-in-their-own.html and it makes some good arguments - perhaps SOAP can play after all, and the OGC have ebRIM :-) I have been enjoying the discussion of REST on this blog - thanks!

Re: Last REST Post of 2007

Author: Allan Doyle

Ganesh talks a good game. But remember that HTTP is itself a dumb-network/smart endpoint protocol that rides on top of TCP (and thus can also make use of IPsec). SOAP tends to ride on top of HTTP, and is generally run at a higher level on the networking stack than HTTP. True, SOAP can run on SMTP, or any other protocol, but as implemented it's on HTTP probably 99+% of the time. Ergo, there is no protocol advantage. The entire smackdown is moot.

Re: Last REST Post of 2007

Author: Allan Doyle

And, just one more thing... it strikes me that if you were to apply REST principles to SOAP you might just come up with something approaching "SOAP message with WS-Addressing headers". In other words, REST is a set of architectural constraints that can be embodied on any distributed system that has the ability to provide unique IDs for resources which can themselves contain links to other resources.

Re: Last REST Post of 2007

Author: Keyur

Last REST post? 13 days is a lifetime.

Knowhere Project Wiki

I made a new wiki page for Knowhere. It's my Christmas present to everyone. What, you were expecting something other than open source software?

The buildout failed on my PowerPC iBook, which has gcc 4.0.0. Howard Butler has been successfully building the spatialindex library on his servers with gcc 4.0.1, so it may build on newer OS X systems as long as you have Python 2.4.

RESTful Marketecture

That's a great term. Bill de hÓra:

Ascribing REST to the SimpleDb API might be down to lack of knowledge or marketecture; hopefully they'll get the API* sorted out soon, before it gets used heavily.

The GIS industry leader is soon to release new REST stuff. We'll see whether it abuses the architectural style like Amazon does or if it's the real deal.

Update (2007-12-17): from Assaf Arkin:

1999 called, they want their CGI scripts back.

That's more or less how I feel about the OGC WxS service architecture.

Comments

Re: RESTful Marketecture

Author: James Fee

I think many folks have great hope that Amazon actually nails this SimpleDB. It will be interesting to see if hey can.

Re: RESTful Marketecture

Author: Paul Ramsey

I hope so, but the problem just got a lot harder for them. In addition to delivering a better technological solution, now they also have to surmount the psychological hurdle of admitting they got it wrong the first time. That's a high bar for most organizations to clear.

Re: RESTful Marketecture

Author: James Fee

How about this Sean? A RESTful version of Amazon's SimpleDB

Re: RESTful Marketecture

Author: Sean

Yep, I saw that one. Assaf Arkin did something similar, with code. People are mystified that Amazon could get this wrong after they got S3 right.

Grok, the Paleolithic Geographer

Update (2008-02-19)

The batlas demo is gone. Please see instead http://sgillies.net/kw/demo.

I've wanted to make an easily deployed and transparent application to demonstrate what can be done with zgeo.geographer and friends, the new geospatial packages for Zope, and finally inspiration struck. I chose to write a Grok project using the aptly named grokproject. Grok aims to simplify Zope to the degree that a caveman could use it. Convention over configuration and all that. It certainly allows a developer a clear view of the zgeo.* packages in action.

The paleolithic geographer isn't concerned about fancy geographic information systems. He wants a know-where-things-are system; hence the name of my demo app: Knowhere (pronounced "Oog"). Knowhere annotates resources with geographic location metadata and publishes them using GeoRSS (Atom, specifically) or KML so that a even a caveman can visualize them in the context of other geographic information on the Web.

Building and launching Knowhere is a 5 step process that requires Python 2.4, gmake, gcc, etc. Within 15 minutes you can be publishing geo-referenced content. Start by getting the code from the PrimaGIS Subversion repository:

$ svn co http://svn.gispython.org/svn/zope/Knowhere/trunk Knowhere

Cd into that new directory and bootstrap the project:

$ /usr/bin/python2.4 bootstrap.py

Build it out:

$ ./bin/buildout

This downloads and builds 2 C/C++ libraries (GEOS and spatialindex), and fetches a heap of eggs. It will take a few minutes.

When the build out is complete, edit 2 configuration files. In parts/zopectl/zope.conf, set the port number to an open value. In parts/app/site.zcml, change the management password from "admin" to one less obvious. That done, start up the application:

$ ./bin/zopectl fg

Browse to http://localhost:8080 (let's presume that's where it's at), authenticate, and you should see a Grok management page like this:

http://sgillies.net/images/knowhere-apps.jpg

Add a new application named "cities" as shown, and then browse to http://localhost:8080/cities where you'll be faced with one of the ugliest forms ever seen. Fill it out like so

  • name: fort-collins
  • title: Fort Collins
  • description: Fort Collins, Colorado, USA
  • text: The little geospatial hothouse on the prairie
  • geometry type: Point
  • coordinates: [-105.08, 40.59]

using a GeoJSON coordinate representation, and submit. Click through to the new placemark page, which should be something like this. There are 5 fields on that page. Look at the Knowhere model (src/knowhere/app.py) and you'll see that the Placemark class actually only has one attribute:

class Placemark(grok.Model):

    def __init__(self, text=None):
        self.text = text

Where are the other fields? Grok gives you Dublin Core metadata annotations automatically. No configuration required. Peek into the model's configuration (src/knowhere/configure.zcml) and you'll see that adding geographic location metadata from zgeo.geographer to any model class is as simple as:

...
<include package="zgeo.geographer"/>

<class class=".app.Placemark">
  <implements interface="zgeo.geographer.interfaces.IGeoreferenceable"/>
</class>
...

The zgeo.kml package provides KML representations of containers of georeferenceable objects. Browse to http://localhost:8080/cities/@@kml-document for an example. The zgeo.atom package provides Atom entry and feed document views with GeoRSS. For examples, browse to http://localhost:8080/cities/fort-collins/@@atom-link-entry and http://localhost:8080/cities/@@atom-search-feed.

Knowhere also includes an R-Tree spatial index from zgeo.spatialindex. Containers of the model can be made indexable as easily as:

...
<include package="zgeo.spatialindex"/>

<class class=".app.Demo">
  <implements interface="zgeo.spatialindex.interfaces.ISpatiallyIndexable"/>
</class>
...

Browse to http://localhost:8080/cities/@@manage-index to see the index management form. Future versions of Knowhere will include georeferencing event handlers, but for now you must rebuild the index manually after adding placemarks. Lame indeed, but this is just a demo (I already have a working handler for Plone AT content types). The management screen in the image below is from my demo application at http://sgillies.net/kw/batlas, which had 2 placemarks at the time I wrote this post.

http://sgillies.net/images/knowhere-zmi.jpg

The spatial index allows fast bounding box queries. The atom search feed accepts a bbox parameter of the form minimum_longitude,minimum_latitude,maximum_longitude,maximum_latitude. For example, see http://sgillies.net/kw/batlas/@@atom-search-feed?bbox=-5,50,5,55 [map].

http://sgillies.net/images/knowhere-ge.jpg

As you can see in the image above, zgeo.* can handle polygon geometries as well as lines and points. View that demo application in Google Earth using http://sgillies.net/kw/batlas/@@kml-document.

There's more work to do on Knowhere and zgeo.*. The index might be better as a local utility, and I should be using an integer id utility instead of hashing content names. The HTML is wretched and the URLs cry out for rewriting (which Grok supports right out of the box). Any Zope user that can look past those warts might start to get some good ideas from Knowhere.

The Pleiades Fitness Program

http://sgillies.net/images/bikeski.jpg

Lee Martinez Park and the Poudre River trail are just a short bike ride from the AWMC West Annex. The snow was getting a little thin yesterday, but we got another 5 inches overnight. You must pounce on the snow here in town because it doesn't last long. Last winter was anomalous; I'll probably be back out in running shoes in a week or so.