2008 (old posts, page 3)

Beyond the RDBMS

Martin Davis's post reminds me that the GIS industry, or at least the open source corner of it, still trails the Web community in thinking about data. I teased Paul Ramsey about baking in the assumption of SQL last summer. There wasn't a presentation at FOSS4G 2007 that questioned the RDBMS paradigm. N = 1 thinking prevails.


Re: Beyond the RDBMS

Author: Stefan

Sean, You wrote "There wasn't a presentation at FOSS4G 2007 that questioned the RDBMS paradigm". Well, it depends on what you mean by presentation: There was at least a poster about the object database db4o (http://www.foss4g2007.org/presentations/posters/). My impression there was, that most people - even from open source - either ignore or don't really consider to go beyond RDBMS but only want a confirmation that there's nothing beyond RDBMS... :->

Re: Beyond the RDBMS

Author: Sean

I missed that poster. Now I know that there was at least one other object database user there besides me.


James Fee has been having fun with buzzwords and phrases. "GeoWeb" (or "Geospatial Web", or "Geospatial Semantic Web") is the one that really gets to me. I've used it a bit, and always feel like a snake oil salesman when I do. It's marketing: anybody who uses it without scare quotes is probably trying to sell you something.

The meaning of the term "World Wide Web" has also become diluted with time, but it does have a documented architecture that geeks like me can rally to. I know people who hate to read this, and feel that it verges on FUD, but the GIS industry has not been big on Web architecture. Our services are essentially (I'm singling out WFS here) relational database create, read, update, and delete (CRUD) operations wrapped in XML, tunneling over HTTP. True, a lot of the conventional non-geo Web is like this too, but it's the resources that link and provide hypertext navigation that make the Web into a singular application.

On the brighter side, I do think that KML is putting the "Web" back in "GeoWeb" by encouraging people to develop geospatial applications that use the architecture of the Web. Google obviously benefits from growing the Web (there's the marketing element again), but a return to Web architecture is also a win for the rest of us.


Frugosapalooza is a series of open source GIS meetups on the Front Range of Colorado. Brian Timoney has one set for Denver (19 Feb), I have a tentative date (26 or 27 Feb) for one here in Fort Collins, and Boulder and Colorado Springs are the other likely candidates. Watch the wiki page for details.


Re: Frugosapalooza

Author: James Fee

Keep rubbing in how great the Front Range is and how bad the Valley of the Sun is. I might actually try to stop by for either Denver or Fort Collins to see how you guys roll.

Re: Frugosapalooza

Author: Sean

I'm coming to Denver for BT's meetup. Would be fun to meet you there or up here in Fort Agile.

Re: Frugosapalooza

Author: James Fee

Denver would be easier as we have an office in Boulder. I can find a reason to be there I'm sure.

There is More to REST Than HTTP + POX

It's great to see more people in the geospatial community thinking outside the SOAP box. Still, there's more to REST than HTTP and plain old XML (POX) or JSON. Fuzzy's service architecture, typified by requests like:

GET v1/ws_geo_getextent.php?geotable=places&srid=4269&parameters=id=10 HTTP/1.1
Host: webservices.example.com

is the venerable REST-RPC hybrid explained in Chapter 1 of RESTful Web Services. This architecture is okay for read-only services, but doesn't easily accommodate creation, update, or deletion of resources (it doesn't actually expose any resources at all), and sets you up for troubles like those of SimpleDB. Even if you dodge the pitfall of:

GET v1/ws_geo_deleterow?geotable=places&parameters=id=10 HTTP/1.1
Host: webservices.example.com


by properly using HTTP DELETE, you've lost uniform interface to your resources and no longer have a RESTful architecture.

Seems like a cool project. A little restraint in hyping REST is all I'm suggesting.


Re: There is More to REST Than HTTP + POX

Author: Dave Smith

Downright scary for transactional purposes...

Re: There is More to REST Than HTTP + POX

Author: Sean

Technical merits of the web services aside: I'd just rather that "REST" didn't become the new "SOA", a term taken over by marketing and stripped of all meaning.


Speaking of Django: EveryBlock, a new startup. Uses OpenLayers. Like outside.in, no GeoRSS or KML (yet?). Via Simon Willison, naturally.


Re: EveryBlock

Author: Christopher Schmidt

And TileCache, of course. (With what looks like Mapnik styling. I sometimes forget how pretty Mapnik can be.)

Re: EveryBlock

Author: Artem Pavlenko

EveryBlock is indeed Mapnik rendering and thanks a lot for the compliment. Chris, nice to know you appreciate Mapnik even if you wouldn't always "personally" recommend it ;) Building Mapnik usually doesn't pose a problem but if you're having trouble just give me a shout!

Re: EveryBlock

Author: Sean

Indeed it does look nice, Artem.

Re: EveryBlock

Author: Paul Smith

OpenLayers + TileCache + Mapnik and a bit of Shapely ;) We're working on improving our feeds and we'll have consumable geo data very soon. Thanks for checking us out.

Re: EveryBlock

Author: Sean

I love seeing Shapely in action!

Rtree 0.4 and Spatialindex 1.3

I'm pleased to announce the releases of Rtree 0.4 and spatialindex 1.3.0. The only thing standing between us and Rtree 1.0 is the Windows support coming in spatialindex 1.4.

Like Shapely (1.0 release last Friday; toot, toot!), Rtree is designed to be a specialized, highly-reusable Python interface to an industrial-strength library. It doesn't do formats. It doesn't do projections. It's not a CGI program. It's a building block that does one thing well and otherwise stays out of your way. It indexes spatial data and provides query mechanisms, and that's all it does.


Re: Rtree 0.4 and Spatialindex 1.3

Author: Matthew Giger

That's a great tool! It would be truly awesome if someone could put Rtree support into SQLite, making it a true competitor to Postgresql.

Re: Rtree 0.4 and Spatialindex 1.3

Author: Sean

Thanks. They'd probably want to use the spatialindex lib instead of the Python package.

Rtree, Shapely, and WorldMill: Jamming Econo

Update (2008-01-26)

I've added Python 2.5 to the buildout. It adds a minute or two to the build time, but gives you a much more isolated environment in which to jam. I also found a work-around for the issue reported in zc.buildout bugs 110133 and 138260: building WordMill requires Cython in the working set of the custom python, something that cannot be accomplished using zc.recipe.egg. What I've done is used Kai's hexagonit.recipe.download to fetch the Cython source, and then used iw.recipe.cmd to install it into the buildout's custom python. See the cython-src and cython-install sections in buildout.cfg. If you've already fetched Gdawg once, I recommend you discard it and clone a fresh copy.

I just added Shapely 1.0 and Rtree 0.4 to the Gdawg buildout, where they join WorldMill 0.1. Together they create a friendly environment on the C Python platform where you can read GIS feature data, spatially index it, and manipulate its geometries. (Sorry, Windows users are out of luck until the next Rtree and WorldMill releases. Patches are welcome.)

Again, getting Mercurial (Hg) is as easy as

$ apt-get install mercurial

on a Debian/Ubuntu system. Check the downloads page for other installers (Gdawg does build on a Mac). After you've installed Hg, clone my repo, and build it out:

$ hg clone http://sgillies.net/sgillies/hg/gdawg my-gdawg
$ cd my-gdawg
$ python bootstrap.py
$ ./bin/buildout

It could take up to 15 minutes to build GEOS and GDAL. In the meanwhile, grab some data to play with. I downloaded the Zillow Colorado neighborhoods and extracted them into /tmp to see if they lived up to the hype. When the buildout script finishes, start up the custom Python interpreter

$ ./bin/gdawgpy

I need to make a funny preamble someday, but there's the Python prompt. To begin, let's create a WorldMill workspace:

>>> from mill import workspace
>>> ws = workspace('/tmp/ZillowNeighborhoods-CO.zip_FILES')
>>> ws
<mill.workspace.Workspace object at ...>

Which allows me to use the only Italian I haven't yet forgotten: va bene. A workspace is a mapping of collections, as you can see here:

>>> ws.items()
[('ZillowNeighborhoods-CO', <mill.collection.Collection object at ...>)]

Access the neighborhoods collection and inspect it briefly:

>>> co = ws['ZillowNeighborhoods-CO']
>>> co
<mill.collection.Collection object at ...>
>>> len(co)
>>> co.schema
[('STATE', 4), ('COUNTY', 4), ('CITY', 4), ('NAME', 4), ('REGIONID', 2)]

95 neighborhoods (of note) in Colorado, 5 attributes per feature, all of them strings except for REGIONID, which is an int. Let's look now at the first neighborhood feature:

>>> x = co['0']
>>> x
<mill.feature.Feature object at ...>
>>> x.id
>>> x.properties['NAME']
>>> x.properties['CITY']

Hmm, they misspelled "Shelbyville". If you were to access x.geometry at the prompt, you'd get a small binary flood. By default, unless an object hook has been specified, feature geometry is expressed as WKB (Long/Lat). Let's now set an object hook for this geometry so that we get features with Shapely geometries:

>>> from mill.feature import Feature
>>> from shapely.wkb import loads
>>> def shapely_feature(id, properties, wkb):
...     return Feature(id, properties.copy(), loads(wkb))
>>> co.object_hook = shapely_feature

The only requirement on the object hook is that it be a callable with 3 positional parameters. Now, get the first feature again:

>>> x = co['0']
>>> x.id
>>> x.geometry
<shapely.geometry.polygon.Polygon object at ...>
>>> x.geometry.bounds
(-105.26320656676199, 40.010885702215496, -105.243224298828, 40.038423410870699)

And that's how you integrate Shapely and WorldMill. Now, how about spatially indexing the neighborhoods using Rtree? First, create a named index that will be persisted on disk next to the shapefile data:

>>> from rtree import Rtree
>>> index = Rtree('/tmp/ZillowNeighborhoods-CO.zip_FILES/ZillowNeighborhoods-CO')

Then iterate over features in the collection, adding each to the index in turn:

>>> for feature in co.all:
...     index.add(int(feature.id), feature.geometry.bounds)

Pretty fast, eh? Now let's find some of the "Crossroads" feature's neighbors by putting its bounding box back into an intersection query:

>>> b = co['0'].geometry.bounds
>>> index.intersection(b)
[42L, 32L, 0L, 49L, 15L, 64L, 66L]

The index returns Python longs, but we can get the corresponding features from the collection like so:

>>> neighborhoods = [co[str(uid)] for uid in index.intersection(b)]
>>> from pprint import pprint
>>> names = [n.properties['NAME'] for n in neighborhoods]
>>> pprint(names)
['North Boulder',
 'Colorado University',
 'Southeast Boulder',
 'East Boulder',
 'Palo Park',
 'Central Boulder']

Incidentally, there don't appear to be any Fort Collins neighborhoods:

>>> index.intersection((-105.09, 40.58, -105.08, 40.59))
>>> [f.id for f in co.all if f.properties['COUNTY'] == 'Larimer']

which is fine because we don't really need anybody else moving here unless they are going to open a nice old world bakery or cheese shop downtown.

If I may say so, Shapely, Rtree, and WorldMill are just about the best trio since D. Boon, George Hurley, and Mike Watt.

Django People Map

By Simon Willison: Django People. Using GeoDjango, maybe? I'm tempted to smash together a Grok version.


Re: Django People Map

Author: James Fee

Have you been working with Django Sean?

Re: Django People Map

Author: Sean

I've dabbled, nothing in production. Blogged about it in 2005, for what it's worth: http://zcologia.com/news/69/django-and-mapserver/.

Re: Django People Map

Author: Andrew Turner

no syndication via GeoRSS or KML. :p

Re: Django People Map

Author: Sean

Could be that the Django People community doesn't need them quite yet, or at least not as much as they need other features. I'd like to see them choose the GML profile over the Simple one when the time comes.

This Blog is for the Birds

Oh yeah, we get White-crowned Sparrows too, for a week or two later in the Spring.

I want to reassure Paul that I am in fact feeding seeds to the little birds, so here is a picture of McNutty the Red-breasted Nuthatch, which I took just a few minutes ago with my little PowerShot. He's giving me that annoyed look because he spent the night outside at -5F and I'm sticking the camera right in his face. 4 inches away (or approximately 20 centimeters according to my NASA Mars Mission unit conversion table). These little birds are fearless. If I had their agility I probably would be too. Nevermind Spiderman: Nuthatchman could easily kick his creepy arachnoid ass.



Re: This Blog is for the Birds

Author: Jason Birch

Mars Mission unit conversion? I watched Roving Mars with my family at the IMAX in Victoria last weekend. http://disney.go.com/disneyvideos/liveaction/rovingmars/ Much better than Titanica :) I was interested that all of the speeds and distances that they were calling out during one of the launch sequences were in miles. No conversion required for those missions apparently... except for those darn foreign audiences. -J