Plone R-Tree Spatial Index

At the 2006 Plone Conference sprint, Shaun Walbridge and I wrote a Quadtree-based spatial index for Plone. Unlike the portal catalog, it was a localized index, turning a Plone folder of georeferenced content into a shapefile of sorts. It was a nice proof of concept, but was limited by lack of persistence. A few months ago Howard Butler made it possible to persist Rtree indexes on the filesystem, and last week I finally made the time to rewrite the original Plone product into a persistent R-tree index for Zope/Plone data.

SpatialIndex keeps its original design. Adapting a Plone folder to Products.SpatialIndex.interfaces.ISpatialIndex creates an index on disk alongside the ZODB data, with a name that corresponds to the folder's physical path. Content objects can then be adapted to Products.PleiadesGeocoder.interfaces.IGeoItemSimple and added to the index. Ultimately, the index may be queried for the records of items that intersect with a bounding box. The capabilities are summarized in the session below, using a parks folder that contains a lee-martinez document:

>>> parks = app['plone']['parks']
>>> document = parks['lee-martinez']
>>> from Products.SpatialIndex.interfaces import ISpatialIndex
>>> index = ISpatialIndex(parks)
>>> from Products.PleiadesGeocoder.interfaces import IGeoItemSimple
>>> geoitem = IGeoItemSimple(document)
>>> geoitem.setGeoInterface('Point', (-105.08442, 40.59512))
>>> index.add(geoitem)
>>> hits = index.intersects((-106, 40, -105, 41))
>>> [h for h in hits]
[('lee-martinez', (-105.08442, 40.59512, -105.08442, 40.59512)]

SpatialIndex depends on

You should get PleiadesGeocoder 1.0a1 and SpatialIndex 1.0a1 from the repositories:

$ svn co http://icon.stoa.org/svn/pleiades/PleiadesGeocoder/tags/rel-1.0a1\
 PleiadesGeocoder
$ svn co http://svn.gispython.org/svn/primagis/SpatialIndex/tags/rel-1.0a1\
 SpatialIndex

Additionally, SpatialIndex provides a yet-under-construction index management view through which you count the indexed items and reindex folders, and another public view (@@spatialindex) that can be used in various custom forms and pages.

http://sgillies.net/images/manage-index.jpg

Comments

Re: Plone R-Tree Spatial Index

Author: Yves Moisan

This morning on slashgeo an article points at http://www.directionsmag.com/press.releases/?duty=Show&id=19542&trv=1 : "... CartaLens, an innovative geospatial digital asset management ... Unlike other digital asset management solutions that only manage structured content, CartaLens is able to search and retrieve location-based information from both structured content and a broad base of digital content such as photos, video, audio and documents, ..." You mean one can map progress reports like little Word of PDF icons on a map ;-). I don't think it is emphasized enough that the stack you and Kai are building allows all sorts of structured/semi-structured/unstructured content to be viewed on maps now (and without the ArcGIS requirement). It will be interesting to see what data CartaLens sites typically provide and how close we can be with the Zope/Plone/Sean*/Kai* platform. Cheers, Yves

Re: Plone R-Tree Spatial Index

Author: Sean

I've always assumed that one turned the MetaCarta engine loose on networks or storage systems to find information, and that it was rather different from the software I'm developing.

Re: Plone R-Tree Spatial Index

Author: Yves Moisan

I must admit I know nothing about MetaCarta. It just striked me as not particularly groundbreaking news it was to be able to view unstructured content on maps, as I've been doing that (I think ??) with Plone and its various "location products" for many months now. But I may well be underestimating MetaCarta's engine and overestimating Plone.

Re: Plone R-Tree Spatial Index

Author: brentp

sean, somewhat related: is it possible to add an rtree index to a group of shapely objects? it could sidestep a lot of conversions back and forth from python to postGIS if one could have a persistent (pickle/sqlite) group of shapely object with an also persistent index.

Re: Plone R-Tree Spatial Index

Author: Sean

Yes, that can be done. Shapely geometries are pickleable. Make a dict, for example, and store geometries with unique keys. Then index all the stored geometries, hashing the keys to get integer ids for the R-Tree: 
  >>> import pickle
  >>> import rtree
  >>> import shapely
  >>> store['geom0'] = shapely.geometry.Point(0, 0)
  >>> index = rtree.Rtree('the_points')
  >>> for key, geom in store.items():
  ...     index.add(hash(key), geom.bounds)
  >>> f = open('the_points.pik', 'wb')
  >>> pickle.dump(store, f)
  >>> f.close()
That would produce a Python shapefile of sorts: the_points.pik, the_points.idx, and the_points.dat.

Re: Plone R-Tree Spatial Index

Author: brentp

ah, too easy! thanks. i'll try it out.