2007 (old posts, page 27)

PleiadesGeocoder 1.0b1

Here it is, the first 1.0 beta release of the ultimate content geo-annotation plugin for Plone 2.5: PleiadesGeocoder-1.0b1.

It's deployed on the Pleiades site (see links on the project wiki), and also at telascience.org: append 'georss' or 'kml' to any of the channel URLs to get an alternative representation of those aggregated feeds. Earlier versions of this software were inspired by the simple profile of GeoRSS, but 1.0 is aligned with GeoJSON. You can see it in the primary interface:

class IGeoItemSimple(Interface):

    """A simple georeferenced object, analogous to a GeoRSS entry, a KML
    placemark, or a GeoJSON feature.

    The spatial reference system is implicitly lat/long WGS84.
    """

    geom_type = Attribute(
        """Name of geometry type: 'Point', 'LineString', or 'Polygon'"""
        )

    coords = Attribute(
        """GeoJSON-style coordinates tuple(s) with long, lat ordering."""
        )

    __geo_interface__ = Attribute(
        """A mapping that provides Python geo interface. See
        http://trac.gispython.org/projects/PCL/wiki/PythonGeoInterface.
        """
        )

    def setGeoInterface(geom_type, coords):
        """Set type and coordinates, following the geo interface spec."""

    def isGeoreferenced():
        """Returns True if the item is "on the map", meaning that the item
        has a geometry type and coordinates."""

PleiadesGeocoder allows annotation of content types derived from Archetypes right out of the box. Plone developers who know their way with Zope and Five should find it easy to adapt non-AT content.

Taming the OGR

This evening I made a protoype of a smoother, simpler interface to the industrial-strength vector data functions in libgdal. The new entities in Refinery are workspaces, collections, and features. Workspace.collections is a dict of Collection objects, which are approximately equal to OGR layers. Collection.features is an iterator over GeoJSON-style objects:

>>> workspace = refinery.workspace('/var/gis/data/world')
>>> workspace.collections.keys()
['world_borders']
>>> borders = workspace.collections['world_borders']
>>> for feature in borders.features:
...     print feature.id, feature.properties['CNTRY_NAME']
world_borders/0 Aruba
(etc)

Use of the refinery package reduces the size of my canonical matplotlib script from 17 to 12 lines:

import pylab
from numpy import asarray
import refinery
from shapely.wkb import loads

workspace = refinery.workspace('/var/gis/data/world')
borders = workspace.collections['world_borders']

fig = pylab.figure(1, figsize=(4,2), dpi=300)

for feature in borders.features:
    shape = loads(feature.geometry)
    a = asarray(shape.exterior)
    pylab.plot(a[:,0], a[:,1])

pylab.show()

That's starting to feel civilized.

This started off as recreational programming, but I think it has some promise. Get the code [Refinery-0.0.tar.gz] if you're curious and make a trial workspace from your own data. Next: a look at the GeoDjango GDAL wrappers to see what Justin and I are doing differently.

Comments

Re: Taming the OGR

Author: Matt Perry

Very cool. Some nice syntactic sugar on top of the unpythonic (but useful) OGR API. Out of curiosity, why "collections" rather than the more standard terminology of "layers"?

Re: Taming the OGR

Author: Sean

Thanks, Matt. My other favorite feature is no dependence on SWIG. This uses the Python ctypes module, which is standard in 2.5. The term 'collection' is inspired by WFS, AtomPub, and GeoJSON. Is 'layer' really standard outside the GDAL community?

Re: Taming the OGR

Author: Matt Perry

Ah! I assumed it was a thin layer on top of OGR SWIG bindings.. ctypes looks like the future of FFI and it's nice to see it being put to such good use! Layer is pretty much the universal word across all GIS camps for a group of vector features. This is the first I've ever heard of "collections" in reference to spatial data (apparently I don't delve too much into the web-based world too often these days ;-).

Re: Taming the OGR

Author: Paul Ramsey

I think "Collection" is a better term for a set of features. "FeatureCollection" is somewhat more explicit. A "Layer" is a styled "FeatureCollection", it's the thing you see in your map, it's not the data itself, and one "FeatureCollection" can be the basis for multiple "Layers".

Re: Taming the OGR

Author: Matt Perry

I like that. A workspace contains collections of features. Each collection can be paired with one or more styles to create layers. The only reason I'm harping on this is that anyone with a traditional GIS background will be flummoxed by the "collection" terminology at first. The last thing we need in the geospatial world is more confusion over vocabulary ... I'm still dealing with the whole "coverage" debacle!

More fun With Curl and AtomPub

GData is AtomPub, more or less. The GML and GeoRSS namespaces appear in the examples. Have I ever mentioned that I think it's extremely interesting and significant that Google isn't using WFS-T as a gateway for geo-tagged Picasa items?

Plone R-Tree Spatial Index

At the 2006 Plone Conference sprint, Shaun Walbridge and I wrote a Quadtree-based spatial index for Plone. Unlike the portal catalog, it was a localized index, turning a Plone folder of georeferenced content into a shapefile of sorts. It was a nice proof of concept, but was limited by lack of persistence. A few months ago Howard Butler made it possible to persist Rtree indexes on the filesystem, and last week I finally made the time to rewrite the original Plone product into a persistent R-tree index for Zope/Plone data.

SpatialIndex keeps its original design. Adapting a Plone folder to Products.SpatialIndex.interfaces.ISpatialIndex creates an index on disk alongside the ZODB data, with a name that corresponds to the folder's physical path. Content objects can then be adapted to Products.PleiadesGeocoder.interfaces.IGeoItemSimple and added to the index. Ultimately, the index may be queried for the records of items that intersect with a bounding box. The capabilities are summarized in the session below, using a parks folder that contains a lee-martinez document:

>>> parks = app['plone']['parks']
>>> document = parks['lee-martinez']
>>> from Products.SpatialIndex.interfaces import ISpatialIndex
>>> index = ISpatialIndex(parks)
>>> from Products.PleiadesGeocoder.interfaces import IGeoItemSimple
>>> geoitem = IGeoItemSimple(document)
>>> geoitem.setGeoInterface('Point', (-105.08442, 40.59512))
>>> index.add(geoitem)
>>> hits = index.intersects((-106, 40, -105, 41))
>>> [h for h in hits]
[('lee-martinez', (-105.08442, 40.59512, -105.08442, 40.59512)]

SpatialIndex depends on

You should get PleiadesGeocoder 1.0a1 and SpatialIndex 1.0a1 from the repositories:

$ svn co http://icon.stoa.org/svn/pleiades/PleiadesGeocoder/tags/rel-1.0a1\
 PleiadesGeocoder
$ svn co http://svn.gispython.org/svn/primagis/SpatialIndex/tags/rel-1.0a1\
 SpatialIndex

Additionally, SpatialIndex provides a yet-under-construction index management view through which you count the indexed items and reindex folders, and another public view (@@spatialindex) that can be used in various custom forms and pages.

http://sgillies.net/images/manage-index.jpg

Comments

Re: Plone R-Tree Spatial Index

Author: Yves Moisan

This morning on slashgeo an article points at http://www.directionsmag.com/press.releases/?duty=Show&id=19542&trv=1 : "... CartaLens, an innovative geospatial digital asset management ... Unlike other digital asset management solutions that only manage structured content, CartaLens is able to search and retrieve location-based information from both structured content and a broad base of digital content such as photos, video, audio and documents, ..." You mean one can map progress reports like little Word of PDF icons on a map ;-). I don't think it is emphasized enough that the stack you and Kai are building allows all sorts of structured/semi-structured/unstructured content to be viewed on maps now (and without the ArcGIS requirement). It will be interesting to see what data CartaLens sites typically provide and how close we can be with the Zope/Plone/Sean*/Kai* platform. Cheers, Yves

Re: Plone R-Tree Spatial Index

Author: Sean

I've always assumed that one turned the MetaCarta engine loose on networks or storage systems to find information, and that it was rather different from the software I'm developing.

Re: Plone R-Tree Spatial Index

Author: Yves Moisan

I must admit I know nothing about MetaCarta. It just striked me as not particularly groundbreaking news it was to be able to view unstructured content on maps, as I've been doing that (I think ??) with Plone and its various "location products" for many months now. But I may well be underestimating MetaCarta's engine and overestimating Plone.

Re: Plone R-Tree Spatial Index

Author: brentp

sean, somewhat related: is it possible to add an rtree index to a group of shapely objects? it could sidestep a lot of conversions back and forth from python to postGIS if one could have a persistent (pickle/sqlite) group of shapely object with an also persistent index.

Re: Plone R-Tree Spatial Index

Author: Sean

Yes, that can be done. Shapely geometries are pickleable. Make a dict, for example, and store geometries with unique keys. Then index all the stored geometries, hashing the keys to get integer ids for the R-Tree:
  >>> import pickle
  >>> import rtree
  >>> import shapely
  >>> store['geom0'] = shapely.geometry.Point(0, 0)
  >>> index = rtree.Rtree('the_points')
  >>> for key, geom in store.items():
  ...     index.add(hash(key), geom.bounds)
  >>> f = open('the_points.pik', 'wb')
  >>> pickle.dump(store, f)
  >>> f.close()
That would produce a Python shapefile of sorts: the_points.pik, the_points.idx, and the_points.dat.

Re: Plone R-Tree Spatial Index

Author: brentp

ah, too easy! thanks. i'll try it out.

Lines and Polygons in Plone

Update (2007-10-17): the main Pleiades site has the update. See the Barrington Atlas map feed and KML. If you're interested in trying this out, be sure to get revision 1185.

I finally implemented lines and polygons in our PleiadesGeocoder software not so much for display of ancient roads as for making georeferenced feeds that show where we will have new data coming online. Here's an example feed and map, and a snapshot of the KML in Google Earth below:

http://sgillies.net/images/pgc_polys.jpg

Also new is a form for setting the location of any Plone content. Location is still stored in GeoRSS (Simple) form in PleiadesGeocoder (mostly to delay content migration), but the form takes GeoJSON. The new code I'm working on for Plone will be storing locations as GeoJSON geometries.

http://sgillies.net/images/where_tab.jpg

After we get a few bugs out, this will be released as PleiadesGeocoder 0.7 for Plone 2.5, and I'll get back to work on the next generation for Plone 3.

Comments

Re: Lines and Polygons in Plone

Author: Yves Moisan

Can't wait for the Plone 3 work :-). I just got myself an OpenID account to start playing with on a Plone 3 site. Maps next. Cheers,