Sean Gillies (Posts about zope)

Migration of my blog

Sean Gillies — Sun, 29 Nov 2015 07:00:00 GMT

A while back I switched my blog from a database-backed dynamic site to static HTML generated by Sphinx and Tinkerer. This weekend I finished migrating eight years of old posts (with their comments) to the new blog. And I configured my new blog and home page to support HTTPS. Hello, 2015.

You may, when visiting old posts, experience browser warnings because of embedded HTTP resources. Images mainly, but also the odd script here and there. I have also not yet implemented the redirect rules that will get you from old style number/slug URLs to the new YYYY/MM/DD/slug URLs, though I have mapped them all out. Please bear with me, I'll address these issues soon.

Doing this lets me retire my last dedicated server. My blog now runs on a very cheap ($5 per month) virtual server provisioned (Nginx, SSL certs and keys, the works) using Ansible. I build my blog HTML on my laptop and rsync it with the server also using Ansible.

I'm pretty sure this blog migration is the final chapter of my Zope story. The following is very likely my final Zope external method, ever.

import json


def dumper(context):
    """Dump Python blog posts to JSON"""
    for result in context.searchResults(
            meta_type='COREBlog Entry', sort_on='created',
            sort_order='descending'):
        post = result.getObject()

        doc = {}
        doc['id'] = post.id
        doc['slug'] = post.entry_slug()
        doc['title'] = post.title_or_id()
        doc['body'] = post.body
        doc['categories'] = post.categories()
        doc['published'] = post.published()
        doc['html'] = (post.format == 2)

        coms = []
        for comment in post.comment_list():
            com = {}
            com['title'] = comment.title
            com['author'] = comment.author
            com['url'] = comment.url
            com['body'] = comment.body
            coms.append(com)
        doc['comments'] = coms

        date = post.published().split('T')[0]
        date = date.replace('-', '')

        outfilename = '/tmp/blog/{0}-{1}-{2}.json'.format(
            date, post.id, post.entry_slug())

        with open(outfilename, 'w') as f:
            json.dump(doc, f, indent=2)

Lessons learned from Zope

Sean Gillies — Wed, 21 Dec 2011 07:00:00 GMT

Chris McDonough:

Most developers are very, very risk-averse. They like taking small steps, or no steps at all. You have to allow them to consume familiar technologies and allow them to disuse things that get in their way.

The allure of a completely integrated, monolithic system that effectively prevents the use of alternate development techniques and technologies eventually wears off. And when it does, it wears off with a vengeance.

The natural reaction to the problems caused by monolithic systems, which is to make anything and everything uniformly "pluggable" is also a net lose.

Tests are very important.

Documentation is even more important than tests.

When trying to replace a complex system, make it simpler, don't just push the complexity around.

Brands are important. You, ideally, must make sure that the question "what is 'noun'" have one answer. Sublety and branding do not mix; they produce a toxic haze of confusion.

My story with Zope parallels Chris's, though not as nearly as deep, and trailing by about a year and a half. I've been using his software and docs for about a decade now. For the past 4-5 years I've been trying to apply the Zope lessons above to GIS software; Shapely and Fiona are thoroughly steeped in them.

Versioning in Pleiades scripts

Sean Gillies — Thu, 30 Dec 2010 07:00:00 GMT

In preparing for a few rounds of large Pleiades coordinate updates in the next few weeks, I've been rediscovering zopectl's run command. It lets you run a script in the context of your database:

$ zopectl run script.py

much like psql (for example) lets you run a script on your Postgres database:

$ psql -f script.sql

The main differences are that with zopectl your script is Python, not SQL, and that the data is a arbitrarily large and nested mapping bound to the name app instead of tables. Imagine your dataset is a giant JSON structure: that covers much of what developing with the ZODB (Zope object database) is like. The ZODB is stored as an append-only B-Tree and enjoys many of the benefits explained in the CouchDB Guide. The essential similarities and differences between ZODB and CouchDB were summarized two years ago by Chris McDonough here.

A key feature of Pleiades is resource versioning. There's a repository tool in which we track changes to places, names, and locations; it even provides diffs between versions. I found out before the holiday that a Plone developer (Pleiades uses Plone 3.2, and your mileage may vary) has to take a few extra steps to setup supporting utilities for the repository tool in the case of an offline script.

from Products.CMFCore.utils import getToolByName
from Products.CMFUid.interfaces import (
    IUniqueIdGenerator, IUniqueIdAnnotationManagement, IUniqueIdHandler)

from zope.component import provideUtility

def setup_cmfuid(site):
    provideUtility(
        getToolByName(site, 'portal_uidgenerator'), IUniqueIdGenerator)
    provideUtility(
        getToolByName(site, 'portal_uidannotation'),
        IUniqueIdAnnotationManagement)
    provideUtility(
        getToolByName(site, 'portal_uidhandler'), IUniqueIdHandler)

Now I call this function in my script and the repository tool is ready for use. The login() function makes sure that all changes are made as the specified user (me, in this case) and bulk_update_locations() updates the coordinates from a number of CSV records within a single transaction, annotating each version change with a message.

from pleiades.bulkup import login, setup_cmfuid

site = app['plone']
setup_cmfuid(site)
login(site, user)
bulk_update_locations(site, reader, columns, message)

Mikko Ohtamaa's Command-line interaction and scripting helped me learn how to set the effective user (or login), but didn't explain why the 3 Plone tools in my case weren't being automatically registered.

Shapely 1.0.8

Sean Gillies — Sat, 01 Nov 2008 06:00:00 GMT

Paul Winkler and the OpenGeo team found a problem with Shapely and Zope's web server on their 64-bit system, provided the fix, and now here's Shapely 1.0.8. An upgrade is recommended if you're using Zope or a similar framework.

Comments

Re: Shapely 1.0.8

Author: Paul Winkler

Hi Sean, thanks for the kind words and getting an official fix out so quickly. I must give credit where due: All the hard gdb sleuthing was led by my coworkers David Turner and Douglas Mayle. I don't have the necessary C/C++ chops to have solved this on my own. Oh, and to clarify, the problem had nothing to do with Zope (or Grok in our case), this upgrade should be good for anyone running on a 64-bit system. I was eventually able to write a standalone script that demonstrated the segfault. The same problem could afflict any python package that uses Ctypes on 64-bit systems without explicitly marking argument and return types.

Adding pyproj to a buildout

Sean Gillies — Thu, 18 Sep 2008 06:00:00 GMT

Pyproj is Jeffrey Whitaker's Python interface to PROJ.4. I'm no longer interested in other Python projection packages. Its most interesting feature is interoperability with packages (such as Shapely) that use the Numpy array interface. It depends on Cython, which makes it a bit tricky to include in a buildout: you must install Cython into your buildout's python, not as an egg, and make the pyproj egg only after this step is finished. Like this:

System Message: ERROR/3 (<string>, line 3)

Cannot find pygments lexer for language "none"

.. sourcecode:: none

  [buildout]
  parts =
    cython-src
    cython-install
    pyproj

  [cython-src]
  recipe = hexagonit.recipe.download
  url = http://cython.org/Cython-0.9.8.1.1.tar.gz

  [cython-install]
  recipe = iw.recipe.cmd
  on_install = true
  cmds =
    cd ${buildout:directory}/parts/cython-src/Cython-0.9.8.1.1
    ${python:executable} setup.py install

  [pyproj]
  recipe = zc.recipe.egg:eggs
  index = http://atlantides.org/eggcarton/index
  eggs = pyproj

You might be able to pull pyproj off PyPI, but here I am using my own index. Once built, you can try it out using zopepy:

>>> from pyproj import Proj
>>> defn_900913 = """
... +proj=merc +a=6378137 +b=6378137
... +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0
... +units=m +nadgrids=@null +no_defs
... """
>>> proj_900913 = Proj(defn)
>>> lonlat = (25.0, 25.0)
>>> proj_900913(*lonlat)
(2782987.269831839, 2875744.6243522423)

I like that projection definitions can be split across lines of the screen.

AtomPub for zgeo.atom

Sean Gillies — Fri, 14 Mar 2008 06:00:00 GMT

I've rewritten Knowhere's AtomPub implementation, dropping the Grok skin traversal mechanism in favor of a custom publisher and customized traverser. I've gained harmony among the various resource URLs and learned a ton about publishing in Zope. Now, zgeo.atom has an incomplete, but functional Atom publishing protocol. A collection is currently online at

http://sgillies.net/kw/demo/atompub-collection

The curious are free to add new placemark entries and edit them following the steps below.

First, create a file called test.atom with contents something like this:

<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:georss="http://www.georss.org/georss"
       xmlns:gml="http://www.opengis.net/gml">
  <title>Test</title>
  <summary>Testing placemark</summary>
  <content type="text/html">
    &lt;p>600 N Sherwood St, Fort Collins, CO&lt;/p>
  </content>
  <georss:where>
    <gml:Point>
      <gml:pos>-105.0842514037999962 40.5944633483999979</gml:pos>
    </gml:Point>
  </georss:where>
</entry>

Use your own location for extra points. Add to my collection using everyone's favorite RESTful client, curl:

sean@lenny:/tmp$ curl -i -X POST \
  -H "Authorization: Basic YWRtaW46OGZjOGFkZmM=" \
  -H "Content-Type: application/atom+xml;type=entry" \
  -H "Slug: 600 N Sherwood" \
  -d@test.atom \
  http://sgillies.net/kw/demo/atompub-collection

Use a different slug if you like. Here are the response headers:

HTTP/1.1 201 Created
Date: Fri, 14 Mar 2008 04:32:33 GMT
Server: Twisted/2.5.0 TwistedWeb/[twisted.web2, version 0.2.0]
Content-Length: 744
X-Powered-By: Zope (www.zope.org), Python (www.python.org)
Accept-Ranges: bytes
Location: http://sgillies.net/kw/demo/600-n-sherwood/atom-entry
Content-Type: application/atom+xml;type=entry

Browse to that location and you'll get an Atom entry document like this:

<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:georss="http://www.georss.org/georss"
       xmlns:gml="http://www.opengis.net/gml">

    <title>Test</title>
    <link href="http://sgillies.net/kw/demo/600-n-sherwood/atom-entry"
          type="application/atom+xml;type=entry" rel="edit"/>
    <link href="http://sgillies.net/kw/demo/600-n-sherwood"
          type="text/html" rel="alternate"/>
    <id>urn:uuid:dfa47428-e9ce-41b4-9f42-c2a3cad9037a</id>
    <updated>2008-03-14T04:32:33Z</updated>
    <summary>Testing placemark</summary>
    <georss:where>
      <gml:Point>
        <gml:pos>-105.084251 40.594463</gml:pos>
      </gml:Point>
    </georss:where>
</entry>

The newly created placemark can be edited by PUT of a new representation to its "edit" link. Edit your test.atom file, changing the title, or summary, or location, and use curl once again:

sean@lenny:/tmp$ curl -i -X PUT \
  -H "Authorization: Basic YWRtaW46OGZjOGFkZmM=" \
  -H "Content-Type: application/atom+xml;type=entry" \
  -d@test-edit.atom \
  http://sgillies.net/kw/demo/600-n-sherwood/atom-entry

The response: HTTP/1.1 200 OK.

DELETE works too. Go ahead and delete your own placemark if you wish. It's not as exciting as you might think. The Knowhere map app uses the same AtomPub links to add placemarks. Editing and deletion through OpenLayers are next up.

Comments

Re: AtomPub for zgeo.atom

Author: Brian Flood

hi sean what are your thoughts on APP feeds with multiple geometry types? (like the example above). At first I thought we should enforce a single geometry type per feed but this is probably impracticable in the real world. Client apps that tried to enforce this would be precluded from using (what could be) the vast majority of feeds that don’t care about enforcing similar geometries (for example I could see GMaps MyMaps doing this). How do you see the OL APP client working? a free for all like cschmidt’s original demo or something more GIS like with a single geometry? for our ArcMap client, I’m now creating point,line,and polygon featureclasses for each feed to serve as the local store. users will need to switch between layers when editing but all updates will be posted to the same feed. thoughts? cheers brian

Re: AtomPub for zgeo.atom

Author: Yves Moisan

Neat. Made me discover curl on Windows. I typed a bit too quick and inadvertently edited the title of your 600 N sherwood entry :-(. I redid it after for the entry I made for "Mont Orford" and refixed yours using the info in your edit.atom example :-). Phew ! Great work again.

Re: AtomPub for zgeo.atom

Author: Sean

Brian, I'm mainly going to use collections of a single geometry type for Pleiades, separating settlements and ethnic regions for example, but there are uses for the free for all collection.

Re: AtomPub for zgeo.atom

Author: Sean

Thanks, Yves. Hopefully I can turn some other Zope/Plone developers on to this. The scope of zgeo.atom is rather narrow now, but could be generalized to non-geo content.

Re: AtomPub for zgeo.atom

Author: Yves Moisan

Hopefully I can turn [INTO a] Zope/Plone developer [soon]. I'm more of a give-it-a-quick-go lurker for the time being.

Knowhere Project Wiki

Sean Gillies — Mon, 17 Dec 2007 07:00:00 GMT

I made a new wiki page for Knowhere. It's my Christmas present to everyone. What, you were expecting something other than open source software?

The buildout failed on my PowerPC iBook, which has gcc 4.0.0. Howard Butler has been successfully building the spatialindex library on his servers with gcc 4.0.1, so it may build on newer OS X systems as long as you have Python 2.4.

Grok, the Paleolithic Geographer

Sean Gillies — Sat, 15 Dec 2007 07:00:00 GMT

I've wanted to make an easily deployed and transparent application to demonstrate what can be done with zgeo.geographer and friends, the new geospatial packages for Zope, and finally inspiration struck. I chose to write a Grok project using the aptly named grokproject. Grok aims to simplify Zope to the degree that a caveman could use it. Convention over configuration and all that. It certainly allows a developer a clear view of the zgeo.* packages in action.

The paleolithic geographer isn't concerned about fancy geographic information systems. He wants a know-where-things-are system; hence the name of my demo app: Knowhere (pronounced "Oog"). Knowhere annotates resources with geographic location metadata and publishes them using GeoRSS (Atom, specifically) or KML so that a even a caveman can visualize them in the context of other geographic information on the Web.

Building and launching Knowhere is a 5 step process that requires Python 2.4, gmake, gcc, etc. Within 15 minutes you can be publishing geo-referenced content. Start by getting the code from the PrimaGIS Subversion repository:

$ svn co http://svn.gispython.org/svn/zope/Knowhere/trunk Knowhere

Cd into that new directory and bootstrap the project:

$ /usr/bin/python2.4 bootstrap.py

Build it out:

$ ./bin/buildout

This downloads and builds 2 C/C++ libraries (GEOS and spatialindex), and fetches a heap of eggs. It will take a few minutes.

When the build out is complete, edit 2 configuration files. In parts/zopectl/zope.conf, set the port number to an open value. In parts/app/site.zcml, change the management password from "admin" to one less obvious. That done, start up the application:

$ ./bin/zopectl fg

Browse to http://localhost:8080 (let's presume that's where it's at), authenticate, and you should see a Grok management page like this:

Add a new application named "cities" as shown, and then browse to http://localhost:8080/cities where you'll be faced with one of the ugliest forms ever seen. Fill it out like so

name: fort-collins
title: Fort Collins
description: Fort Collins, Colorado, USA
text: The little geospatial hothouse on the prairie
geometry type: Point
coordinates: [-105.08, 40.59]

using a GeoJSON coordinate representation, and submit. Click through to the new placemark page, which should be something like this. There are 5 fields on that page. Look at the Knowhere model (src/knowhere/app.py) and you'll see that the Placemark class actually only has one attribute:

class Placemark(grok.Model):

    def __init__(self, text=None):
        self.text = text

Where are the other fields? Grok gives you Dublin Core metadata annotations automatically. No configuration required. Peek into the model's configuration (src/knowhere/configure.zcml) and you'll see that adding geographic location metadata from zgeo.geographer to any model class is as simple as:

...
<include package="zgeo.geographer"/>

<class class=".app.Placemark">
  <implements interface="zgeo.geographer.interfaces.IGeoreferenceable"/>
</class>
...

The zgeo.kml package provides KML representations of containers of georeferenceable objects. Browse to http://localhost:8080/cities/@@kml-document for an example. The zgeo.atom package provides Atom entry and feed document views with GeoRSS. For examples, browse to http://localhost:8080/cities/fort-collins/@@atom-link-entry and http://localhost:8080/cities/@@atom-search-feed.

Knowhere also includes an R-Tree spatial index from zgeo.spatialindex. Containers of the model can be made indexable as easily as:

...
<include package="zgeo.spatialindex"/>

<class class=".app.Demo">
  <implements interface="zgeo.spatialindex.interfaces.ISpatiallyIndexable"/>
</class>
...

Browse to http://localhost:8080/cities/@@manage-index to see the index management form. Future versions of Knowhere will include georeferencing event handlers, but for now you must rebuild the index manually after adding placemarks. Lame indeed, but this is just a demo (I already have a working handler for Plone AT content types). The management screen in the image below is from my demo application at http://sgillies.net/kw/batlas, which had 2 placemarks at the time I wrote this post.

The spatial index allows fast bounding box queries. The atom search feed accepts a bbox parameter of the form minimum_longitude,minimum_latitude,maximum_longitude,maximum_latitude. For example, see http://sgillies.net/kw/batlas/@@atom-search-feed?bbox=-5,50,5,55 [map].

As you can see in the image above, zgeo.* can handle polygon geometries as well as lines and points. View that demo application in Google Earth using http://sgillies.net/kw/batlas/@@kml-document.

There's more work to do on Knowhere and zgeo.*. The index might be better as a local utility, and I should be using an integer id utility instead of hashing content names. The HTML is wretched and the URLs cry out for rewriting (which Grok supports right out of the box). Any Zope user that can look past those warts might start to get some good ideas from Knowhere.

New Geospatial Packages for Zope

Sean Gillies — Sun, 02 Dec 2007 07:00:00 GMT

As part of my work on migrating Pleiades to Plone 3, I'm distilling Zope packages from the original old-style Plone products. The zgeo.geographer introduces the conventional GIS data model of geometries, features, and collections (ala GeoJSON) to Zope and describes an interface for annotating objects with geographic location metadata. The zgeo.spatialindex provides a container-local R-Tree spatial index for geographically annotated objects, enabling fast spatial bounding box searches.

Comments

Re: New Geospatial Packages for Zope

Author: J.F.

Excellent news!

Grok Does AtomPub

Sean Gillies — Fri, 09 Nov 2007 07:00:00 GMT

Nice. I'd like to bend Grok's REST components to fit Pleiades after our move to Plone 3.