Python packaging and builds

I don't follow the django-developers list so I might have missed Kevin Teague's great overview and analysis of Python packaging, dependencies, and builds (via Simon Willison). I've referred to setuptools, easy_install, and zc.buildout a number times, but never explained them quite as thoroughly or as well.

Setuptools is the not-quite-standard replacement for Python's distutils module. It has a big set of features, the one of which that helps me the most is versioning of dependencies. I can specify that zgeo.plone.kml 0.2 requires version 0.1 or higher of zgeo.plone.geographer, which in turn requires version 0.3 or higher of zgeo.geographer, and easy_install of the first one of those packages gets the proper dependencies in a cascade. I'm finding this to be better for my sanity than use of the svn:externals property to pull in tagged revisions as we used to do.

Despite what Teague says (update: I misinterpreted, see clarification in comments), I'm finding zc.buildout to be a big help with managing the C/C++ components in Pleiades. Because the GEOS project is -- how shall I say it? -- cautious about bug fix releases, and version 3.0 absent from Linux distros, I'm using Kai Lautaportti's hexagon.recipe.cmmi buildout recipe to download the GEOS 3.0.0 tarball, patch it, compile, make, and make install into an isolated environment for our web application with no more configuration than this:

[libgeos]
recipe = hexagonit.recipe.cmmi
url = http://geos.refractions.net/downloads/geos-3.0.0.tar.bz2
patch-options = -p1
patches =
  ${buildout:directory}/patches/geos-3.0.0-conditional.patch

Works like a charm.

To avoid polluting the Python package index with software of no use outside Pleiades, I've made our own specialized index. Easy_install of pleiades.policy (my obvious cargo cult of Martin Aspeli's example) pulls in pleiades.kml, and then the appropriate versions of zgeo.plone.kml and zgeo.kml. I'm happy to explain how to make such an index to anybody who's interested.

Do read Teague's post to the django-developers list if you're new to the Python software ecosystem, and if you develop any curiosity about how Zope works, read his excellent article about its component architecture.

Comments

Re: Python packaging and builds

Author: Kevin Teague

Thanks :) To clarify, I meant that buildout isn't useful for the problem of source code recompilation. It works great for source code compilation. The distinction between compilation and recompilation is the later is designed to help someone developing in a compiled language where you have an edit-compile-test cycle, a recompilation tool will do fancy things like look at timestamps to decide if an individual file needs to be recompiled, or manage compilation on multiprocessers or machines. If you're assembling parts of an application from source though, a one time 5 minute compilation step is no big deal. But if you have to go through that compilation every time you want to test a change in your code it's a pain. That being said, I guess it'd be possible to develop a buildout recipe which is useful for this type of workflow.

Re: Python packaging and builds

Author: Sean

Ah, yes I agree with you. Developing that GEOS patch within the buildout would have been painful.