How to lay out Python project code

A few years ago, a colleague asked me how to lay out a Python project. He was asking about web projects specifically, coming from a PHP background with an established tradition of file and directory structure. I didn't have a recommendation at the time other than to follow the practices of other successful projects. As Python use increases in GIS, and shops move beyond one-off scripts to reusable packages of standard processing code designed to be distributed around their systems, more GIS developers will be asking the same question. My answer is the same, but now I recommend a particular tool for arriving at the answer: paster and Paste Script. In its author's words:

paster is a two-level command, where the second level (e.g., paster help, paster create, etc) is pluggable.

Commands are attached to Python Eggs, i.e., to the package you distribute and someone installs. The commands are identified using entry points.

Paste Script has many more features, but is worth getting for the project creation feature alone. Get it from the Python Package Index:, or by using easy_install:

$ easy_install PasteScript

The paster program comes with one template for a basic Python package. When executed, it prompts you for package name and metadata. A package of code that put a nice wrapper around arcgisscripting could be created like this:

sean@lenny:/tmp$ paster create -t basic_package
Selected and implied templates:
  PasteScript#basic_package  A basic setuptools-enabled package

Enter project name: biarcgis
  egg:      biarcgis
  package:  biarcgis
  project:  biarcgis
Enter version (Version (like 0.1)) ['']:
Enter description (One-line description of the package) ['']:
Creating template basic_package
Creating directory ./biarcgis
  Recursing into +package+
    Creating ./biarcgis/biarcgis/
    Copying to ./biarcgis/biarcgis/
  Copying setup.cfg to ./biarcgis/setup.cfg
  Copying setup.py_tmpl to ./biarcgis/
Running /usr/bin/python egg_info

The result:

|-- biarcgis
|   `--
|-- biarcgis.egg-info
|   |-- PKG-INFO
|   |-- SOURCES.txt
|   |-- dependency_links.txt
|   |-- entry_points.txt
|   |-- not-zip-safe
|   |-- paster_plugins.txt
|   `-- top_level.txt
|-- setup.cfg

2 directories, 10 files

Let's say your shop has a standard license and authorship policy, wants more package metadata, wants to make sure all packages have a standard test suite, or just prefers a different layout. You can write your own project templates to meet these needs. I like the Pylons wiki example.

Update (2008-12-02): Kurt Schwehr wants to know where is the ChangeLog. The basic package template doesn't provide one, perhaps because the author doesn't want to presume what you want to name your log of changes. The basic Zope and Plone templates from ZopeSkel do provide a docs directory and a HISTORY.txt file for changes. The former even provides a test suite. ZopeSkel is an example of a paster template package that a shop (or community, like Zope's) might make to standardize its Python software.


Re: How to lay out Python project code

Author: Stefano Costa

It's for sure a coincidence that I was looking for an answer to the very same question during the past 3 days, but at least it's a good one.

My concern about using PasteScript is whether it is good also for an application that has nothing to do with web projects. Would you recommend it also for a generic Python application?

Re: How to lay out Python project code

Author: Sean

I did. Paste Script's basic package template is for Python packages, not web applications.

Video, Namespacing and Zopeskel

Author: Christopher Perkins

I actually have a video on showmedo that shows how to use all this stuff, and integrate testing. Personally, I don't start a new Python project without a fresh virtualenv. The video is a really simple how-to on how to get started with this stuff. I also blogged about the motivation for, and how to set up a namespaced project a few months ago here: Hope this information is useful to you. cheers. -chris

Re: How to lay out Python project code

Author: Sean

Thanks, Chris. I've been using ZopeSkel's namespace package template for my zgeo projects. I'm a virtualenv user too, yet wary of recommending it to a former Avenue programmer in a GIS shop. I did some hand waving last year about how virtualenv could potentially replace FWTools.

Re: How to lay out Python project code

Author: Sean

Do watch the video Chris made if you're new to this and curious.