How to lay out Python project code

A few years ago, a colleague asked me how to lay out a Python project. He was asking about web projects specifically, coming from a PHP background with an established tradition of file and directory structure. I didn't have a recommendation at the time other than to follow the practices of other successful projects. As Python use increases in GIS, and shops move beyond one-off scripts to reusable packages of standard processing code designed to be distributed around their systems, more GIS developers will be asking the same question. My answer is the same, but now I recommend a particular tool for arriving at the answer: paster and Paste Script. In its author's words:

paster is a two-level command, where the second level (e.g., paster help, paster create, etc) is pluggable.

Commands are attached to Python Eggs, i.e., to the package you distribute and someone installs. The commands are identified using entry points.

Paste Script has many more features, but is worth getting for the project creation feature alone. Get it from the Python Package Index: http://pypi.python.org/pypi/PasteScript/1.7.2, or by using easy_install:

$ easy_install PasteScript

The paster program comes with one template for a basic Python package. When executed, it prompts you for package name and metadata. A package of code that put a nice wrapper around arcgisscripting could be created like this:

sean@lenny:/tmp$ paster create -t basic_package
Selected and implied templates:
  PasteScript#basic_package  A basic setuptools-enabled package

Enter project name: biarcgis
Variables:
  egg:      biarcgis
  package:  biarcgis
  project:  biarcgis
Enter version (Version (like 0.1)) ['']:
Enter description (One-line description of the package) ['']:
...
Creating template basic_package
Creating directory ./biarcgis
  Recursing into +package+
    Creating ./biarcgis/biarcgis/
    Copying __init__.py to ./biarcgis/biarcgis/__init__.py
  Copying setup.cfg to ./biarcgis/setup.cfg
  Copying setup.py_tmpl to ./biarcgis/setup.py
Running /usr/bin/python setup.py egg_info

The result:

biarcgis
|-- biarcgis
|   `-- __init__.py
|-- biarcgis.egg-info
|   |-- PKG-INFO
|   |-- SOURCES.txt
|   |-- dependency_links.txt
|   |-- entry_points.txt
|   |-- not-zip-safe
|   |-- paster_plugins.txt
|   `-- top_level.txt
|-- setup.cfg
`-- setup.py

2 directories, 10 files

Let's say your shop has a standard license and authorship policy, wants more package metadata, wants to make sure all packages have a standard test suite, or just prefers a different layout. You can write your own project templates to meet these needs. I like the Pylons wiki example.

Update (2008-12-02): Kurt Schwehr wants to know where is the ChangeLog. The basic package template doesn't provide one, perhaps because the author doesn't want to presume what you want to name your log of changes. The basic Zope and Plone templates from ZopeSkel do provide a docs directory and a HISTORY.txt file for changes. The former even provides a test suite. ZopeSkel is an example of a paster template package that a shop (or community, like Zope's) might make to standardize its Python software.