Multiprocessing with Rtree

2008-10-30T19:49:11Z in python, the lab

I came up with an example of how Python's multiprocessing package -- standard in 2.6, and recently backported to 2.4 and 2.5 (on PyPI) -- could be used to set up a simple R-Tree index server:

from multiprocessing.managers import BaseManager

class RtreeManager(BaseManager):
    pass

RtreeManager.register('add')
RtreeManager.register('intersection')

if __name__ == '__main__':

    from rtree import Rtree

    class NoisyRtree(Rtree):
        def add(self, i, bbox):
            print "Adding: %s, %s" % (i, str(bbox))
            Rtree.add(self, i, bbox)
        def intersection(self, bbox):
            print "Searching: %s" % str(bbox)
            return Rtree.intersection(self, bbox)

    index = NoisyRtree('processing')

    RtreeManager.register('add', index.add)
    RtreeManager.register('intersection', index.intersection)

    manager = RtreeManager(address=('', 50000), authkey='')
    server = manager.get_server()
    print "Server started"
    server.serve_forever()

Run that module (man.py) as a script to start the server, and access the index from Python in a new process like this:

>>> from man import RtreeManager
>>> manager = RtreeManager(address=('', 50000), authkey='')
>>> manager.connect()
>>> print manager.intersection((-20.0, -20.0, 20.0, 20.0))
[1L, 2L, 5L]

Three items were already in my index, persisted on disk. One more can be added like this:

>>> manager.add(4, (-10.0, -10.0, -9.0, -9.0))
<AutoProxy[add] object, typeid 'add' at 0x-483da894>
>>> print manager.intersection((-20.0, -20.0, 20.0, 20.0))
[1L, 2L, 5L, 4L]

The manager synchronizes access so additions and queries from different processes don't clobber each other.

Comments are closed after 13 days.

about archive feed search

Some rights reserved by Sean Gillies.