The first MapServer 4.6 beta is out, the first of several on our way to a final in June. 4.6 will be an important release for mapscript users, and we need beta testers. Please download and try it on one of your development platforms. Only one notable adjustment to my 4.6 targets: I won't be adding non-square pixel support to mapscript.

The workshop that Hobu and I are running is full, so now it's time for the first homework assignment. If you are new to the Python language and platform, please spend a bit of time with Mark Pilgrim's Dive Into Python. Chapters 1-6 are essential background. Chapters 7-11 are not essential reading; we'll cover the important aspects as needed. Chapters 12 and higher are outside the scope of the workshop. We plan to spend about 20-30 minutes summarizing from Dive Into Python before we get into the GIS hacking.

According to Tim O'Reilly, sales of PHP books are up 16% in the last year. C# sales are also up slightly. While sales of other language books are down, Python continues to gain on Perl. My sense is that trends in the MapServer community roughly parallel the trends in computer language book sales.

Perl was the original mapscript language, followed soon after by Tcl and PHP (3). Use of Tcl mapscript has, as far as I can tell, ceased. O'Reilly makes no mention of Tcl in his article. Over the last year use of Perl mapscript has eroded (measured by user and developer list traffic), with users moving to PHP (4), Python, and Ruby. Use of Java mapscript continues to grow very slowly. MapServer's C# interface is attracting much more interest in the past year. This appear to be an entirely new set of users that are not crossing over from another mapscript-ing language.

Interest in PHP mapscript, already huge, continues to grow despite the lack of clear breakthroughs for PHP 5 or DSO support. It's easy to understand; PHP is pervasive, well documented and promoted, and (my bias may be showing) how is a PHP shop going to migrate its applications anyway? To Python? How would they pick from the many frameworks? To Java? They've chosen PHP because it is easy and flexible, and would struggle to shoe-horn their apps into a Java framework. There are new PHP applications and there are legacy PHP applications, but I can't imagine any moving away from PHP.

In the next year, I expect the Perl -> PHP trend to continue for MapServer. Java mapscript users will probably be switching to packages from the maturing GeoTools/GeoServer/uDig nexus. Python users will probably be switching from mapscript to PCL. C# developers, who now seem to be entirely in the Microsoft rather than Mono camp, may be picking the Carbon project over MapServer. I'm looking forward to the meeting this summer and the chance to hear other takes on the linguistic landscape of the future.

The Cartographic Objects for Zope 0.5 release is now available. There are new features associated with styles and rules, improved spatial reference systems, and support for richer scripting with the Python Cartographic Library. ZCO-0.5.0.tar.gz (sourceforge) requires either the PCL-0.8.1.tar.gz (sourceforge) source release or the PCL-0.8.1-win32.zip (sourceforge) windows binary release.

Have I mentioned how impressed I am by matplotlib? Swapping out the RendererAgg in my prototype mapping engine for a RendererSVG from matplotlib's SVG backend in no time yields a map as image/svg+xml: nwmed.svg (warning: 500K file). I downloaded an SVG-enabled Firefox CVS build and it displays a nice looking map of the Northwest Mediterranean.

As more MapServer users begin to explore the Java and C# mapscript modules, there are more questions about thread safety. I spent a little time yesterday afternoon gathering what is known about MapServer's thread safety and summarizing it in the form of answers to frequently asked questions.

Q: Is MapServer thread-safe?

A: Generally, no (but see the next question). Many components of MapServer use static or global data that could potentially be modified by another thread. Under heavy load these unlikely events become inevitable, and could result in sporadic errors. Furthermore, instances of mapscript classes should never be shared between threads.

Q: Is it possible to safely use any of MapServer in a multi-threaded application?

A: Some of it, yes, with care. Or with Python :) Programmers must either avoid using the unsafe components of MapServer or carefully place locks around them. Python's global interpreter lock immunizes against MapServer threading problems; since no mapscript code ever releases the GIL all mapscript functions or methods are effectively atomic. Users of mapscript and Java, .NET, mod_perl, or mod_php do not have this extra layer of protection.

A: Which components are to be avoided?

Q: Below are lists of unsafe and unprotected components and unsafe but locked components.

Unsafe:

• OGR layers: use unsafe CPL services

• Cartoline rendering: static data

• Imagemap output: static data

• SWF output: static data and use of unsafe msGetBasename()

• SVG output: static data

• WMS/WFS client connections: potential race condition in Curl initialization

• WMS/WFS server: static data used for state of dispatcher

• Forcing a temporary file base (an obscure feature): static data

• MyGIS: some static data

• ArcSDE support: connection information stored as static data

• PostGIS support: assumes all connections have the same byte order, which is stored as static data

Unsafe, but locked. The specific issue is noted:

• Map config file loading: global parser

• Setting class and and layer filter expressions: global parser

• Class expression evaluation: global parser

• Setting map and layer projections: PROJ.4 initialization

• Raster layer rendering and querying: GDAL library uses unsafe CPL services

• Database Connections: static connection information

Rather coarse locks are in place for the above. Only a single thread can use the global parser at a time, and only one thread can access GDAL raster data at a time. Performance is exchanged for safety.

So what can be done safely? Rendering features from shapefiles using the GD driver is safe. Raster data via GDAL is safe. PostGIS connections are fine so long as all return data with the same byte order. Projecting layers is safe. I'm working around the WMS client issue in several applications by fetching imagery using urllib and loading it into a MapServer imageObj with good results. This is good enough for my needs (PCL and ZCO). Most of the unsafe whizz-bang stuff (SWF, SVG) isn't implemented in a way useful to mapscript anyhow.

I couldn't resist writing a real matplotlib/Agg mapping engine for PCL to see how it could handle real-world features. In this example I am using the VMAP0-based world borders data available from http://mappinghacks.com. Usually available, that is -- Schuyler's site is offline today. The world borders shapefile has 3784 polygons and roughly 400,000 vertices in all.

My prototype engine is about 9x slower than the existing MapServer-based engine. I actually had expected it to be even slower than this because I am doing some expensive operations in Python rather than C. Turning PCL geometries into a Python list of vertex tuples for matplotlib and transforming from world to image coordinates are costly. This mere factor of 9 gives me hope that a mapping engine using Agg's C++ API could be at least as fast as MapServer's GD renderer.

The result is below. Its color scheme is from Color Brewer, courtesy of Cindy Brewer. The grey blobs around coastlines are polygon stroke noise; the data is too detailed for a map of this scale.

Below, for comparison, is a map using the same data rendered by MapServer (and GD). It renders small features such as the Maldives rather poorly compared to matplotlib and Agg. Indonesia and the Philippines are messy as well.

Many minor bugs are fixed in the 4.4.2 source release. The bugs are mostly related to raster layers, styled layer descriptors, and filter encoding. If you are using MapServer as a WMS or WFS server, particularly with FastCGI, an upgrade is recommended.

Matplotlib (http://matplotlib.sourceforge.net), a 2D plotting library, is attracting well deserved attention from Python users. It is a readily installed and rich environment for 2D plotting and visualization with a fat users guide and many nice examples. There even exists a cartographic module to generate background basemaps for plots of geophysical data. What intrigues me the most is its Agg backend and API that provides an easy way to learn about anti-grain geometry. It seemed like it wouldn't take long to find out if matplotlib and Agg were a good fit to PCL's mapping classes.

I began by revisiting the two slightly overlapping triangle polygons from a previous entry, then creating a polygon symbolizer, and defining map image output parameters. PCL polygon symbolizers have a default opacity of 50% -- we'll see the overlap effect in the output image:

from cartography import mapping, spatial
from cartography.spatial import Point, LinearRing, Polygon

t1wkt = 'POLYGON ((-1.0 50.5, -0.5 51.2, 0.3 50.9, -1 50.5))'
triangle1 = spatial.Polygon(wkt=t1wkt)

t2wkt = 'POLYGON((-0.7 50.3, 0.1 51.0, 0.6 50.1, -0.7 50.3))'
triangle2 = spatial.Polygon(wkt=t2wkt)

# Symbolizer for triangles
symb = mapping.PolygonSymbolizer(fill={'fill': '#FF8080'})

# Define output map parameters
epsg4326 = spatial.SpatialReference(epsg=4326)
view = mapping.View(epsg4326, spatial.Point(-1.2,51.25),
                    spatial.Point(1,50.05))

width = 550
height = 300

Define two utility functions: hex_to_rgb() converts PCL colors to matplotlib's normalized tuple, and transform_point() maps PCL points in world units to matplotlib image units. Image coordinates in matplotlib have their origin at the lower left corner, contrary to convention:

def hex_to_rgb(color):
    """convert a hex color such as #FFFFFF to a red, green, blue tuple"""
    c = eval('0x' + color[1:])
    r = (c >> 16) & 0xFF
    g = (c >> 8) & 0xFF
    b = c & 0xFF
    return (r, g, b)

def transform_point(point, view, width, height):
    """transform from world to image coordinates"""
    dx = (view.lr.x - view.ul.x)/width
    dy = (view.lr.y - view.ul.y)/height
    p = (point.x - view.ul.x)/dx
    l = (point.y - view.lr.y)/dy
    return (p, l)

Import the matplotlib module and render the two triangles. The matplotlib graphic context 'gc' is configured from properties of the PCL polygon symbolizer:

import matplotlib
matplotlib.use('Agg')
from matplotlib.backends.backend_agg import RendererAgg
from matplotlib.transforms import Value

dpi = Value(72.0)
o = RendererAgg(w, h, dpi)
gc = o.new_gc()

for polygon in [triangle1, triangle2]:
    points = [transform_point(p, view, width, height) for p in polygon[0]]

    # Fill
    gc.set_alpha(symb.fill['fill-opacity'])
    rgb = hex_to_rgb(symb.fill['fill'])
    face = (rgb[0]/255.0, rgb[1]/255.0, rgb[2]/255.0)
    o.draw_polygon(gc, face, points)

    # Stroke
    gc.set_alpha(symb.stroke['stroke-opacity'])
    gc.set_foreground(symb.stroke['stroke'])
    gc.set_linewidth(symb.stroke['stroke-width'])
    o.draw_polygon(gc, None, points)

o._renderer.write_png('agg_rendered.png')

the result, agg_rendered.png:

My impressions? So far, I like what I see. The matplotlib API is better suited to PCL's SLD-ish mapping objects than MapServer's mapscript module. That's a big plus for me. I don't yet appreciate why matplotlib excludes shape fill colors from the graphic context, or the quirky choice of lower left for image coordinate origin.

Clearly, the code within the loop over triangles needs to be written in C++ (like Agg) rather than Python to get decent performance when rendering hundreds or thousands of polygons at a time. I haven't written any C++ extensions for Python yet (C only), but this might be my first opportunity.

PCL-0.8-win32.zip (sourceforge) contains windows installer and 'dumb' binaries for Python 2.3. Do remember that this requires the MapServer mapscript module and Python modules from GDAL/OGR. I recommend the FWTools bundle as a source for those modules.