Yo, Lazyweb: is there a publicly available WMS providing maps of the Pleistocene world, or some layers thereof?

(Image from USGS pub http://pubs.usgs.gov/gip/continents/.)

It feels good to read about Shapely being put to use in EveryBlock alongside OpenLayers, Mapnik, PostGIS, OGR, and TileCache.

RSS 1.0 is a format, RSS 2.0 is a format, Atom is a format. GeoRSS is not a document format as such: it is a trio (one deprecated) of ways to format geospatial metadata within documents.

This week I finally got around to working on a task that's been haunting me: OpenLayers as an AtomPub client. Chris Schmidt has done much of the work already, but I'm going to see it all the way through over the next few months.

A quick review of AtomPub and how it relates to geospatial applications is probably needed here. There are 3 different types of resources in the Atom Publishing Protocol (omitting for now workspaces): services, collections, and entries. the analogous components in a GIS architecture are: services, featuretypes (or coverages), and features. An AtomPub service document is in some ways like an OWS capabilities document, but simpler because AtomPub doesn't fool around with non-HTTP semantics and bakes more metadata into its single protocol. An Atom feed document is rather a lot like a WFS feature collection document, and an Atom entry is also much like a GML feature, but much more standardized. OpenLayers does WFS already, and needs little modification to be a good AtomPub client.

Here's the little modification that I have done: a new OpenLayers.Format.Atom lets me ignore the jumble of namespaces and elements that is RSS 1 and 2. This Atom.js is simpler than GeoRSS.js. The Atom entry atom:content element is the prime avenue for extending Atom, and I've also added support for that. Best practice for georeferencing Atom entries is to use a GML simple features geometry (see OGC 06-049r1) within a georss:where element and OpenLayers.Format.GMLSF supports simple features GML within my new Atom format. The GMLSF format writes gml:pos and gml:posList only, no gml:coordinates, and uses "exterior" and "interior" as ring element tags. On the other hand, it is just as forgiving as OpenLayers.Format.GML when reading. So now an OpenLayers app can write well calibrated feature entries to be POSTed to an AtomPub-style collection like the one at http://sgillies.net/kw/++rest++knowhere/demo. (I'll explain the odd URL in a future post).

That collection is live. I've copied over a few entries from my Hammock app like so:

sean@lenny:~$ curl -X GET http://sgillies.net/hammock/places/6.atom > 6.atom
sean@lenny:~$ curl -X POST -v \
-H "Content-Type: application/atom+xml" \
-H "Slug: Navarro Vineyards" \
--data @6.atom \
http://sgillies.net/kw/++rest++knowhere/demo
* About to connect() to sgillies.net port 80
*   Trying ...... connected
* Connected to sgillies.net (...) port 80
> POST /kw/++rest++knowhere/demo HTTP/1.1
> User-Agent: curl/7.15.5 (i486-pc-linux-gnu) ...
> Host: sgillies.net
> Accept: */*
> Content-Type: application/atom+xml
> Slug: Wehlener Sonnenuhr
> Content-Length: 1301
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 201 Created
< Date: Sun, 17 Feb 2008 20:46:21 GMT
< Server: Twisted/2.5.0 ...
< Content-Length: 73
< Accept-Ranges: bytes
< X-Content-Type-Warning: guessed from content
< Location: http://sgillies.net/kw/++rest++knowhere/demo/wehlener-sonnenuhr
< Content-Type: text/plain;charset=utf-8
Connection #0 to host sgillies.net left intact
* Closing connection #0
Location: http://sgillies.net/kw/++rest++knowhere/demo/wehlener-sonnenuhr

(The ability to copy resources is built into HTTP.) If you browse to http://sgillies.net/kw/demo/map (and wait a few seconds for a pile of javascript to download) you'll see this placemark displayed among others as a vector layer in an OpenLayers map of the demo collection. Toward implementing AtomPub, I have a second vector to serve as a buffer for to-be-posted placemarks. Draw a new geometry in the map using the OpenLayers tools and you'll see it in green. Then set a title, summary, text, a placemark name slug, and click "post". If you've picked a slug that collides with an existing placemark name, you'll see an error message. Otherwise, you'll see the location of the newly created placemark.

Still yet to do are placemark edit and deletion as Chris's AtomPub demo already does. I think that getting this right will require a new OpenLayers.Layer.AtomPub, one that behaves much like the existing WFS layer, but interacts with the server using GET/PUT/DELETE/POST. The protocol implementation, such as it is, currently exists in this page template. The only real novelty in it is the use of the HTTP Slug request header to make suggestions for resource names.

function postPlacemark() {
  var feature = buffer.features[0];
  feature.attributes.title = $("pm-title").value;
  feature.attributes.description = $("pm-summary").value;
  feature.attributes["content"] = $("pm-content").value;
  var atom = format.write(feature);
  var options = {
    method: "post",
    contentType: "application/atom+xml",
    postBody: atom,
    onSuccess: updatePage,
    onFailure: function(xhr) {
      update_status("Failed post (status code "+xhr.status+"). Check your URL.");
    },
  };
  var slug = $("pm-slug").value;
  if (slug.length > 0) {
    options.requestHeaders = ["Slug", slug];
  }
  new OpenLayers.Ajax.Request(collection_URL, options);
}

My server has the final say, and currently just lowercases the slugs and replaces whitespace with a dash.

I submitted the new Atom and GMLSF formats to OpenLayers (#1366) and look forward to working with the developers to get them into an upcoming release.

Some guy who either can't give up the straw man argument or still doesn't get it writes:

Just like a biker ends up carrying more or less the same weight -- whether as a bike or as a lock, the GIS system users end up paying, one way or another. They pay the software vendor if they choose a commercial platform, or they pay the implementer (in staff time or consultant fees) to glue together the pieces of the “free” solution.

And who exactly is arguing otherwise? Open source is about freedom, not about cost-free operation. How many times does this have to be said?

Raj Singh writes this about growth of OGC services and standards:

I was over at Harvard yesterday talking to people from the GSD, the Herbarium, the Library, the new Center for Geographic Analysis, and MassGIS. It was great to see some old friends and make some new ones. One topic that came up was why you don’t see OGC standards in widespread usage. I argued that the geospatial Web is where the regular Web was in 1996, when the tech industry thought that HTML was so easy and powerful that everyone would build their own Web sites – not by writing HTML but by using tools like Dreamweaver. As it turns out, that was still way too high a barrier to entry. People didn’t want the hassle of designing a site from scratch. They wanted to post a blog entry or a MySpace page. That’s when the Web saw a real quantum leap in content.

In 1995, the number of web sites increased from about 10,000 to about 100,000. By the end of 1996, there were about 600,000 web sites. If there are this many WxS servers out there, and the number is growing at such a rate, I'm just not seeing it. My own take is that the WxS-based "GeoWeb" is more like Gopher circa 1993, before the advent of the Web: useful, technically adequate for many purposes, but ultimately just an experiment in web-building. There are design defects in OGC protocols (which use HTTP while keeping it at arm's length) that Raj does not acknowledge here: it's not just a matter of making better geo-editors. WxS was a worthy experiment, but ultimately not the way forward. The explosive growth that Raj is looking forward to is going to happen instead on the new axis of KML (with GML) and HTTP.

Update (2008-02-05): Is it a meme yet? Gary Sherman is thinking outside the RDBMS box too.

Interesting question from Kirk Kuykendall:

What I’m proposing here sure sounds a lot like what Hadoop does. I wonder if we will ever see the day when we can use something like Hadoop for geoprocessing with ArcObjects on .NET?

The answer is yes, right now. Maybe on windows, for sure on Linux. The why you'd want to put proprietary per-server-licensed software in the mix – when the point of Hadoop is to leverage the combination of commodity hardware and open source – escapes me. Streaming mappers using JTS or GEOS (or Shapely) seem like the way to go. I'd almost bet that somebody somewhere is already doing this, perhaps with their own proprietary geoprocessing software instead of open source. Now what we really need is spatial indexes for HDFS ... maybe Paul Ramsey is interested in tackling this at his new gig?

OGC update sequence? Why bother when HTTP/1.1 already specifies ETag?

Geoff Zeiss writes:

Graduating OSGEO Incubation includes requirements for open community operation, a responsible project governance model, code provenance and license verification and general good project operation. Graduating incubation is the OSGeo seal of approval for a project and gives potential users of the project added confidence in the viability and safety of the project.

It looks like it is up to cranky old me to point out that of FDO's 17 committers, "haris" is the only one who is not an employee of or contractor to Autodesk. I'm not counting "timstrang", who has 1 commit for 0 lines of code. 13 Autodesk employees, 2 Autodesk contractors, and 1 independent committer.

Don't get me wrong, FDO looks like a technically excellent project, and they've done their best to do project business in the open and recruit new people. But it seems to me that it's failed to attract a diverse group of contributors. It probably would not have been graduated yet from an incubator like Apache's. Do non-diverse open source projects suck? No. I'm almost sole committer on several projects that kick ass, but I'll be the first to admit that they wouldn't be ready to graduate from the Apache incubator (even if they were accepted).

Maybe the real issue is whether the OSGeo brand is to represent standards as high as Apache's. It might represent lower standards, and that would be fine. Many entities happily used GDAL and GRASS before they were OSGeo branded. However, if the brand is supposed to be the equal of Apache's, OSGeo's "A for effort" incubation policy needs to be reconsidered.

Update: a reader says I was wrong to count mloskot as an Autodesk contractor. That would make 2 independent committers.

Howard Butler gives me something to pin on the Pleiades bulletin board (nothing motivates quite like adversity):

[...] I do have a lot of experience with Zope/Plone, and GeoDjango fills some big holes in the Zope stack when it comes to geospatial stuff that I don’t see ever being filled.

and then goes on to write all kinds of sensible stuff about GeoDjango. I'm not a GeoDjango user, but I've benefited from interaction with the developers. Justin Bronn inspired me to apply ctypes to GEOS (Shapely is in some sense a rip-off of GeoDjango, which was itself inspired partly by PCL's geometry package), and meeting him was one of my FOSS4G 2007 highlights. It's a worthy project.

The biggest reason to go with Django over Zope is, without a doubt, PostGIS: as a open source GIS developer/consultant you've already invested huge amounts of data and brain cells in Paul Ramsey's killer app while developing MapServer maps, and every single bit of this investment carries over into Django application development. You could say the same for Rails. None of your MapServer skills translate into Zope skills.

Of course, PostGIS can't help you at organizational boundaries, and Django doesn't provide "GeoWeb" services any more than Zope does. It's up to GeoDjango to develop WxS (or alternative) implementations.

Here's a disturbing thought: if only MapServer (the CGI program) implemented the Model-View-Controller pattern ... you wouldn't need GeoDjango at all. Or Django, for that matter. Use OGR as your connector to data (geospatial or not), and make entire dynamic "GeoWeb" sites entirely through mapfile configuration and CGI query string parameters.