In preparing for a few rounds of large Pleiades coordinate updates in the next few weeks, I've been rediscovering zopectl's run command. It lets you run a script in the context of your database:

$ zopectl run script.py

much like psql (for example) lets you run a script on your Postgres database:

$ psql -f script.sql

The main differences are that with zopectl your script is Python, not SQL, and that the data is a arbitrarily large and nested mapping bound to the name app instead of tables. Imagine your dataset is a giant JSON structure: that covers much of what developing with the ZODB (Zope object database) is like. The ZODB is stored as an append-only B-Tree and enjoys many of the benefits explained in the CouchDB Guide. The essential similarities and differences between ZODB and CouchDB were summarized two years ago by Chris McDonough here.

A key feature of Pleiades is resource versioning. There's a repository tool in which we track changes to places, names, and locations; it even provides diffs between versions. I found out before the holiday that a Plone developer (Pleiades uses Plone 3.2, and your mileage may vary) has to take a few extra steps to setup supporting utilities for the repository tool in the case of an offline script.

from Products.CMFCore.utils import getToolByName
from Products.CMFUid.interfaces import (
    IUniqueIdGenerator, IUniqueIdAnnotationManagement, IUniqueIdHandler)

from zope.component import provideUtility

def setup_cmfuid(site):
    provideUtility(
        getToolByName(site, 'portal_uidgenerator'), IUniqueIdGenerator)
    provideUtility(
        getToolByName(site, 'portal_uidannotation'),
        IUniqueIdAnnotationManagement)
    provideUtility(
        getToolByName(site, 'portal_uidhandler'), IUniqueIdHandler)

Now I call this function in my script and the repository tool is ready for use. The login() function makes sure that all changes are made as the specified user (me, in this case) and bulk_update_locations() updates the coordinates from a number of CSV records within a single transaction, annotating each version change with a message.

from pleiades.bulkup import login, setup_cmfuid

site = app['plone']
setup_cmfuid(site)
login(site, user)
bulk_update_locations(site, reader, columns, message)

Mikko Ohtamaa's Command-line interaction and scripting helped me learn how to set the effective user (or login), but didn't explain why the 3 Plone tools in my case weren't being automatically registered.

Update (2011-01-12): Better: http://sgillies.net/blog/1065/thats-more-like-it

A reader suggested to me in comments that I should approach the words on the FOSS4G 2011 site (which I quoted previously) from a different angle and try to appreciate that they're aimed at folks outside the FOSS4G community. Speaking to them using their own language including, apparently, their own unflattering stereotypes of open source users in an anything goes effort to lure them in to the event. I suppose this makes some twisted sense if you ignore an eventuality: that this corporate audience will at some point discover themselves fully surrounded by these very same wild-eyed open source hippy freetard philosophers. But nevermind that for now, I'm fully onboard with the FOSS4G rhetoric and have some other stereotypes that the marketers are free to use or repurpose if the original

Many early adopters of FOSS solutions chose them based on "philosophical" reasons, but ...

starts to wear thin. How about this one, which uses unnecessary quotes to maintain the right style?

Many early adopters of FOSS solutions were into GIS as a "fun" hobby, but ...

That will have strong appeal for certified professionals.

Many early adopters of FOSS solutions felt "everything" should be free, but ...

The conference is coming back to the USA, and nothing is more American than using Communists as bogeymen.

Many early adopters of FOSS ranted about the "difference" between freedom and free beer, but ...

Always with the damn philosophizing, those freetards.

Many early adopters of FOSS were from "academic" backgrounds, but ...

I almost forgot the Ivory Tower! Nobody is less pragmatic than an academic researcher, right? Hope this helps, and Merry Christmas.

Bullshit.

Recent years have seen substantial changes in the geospatial industry. One of those changes has been the growth in maturity and adoption of free and open source solutions. Many early adopters of FOSS solutions chose them based on "philosophical" reasons, but increasingly large enterprises and government organizations are choosing these solutions for pragmatic business reasons. In many cases organizations are using a mixture of open and closed source solutions.

Emphasis above is mine. What the hell is up with the scare quotes and the impractical, ideology-driven, early adopter strawman? If you were there in the early FOSS4G days, you'll remember that it was in fact about pragmatic solutions: proprietary software often lacked features that we needed to do our jobs or was riddled with bugs that we suspected wouldn't be fixed any time soon. The need to get shit done was the driving force behind GDAL, MapServer, PostGIS, and friends. Driving a stake through the heart of ArcIMS was just gravy.

Update (2011-01-12): Better: http://sgillies.net/blog/1065/thats-more-like-it

James Clark, Technical Lead of the XML working group and author of expat recently wrote an interesting post about XML, JSON, and web development titled "XML vs the Web":

So what's the way forward? I think the Web community has spoken, and it's clear that what it wants is HTML5, JavaScript and JSON. XML isn't going away but I see it being less and less a Web technology; it won't be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire.

This isn't JSON triumphalism; Clark has well expressed mixed feelings about the turn.

If our pre-WhereCamp5280 meetup is indicative of how cutting edge developers are evaluating and selecting technology, JSON is definitely the future of sending features to geospatial apps running in the desktop, laptop, or mobile browser. But there's more to the web than browsers. I get a little concerned sometimes that going all in for JSON and HTML5 tech like WebSockets takes us even further off the web as integration platform than we were with GML and W*S.

Tomorrow, November 18, is the date for the pre-WhereCamp5280 open source dynamic language geospatial programming hackfest/unworkshop. Please do sign in (or out if you must) here – we'll get to see more of Chris if he doesn't have to scramble to find a bigger room tomorrow morning. It's fun that this ended up being scheduled on the day after "GIS Day"; A bunch of us are definitely going to be working on what comes next after GIS.

For extra adventure, I'm going to attempt something I've never done before: travel from downtown Fort Collins to downtown Denver using only my legs and public transportation. In theory, Transfort/Flex and RTD connect in Longmont, and I've received some confirmation of this and other good tips from Front Range riders. We often went by car last year in France, but also did a lot of inter-city train trips and bus trips around the Montpellier Agglomeration. Other than riding to DIA, I've never done this kind of bus trip in Colorado. It will be interesting to compare the experiences.

I tried a new approach in explaining Pleiades last week:

Are you familiar with "Google Places"?

Yeah.

Pleiades is like that for the ancient world.

Cool.

The analogy is only a little bit bogus, and I'm going to try explain that now. Google's Places (and Facebook's and Foursquare's) are about linking the concept of being at a store, or venue, or landmark with advertisements, reviews, deals, reservations, media, nearby places, resources (including maps) that reference them – largely for commercial purposes. Resources that link to other resources. Pleiades places, on the other hand, are about linking the concept of studying an ancient city, road, or region with nearby places and references to supporting articles, books, maps, digital editions of inscriptions, and archaeological GIS. These are scholarly, not commercial nodes in the web.

That's not to say that there is no data in commercial location sites worthy of research or that there's nothing in them worth emulating. While I'm not personally interested in "playing" Foursquare, Gowalla, or the like, I think we'd be completely off track if Pleiades didn't help make something like a visualization of Alexander checking in, ousting mayors, and gaining badges across Asia trivially possible.

In a abstract just submitted to Digital Humanities 2011, I labeled Pleiades an "Un-GIS". I feel it's important for users and watchers in the humanities, which is going gang-busters for GIS technology, to understand the differences between Pleiades and a ESRI geodatabase, an OGC-style feature/map service, or a conventional digital gazetteer. I don't think it's useful to try to precisely define "Un-GIS", but here are a few qualities that I think distinguish Pleiades from a typical geographic information system or spatial data infrastructure:

• Aggregation of temporally varying features into conceptual places or spaces that reflect ancient practice or modern scholarly method.
• Rich toponyms with ancient spellings, transcription details, temporal scope, and links to primary sources and scholarly literature.
• Identification and representation of geographic features that have no known locations, or that can be located only vaguely, roughly, or in relationship to each other.
• Embrace of the uneven distribution in quality and density of data that is inherent in ancient world studies.
• Embrace of web architecture.

Pleiades distinguishes between place and space [1]. In writing "Ephesus demonstrates the potential complexity of ancient Mediterranean urban centers" (an example from the DigitalClassicist wiki), a scholar would not be referring to the coordinates of the footprint of Ephesus, but to a historical entity and also the body of work of which it is the subject.

Primacy of space might be the defining characteristic of GIS, but it's the names of places, not their coordinates, that occur in ancient texts and inscriptions. We model toponyms carefully so that we'll be able to serve researchers mining ancient texts for new insights into ancient geography. We're even going to keep track of what the Barrington Atlas calls "false toponyms": place names attested to in ancient or modern works that are now considered to be erroneous. These include names from Avienus' "Ora Maritima" that are regarded to be not just wrong, but fictitious [2].

More common than these false toponyms are names for unlocated places such as Kritalla, the marshalling point of Xerxes' army [3]. More common yet are places with fuzzy or non-determinable boundaries like the territory of the Salluvii or the Aegean Sea. Pleiades can identify and represent places for which boundary lines would be misleading. The boundaries of the Roman province of Aegyptus shown on map sheet 100 of the Barrington Atlas, for example, are clearly noted by the editors as rough and approximate [4].

The compilers of GIS datasets (a population that once included me) usually aspire to uniform density and quality of data, and for good reason. The ancient world, however, doesn't give up its secrets like that. Data about it is spotty. Some places are truly lost, some are less accessible. Scholars, too, choose the places that interest them whether they fill in the gaps or not. Pleiades embraces the inherently unfinished nature of ancient studies; instead of waiting on precise coordinates from partners, we're rolling out places with approximate locations that will be refined, live, as we get better locations. I think we're lucky to be doing this now after Wikipedia and OpenStreetMap have completely changed the nature of content and geodata creation – not just in opening it up to non-experts, but in freeing data to be improved incrementally.

Throughout this post, I've provided links to Pleiades using the URIs of places. The representations at the other end themselves carry links to names and locations and neighboring places, and soon (we hope) to digital editions of ancient and modern works served from other domains. We take GIS seriously in Pleiades, but whenever there is a question of "what is the GIS approach" vs "what is the web approach" to a problem, we go with the latter.

Most of all, I offer "Un-GIS" as a starting point for interesting discussions at the conference about how GIS technology and methods do and don't directly apply to historical geography.

Web Linking is RFC 5988. Via Assaf Arkin's Rounded Corners 258 I came on to an example (for Firefox and Opera) that assigns a stylesheet using .htaccess. This could also be used to turn "dumb" image files into hypermedia; a map tile JPEG could link (in its headers) to metadata, to neighboring tiles, to data for the features depicted within.

Update (2010-11-04): more from Erik Wilde.

Geospatial programmers inclined toward interpreted languages and open source don't get together enough on the Front Range. Please help Chris Helms, Tyler Erickson, and I correct this unfortunate state of affairs on November 18 at the once Colorado Brewery, now Nationally Registered Historical Place known as CU Denver's Tivoli Student Center. WhereCamp once piggybacked on Where 2.0 and now we're hooky bobbing on WhereCamp5280. See the announcement by Chris on the FRUGOS list for details and spread the word.

Update (2010-10-26): Please sign up on the wiki if you're planning to attend.

Earlier this week, Raj Singh graded the OGC's services against REST and then graded the REST architectural style itself:

The fundamental problem with REST and GIS in my opinion, is that REST optimizes for data access, and OGC services are optimized for data processing. I consider GIS a type of OLAP system, and we as an industry will continue to resist REST because it would be counter-productive to expose data via atomic resources that all had their own URL (and metadata!?!) because everything we really want to do with the data would get harder.

In comments, Stu Charlton points out that REST is in fact optimized for data processing. Myself, I think it's simply in a different style, one that looks more goal oriented or agent oriented; I've linked to a past post by Charlton on hypermedia agents that has many good comments of its own.

Is it true to that the GIS industry resists REST? Some companies and consortia do, others don't. It seems like there's a press release about a new Geospatial REST API every other day. We've just seen a "REST API standard" document released by the 800 lb gorilla itself. A lot of this is just marketing, but there is also a growing, honest interest in resources instead of endpoints and HTTP as an application protocol.

I'm not sure grading OGC services benefits anybody, but if you absolutely have to grade your services against a REST maturity model it's probably best to use a standard model.