2007 (old posts, page 16)

On Expertise and Revolution

If you're interested in the expert/amateur debate, you must read the latest from Clay Shirky:

Digital and networked production vastly increase three kinds of freedom: freedom of speech, of the press, and of assembly. This perforce increases the freedom of anyone to say anything at any time. This freedom has led to an explosion in novel content, much of it mediocre, but freedom is like that. Critically, this expansion of freedom has not undermined any of the absolute advantages of expertise; the virtues of mastery remain are as they were. What has happened is that the relative advantages of expertise are in precipitous decline. Experts the world over have been shocked to discover that they were consulted not as a direct result of their expertise, but often as a secondary effect -- the apparatus of credentialing made finding experts easier than finding amateurs, even when the amateurs knew the same things as the experts.

This applies to mainstream GIS as well. Who was the go-to person for GIS data formats before blogs? Frank Warmerdam. Who's the go-to person today? Frank Warmerdam. A million amateurs typing in their blogs about the pros and cons of various formats hasn't changed that in the least. Likewise, a million amateur heatmaps won't reduce the value of a sharp spatial statistician. What has changed is the situation for people with industry credentials but only marginal expertise.


Re: On Expertise and Revolution

Author: FantomPlanet

Good grief, Sean. Now I'm mediocre? :)

REST on the Conference Circuit

For what it's worth, here's the result of a quick search for REST on the agendas of some of this year's GIS conferences:

  • ESRI User Conference: 1;
  • GeoWeb: 0 -- exclusively OGC, SOA (meaning SOAP), or OGC-SOA;
  • FOSS4G: 2 (so far) -- I figure the OSM talk will broach the subject -- and at least 9 presentations and workshops based on OGC, or SOA.

Bare Minimum RESTful WFS

I get glimpses of discussion on the non-public WFS Simple/Basic email list about a RESTful makeover for the protocol. We're going to see more and more of this as the developers of all public-facing services realize that they need at least a minimal RESTful aspect. Here's what that looks like for WFS:

  • To read (or query) use only HTTP GET
  • To publish use only POST and PUT
  • Always put request scoping parameters in the URL
  • Always express error conditions in HTTP response headers

What about complex filters with OGC XML expressions too long for the query string? First of all, this is a WFS bug and web server bug, not a REST bug. I've got some ideas about how to solve this using resource-oriented design (future post), but will stick to basic REST here.

RFC 2616 states:

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line. POST is designed to allow a uniform method to cover the following functions:

  • Annotation of existing resources;
  • Posting a message to a bulletin board, newsgroup, mailing list, or similar group of articles;
  • Providing a block of data, such as the result of submitting a form, to a data-handling process;
  • Extending a database through an append operation.

Does this invite us to POST filters after all? No. The WFS filters help describe the scope of the request. They are not part of the representation of a new resource, or data to be processed. Splitting request scoping between the URL (typename, bbox) and the request body (posted filters) seems to me to be the worst of both worlds. Better to put all scoping in the HTTP request body (if you must) and call it XML-RPC or SOAP, or put it all in the URL and call it RESTful.

Similarly, a RESTful WFS has to take advantage of the response status header. Its clients are HTTP clients first and foremost, not WFS clients. If the client sends a mangled request, reply with a 400 Bad Request. If the WFS filters screened out everything, reply with 204 No Content. By all means, do put a more verbose message in the response body, but the status header is more important to an HTTP client.

Ignoring service capability and feature type metadata for the moment, the above is all that's needed to produce a basic REST-RPC hybrid service.


Re: Bare Minimum RESTful WFS

Author: Bill Thorp

Whats the return type of your RESTful GET? Can it be JSON, so that browsers can eat WFS's across domains via script-tag injection? Maybe a thin JSON wrapper of SVG, VML, or canvas? At what point to do stop catering to client implementation?

Re: Bare Minimum RESTful WFS

Author: Sean

Good question. I think one flavor of JSON + GML would cover a lot of bases. The former is handy for browser clients, and the latter can be transformed to almost anything else by more capable clients.

Re: Bare Minimum RESTful WFS

Author: Jason

How's this: the OGC Filter can be POSTed -- it is a representation of a subordinate resource to the GetFeatures resource. The GetFeatures operation re-done RESTfully is a "features" resource -- all the features available. A Filter is a representation that, er, represents a sub-set (subordinate) of the "features" resource. It's not the only representation. There's also the GetFeatureById (I think) operation that represents a subset of "features" just using a different representation...

Re: Bare Minimum RESTful WFS

Author: Sean

Yes, that's pretty much what I was going to suggest in a future entry. Put filters to a subordinate resource like /wgis/{typename}/queries/{name}, and then read them like filtered sub-types. It's not a foreign concept at all. Many of us use watchlists or saved queries in other web applications. The trick would be to do it without keeping client state on the server.

Re: Bare Minimum RESTful WFS

Author: Jason

You could avoid Filter and attempt CQL, but I don't think CQL has much traction with implementers(?). CQL would fit nicely in a HTTP/GET query string, tho. Still bothering me is the fact that POSTing a Filter to a "features" resource, ought to result in the creation of a new (sub-ordinate) resource. But this never happens. The result set is created, sent back to the client and forgotten about usually. Does this violate the whole POST/Filter strategy?? Alternatively, the GetFeatures operation could accept a URL that points to a Filter instance (on my server -- my shared/saved/bookmarked filters) instead of me sending it. That Filter URL could be used with HTTP/GET more easily than CQL perhaps. The stored query works nice in Catalog (maybe WFS has this too?). Basically a pre-cooked Filter on the server waiting to be parameterized with the HTTP/Get query string. This seems good n' RESTful, but that stored query must pre-exist on the server. A small number of stored queries would actually satisfy most search use-cases, but the primary mode of today is to just sending along big/ugly filters every time. Perhaps, if this shifts -- user-agents using the stored query as a rule not the exception -- then POSTing a Filter would actually mean something like: "Oh, you want ME, the server, to support a new Filter? Well, OK. Here's my response that contains your freshly baked URLs that you may use to execute that Filter". This way, the POSTing would create a new HTTP/GET URL for the resource (features) satisfying the filter that was previously POSTed.

Geo-Web-REST Group

I've convinced the people with whom I discuss REST on several IRC channels to try a discussion list: Geo-Web-REST. Totally public, and devoted to RESTful and resource-oriented GIS/geo architectures. To kick things off, I've posted about my ongoing implementation of the Atom Publishing Protocol using GeoRSS.

Diving Into Python

Update: speaking of Python, read Guido van Rossum's Python 3000 update.

Let's say you, like Tom Kralidis, are getting into Python because it's the best environment for combining MapServer and GDAL/OGR. Not only in scripts, but at the interpreter prompt. You'd like to not only learn to do the same old stuff in a new language, but also to learn some new tricks and increase your programming Fu. You're in luck, because Python is a great environment for knowledge acquisition.

Tom writes:

So here's what I typically start off with when doing development:

  • HTTP / URLs / Forms: for Perl, I use CGI; for PHP, I use the native, built in functions

My recommendation is to leave CGI at the door. If you want to learn about Python web programming, start with WSGI. Joe Gregorio calls it the "Java servlet API for Python", but to me it also feels a lot like good old CGI. There is a reference implementation in the Python 2.5 standard library, and wsgiref is also easy to acquire for Python 2.4. For web page templating, I recommend Genshi.

  • XML Processing: for Perl, I use XML::Simple; for PHP, I use SimpleXMLElement

Take the opportunity to learn how to use XPath. For example, see how WxS metadata is parsed in OWSLib using element.find and element.findall.

  • Database: for Perl, I use DBI; for PHP, I use the compiled in mysql support, for example

You already know how to execute SQL in a scripting context, so why not try out one of the interesting object-relational APIs for Python such as SQLAlchemy.

Joe Gregorio's Robaccia tutorial covers everything you need to know about modern Python web programming. Read it, code it, run it. Then try a community supported framework like Django. Then think about using these new tools in some geo applications. My toy not-a-WFS Hammock has some good examples, and MetaCarta's TileCache also works with any WSGI server.

The name of this post is a reference to Mark Pilgrim's Dive Into Python, a great resource. Just don't follow Pilgrim into xml.dom or unittest, use ElementTree or lxml, and doctest instead.

At any rate, do experiment with Python community packages. The SWIG or SIP-generated stuff that comes out of our GIS world tends to obscure the goodness of Python.


Re: Diving Into Python

Author: Allan

I second the recommendation of Dive Into Python.

Hammock Update

Update (2007-06-22): No DELETE, but you can POST an Atom entry to /features.atom and PUT to any of the individual entry edit URLs. I also verified the service document and collections using Joe Gregorio's APP Test Client.

I haven't added PUT or DELETE yet, but Hammock is close to satisfying the read-only aspects of the Atom Publishing Protocol. The service document links to the single collection represented as a feed. Every resource now has alternate XHTML representations, which are created using Genshi: service, collection, and item.

GeoSummit Conclusion

Update: from Peter Batty.

Our unconference went well. We even had parallel sessions twice. It was a pleasure to meet fellow bloggers (I'm joking, I don't think any of us identify primarily as "bloggers") Peter Batty, Chris Haller and John Spinney, who had great answers to my dumb questions about local positioning and mobile networks. G. H. Chinoy gave us all a detailed, candid breakdown of .NET and Java WorldWinds, from which we made tangential forays into the pros and cons of government-lead open source development. Brian Timoney shared some killer tips on KML applications served from S3 and demonstrated web-based spatial analysis linked to Google Earth that has serious business value. Who cares if it's not "real" GIS? His workshop at FOSS4G is not to be missed.

Thanks to everyone for coming out, and an extra big thanks to Tom Churchill for hosting. I'm still partially mesmerized by his last demo of the day.

Role for JP2 in RESTful Web Services?

Recently, I wrote about a maximally connected resource-oriented alternative to the OSGeo Tiled Map Service (TMS). Readers mainly objected to the double request: one to get a map image tile, one more to get links to its neighbors or links to higher/lower resolution. Is this prohibitive? Without trying, I can't say, but it does occur to me that more sophisticated enterprisey systems might use JPEG 2000, with URIs of neighboring tiles stored in the JP2 XML box. One request per tile, and I'm still using hypermedia as the engine of application state.

If JP2 + XML is good enough for T.J. Hooker, Jr. (top of the article's lead-off graphic), it's good enough for me.


Re: Role for JP2 in RESTful Web Services?

Author: Matthew Giger

Actually you could probably put xml in the comments fields of standard jpegs and png files and have the same thing. The overhead makes it not such a good tiling format, unless the tiles are like 8k by 8k images. JP2 is far better for distributing large images with the ability to efficiently pan within.