2009 (old posts, page 4)

Look what I found in KML

Look what I found in KML while looking for hyperlinks: a RESTful service description format. A neat find because we need more working examples of service descriptions.

KML's "NetworkLink" isn't the most well-named element. It's not so much of a link, really, but a slot for feature data that is filled in during a user session. The data is provided by a read-only web service described by the content of the network link's "Link" element. There are 3 service flavors. A link with nothing other than a "href" element tells the client to fetch the resource identified by link[@href]. A link with an "onStop" view refresh mode tells the client to append a bounding box parameter to the service's resource identifier. Lastly, a link may give the client a URI template of sorts via the "viewFormat" element.

This is the shape of RESTful service description. The server tells the client how to make requests for feature data in the context of a particular KML document. It might use a slightly or radically different description in a different context. It might evolve the description over time within the same KML context to meet changing client needs or engineering constraints.

KML was on the cutting edge here, but has diverged from the web. Using standard links with the same semantics of those in HTML and Atom instead of an "href" would be good. Standardizing on URI templates might be a nice feature for the future, as would better separation of concerns in "Link". Specification of on what event and how often the "network link" is refreshed is mixed up with the service description. Some of the elements seem to me to belong in the network link rather than the link.

I often gripe about the KML format ("Snippet"!), but there's much good in it – the RESTful service description pattern in particular.

Update (2009-05-13): all aboard the snippet train.


Re: Look what I found in KML

Author: Bent

<Icon> has the same semantics and refresh modes as <Link>

(And, <Link> in the context of <Model> is for the collada data type).


More literally <NetworkLink> could perhaps have been <KmlLink>.

Personally I find onRegion the most interesting and most restful

mode of all...

New GIS and Python programming blog

I stumbled onto Solving GIS Problems today. It has an ESRI bent, but it's Python, and there's code – a rare thing in the geoblogosphere. And we agree on a bunch of things such as how logging wins and arcgisscripting cursors fail. Subscribed.


logging vs testing

Author: Jonathan Hartley

Hey there.

Is logging still relevant when you have unit and functional tests? I suspect not, but I'm asking because I genuinely don't know.

I used to use logging in previous projects. My current one is heavily tested, which seems to work really well for us, and we don't ever seem to need any logging, so we've stopped putting any in. Are the two factors correlated?

I think so, but I can't prove it. Clearly the two techniques don't do the same thing, but they do provide similar information (eg. what the code did on one particular run, verses what the code does under all the circumstances that the test writer could think of)

Anyone got any insight to add?

Re: New GIS and Python programming blog

Author: Simon Norris

Thanks for posting this. An ESRI bent can be useful!

Re: New GIS and Python programming blog

Author: sharkinsGIS

Thanks for posting your discovery. You may want to check out my latest post at http://foldingspace.wordpress.com/ as well.

Re: New GIS and Python programming blog

Author: Sean

Unexpected things happen even if your code is well tested: inaccessibility of network resources, topologically invalid user data, you name it. You'll want to log those events and move on when you're processing batches of data or providing a web service.

Philosopher's Stone

Jeff Harrison tries to flip the alchemist label back onto the web wonks:

Maybe what we should be thinking about is how to advance GeoWeb platforms NOT as proprietary implementations using an alchemy of REST, JSON, RSS, KML - but as Community GeoWeb Platforms that combine international standard geospatial web services with Web 2.0 technologies.

Nice try, but no. We agree about the proprietary flavor of some of the "GeoWebs" out there. Google, in particular, promotes a "GeoWeb" that is dominated by its proprietary browser and its proprietary search. Our role in this "GeoWeb" is to make KML files and expose them through sitemaps, increasing the value of Google's proprietary index in exchange for some return traffic through Google's proprietary browser. Sharecropping, in other words. Still, the OGC's SOA isn't appropriate for a "GeoWeb". There's no web there. Principles of web architecture, standard protocols like HTTP/1.1, and standard formats like JSON, Atom – and even KML – remain the way to geographically enrich the existing web.

Generic geometry library

The Generic Geometry Library (via Mateusz Loskot) looks interesting. Algorithms operating on coordinate arrays. Optional geometry classes. Shapely's new plugin framework was motivated by my interest in Python (or Java or .NET) implementations, but a plugin using a C or Cython interface to ggl could be neat.


Re: Generic geometry library

Author: Howard Butler

GGL doesn't really do squat right now. A CGAL or LEDA backend for Shapely would be much more interesting and immediately useful.

Re: Generic geometry library

Author: Sean

It doesn't? Too bad, because I liked the idea algorithms without a lot of class baggage.

Re: Generic geometry library

Author: Mateusz Loskot

I'm not sure what hobu means it doesn't do squat, probably due to lack of proper english understanding ;-)

But answering to the Python thing, I suppose that if GGL will get Python bindings it will be based on Boost Python as GGL is going to be submitted for review and if approved it will join the Boost Libraries.

GeoRSS media type?

As Randy George finds out, the notion of a "GeoRSS format" is confusing:

There seems to be some confusion about GeoRSS mime type - application/xml, or text/xml, or application/rss+xml, or even application/georss+xml show up in a brief google search? In the end I used a Virtual Earth api viewer to consume the GeoRSS results, which isn’t exactly known for caring about header content anyway. I worked for awhile trying to get the GeoRSS acceptable to OpenLayers.Layer.GeoRSS but never succeeded. It easily accepted static .xml end points, but I never was able to get a dynamic servlet endpoint to work. I probably didn’t find the correct mime type.

There is no GeoRSS media type. Atom's is application/atom+xml. RSS 2.0 uses application/rss+xml, and RSS 1.0 might use application/rdf+xml. You should use one of those depending on whether your GeoRSS elements appear in an Atom, RSS 2.0, or RSS 1.0 feed. Why? Atom has its own processing model. When you request a feed, the server writes application/atom+xml in the response's content-type header to inform you that you'd better use the Atom content processing rules if you want to make sense of the data.

Is there anything to be gained by having a special GeoRSS media type that overrode the Atom or RSS media types?


Re: GeoRSS media type?

Author: Peter Rushforth

Given that "GeoRSS" is like Kleenex, ie there are lots of formats which can be called that, would it not be helpful to have something like application/atom+gml, application/atom+geo etc to distinguish those brands?

Re: GeoRSS media type?

Author: Sean

Nevermind that +gml and +geo media types aren't standardized -- what would they mean to a processor that's not already said by one of the standard XML media types like text/xml, application/xml, and application/*+xml?

After a read of http://tools.ietf.org/html/rfc3023, I'm inclined to say that GML itself should be using something like application/gml+xml. GML is based on XML, yes, but rather opaque when displayed as text.

REST vs SOAP at ESRI DevSummit

Yesterday, I saw a lot of links to David Chappell's ESRI Developers Summit keynote. I tried and failed to stick with the video, but I did read through the slides. They're largely good, but there are some errors. Before his presentation becomes canon in the ESRI user community, I'd like folks to consider:

The watering down of REST constraints on slide 14 into optional principles that you apply or not "whenever possible", and disregard for the hypertext constraint. In fact, there's no mention of the hypertext constraint at all in the slides. Does Chappell not understand it? Not believe in it? It's impossible to say from the slides; maybe he went into it more, live. At any rate, substitute "when you want certain derived properties" for "whenever possible" and you get closer to the essence of REST. If you want the property of cacheability, then you add the uniform interface and stateless communication constraints. If you want the property of loose coupling, you add the hypertext constraint.

On slide 18, Chappell fails to mention an option that's familiar to all of us even if we're not entirely aware of it: code on demand. The client library provided by the service can be downloaded by clients at the time the service is accessed. On demand. You're probably already using OpenLayers (or something like it) in a way that's close to an on-demand library for negotiating with a service. Chappell's second option on that slide, while useful enough, has very little to do with REST.

On slide 20, Chappell confuses REST with the half-baked "REST APIs" that don't use service descriptions or any sort of hypertext representations. If you stick to standardized representation formats, there's no more dependence on written documentation than in the SOAP case. Developers of SOAP tools had to read a pile of specs. Developers of RESTful HTTP tools will have to read a pile of specs too.

Finally, there's a lot of slides spent trying to explain how to choose between SOAP and REST. Ultimately, it comes down to this: if you want your system to have the properties that derive from REST constraints, you choose the REST style. If you want to integrate in the Web, you choose REST. If you want a system that tries to abstract away the network, letting your developers program with remote services as though they were local objects, you choose SOAP. Pick your architectural style. I don't think it has to be any more complicated than that.

Update (2009-04-09): Pete (below) has shamed me into watching the video all the way through. I'm glad I did. I knew this stuff already, but it's a very good presentation for the ESRI user/developer. Unfortunately, despite Pete's claims, my quibbles with his slides stand. Chappell didn't cover the hypertext constraint (though he did an excellent overview of the wins from uniform interface and universal identifiers). An adequate treatment would have consumed too much time, granted, but some mention would be nice. The hypertext constraint has indeed been a matter of debate, but the wins are becoming clear. Without links in content, well-defined formats, and code on demand, developers are indeed dependent on service provider libraries, but this state of affairs isn't a fault of the REST style. Still, to use SOAP in your enterprise behind the firewall, to use the REST style outside on the Web is the obvious conclusion. Good answers to questions, too, including comments about AtomPub and the issue of snowflake APIs.


Re: REST vs SOAP at ESRI DevSummit

Author: Pete

I like how you are trolling a presentation you didn't even bother to watch.

As you say yourself: "It's impossible to say from the slides".

If you had watched the presentation, you would know that he deals with most of your arguments.

Re: REST vs SOAP at ESRI DevSummit

Author: Sean

I found it hard to watch. Too much emoting. Not that I'm better (surely not), but it was not to my taste. How did he deal with these arguments? Or at what times? Maybe I could fast-forward to them.

Keytree 0.2.1

Keytree provides some utilities for manipulating KML using the ElementTree API. I've added factories for KML placemark and geometry elements, input being geojson or Shapely objects:

>>> from geojson import Feature
>>> f = Feature('1',
...             geometry={
...                 'type': 'Point',
...                 'coordinates': (-122.364383, 37.824663999999999)
...                 },
...             title='Feature 1',
...             summary='The first feature',
...             content='Blah, blah, blah.'
...             )

Any object that provides the Python geo-feature interface will do. Next, you need a KML document as context for a new placemark:

>>> data = """
... <kml xmlns="http://www.opengis.net/kml/2.2">
...   <Document>
...   </Document>
... </kml>
... """
>>> from xml.etree import ElementTree
>>> kml = ElementTree.fromstring(data)

Make a placemark element using the element factory:

>>> elem = keytree.element(kml, f)
>>> import pprint
>>> pprint.pprint((elem.tag, elem.text, list(elem)))
 [<Element {http://www.opengis.net/kml/2.2}name at ...>,
  <Element {http://www.opengis.net/kml/2.2}Snippet at ...>,
  <Element {http://www.opengis.net/kml/2.2}description at ...>,
  <Element {http://www.opengis.net/kml/2.2}Point at ...>])
>>> pprint.pprint(list((e.tag, e.text, list(e)) for e in elem))
[('{http://www.opengis.net/kml/2.2}name', 'Feature 1', []),
 ('{http://www.opengis.net/kml/2.2}Snippet', 'The first feature', []),
 ('{http://www.opengis.net/kml/2.2}description', 'Blah, blah, blah.', []),
  [<{http://www.opengis.net/kml/2.2}Element coordinates at ...>])]

This element could be appended to the Document element, or you could use the subelement factory:

>>> elem = keytree.subelement(kml[0], f)

More at http://pypi.python.org/pypi/keytree/.

Now, I'm trying to decide if something similar would be useful for Atom with GeoRSS.

Sensible observation services, part 2

Another good question from the OGC REST discussion:

In essence, within REST, how do I ask for a collection of measurements between any time1 and time2? This is simple with SOA, but seems to require predefined granularization by the service in ROA or perhaps uses an adaptation of SOA-like specifications as Pat has suggested in the past.

SOA didn't say how to spatially or temporally slice data. SOA said "have services". Services with well-defined interfaces. It's up to communities to define those interfaces. It's the same for RESTful architectures. REST just says use universal identifiers for entities, use uniform methods, and use hypertext as the engine of application state. The entities and the hypertext formats are up to the community.

How would you ask a sensible observation service for measurements between two times? This device is blogging away, adding observations to the head of a feed every N microseconds. If you're interested in historical data – and everything becomes historical quickly if it's sampling very frequently – the service should allow you to slice temporally, returning feeds that begin and end at or near specified times. The Atom format spec doesn't specify how to do this. It was developed in the tradition of layering specifications and recommendations. Search is to be layered on top of Atom like feed paging (RFC 5005) is layered on top of Atom. One possible way to do this is with XHTML forms.

Consider a sensible observation service identified by the URI http://example.com/sosa. Dereferencing that gets you an Atom feed of N entries, ideally containing links to the next "page" and a link to a search form:

<feed xmlns="http://www.w3.org/2005/Atom">
  <link rel="search"

The "search" relation isn't standardized yet, but something like it will be. The exact URI of the search form isn't important at all, only that it can be found via a link. Its representation would be something like this:

<html xmlns="http://www.w3.org/1999/xhtml"
    <form id="sos-search:form"
      <input id="sos-search:start" type="text" name="start"/>
      <label for="sos-search:start">Minimum bounding timestamp, RFC 3339</label>
      <input id="sos-search:stop" type="text" name="stop"/>
      <label for="sos-search:stop">Maximum bounding timestamp, RFC 3339</label>
      <input type="submit" name="search"/>

The "start" and "stop" inputs are borrowed from Python's sequence slicing syntax. A community that standardized on form element ids could have a self-describing search interface, one that was also helpful to human users (at least) outside that community. Any "sos-search" aware client could recognize from this form, by parsing elements with "sos-search" ids, that URIs like http://example.com/sosa/search?start=2003-12-13T18:30:02Z&stop=2003-12-13T19:30:02Z identify Atom feeds of temporally sliced data.

I'm not sure XHTML forms are the best solution, but it's illustrative of one way to do RESTful search interfaces.

REST in reality

How to RESTfully change the state of power lines and poles? From a thread on the OGC's mass-market-geo list (archives available only to subscribers, for shame):

Actually, I am really talking about poles and wires. Specifically power poles and lines. My example is directly derived from real life. A local electric system has tons of old wood poles that it wants to replace with concrete poles. It is doing this by installing the concrete pole 1m behind the existing pole and then restringing the power lines.

So, the client has a feature type POLES (Oracle table) with all the poles and a feature type LINES (Oracle table) with all the power lines. Not the actual named but you get the idea.

For each pole that is replaced, they update the existing pole record since the new pole will have the same pole id as the pole it replaces but the new pole will be in a slight different position.

Using the current WFS specification, I can make these changes in a single request. I simply POST the following the the URL of the server:


The question of how to do this in a single transaction is a big juicy red herring that has list subscribers rather distracted. The process of disconnecting and taking down wires takes some time, and presumably has to be done before you can take down the pole they connect to. Certainly the new pole can't be erected before the old one is torn down, and only after that happens can new lines be brought in to connect to the new pole. This is not an atomic operation. Bad weather, equipment failure, injury, or any mishap might interrupt the process and leave it an unfinished state for some time. Treating it as atomic in your information system seems more hopeful than realistic, and perhaps even harmful. Once the lines and poles come down, you can't be giving anyone the false impression that they're still up.

Let's say you've modeled your power system RESTfully. You've got pole resources like http://example.com/power/poles/1. You've line resources like http://example.com/power/lines/1 and http://example.com/power/lines/2. These resources aren't poles and lines by virtue of the strings in the URI, of course, that's just a convenient design. Lines 1 and 2 connect at pole 1. A utility crew keeps your information system up to date as they work like this:

This isn't finance. Sometimes non-transactional is more honest.


Re: REST in reality

Author: Gary Sherman

In reality the new poles will all be installed (1 m behind existing), then the lines will be moved all at once.

Re: REST in reality

Author: Sean

I was imaging the lines would be upgraded too. If not, even easier: PUT new state (new location, new material attributes, etc) to http://example.com/power/poles/1 when the lines are hung on the new pole. Don't change the state of the line at all.

Re: REST in reality

Author: Jason Birch

Keeping the same id feels a bit contrived. Normally, you'd want to retain the old ID in a "removed" state for asset managment reporting on upgrade history, service lifetime, etc, etc. So, in this case, I think you'd PUT the new state (retired) to the old pole, and POST a new pole.

Typically a line is connected to many poles, and a pole can have many lines strung on it. Assuming this relationship was modeled by GETing lists of hyperlinks from the following URIs (don't know if this is "correct"):

GET http://example.com/power/poles/1/lines

GET http://example.com/power/lines/1/poles

How would you move the lines from one pole to another?

DELETE http://example.com/power/poles/1/lines/1

POST http://example.com/power/poles/2/lines/1

That seems a bit odd to me, because you wouldn't

GET http://example.com/power/poles/2/lines/1

Separate "service"?

DELETE http://example.com/power/pole/1/lines {content: 1,2,5,61}

POST http://example.com/power/pole/2/lines {content: 1,2,5,61}

Re: REST in reality

Author: Cedric Moullet

Jason's use case is quite interesting. I was thinking to another one: usually, when line modification occurs, a mailing is done in order to inform the customers of the outage (typical question: provide me the list of all customers affected by the deletion of this line).

What would be the correct URI for this kind of question ?

Re: REST in reality

Author: Sean

I'm not sure what you mean by URI for the question. If you designed a system such that the customers for each line (or segment of the power grid? I'm getting out of my depth, here) were enumerated in a resource like http://example.com/power/lines/1/customers, you might have a form to which you could POST a message like http://example.com/power/lines/1/customers/notice (delivered by email and snail mail).