REST Can't Handle Rasters and Coverages?

That's what Raj Singh seems to be saying here:

For example, most of the REST material I've read treats images as indivisible, binary resources. This won't work in the geo field, where images are in the gigabyte and terabyte range and you've got to design services that provide access to portions of those images. However, aside from imagery I don't see much of a problem with using a resource-oriented approach to information access. In fact, in the sensor web arena I think a REST approach makes a ton of sense.

Raj, you're reading the wrong material. No REST architect accepts that the image "file" is the only possible resource. If I had a global sea-surface temperature coverage resource at /sst/current/, I could easily treat individual pixels as resources of their own: /sst/current/{pixel},{line} is just one possible URL form. I could even interpolate to fractional positions within the coverage, or do lat/long: /sst/current/{latitude},{longitude}. Modes of an empirical orthogonal function decomposition (hot topic when I was in school) could be modeled as resources: /sst/current/eof/{n}. Simple temperature slices are another possible set of algorithmic resources: /sst/current/?tmin=28 would be the warm waters that spawn tropical storms. All of the above are RESTful and no client needs to know whether I have a single file or multiple databases backing the resources. Furthermore, a resource-oriented approach scales better than an endpoint-based protocol like WCS. As the number of resources grows, you simply spread them over more servers like Google has done with its Earth and Maps tiles.

There may be geospatial problems that REST can't tackle, but access to arbitrary regions of a coverage is not one of them.

I'm also a bit disappointed to read this:

After all this [research], I think I know what's going on, but I don't think there's any one clear explanation (despite some nice pieces of the puzzle here [1] and here [2]) available, and there has certainly been little effort to analyze the REST architecture in relation to geographic information systems theory, so that's what I'll try to do now.

Of all the great posts and email threads which break GIS web services down in RESTful terms, he picks that muddled Directions article and another that only quotes one of my emails? This trivializes the geospatial community's conversation about REST, one that has been enlightening and amusing us for about a year now. The fact that there will be three REST related presentations at FOSS4G hints at the maturity of the topic.

Comments

Re: REST Can't Handle Rasters and Coverages?

Author: Frank Steggink

Sean, although I appreciate, almost even admire, the work you're doing on REST, I would like to say something. Yes, you don't know me, and yes, you might not like it what I'm going to say, but like you, I'm just using my right to speak freely. It sounds like you're on a crusade again! Can you please tone down the posts? This way it sounds like you're far in the "glass half empty" camp. I, on the other hand, find it very encouraging that Raj is posting on REST. That indicates that not only the OGC members (like Ron Lake, etc.) follow REST with much interest, but even the OGC itself. So, all the nudging you did (and do!) shows off its value! Yes, Raj might have better linked to other posts than the ones he did. Yes, he might not understand REST as much as you do. But guess what, that is true for 99% of the people. Everybody makes mistakes, doesn't understand things the right way immediately. (I'm also struggling with understanding REST, and its implications on the geo web, and it is not that easy.) The reason: we're coming from a different mindset (freely speaking on behalf of Raj as well), and change is always hard. You are one of the guys leading the pack. But all that endless one-sided bickering costs a lot of energy, which can be spent much better. Making nice demo's for instance, writing some good intro's (especially on the geospatial field), thinking how you would implement WFS in a RESTful manner. And above all, use your energy to use the momentum the OGC is showing with regard to REST. I would rather prefer to read and even engage in constructive dialogs and interesting discussions with you, with Raj, with others. So, please get over whatever raises your hackles at the OGC, and go into a more cooperative mo(o)d(e). Spicey food tastes great, but too much just burns your mouth. Frank

Re: REST Can't Handle Rasters and Coverages?

Author: Sean

When our industry leaders speak from authority and get it wrong, I'm going to say so. I try to keep it above the personal level (call me on this if I slip), but I'm not going to sugar-coat it. Sorry. At any rate, it would be a big mistake to tune out REST just because of one shrill voice. I don't think anybody at the OGC will. Now, back on topic.

Re: REST Can't Handle Rasters and Coverages?

Author: Jason Birch

I can't really say that I know Sean either, and I certainly don't always agree with what he posts or his tone - I'm more conciliatory myself. However, I see immense value in a strong critical voice, especially considering how disruptive REST is to traditional GIS. If the voices of folks (like Sean) who have a clear vision of how things like REST and AtomPub can work in our field are ignored, then we are doomed to a set of officially-sanctioned sub-optimal standards. Which will in turn be ignored as mainstream geospatial rolls over top of traditional GIS without even feeling the bump. Jason

Re: REST Can't Handle Rasters and Coverages?

Author: mpg

Perhaps like Raj, I'm still trying to get my head around the idea of specifying arbitrary BBOXes for image requests in REST format: seems utterly uncacheable, and thus against one of the REST goals. That aside, though, I see no problem with using REST for images. Scales/resolutions, reprojections, band slicing, etc, all seem to work perfectly nicely in the whiteboard-level thoughts I've been having. I'd be perfectly happy to see all coverage data presented as a REST-service-of-tiles instead of the montage of monolithic files we deal with today. The backend storage might be file-based or indeed a number of other options (including wavelets), but key point is that the lives of clients just gets easier as you mandate simple tile schemes. -mpg (totally not speaking for my employer...)

Re: REST Can't Handle Rasters and Coverages?

Author: Raj

If a client--such as a Web browser--wanted a 500x500 pixel image, would this require 250,000 URL requests? Or would image requests be more like queries, and look very much like WMS does now, but every pixel would have a canonical URL if you really needed it?

Re: REST Can't Handle Rasters and Coverages?

Author: Sean

I'm just showing off the freedom we have to model coverage resources. The resources that you actually expose should be the ones that make the most sense for your application.

OGC, GeoDRM, and Me

My hackles rise every time I read that GeoDRM is one of the OGC's advanced technologies. I simply do not find it easy to look past it to the other more sensible working groups and standards. It's as though I'm in a strange city, getting directions from a helpful man. He's sharply dressed, Wall Street Journal tucked under his arm, speaking authoritatively about the neighborhood ... and then I notice that he's surreptitiously carrying a smoldering crack-pipe. When I ask, he's "just holding it for some other guy". Right.

Or maybe I'm just the cook (near the end) in this classic ;)

Shorter Sebastian Good

Let's you and him fight. Nevertheless, I always enjoy Sebastian's writing.

Comments

Re: Shorter Sebastian Good

Author: Sebastian Good

I suppose some of my, er, florid prose, makes it sound like it's an either/or Galdos/XML versus PROJ4/REST thing sometimes, like the rest of the overhyped SOAP/REST fracas. But really the point is there are lots of spatial reference formats flying around all the time, and a global service that admits that seems more useful to me than a global service that provides perfect GML to an imaginary projection engine. Even nicer would be for the Illuminatii (OGC, EPSG, ISO) (does Illuminati have a plural...?) to hang out at some of the seedier open source bars and build an infrastructure that everyone can use... so the GIS department can go the way of the VARCHAR2 department and it's easy for programmers to get things right geodetically. Seriously, how expensive could it possibly be to build a new projection engine and give it away to the world?

Re: Shorter Sebastian Good

Author: Sean

VARCHAR department! That still cracks me up.

Stop Using Mapscript: Finally

MapServer 5.0 has some nice new features. Christopher Schmidt has been blogging the hell out of one. Dynamically allocated arrays (culmination of my work on a container API -- msInsertLayer and friends) is a big one that doesn't get a lot of notice. I think my favorite is this: now you can pretty much stop using mapscript.

Steve Lime did the work. I did the nagging, testing, and the final mapscript touch. Here's a Python usage excerpt from the tests:

>>> import mapscript
>>> mo = mapscript.fromstring("""MAP NAME "test" END""")
>>> mo # doctest: +ELLIPSIS
<mapscript.mapObj; proxy of C mapObj instance at ...>
>>> mo.name
'test'

I've explained the motivation previously. You can now transform the problem of writing intricate, error-prone mapscript code into a more tractable template interpolation problem.

KML Module: Atom

Since my employer isn't a member of the OGC, I'm stuck with blogging my ideas about KML in the hopes that they strike a chord with some insider and thereby tunnel through into the smoke-filled rooms where our industry standards are made.

Andrew Turner wrote the other day about KML modules and listed five: core, styling, 3d, metadata, and services. I'd like to see the core divided one more time. All the KML elements that overlap with Atom need to be factored into a module of their own. This means name, description, atom:author, and atom:link. An Atom module would help developers implement Atompub clients and services by marking a clear boundary between media and metadata elements.

Comments

Re: KML Module: Atom

Author: Jason Birch

Well, there's always #kml ... but you might end up talking to yourself since it hasn't been advertised :)

OWSLib 0.2.1

Kai Lautaportti just released OWSLib 0.2.1 [Cheeseshop]. It now works with no external dependencies on Python 2.5. Get it from the Cheeseshop:

$ easy_install OWSLib

or our Subversion repository:

$ svn co http://svn.gispython.org/svn/gispy/OWSLib/tags/rel-0.2.1 OWSLib

Comments

Re: OWSLib 0.2.1

Author: Kurt

Cool. Updated in fink.

Atompub and KML Demo

Update (2009-05-20): Hammock has been retired, replaced since December 2007 by Knowhere.

I have repurposed my Hammock application into a demonstration of the Atompub, KML, and Google Earth integration. Now, from an Atom perspective there is a service document and one collection at:

http://sgillies.net/hammock/index.atom

http://sgillies.net/hammock/places/

Subscribe to that second URL, it's an Atom feed. Any newsreader should be able to use it to monitor changes to the collection. Next, start Google Earth and in your temporary places container create a new network link to this URL:

http://sgillies.net/hammock/places.kml

The same (very simplistic) data model supports both the Atom feed and the KML document.

Ready to publish data to the Web, Atompub-style? All you need is a terminal, curl or another moderately sophisticated HTTP client, and a little make-believe. Imagine that Google Earth or another future geo-browser is doing all the POST and PUT dirty work for you.

In the temporary places container, create a new point placemark anywhere. Give it a name and a plain text description. Save it to your filesystem as hammock.kml. To create a new, publicly-shared place within the collection based on your new placemark you send the contents of that file in the body of a request to the collection URI. I showed that URI above, but the sure place to find it is in the atom:link element of the KML file. View the source on that file and verify that there's a link to http://sgillies.net/hammock/places/. Now send a request like:

POST /hammock/places/ HTTP/1.1
Host: sgillies.net
Content-type: application/vnd.google-earth.kml+xml

<kml>
...
</kml>

You can use curl to do it like this:

$ cat hammock.kml | curl -d @- --header
Content-type:application/vnd.google-earth.kml+xml
http://sgillies.net/hammock/places/

It's very important to specify the proper content type. If you've typed carefully, you will get response with a Location header containing the URL of the newly created resource. If you refresh the Atom feed and network link, you'll see the new resource reflected in each. You've seen this before in other apps like FeatureServer. What's new is that the KML document itself carries a link to the collection URI, the factory for new placemark resources. That's Atompub in action.

Furthermore, any placemark resource represented in http://sgillies.net/hammock/places.kml can be edited via its "edit" or "edit-media" (they are equivalent in this demo) atom:links using the HTTP PUT method. View the source to find the edit link for your newly created resource, and copy it (remember, one day Google Earth should do all this for you). Now, move the placemark in Google Earth or create a new one, and save it to the filesystem. Finally, PUT the saved placemark to the edit URI you copied using curl like:

$ cat edit.kml | curl -X PUT -d @- --header
Content-type:application/vnd.google-earth.kml+xml
http://sgillies.net/hammock/places/3

In this case, I've edited the third place. Note the use of the PUT method. If you typed carefully, the changes you made will be reflected in your newsreader and in the Google Earth network link.

It almost seems too simple to work: serve KML to users that provides the very links by which users can add to, or modify, the web resources represented in the document. I'd be surprised if there is a tidier solution. I hope this simple demo can convince a few people to take a closer look at using Atompub for geospatial services.

Comments

Re: Atompub and KML Demo

Author: Christopher Schmidt

The KML doesn't use the Atom namespace at all, so far as I can tell. Is that intentional? Seems odd to me...

Re: Atompub and KML Demo

Author: Sean

Fixed.

Re: Atompub and KML Demo

Author: Christopher Schmidt

So, I'm confused on one thing: you say '... can be edited via its "edit" or "edit-media" (they are equivalent in this demo) atom:links ...' What is the difference? It seems like in general, the edit-media URLs allow you to edit the different representations of the same resource. For FeatureServer, it seems like this means that there could (theoretically) be edit-media URLs for each of the supported input services (Atom, KML, JSON, etc.). Is that right? Is there a reason these URLs have to be different, or is differentiation between 'Content-Type' on the PUT, as defined by the 'type' attribute on the <atom:link> sufficient?

Re: Atompub and KML Demo

Author: Christopher Schmidt

For the record, I've added a 'atom:link rel="self"' and rel='edit-media' to featureserver's KML output. I suppose I should add the same to the atom feeds? Is there any atom-pub supporting client I could test out?

Re: Atompub and KML Demo

Author: Sean

Christopher, I think that anything we made would be the first truly geospatial Atompub client. I can see myself contributing to one to for OpenLayers. Pete Lacey's appfs is a really intriguing client that might support geo use cases. Seems like a lightweight WebDAV. The canonical example for media resources and edit-media links is an image collection (ala Flickr). Posting a new image to the collection creates both an image (media) resource and a media link resource. That media link resource contains metadata for the image resource. The resources in the canonical example are completely normalized; there's no overlap between the media and metadata (media link). You update the image by a PUT to the collection entry's "edit-media" URI, and update the metadata by a PUT to the entry's "edit" URI. What's the rationale for explicit "edit-media" links? Here's one case: completely separate backends for images and metadata/annotations. On the other hand, a KML placemark and Atom entry overlap significantly, and I've limited the scope of my demo to the point where the overlap is complete. You can PUT KML to the "edit-media" URI, or an Atom entry to the "edit" URI, but the effect is the same. That's just a detail of my demo's implementation, not a recommendation. Andrew Turner's work on factoring KML into modules could tie into Atompub very well. An Atompub + KML client could PUT the metadata module's elements (as Atom) to an "edit" URI, and PUT everything else (as KML) to the "edit-media" URI.

The Shapely Alchemist

Following the example at byCycle.org I've figured out how to use Shapely geometries with SQLAlchemy and PostGIS. Here's the custom type:

from sqlalchemy import types
from shapely.wkb import loads

class Geometry(types.TypeEngine):

    def __init__(self, srid, geom_type, dims=2):
        super(Geometry, self).__init__()
        self.srid = srid
        self.geom_type = geom_type
        self.dims = dims

    def get_col_spec(self):
        return 'GEOMETRY()'

    def convert_bind_param(self, value, engine):
        if value is None:
            return None
        else:
            return "SRID=%s;%s" \
            % (self.srid, value.wkb.encode('hex'))

    def convert_result_value(self, value, engine):
        if value is None:
            return None
        else:
            return loads(value.decode('hex'))

Example:

>>> db = create_engine("postgres://localhost/the_db")
>>> metadata = MetaData(db)
>>> places = Table("places", metadata,
...     Column("the_geom", Geometry(4326, "POINT"))
...     )
>>> result = places.select().execute()
>>> row = result.fetchone()
>>> row
(<shapely.geometry.point.Point object at 0xb771490c>,)
>>> row.keys()
['the_geom']
>>> row.the_geom
<shapely.geometry.point.Point object at 0xb771482c>
>>> row.the_geom.wkt
'POINT (-106.0000000000000000 40.0000000000000000)'
>>> row.the_geom.x
-106.0
>>> row.the_geom.y
40.0