2009 (old posts, page 3)

OpenLayers constrained by hypertext

I go on at times about REST and its hypertext constraint. I'm sure some people wonder if I ever actually apply these principles or just like to argue about them. Much of my work involves Plone, which is a poor environment for doing things in a RESTful manner, but I try to find good patterns where I can. One of them involves OpenLayers, puts the REST hypertext constraint in a familiar context, and isn't something that I've seen in the OpenLayers examples or gallery.

The site I'm working on has resources about some places of the ancient world, such as the the place known as Aphrodisias. The URI http://pleiades.stoa.org/places/638753 leads to an HTML representation. There are also Atom, JSON, and KML representations, and these are linked from the HTML page as shown below:

<link rel="alternate"
    href="http://pleiades.stoa.org/places/638753#this" />
<link rel="alternate" type="application/atom+xml"
    href="http://pleiades.stoa.org/places/638753/atom" />
<link rel="alternate" type="application/json"
    href="http://pleiades.stoa.org/places/638753/@@json?sm=1" />
<link rel="alternate"
    type="application/vnd.google-earth.kml+xml"
    href="http://pleiades.stoa.org/places/638753/kml" />

The HTML page has an OpenLayers map of ancient world features contained in this place. You'll see one, the feature digitized from the Barrington Atlas map sheet, unless you're a authenticated user that can view Tom Elliot's unpublished features. Previous incarnations of these map pages called on dynamically generated scripts that contained the data, but I'm preferring now to get data from the alternate application/json representation of the place resource. It's trivial, more loosely coupling, and more scalable (at least in my Plone setup) to separate code and data, parsing the JSON link href out of the document and using it in an OpenLayers "GML" layer:

function getJSON() {
    var documentNode = document;
    var linkNode = documentNode.evaluate(
                    '//link[@rel="alternate" and @type="application/json"]',
                    documentNode,
                    null,
                    XPathResult.FIRST_ORDERED_NODE_TYPE,
                    null
                    ).singleNodeValue;
    var jsonURI = linkNode.getAttribute("href");
    return jsonURI;
}

function initMap() {
    var jsonURI = getJSON();
    vectors = new OpenLayers.Layer.GML(
                "Place",
                jsonURI,
                {format: OpenLayers.Format.GeoJSON}
                );

All data the OpenLayers code needs to render a map of the place is now discoverable through HTML links without having to go out of band.

Comments

Re: OpenLayers constrained by hypertext

Author: Vish

Hi Sean,

So, does that mean that you are pulling down the same data from the server to the browser twice? First in HTML and then JSON?

Thank You,

Vish

Re: OpenLayers constrained by hypertext

Author: Sean

The same data twice? No. There's a small overlap between the text/html and application/json representations, yes. A title string, an id, a URL. 100 bytes or so. There's no geometry or coordinates in the HTML page, and so the fractional overlap diminishes as the number of line and polygon features increases.

Re: OpenLayers constrained by hypertext

Author: Vish

Hi Sean,

I was just curious as to what your HTML representation looked like. Also, not to pick on things here, i am just thinking out loud...

1) Does your HTML representation not contain any attribute information about the feature or are u omitting it along with the geometry?

2) What are your thoughts on omitting the geometry for the feature in its HTML representation? Should it be in there as SVG/VML? Is it alright to have only subsets of the feature info in the different format? Or should they all have the same info?

3) If your HTML only contains the 100 bytes of info you mention... then why have a HTML representation at all? REST doesn't say anything about HTTP/HTML. Is it only for the web crawlers?

Thank You,

Vish

Re: OpenLayers constrained by hypertext

Author: Sean

Vish, I meant 100 bytes of overlap between the HTML and JSON representations. How much the representations overlap is totally up to the application (and me). Currently, the JSON representation is primarily in the service of the HTML pages. If users wanted richer JSON data, we'd consider providing it.

Spring is officially here

The warm weather and blossoms have been hinting at it, but now it's official.

http://farm7.staticflickr.com/6094/7016282337_05a6ed4bdd_z_d.jpg

This turkey vulture arrived at 920 West Mountain Avenue between 2 and 5 PM today. The first one came also on the 25th of March in 2008 and 2009. Before that I wasn't keeping close track, but I'm sure it was within a couple days of the 25th. I was in France in 2010 and then in London last year and missed the big event. Is the vernal equinox the signal for them to head north from at least 4 days away?

Good because it's good

Here's a quote from Tim Bray that's relevant to the WMTS discussion:

REST isn’t good because it’s REST, it’s good because it’s good.

I get the impression that the OGC is keen to have a REST standard only because it fears missing out on the booming interest in REST and wants to keep in control of geospatial protocols. It perceives REST as a ripe emerging market rather than a distinct architectural style. This is reflected in the WMTS "KVP, SOAP, or REST" offerings, which seem like "Bud Light, Bud Dry, or Bud Ice" to me. REST isn't good because it's trendy, it's good because it's good. Loose coupling. Scalability. Evolvability. Serendipitous reuse. A real alternative to RPC.

Commenting on OGC WMTS

I finally caught wind of the OGC's request for comments on the Candidate Web Map Tiling Service Standard via Geoff Zeiss with 11 days remaining to comment. Not for lack of OGC press releases, it's just that I don't follow that particular media very closely. I have sent in a "public comment" advising the authors on how to better follow the REST style. To be honest, I'd rather the OGC stayed away from REST, but if it won't, I'll insist it's done properly and doesn't misinform mainstream GIS developers. I'll even try to help as much as the OGC's closed process will allow. Supposedly, the OGC's comment list is open to the public. I've registered, but it's coming up on 24 hours and I have received neither confirmation of receipt of my comment nor have I been subscribed to the list. Frankly, the public comment process isn't very smooth. Does the OGC actually expect comments? Will there be any others? I'm curious to find out. Meanwhile, I've started a discussion on geo-web-rest by posting my comment verbatim, and will write a bit more about the candidate standard here.

Update (2009-03-21): I'm subscribed to the list, finally, thanks to intervention by the manager. I'm not sure whether the normal subscription process is working or not.

Comments

Re: Commenting on OGC WMTS

Author: Allan Doyle

My comments on the geo-web-rest are a bit of a shot in the dark. I'm not clicking "Accept" on the OGC click-through license. I've tried many times to get OGC to stop using them but it's a bit like tilting at windmills.

Implications of WMTS for S3 tiles

Check out this interesting post by Randy George concerning S3 map tiles for DeepEarth:

The project also includes an example showing how to set up a local tile set. The example uses 256×256 tiles but not in the OSGeo TMS directory structure. Here is an example using this DeepEarth local example TileProvider DeepEarth BlueMarble. You can see the tiles spin in from a local directory store on the server. The resolution is not all that great, but a full resolution BlueMarble isn’t that hard to get from BitTorrent. The alternative selection “Blue Marble Web” tiles are full 500m resolution hosted on Amazon S3 courtesy of ModestMaps.org. The Amazon S3 bucket is a flat structure, in other words the buckets don’t have an internal directory tree, which is why the tiles are not stored in a TMS directory tree.

The DeepEarth local TileProvider was easily adapted to suit the TMS directory so I could then directly pull in my El Paso tiles and show them with a DeepEarth interface. However, if I wished to take advantage of the high availability low latency S3 storage, I would need to flatten the tile tree. In S3 subdirectory trees are hiddeen inside buckets as file names. In the S3 case the tile names include the zoom level as a prefix: http://s3.amazonaws.com/com.modestmaps.bluemarble/5-r8-c24.jpg The 5 on the 5-r8-c24 nomenclature is the zoom level while row is 8 and column 24 at that level. TMS would encode this in a tree like this: ../bluemarble/5/24/8.jpg. The zoom level= subdirectory, row = subdirectory name, and column = name of tile. The beauty of TileProvider class in DeepEarth is that minor modifications can adapt a new TileProvider class to either of these encoding approaches.

Performance and reliability is a lot nicer on an Amazon S3 delivery, especially with heavy use. Once in S3 a map could also be promoted to a CloudFront edge cache without much difficulty. I imagine that would only make sense for the upper heavily used zoom levels, say 1-5 for BlueMarble. Once down in the level 6 part of the pyramid the number of tiles start escalating dramatically and repetitive hit rates are less likely.

The OGC's proposed Web Map Tiling Service standard's strict tile resource URL specification (in effect, just another form of RPC) obstructs you from using S3 tiles directly. That's a problem.

Update (2009-03-20): It's pointed out in comments that I'm wrong about the obstruction, and that you can in fact simulate the proposed WMTS tile resource hierarchy in S3. I'm not wrong about the RPC-ness of the candidate spec, something that can yet be fixed.

Comments

Re: Implications of WMTS for S3 tiles

Author: Richard Marsden

I've also thought of using S3/Cloudfront for map tiles, and it has been asked about on the forums of my site.

It would be a relatively simple and cost effective way of creating a tile server that didn't involve the hassles of building & installing something like MapServer.

System Message: WARNING/2 (<string>, line 30)

Explicit markup ends without a blank line; unexpected unindent.

</p><p> The proposed URL standard could be a problem, but would this simply cause a pragmatic variation of the standard to develop on its own?</p>

Re: Implications of WMTS for S3 tiles

Author: Michal Migurski

I'm surprised they're not providing for arbitrary positions of the x/y/z bits. The S3 URL given as an example above looks the way it does because I didn't want to build a directory tree in the filesystem prior to uploading - there's no reason it can't look like the proposed standard if I understand it correctly, but the "(z)-r(y)-c(x).jpg" has been convenient: http://oakland-1877.s3.amazonaws.com/14-r6331-c2627.jpg, http://oakland-1967.s3.amazonaws.com/14-r6330-c2628.jpg, etc. It's worth remembering that the "/" is a perfectly valid character in S3 object IDs.

We've also used a more VEarth-like quadtree style: http://road.tiles.map.london2012.com/a/03/13/13/13/11/13/13/3.jpg

Re: Implications of WMTS for S3 tiles

Author: Sean

I didn't know about the slashes. That's interesting, but possibly misleading because buckets can't contain other buckets, and objects can't contain other objects. The object foo/bar/x isn't really contained in foo/bar.

The suggestion I've made to the WMTS authors would make it easier to link to your existing "(z)-r(y)-c(x).jpg" tiles.

Re: Implications of WMTS for S3 tiles

Author: Michal Migurski

Thanks for making that suggestion! As far as S3 is concerned, the semantics of the slashes is actually quite fluid. The fact that they give the appearance of containment is completely separate from the implementation behind the scenes. S3 provides ways to browse buckets with a client-provided delimiter, so you can pretend there's a full directory structure there. The more I work with S3, the more impressed I am with their design decisions.

Re: Implications of WMTS for S3 tiles

Author: Joan Maso

The use of a flat structure like

.../5-r8-c24.jpg

in a prerendered very large and detailed layer is not a good idea.

You could end with thousands of files in the same directory. Common

operating systems do not deal well with so many files a in single

directory.

Splitting scales and rows in different directories helps in these situations.

Re: Implications of WMTS for S3 tiles

Author: Michal Migurski

Joan: on S3, objects aren't files, necessarily - I don't actually know what they are, it doesn't seem to be a relevant detail. They aren't files in TileCache's ".../5/24/8.jpg" scheme, either. URL structure doesn't need to bear any resemblance to underlying implementation. In particular, I'm loving Sean's term "hierarchical URL fetishism" in the linked discussion: http://groups.google.com/group/geo-web-rest/browse_thread/thread/fa0633b1dd009c02?hl=en

Re: Implications of WMTS for S3 tiles

Author: Sean

Like Michal says: how tiles are stored is an implementation detail. It's completely orthogonal to REST concerns.

Sensible observation services

I cry like the man in the classic anti-littering PSA whenever I read about implementations of the OGC's sensor "web" specification because environmental observation and real web architecture fit like a hand and a glove. What is an ASOS station or a data buoy if not a very dedicated and precise non-human blogger? Instead of routinely cranking out blurbs about Google Earth imagery updates, ArcGIS service patches, or RESTful architecture, it cranks out regular sets of scalar values: air or water temperature, dew point or salinity, wind speed or wave height.

ObsKML, too, recognizes the fitness of syndication for sensor observations, but KML doesn't really handle this kind of payload very well. Neither does RSS 2.0. Atom does have a general and robust content model that can deal with sensor observations as well as it does imagery or HTML: the observations go in an entry's content element, and all other entry elements serve as metadata.

...
 <entry>
   <id>urn:uuid:a261b59f-3692-4a00-a3bb-320a7448e0d8</id>
   <title>ASOS 89000 Obs</title>
   <updated>2009-03-14T16:00Z</updated>
   <georss:where>
     <gml:Point><gml:pos>-90.0 0.0</gml:pos></gml:Point>
   </georss:where>

   <!-- Human-readable summary -->
   <summary>Temp:-35.0, WindSpeed:5.0/10.0, WindDir:15.0, ...</summary>

   <content type="text">
     <!-- ASOS data here -->
     0187725020200111250051+40690-074170FM-15+0009KEWR...
   </content>
 </entry>

OpenSearch gives you a way to specify how a client slices a logical feed into bite-sized chunks. Goodbye, GetRecords(), GetCapabilites(), and GetObservations(). Hello, observations.

Comments

Re: Sensible observation services

Author: Jeremy Cothran

Thanks for the mention of ObsKML, it might be better in broader practice/acceptance to separate the idea of a simple Obs xml schema from the particular location/time xml wrapper whether it be KML,GeoRSS,etc.

The human/machine bias/divide that I continue to see is that any possible syndication format/wrapper/xml-based host(KML,GeoRSS,Atom,etc) has at its core a human audience focus with free-format text/html as the data payload(content tag in Atom) not providing support for simple/minimal xml schemas leaving the possibility of simple machine-to-machine syndication and automated parsing unaddressed.

As I have not seen any evidence of public interest in simple xml content schemas(public is a consumer of display/styled data, and only a producer in regards again to human-only-readable data in regards to free-form text/html) and no simple xml schema interest/support from technically capable search engine companies,instrument manufacturers,etc then the OGC spec/standards for better/worse seem to be the only game in town.

Thanks

Jeremy

Re: Sensible observation services

Author: Sean

No, Atom supports all kinds of payload, even text/xml, not only HTML.

Content schema isn't really the point here. I'm talking about how you get content/data to clients and how you synch it. Atom is simply better for this than SOS.

Re: Sensible observation services

Author: Jeremy Cothran

Yes, I was incorrect in stating that Atom content tag doesn't support xml schema, think I looked at an old spec page and see obvious examples as listed at http://data.octo.dc.gov Probably a little confused also by your example where the content is type 'text' and some domain-specific text-only format.

I understand and agree with your point that Atom is the bees knees for XML syndication(with an eye on browser performance Javascript/AJAX/JSON-related influence/interaction in the future).

Just adding pointwise that without any popular simpler content standards which might be understandable/implementable by non-OGC experts(the wider public, similar to the popularization of GML as KML), that the content standard and its related syndication are left in OGC's field and influence alone.

Jeremy

Re: Sensible observation services

Author: Sean

And I think that the OGC tends to get the expert formats right. It's the protocols for interacting with the services that suck for the web. Atom with SensorML or GML payloads could be a good solution.

Behind the curtain

James Fee asks:

Curious, does it matter to a RESTful zealot that ESRI’s REST API is a cover for SOAP?

Not terribly. From a client's perspective, as long as there are bookmarkable entry points, solid formats, and links to other application states, everything is fine. A web client shouldn't be aware of what's going on behind the resource representations. When you work with Wikipedia, for example, you're operating on web pages about topics, not chunks of text in database records. That's an implementation detail, and something's broken if it leaks through to the client.

Typically there are other protocols involved in a RESTful service. Your web framework, at the very least, is connecting to an RDBMS using some specialized RPC. ESRI's ArcGIS Server just has one extra layer in there. This design seems inside-out to me, and probably sucks for other reasons, but it doesn't necessarily make things un-RESTful.

Unbearable liteness

If there's anything "GeoWeb" folks understand about REST, it's that REST is "lightweight". While it's possible to achieve goals simply in a RESTful system, lightness is your agenda, not REST's. Check out Roy Fielding's re-enumeration of the constraints you have to follow to get all the good properties of REST in "REST APIs must be hypertext driven". That's about as "lite" as an apprenticeship at Shaolin Temple.

On the other hand, I'm appreciating Steve Vinoski's view on RESTful HTTP as a disruptive technology in IEEE Internet Computing:

RESTful HTTP, on the other hand, has all the makings of a disruptive technology to the RPC market. As RPC systems [ed: Sun RPC and Apollo NCS, DCE, Corba, RMI, J2EE, SOAP, and WS-*] moved up-market and gained capabilities and features over time to continue to satisfy the most demanding customers, they overshot more and more potential users who shunned the complexity and cost of such systems. In RESTful HTTP, which was born in the adjacent market of the World Wide Web and is a sustaining technology there, these overshot users are finding an approach that helps them build solutions that are less expensive, simpler to build, and easier to extend and maintain than what RPC approaches can offer. It's precisely these qualities that make RESTful HTTP a disruptive technology in this context.

It's possible to do things simply using HTTP, without much REST constraint. This is the "lightweight" approach understood by the geospatial community as the "GeoWeb". The reason RESTful HTTP is disruptive is because it addresses the needs of the high end market as well. Systems applying the REST constraints can scale massively and can evolve to meet changing requirements, and that's why I'm uncomfortable with REST being pitched as "lite". Lets you achieve simple goals simply, but not itself a simple thing – is this too wonky a notion to get across?

Transliterating from Greek and Latin

Update (2010-07-08): The latest and greatest is uploaded to PyPI: http://pypi.python.org/pypi/pleiades.transliteration

The code we use to transliterate names from Greek and Latin writing systems in the style of the Barrington Atlas is now available from our package index.

$ easy_install -i http://atlantides.org/eggcarton/index/ pleiades.transliteration

For example, the name Ἀφροδεισιάδος from an edition of a letter from Trajan to Smyrna [IAph 2007 8.33] is transliterated to Aphrodeisiados:

>>> from pleiades.transliteration import transliterate_name
>>> s = u'\u1f08\u03c6\u03c1\u03bf\u03b4\u03b5\u03b9\u03c3\u03b\u03ac\u03b4\u03bf\u03c2'
>>> transliterate_name('grc', s)
'Aphrodeisiados'

My first vimperator script

I wrote my first vimperator plugin script last night. While browsing any of my blog posts I can now type :manage and Firefox opens the "edit" URI found in the page's head (a concept ripped off from AtomPub).

/*
 * Opens the URI of the first link in the document with rel="edit" in the same
 * tab.
 *
 * Usage:
 *   :manage
 *
 * @author Sean Gillies (sean.gillies@gmail.com)
 * @version 0.1
 */

commands.add(["manage"],
  "Open document's edit URI",
  function(args) {
    var documentNode = window.content.document;
    var editNode = documentNode.evaluate(
                    '//link[@rel="edit"]',
                    documentNode,
                    null,
                    XPathResult.FIRST_ORDERED_NODE_TYPE,
                    null
                    ).singleNodeValue;
    var editURI = editNode.getAttribute("href");
    liberator.open(editURI, null, true);
  }
);

Comments

Re: My first vimperator script

Author: jachym

Yeah, vimerator rulez! Imho it is much better, than this strange ubiquity thing...