GeoSummit Agenda

Update: More from Peter Batty, John Spinney, and Chris Haller.

While some topics are being hashed out on our Google page, the actual schedule remains to be decided Saturday on-the-fly. My ultimate goal is to help merge all the cool agendas people bring, but I have a few of my own.

Bank on me to mount my RESTful GIS hobby horse and start tuning up my arguments for the FOSS4G conference. I'd also like to help lead discussion about non-traditional data models and storage in the context of my Pleiades site. I know that I'm not the only one with data that doesn't fit in a single feature table. Maybe this could potentially lead into a data management or stewardship (a hot topic recently) session.

I'm looking forward to reducing my ignorance about mobile GIS and location-based services. I missed the Churchill navigation system demo at a recent Front Range Pythoneers meeting, and hope to get one at the GeoSummit. I hear they're making great use of Google Earth at the NSIDC, and I expect that will be a part of at least one big virtual globe session.

I'll see you there. Saturday, 16 June.

Rome Reborn as a Flash Movie

Mike Shaver's post about rich internet applications, with the great quote

If someone tells you that their platform is the web, only better, there is a very easy test that you can use:

http://sgillies.net/images/is-this-the-web.png

rippled around last month, and I was reminded of it again when I viewed the Rome Reborn 1.0 flash movie (via The Stoa Consortium). It's slick, no doubt, but you can't really call it a web site. This manifestation is practically unusable. Flash movies should complement -- never replace -- addressable, linkable, bookmarkable, and programmable web sites.

See also Daniel Cohen and Roy Rosenzweig in their book Digital History.

Comments

Re: Rome Reborn as a Flash Movie

Author: Bill Thorp

Is this really about Flash, or even web sites? Couldn't I render obtuse SVG in a browser? PDFs? AJAX? Applets? Would you really deep-link their content if you could? Should they want you to? (There are ways: http://www.unfocus.com/projects/)

Re: Rome Reborn as a Flash Movie

Author: Allan

Trouble is, Flash is considered the "easy" way towards cross-platform rich content delivery. There's only one Flash engine that gets ported by one company. Browser engines present a messier landscape to provide content for. Furthermore, Flash apps are probably being built by media design people who don't have a deep appreciation for the implications of non-URL-addressable content that gets buried in the process. I think the solution is not so much to rail against the people who do this but rather to provide examples of what you/we consider better practices. These people are earnestly doing their jobs and when they look around for "how do we build an interactive web site to showcase our stuff" they will have a greater likelihood of finding decent patterns.

Re: Rome Reborn as a Flash Movie

Author: Sean

Ross Scaife (from The Stoa Consortium) found himself unable to link to his favorite image on the Rome Reborn "site", something that we take for granted on the dull, plain old HTML web. I have no problem with Flash, the tool. A site implemented entirely in javascript with no links (no non-human UI) would be almost as bad. It might as well be a glossy paper brochure. Best practices are clearly laid out in Cohen and Rosenzweig's book. The authors were not as concerned about the programmable web as I am, but their advice about URIs and multimedia is as relevant to machine users as human users.

Re: Rome Reborn as a Flash Movie

Author: Bill Thorp

"Flash with Named Anchors" is an HTML publishing template in Flash. This is an awareness issue. The authors of Rome... don't know how to use it, and Mr. Scaife must be totally unaware of HTTP packet sniffing. http://www.romereborn.virginia.edu/imageG8.jpg

Re: Rome Reborn as a Flash Movie

Author: Sean

Good fodder for this weekend, yes?

Python Pain Points

So, there's this meme that you have to be able to enumerate your domain's pain points. For Python and GIS/neogeography these include:

  • the standard library's xml.sax and xml.dom. ElementTree and lxml are the way to go.

  • unittest. I wrote a pile of unittest modules for MapServer's SWIG bindings, aka mapscript. So many that it'll be a while before I slide out of the top 5 all-time MapServer LOC. The tests never really caught on with other developers. The generic xUnit style didn't help seasoned developers cross over, and doesn't help explain usage to new users. These days, I'm much happier using doctest. It's been hard for new Python programmers to find good advice on the right testing framework, and the consequence for open source Python GIS is that people don't write tests.

  • distutils and setuptools. Python's packaging and distribution story is too arcane. Yes, this is intricate stuff, but it needs to be easier. People developing open source GIS for Python rarely bother to figure out the old distutils setup, let alone eggs.

  • str and unicode. Python's unicode story is not bad, and will be even better when unicode is always on.

  • function names. This is a small itch (caused by exposure to Ruby), but I'd love to be able to use self-descriptive Python APIs like:

    >>> if polygon.simple?      # instead of isSimple()
    >>> polygon.rotate!(angle)  # rotate self, not a copy
  • mapscript.py and gdal.py. Many users come to Python via MapServer and GDAL, which is unfortunate because those applications have terrible, un-Pythonic APIs (that ripple through the open source GIS community) and have created the impression that Python is just a binding for C/C++ code.

Comments

Re: Python Pain Points

Author: hobu

MapServer's unittests have slowly gained traction, but recently both myself and Umberto have made commits to them to help with watching the 5.0 release. I agree that doctest is much better and it would be nice to port MapServer's tests to it someday. Re: eggs. Both GDAL and MapServer will use setuptools first if it is available (although current trunk of GDAL will do --single-version-externally-managed by default if you use the 'make install' target). I fricking hate eggs though. In a web-enabled world, having eggs not extract themselves by default in someplace like /tmp that is writeable to processes like Apache causes much pain. Python eggs in their current incarnation seem like an obscene hack to me, with their behind the scenes path munging, unzipping, and other magic. That setuptools also busts compatibility with distutils also causes pain (ie, I can't do --prefix with setuptools but can with distutils). What did setuptools really solve that distutils didn't already again? Python may be fully unicode aware someday soon, but the C/C++ libs that we depend on in the Open Source GIS domain won't be. re unsightliness of gdal.py and mapscript.py... The first one out the gate gets shot in the back :) Do we change it now to make it pretty and break hundreds of thousands of lines of code and piss everyone off who's currently using it?

Re: Python Pain Points

Author: Sean

I like eggs. The development setup target is very handy, and I'm even planning to make more use of the dependency resolution before my projects hit 1.0. Those are my favorite features not found in distutils. I don't have a solution for my mapscript and ogr pain other than to improve Shapely, Rtree, and PCL, and get them to 1.0.

Shapely Geometries for Python

Shapely is to be an LGPL-licensed replacement for PCL's cartography.geometry package. Like its ancestor, it will provide familiar geometry classes with all the spatial operators of GEOS. When done, cartography.geometry can be rewritten as a lightweight layer over the new code. Additionally, Shapely will have a more Pythonic serialization story, and plug and play with other powerful Python packages using the Numpy array interface. Last, but not least, Shapely will be much easier to build, install and deploy.

Shapely serializes and deserializes using the familiar idiom from the pickle module. Here's a pickle stream example (an actual project doctest):

>>> from shapely.geometry import Point
>>> import pickle
>>> point = Point(0.0, 0.0)
>>> data = pickle.dumps(point)
>>> pp = pickle.loads(data)
>>> pp.equals(point)
True

Comments

Re: Shapely Geometries for Python

Author: Bill Thorp

Sean, apologies for this aside... Shapely isn't something you're authoring, is it? I've been meaning to write about my love/hate relationship with certain JSON serializations. '{"type": "Point", "coordinates": [0.0, 0.0]}' is a crazy-horrible waste of space, especially when you consider points are part of just about any geometry. I imagine that most JSON geometry serializers act like this. Mine does, with shorter attribute names. Tripling geometry bandwidth is bad, but using custom JSON (de)serializations just for geometry is also a questionable undertaking. I'm torn between thinking this is a case where KISS just does not apply, and questioning the point of optimizing JSON when large geometries are bunk in browsers anyway.

Re: Shapely Geometries for Python

Author: Sean

JSON is but a small facet of Shapely. Yes, the GeoJSON coming out of our working group is a bit verbose. I like my initial {"type": string, "value": array} better. Anyway, reducing bandwidth versus XML isn't my motivator. I want a non-XML data option, to exchange dicts.

Re: Shapely Geometries for Python

Author: Justin Bronn

You'd still need to hack with OGR since GEOS does not read geometries directly from geographic datasets (e.g., SHP files). That's one of the reasons I implemented GeoDjango's ctypes interface for GDAL/OGR (includes a "pythonic" API).

Re: Shapely Geometries for Python

Author: Sean

Yes, you need GDAL to access legacy data. Your GeoDjango package looks good (and I've definitely learned a lot from your ctypes examples), but it's too tied to Django to be useful to me. On closer examination, looks like one should be able to add contrib.gis to python path and import contrib.gis.gdal after all.

Re: Shapely Geometries for Python

Author: Justin Bronn

On closer examination, looks like one should be able to add contrib.gis to python path and import contrib.gis.gdal after all.
Sean, I've tried to ensure that GeoDjango's external interfaces to GDAL/OGR and GEOS were as loosely coupled as possible. As far as I know, both interfaces are completely independent of Django -- the only requirements are ctypes and a compiled C library of the interface.

Local Bloggers at GeoSummit

In spite of our scheduling conflict with the ESRI UC, the FRUGOS GeoSummit now has most of Colorado's geo-bloggers. Come see Bill "MapWrecker" Thorp tear a stack of road atlases in half with his bare hands! Or better yet, discuss rich internet mapping applications with him.

Bait and Switch

This isn't about Python after all.

Comments

Re: Bait and Switch

Author: Aaron Racicot

Since when is using PyQT and the Python bindings to QGis not considered using Python ?!?!? This is actually pretty cool stuff if you ask me...

Re: Bait and Switch

Author: Sean

The post was entirely about SIP bindings for C++ libs. Not a Python issue in sight. Cool, sure. Python, not really.

Re: Bait and Switch

Author: Gary Sherman

The renderer is written in Python. Using Python bindings. No SIP C++ bindings were created, changed, or harmed in making the renderer. I don't understand how you can write something in Python and it not be Python. Perhaps if your post was more than one sentence it would have been more revealing...

Re: Bait and Switch

Author: Sean

The comments in that thread, all about Qt details, make my point.

Koombaya

One of the fun side-effects of digging into RESTful GIS architecture is finding common ground with people like Keyur Shah. A hardcore Java programmer employed at ESRI and a open source Python hack should be diametrically opposed. Maybe we still are, just converging on some of the same design principles from different directions.

Comments

Re: Koombaya

Author: Keyur

I guess http connects more than just machines...

Richardson and Ruby's RESTful Maps and Gazetteer

I've been sketching out an alternative to the OSGeo Tiled Map Service for most of the spring. Not yet another standard, but an example of another, more RESTful way of providing tiles to a mapping client. I'm just going to scrap it now and point you to Chapter 5 of RESTful Web Services by Leonard Richardson and Sam Ruby.

The OSGeo TMS is composed of a general family of algorithms, parameterized by map scale and image coordinate origin, for finding the URL of specific map tile. The basic algorithm is coded within web map client applications, and the clients query a service for an enumeration of all the variants of the algorithm that it implements. The client application selects the optimal variant of the algorithm, and executes it to obtain the URIs of the map imagery it needs. The advantages of TMS over an OGC WMS are that map tile resources can distributed over any number of servers (it scales, trivially), and that clients can leverage the caching mechanisms built into the Web (If-Modified-Since and ETag, also known as conditional GET) to conserve bandwidth and improve performance.

The service outlined by Richardson and Ruby has all the advantages of TMS and one more: there's no requirement on map clients to possess or execute tile-locating algorithms. A request for imagery to such a service would return a document containing:

  1. a link to the central map tile

  2. links to that tile's neighbors

A map client would simply GET and render the central tile, and then traverse the links to build out a full map mosaic.

This is rather a lot like the RESTful feature services I wrote about in April: simply elegant, but also a hard sell. Mainstream GIS folks recoil in horror at the thought of relying on link traversal to render their maps. I think this is partly based on experience with under-performing OGC WMS instances, partly on discomfort with anything too different than desktop architecture (web-as-filesystem), and on the fact that there are no web-traversing map services in the wild. We're also following Google's lead in this area because its hard to argue with the success of Google Maps. Richardson and Ruby gloss over the fact that Google's map tile service is much more like the OSGeo TMS than their ROA (Resource Oriented Architecture) maps.

Lots of good stuff in the book. I hope the librarians out there appreciate my link to WorldCat instead of Amazon ;)

Comments

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Keyur

One problem I see with this approach is that every tile fetch incurs 2 GETs - one for the document that contains info about the central tile and its neighbors and another to actually fetch the tile image. An approach that I can think of is - one GET to fetch the metadata about the entire tile grid for a given level and then the client constructing URIs based on this metadata and directly GETting tile images. I understand that URI opaqueness is the better practice when it comes to linking resources but when it comes to fetching a resource "grid", constructing URIs is a fair compromise. Both GMaps and VE have gone the path of constructing URIs and it works out quite well.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Sean

I agree, URI construction seems to be the way to go for homogeneous, gridded data. Still, how bad would the double GET be using persistent HTTP connections? Is pure link traversal orders of magnitude slower, or just somewhat slower?

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Allan

I appreciate the RESTfulness. But REST doesn't quite do everthing that's needed, does it? I get back a list of 9 tiles. How would a program know which one goes where? There either has to be just the slightest non-opacity in the URLs that a program can latch on to, or the list of URLs has to be ordered in a way that can be gleaned by the program. One of the tenets (or escape clauses) of REST is that the resource could also provide the Javascript code that knows where the neighbors go. But then the consumer would need a description of the API of the Javascript. Thus there would still be a need to know some a-priori magic that can't be glossed over. I guess I better read the book! It's on my "saved for later" Amazon list. I just bought a Ruby book and a Rails book, so it's going to have to wait.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Allan

Persistent connection or not, the second GET can't happen before the results of the first one are received since you need to do a GET on what came back. You can't have any benefit of pipelining, so to fist order, you incur two round-trip times in the pure REST approach where you would have one plus some small increment in Keyur's approach, and exactly one round trip time in a completely parameterized approach. When you measure ping times of ~50msec or larger cross-country or 200+ in non-broadband parts of the world, round trip times add up.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Sean

Allan, this book is filled with Ruby code. How to assemble the tiles? At the very least, you'd need a microformat
  <a class="nw-neighbor" ...>...</a>
One could argue that extracting these microformatted links is also an algorithm, but it sure is a dirt simple one.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Paul Ramsey

Yes, this is good. The me-and-my-neighbors approach is good, and can be scaled up to an arbitrary NxN metatile, with links to children and parent. It has the same odd performance characteristics as the pure REST feature server though, in that, if I am coming to the service at a high resolution, I have to start at the low resolution root and traverse documents all the way in until I get to the effective resolution I am interested in and can fetch actual imagery. I guess the "problem" with all rest approaches to image services is that a metadata document is pretty much required in order to explicate the structure of the imagery -- unless one uses URL construction. Note that this is basically the same approach as KML <Region>s.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Leonard Richardson

Hi, I don't want to butt into your conversation, but if you look in chapter 8 of RWS you'll see a discussion of constructed URIs vs. links, that shows the tradeoffs in the context of chapter 5's map service. The section is "why connectedness matters".

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Sean

I definitely see how connectedness matters. My major criticism of the so-called "Geo-Web" is how poorly linked everything has been. There's no web there (here, for example). I'm using links extensively in my current project (not a map imagery service). However, the enterprise GIS developer's solution to integrating new, more detailed source data (after all, we can get the road vectors themselves) would be to render it into new gridded map imagery at the same scale, maintaining their powers of two (for example) scheme for URI construction. There are, of course, maps for which no source data exists, but historical GIS is a very small niche. Still, I like the connected map service example, and would love to find the time to implement something like it and compare with a TMS.

Re: Richardson and Ruby's RESTful Maps and Gazetteer

Author: Sean

Paul, while zoom levels are linked together, you're not absolutely required to start at the lowest resolution any more than you are required to start at the root of any website and traverse to its documents. Their RESTful maps can be bookmarked like any other hypermedia.