2009 (old posts, page 8)

Atom feed RSS feed

For spork's sake no

No, Vish (who asked for criticism), no:

I am also hoping that in the near future, ESRI’s SOAP and REST API have the exact same service contracts that are just exposed at different endpoints. Currently, the SOAP API seems to more feature-filled and the REST API seems to be playing catch up. Aren’t SOAP and REST just different ways of accessing the same exact GIS feature set in ArcGIS Server? Alright, I will end the rant right here. :)

REST and RPC are different styles of solutions for different types of problems. Wanting them to be exactly the same misses the point entirely. I'll make a (admittedly Anglo-European biased) dining analogy, because it's getting to be the time of day when I'm pondering dinner: a spork is no match for a spoon when it comes to delivering soup to your mouth, and is equally ill-suited for eating a salad. The difference between spoon and fork is due to good, divergent design, not design gone wrong.

Comments

Re: For spork's sake no

Author: David

Hybrid bike anyone?

Re: For spork's sake no

Author: Sean

Good one. Nothing wrong with a hybrid bike, but two of them? Which one do you take on a long and technical single-track ride? The blue one? The red one? It seems like ESRI users might be making design choices based on similarly wrong facets.

Re: For spork's sake no

Author: bob

I disagree on the analogy, using the car example (in this case a truck example), SOAP (IMHO) is Chevy 1500 pickup and REST is a Ford F150.

Both _can_ carry the same payload, it's just up to the developer to implement it that way. Where I work we pass the same data over all of our transport interfaces. That way the non-SOAP users (typically Unix/Java/Perl groups) can get the same data as the SOAP users (typically Windows/.Net users).

Re: For spork's sake no

Author: Sean

No. My analogy was weak, but yours is completely misbegotten, bob. You're missing the point of having different architectural styles. SOAP (and other RPC) abstract away the network. You trade scalability for productivity. RPC doesn't scale out, but you can apply your familiar object-oriented and MVC practices and tools. In the REST style you trade latency -- more requests, more following of links between loosely coupled components -- and maybe even productivity (fewer fancy tools) for scalability.

NASA SPG REST and SOAP

Thanks to Benjamin Chartier, I'm finally catching up with Allan Doyle's summary of the NASA SPG's recent session on web services. Benjamin and I disagree with the architecture sketched below [1]:

http://farm3.static.flickr.com/2552/3940198525_3a24633dab_d.jpg

I like it. I don't interpret the blue border between "SOAP" and "REST" literally as my firewall, but as a horizon of trade-offs involving scalability. Outside (aka "the web") is where you, in the words of Jim Webber, trade latency for scalability. Inside is where you trade scalability for nice tools and the ease of programming with distributed objects or XML wrappers around distributed objects.

Comments

Re: NASA SPG REST and SOAP

Author: Benjamin Chartier

You're right, I misunderstood the border between SOAP and REST. Thank you for this clarification.

Benjamin

Re: NASA SPG REST and SOAP

Author: Allan Doyle

Indeed, the boundary is not the firewall. The idea is that we have to accommodate systems like ECHO that have been under construction for a long time and represent a huge sunk cost, and that we don't simply want to wrap them, but let them continue on for as long as it makes sense. When you're building a real-world system, it's often better to have points of stability than to keep chasing the latest trend.

That being said, those kinds of systems then can form the basis for adding new interfaces in ways that make them more useful beyond the internal enterprise boundary.

I think a side effect of this is that the people working on the big systems are not pushed into a defensive mode where they feel they have to spend all their energy preserving their system. Instead, they can become partners who try out new ideas and follow the ones that show the most promise.

If anyone is interested in another NASA earth science web services meeting, let me know. There's one coming up in October where I could use a speaker or two...

Reintroducing the GIS and Python Software Laboratory

Comments on a recent post make me think it's time to reintroduce the GIS and Python Software Laboratory. It's a fairly pretentious name for what's little more than a wiki, an issue tracker, a couple of subversion repositories, and a few mailing lists, but a name that describes well what we're doing: experimenting with bringing GIS to the C Python platform, and bringing Python programming idioms and distribution mechanisms to GIS software. We're designing better APIs that feel natural to a Python programmer. We're designing interoperability around Python protocols. We're designing software that can deployed in the standard Python fashion, and also with new tools like virtualenv and pip. Software that's modular, letting users pay only for what they eat.

Using the same dining metaphor, the lab proposes to you the following a la carte menu:

  • Geojson: implementation of the GeoJSON 1.0 spec
  • OWSLib: making W*S suck less
  • Quadtree: quad-tree index for spatial data
  • Rtree: R*-tree indexes for spatial data
  • Shapely: easy access to the GEOS spatial primitives
  • WorldMill: easy access to vector data using OGR

Mix and match. Hack. Build your own stacks, or stack-busting apps. For projections see pyproj. Subscribing to a similar philosophy, it fits well with the GIS-Python packages.

Updates to the Shapely roadmap

Speaking of great Python packages for geoprocessing: the Shapely roadmap has updates. Version 1.0.13 will allow shapely to be imported in py2exe apps. The work on a framework for use of geometry engines other than GEOS that I'd been calling 1.1 is now raised to 2.0. Use of prepared geometries and the re-entrant geos_c API in GEOS 3.1 is planned for Shapely 1.2.0.

Redefining open source?

Is there more to open source than an OSI license? I enjoyed reading Ian Bicking's DjangoCon keynote. By chance, it's coincident with a big press release about (future) open source from TomTom, which adds interesting context to his some of his remarks. Open source watchers should read this: it's a great personal reflection on open source that doesn't push the usual hot buttons.

Where's the book?

Greg Wilson, author of Software Carpentry (blog), asks:

Now, would someone please write “GIS with Python”?

What's my advance?

Seriously, it's a more formidable task than it appears. The Python and GIS landscape is rapidly evolving and segregated. Unlike bioinformatics or natural language processing (with Python programming books listed in Greg's post), GISville is a company town. To most GIS programmers "GIS with Python" means ESRI software scripting now with Python instead of Avenue or AML. If there isn't an ESRI Press book on scripting with Python already, there will be soon, but it'll be bound to proprietary software. Software that's "free" to use in the U's computer lab, but not free to use out in the real world or on the internets. And it won't cover free and open source GIS tools.

The seedy side of GISville could be tackled in an "Open Source GIS with Python", perhaps, but even this is a fragmented place with many competing platforms. The are programmers who script GIS tasks with Python on a MapServer/GDAL + SWIG platform. There are programmers who script GIS tasks with Python on a different Qt + SIP platform. We've even got our own Django-Pylons type divide on the standard Python platform. How would one synthesize the different Python software and practices? I don't see how, but maybe I'm lacking the right perspective.

Update (2009-10-18): See http://groups.google.com/group/python-gis-sig

Comments

Re: Where's the book?

Author: Matthew Perry

Instead of synthesizing all those disparate uses into a typical book, how about a collection of chapters on each, edited together to form a comprehensive overview of the python spatial ecosystem? This way you could have the best authors tackle their primary subject and still provide the big picture.

It could be called "Geospatial Python: Overlapping approaches to spatial programming" :-)

Re: Where's the book?

Author: Eric Wolf

Personally, I'd love to see a Python GDAL/OGR-oriented book that implemented basic geoprocessing. Essentially a "here's how to do geoprocessing without ESRI". It would also be easier to introduce more advanced programming concepts. I've found that GISers and Geographers tend to get stuck in the learning curve when using Python with the arcgisscripting module. The learn to do loops and manage simple variables but frequently have trouble seeing the need for objects and modules.

Ironically, I've found myself temporarily without an ESRI license for the first time in about 10 years. So the project I'm working on will be done in straight Python+OGR. It's actually a little annoying because I'm having to write a slew of computational geometry routines (I know, there are libraries like CGAL, but I like using my own objects).

I'd be game to help write a book...

Re: Where's the book?

Author: Sean

OGR? Here's an example of my perspective problem: I've developed enough loathing for the ogr.py API (since years) that I've written WorldMill and blogged a lot about using it. It prompted Howard to make ogr.py better, but I feel like my Shapely, WorldMill, Rtree stuff remains more usable.

Re: Where's the book?

Author: Sean

I'm on the loosely-coupled "Pylons" side of the divide, if it's not obvious.

Re: Eric

Author: Keith

Eric,

Have you tried Spatialite? It has the core spatial functions - intersect, union, distance, length, centroid, etc. It is so easy to use and has a nice GUI to view the spatial data. The commands are incorporated in SQL. The tutorial is good too.

Keith

Re: Where's the book?

Author: Sean

Keith: Spatialite is a neat data store, but only that. What you use to process data after you take it out of the store (and before you store the results) is the question.

Re: Where's the book?

Author: Peter

Use XSLT!

Re: Where's the book?

Author: Sean

Peter: I like declarative programming, and there are some neat options to combine and balance Python and XSLT in lxml's extension elements, but I imagine that the declarative style works best for geoprocessing in configuring complex pipelines of data processing modules that would be written in Python. Maybe like Paste Script wires up WSGI pipelines.

Re: Where's the book?

Author: Peter

Sean,

There's no doubt (in my mind) the heavy lifting in terms of geoprocessing needs to be done by an indexed datastore, hopefully accessible via spatial XQuery/XPath :). There are good current benefits in terms of simplicity, portability and ownership of your code, to thinking about low-volume, high-fidelity geoprocessing in pipelines, as you say. Some pipeline alternatives for such pipelines may in the future include

xmlsh

, and

XProc

. To provide sufficient performance, a geospatial implementation perspective needs to be baked in, something I would like to see.

Cheers,

Peter

Re: Where's the book?

Author: Keith

Sean, Spatialite just a datastore? I use pysqlite to script and create UserDefinedFunctions (UDF) to extend and reuse functionality. Sqlite is rock solid and fast, with a 2 terabyte database size limit on 32bit machines. All this functionality for free - Transactions, Virtual tables, Spatial indexes, attribute indexes, projections, overlay commands (clip,erase,intersect), aggregation, simplification, routing. For just a data store it sure sounds like a geospatial engine. The datastore aspect is compelling as well. You can store multiple geometry types in the same table. A shapefile will only hold a single geometry type. You can also have multiple geometry columns in a single Sqlite table - various projections, for example or you could store parcels as various geometries - rectangular lot, building footprint, property centroid. The business data is easier to manage in a database too. Gotta love it.

Re: Where's the book?

Author: Sean

Relational databases have a large, but not limitless set of uses. If you're going to process non-relational data such as documents or less structured data, you'll need a general purpose programming language and environment. For example, you're certainly using a general purpose language already (maybe even Python) to model data so that it can be loaded into your database. I think maybe you confuse me for a "NoSQL" partisan. I simply think an RDBMS is better suited to storing models than to doing all kinds of modeling and data processing.

Back to the topic of Python books: it wasn't my intention to throw "stop energy" at a book effort, but to point out that "Python and GIS" is a messy subject and potential readers are somewhat segregated in different camps. I'm hugely in favor of more, excellent narrative documentation.

Re: Where's the book?

Author: keith

I vote for starting a wiki instead of publishing a book. More people can access a wiki via the internet and it is easier/cheaper to maintain and update a wiki. A book might be out of date in a few years as software is updated. There is a huge variety of open source GIS tools that people use. I'd like to see work flow diagrams, recipes, code snippets, etc how real work is getting done. There is also IronPython (dotnet)and Jython (java) to consider along with Python.

Re: Where's the book?

Author: julia

A "Geospatial Python" wiki with evolving content from geospatial python experts that covers many of the various geospatial python implementations, including interacting with spatial data stores, would be a very welcome thing indeed to many people who are not geospatial python experts but need some good resources and examples - so we won't keep getting stuck in that learning curve....

Don't know that I'd characterize Open Source GIS as "the seedy side" of GISville. I think of it more as the "Company Town" side of GISVille being the snooty gated community where entry is controlled both by the size of one's bank roll and one's desire to pay a high entry fee for the "privilege" of feeling like one is part of some exclusive, superior society. The superiority thing is not true, but the "Company" really excels in marketing, so a lot of people believe the hype. The problem is that inside the gated community, the "Company Way" is the only allowed way to do things. Like an H.O.A. gone completely wrong :)

Many of us (who have been forced to live in the gated community for many years by our employers in the public sector) would like to start learning several different ways of doing similar tasks in GIS with python, and not limit ourselves to a single toolset. That same old "geoprocessing" hammer doesn't always do such a great job :) Knowing more than one way to solve the problem, or how to use more than one tool is a big help when deciding what method makes the most sense to use in any given situation. The aforementioned wiki would be very helpful to a lot of people as a learning aid and in helping to compare and contrast the benefits or drawbacks of employing different approaches.

On web sanity

Stefan Tilkov, who has written an excellent REST introduction and an analysis of REST anti-patterns has some criticism for enterprise web apps that applies equally well to "GeoWeb" apps:

In other words: Many web frameworks, especially those within the Java space, try to support the development of desktop applications that are delivered over the Web as opposed to “web sites”, and I believe this idea is deeply flawed: Any Web application should be measured against the criteria that are used to judge web sites.

I think this goes for the "GeoWeb" too. We're delivering too much desktop GIS over the web. It's a point made also by Brian Noyle here, though I see Flex, Silverlight, .NET, and MVC as parts of the problem, not the solution.

Comments

Re: On web sanity

Author: Tim Bowden

Those links are great thanks Sean. I'm developing a desktop app replacement on the web atm, and that material has clarified some of the issues I've been mulling over. I probably won't get it all right this time around, but I'll certainly get closer than I was going to get.

I've previously looked briefly at the idea of RESTful apps, but not in any great detail or real understanding. Now I am, and it's certainly an interesting journey. The app I'm replacing is definitely very much in the traditional desktop space (with a legacy stretching back all the way from its DOS roots; think reverse engineering very old and poorly documented binary data formats). It doesn't *look* like a candidate to be a 'website' type app, as opposed to a 'desktop app on the web'. The requirements *seem* to point that way, that is until you dissect it bit by bit using a RESTful scalpel. I'm not sure how far I'm going to be able to push this model, either because of limitations in my skill and understanding or limitations in the RESTful model, but I'm keen to see where I can go with this.

It's a good thing this is a relatively simple app, as using a RESTful scalpel for the first time isn't so easy. Old mental models of what desktop apps should look like and how they should behave keep getting in the way. I've still not gone very far on this journey (I've not tried implementing any of this yet; next step!), but so far every issue seems to have been answered by sharpening my RESTful scalpel a little more. I think one of the hardest ideas to come to terms with is the idea of “hypermedia as the engine of application state”. All those 'client side' data structures start to melt into very ephemeral constructions. Implementing that is going to be an interesting experience for sure.

Sean, keep pushing the REST barrow. I'm very much a novice, but from my perspective so far, it *seems* to be a winner, despite the mental difficulty of overcoming deeply ingrained ideas from the desktop environment. Hopefully I'll not be too shell shocked when I come out the other side!

Django goes with grilled chicken

Who says I won't try Django?

http://farm3.static.flickr.com/2515/3874713676_67447e79a9_d.jpg

I actually liked this winemaker's "Gispy", a rosé blend of Syrah and Grenache, much better. Cinsault ("Django" is 100%) isn't winning me over, yet.

Comments

Re: Django goes with grilled chicken

Author: James Fee

I'm scared to think what the Plone blend must taste like.

Re: Django goes with grilled chicken

Author: Sean

"The 2009 Plone is staggeringly complex."

Re: Django goes with grilled chicken

Author: Howard Butler

James,

I think the Plone blend requires that you pay a consultant to open the bottle.

Using git to work on OpenLayers

So that I don't forget, and in the hopes of picking up tips from more experienced users, I'm going to write down my notes on using git to work on OpenLayers without any commit access. OpenLayers is a big project, and enhancing it in a significant way takes time and care. It's worth doing this work within a revision control system. The project has SVN sandboxes, but you can't use them offline (which I've been quite a bit this summer), you have to apply for them (I'm impatient), and my OpenLayers guru is annoyed by them. For these reasons, I've gone with a DVCS, and chose git on account of its nice bridge for Subversion: git-svn.

To begin, clone the OpenLayers trunk. As of today, there have been 9623 revisions. Don't fetch all of them! In this example, I'm going to fetch a small revision range just short of the SVN head so that I can illustrate rebasing later.

$ git svn clone -r 9621:9622 http://svn.openlayers.org/trunk/openlayers
...
r9621 = 4f62382011af58e8dd8be3b81a817fff80b4f4c4 (git-svn)
Checked out HEAD:
  http://svn.openlayers.org/trunk/openlayers r9621

Now I have cloned OpenLayers and begin by default in a branch named "master". I'll do my work around http://trac.openlayers.org/ticket/1366 in a new branch named "atom".

$ git checkout -b atom
Switched to a new branch 'atom'

I've already produced a patch using the methods I'm noting here, and will apply it to the new branch for the sake of this demo.

$ patch -p1 < atom-format-2.patch
patching file lib/OpenLayers.js
patching file lib/OpenLayers/Format/Atom.js
patching file tests/Format/Atom.html
patching file tests/list-tests.html
$ git commit -a
[atom 315def2] Apply patch from http://trac.openlayers.org/attachment/ticket/1366/atom-format-2.patch.
 4 files changed, 1171 insertions(+), 0 deletions(-)
 create mode 100644 lib/OpenLayers/Format/Atom.js
 create mode 100644 tests/Format/Atom.html

If I were to run "git diff master" I'd recover that same patch. Now, there's a new attachment on my favorite OpenLayers ticket. A patch would have been nice, but a file is better than nothing. I'll create a new branch named "atom-pwr" based on "atom", copy the attached file and overwrite Format/Atom.js in the new branch.

$ git checkout -b atom-pwr
Switched to a new branch 'atom-pwr'
(copy atom.js)

Running "git diff atom" shows me the difference between atom-pwr and atom branches. I'll commit the changes to atom-pwr.

$ git commit -a
[atom-pwr 29ae464] Incorporate pwr's work from http://trac.openlayers.org/attachment/ticket/1366/atom.js.
 1 files changed, 48 insertions(+), 7 deletions(-)

A patch to the OpenLayers trunk incorporating all these changes is generated with:

$ git diff master > atom-pwr-r9621.patch

Now, lib/OpenLayers/Control/GetFeature.js was added in r9623. To keep up with these changes, use git-svn rebase. First on the master branch:

$ git checkout master
$ git svn rebase
        M       lib/OpenLayers/Control/GetFeature.js
r9623 = 3e4282ccb05e9421d8f078893c5c495c55085621 (git-svn)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to refs/remotes/git-svn.

And then on the development branch:

$ git checkout atom
$ git svn rebase
First, rewinding head to replay your work on top of it...
Applying: Apply patch from http://trac.openlayers.org/attachment/ticket/1366/atom-format-2.patch.

Now a patch against the current head is generated by:

$ git diff master > atom-r9623.patch

As nothing was changed in the files I'm working on between r9621 and r9623, this patch is identical to the one I started with. Up the new patch goes to the OpenLayers issue tracker to make things as easy as possible for an actual committer.

Comments

openlayers on github

Author: whit

Another resource for working with OL using git: ccnmtl is kind enough to mirror OL on GitHub.