Geo + AtomPub

Google Maps Data API. We have been telling you AtomPub would be a pretty big deal for a couple years now. Have you been working on your HTTP chops in that time?

Comments

Re: Geo + AtomPub

Author: Jason Birch

Yeah, it's cool. I like the OpenSearch integration.

I'm not sure I like that public feature feeds need to be wrapped in Atom and accessed via a Google Auth. Maybe I'm missing something and there are other public ways to get straight KML (and other?) representations.

I'd have liked to see a JSON-based alternate--working with XML makes me itchy--but I guess I can't really expect that with the strong investment in GDATA.

Re: Geo + AtomPub

Author: Sean

Well, there are the http://maps.google.com/maps/ms?... URIs for KML via the MyMaps permalinks. You'd think they'd be linked to from the map feeds unless they've something else in the works.

Reactions

Some reactions on the web to the Google Maps Data API announcement ...

Me (paraphrasing):

Dammit, I told you to pay attention to this stuff.

Raj Singh:

Google announced a data API for Maps this morning in San Jose. This is basically a CRUD service for storing geodata in “the cloud” that leverages Atom in lots of ways. That didn’t sound very world-shaking to me at first since there aren’t even any basic spatial query functions, but there are some ways in which this could be a game-changing service — if you trust Google to be your data custodian.

Simon Willison:

I’m disappointed by this one—it’s really just a CRUD store for the KML files used in Google MyMaps. It would be a lot more useful if it let you perform geospatial calculations against your stored map data using some kind of query API—a cloud service alternative to tools like PostGIS.

Brady Forrest:

Geo data can get very large very quickly. Serving it can get expensive. This Data API will help NGOs, non-profits and developers make their data available without breaking the bank. Google's goals for doing this are obvious. If the data is on their servers they can index it easier and make it readily available to their users. There will be concern that Google will have too much of their data, but as long as Google does not block other search engines and allows developers to remove their data I think that this will be a non-issue.

Fake Ed Parsons:

Come and learn how lat49 and geocommons no longer have business models at the Google Booth

Diving into Shapely 1.1

The code that will become Shapely 1.1 is mostly in working (but not by any means production ready) order and ready for testing by folks that are interested in prepared geometries or new plugins. Documentation and doctests are not up to date, though the unit tests are largely passing, and with good coverage of most modules. It's not been tested on Windows at all, but might work with some coaxing.

Coordinates now are stored primarily as Python arrays of floats. GEOS geometries are created as needed and cached, with some positive and negative effects on performance. Python access to coordinates is much more efficient than it has been. Chaining of operations is less efficient because we're unnecessarily creating Python arrays for anonymous geometries. For example, the following code

p = Point(0.0, 0.0)
b = p.buffer(10.0)
x = list(b.exterior.coords)

benchmarks like this (in microseconds):

1.1a1:   873 usec/pass
1.0.12:  957 usec/pass

but this code, with operation chaining

p = Point(0.0, 0.0)
a = p.buffer(10.0).exterior.convex_hull.area

benchmarks:

1.1a1:  1931 usec/pass
1.0.12:  271 usec/pass

I have some ideas about how not to load Python arrays in the case of anonymous geometries that should address this issue.

Shapely 1.1 has a new architecture. Most users won't even notice, but methods of geometries now call on entry points of plugin packages. The default plugin package is shapely.geos, but you can switch via a function in shapely.implementation as you need.

There are a few known bugs and changes to look out for:

  • Adaptation of the geo interface is broken (no asShape()).

  • Heterogeneous geometry collections are broken.

  • Prepared geometry module shapely.prepared needs to be moved to shapely.geos.prepared (only available if you have GEOS 3.1 anyhow).

  • We're now expecting coordinates to come from Numpy as arrays of x's, y's, and z's which means that Shapely integrates with Numpy in the same manner as matplotlib.

To dive in and try it out (with GEOS 3.0+), I suggest you make a fresh Python 2.5 virtualenv and easy_install the shapely.geos package

$ virtualenv --no-site-packages try-shapely
$ cd try-shapely
$ source bin/activate
(try-shapely)$ easy_install -i http://gispython.org/dist/index shapely.geos

Being a dependency of shapely.geos, Shapely is automatically fetched. If you'd like to hack on them (patches welcome!), check the code out and install them into the virtualenv in development mode

(try-shapely)$ mkdir src; cd src
(try-shapely)$ svn co http://svn.gispython.org/svn/gispy/Shapely/trunk Shapely
(try-shapely)$ cd Shapely
(try-shapely)$ python setup.py develop
(try-shapely)$ cd ..
(try-shapely)$ svn co http://svn.gispython.org/svn/gispy/shapely.geos/trunk shapely-geos
(try-shapely)$ cd shapely-geos
(try-shapely)$ python setup.py develop
(try-shapely)$ cd ../..

Despite all the changes inside, the interface (with a few exceptions) is the same as 1.0:

(try-shapely)$ python
Python 2.5.2 (r252:60911, Dec 23 2008, 09:29:43)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from shapely.geometry import Point
>>> p = Point(0.0, 0.0)
>>> p.wkt
'POINT (0.0000000000000000 0.0000000000000000)'

Future for geospatial on App Engine?

Having finally made the time and excuse to develop (virtualenv + Pylons) and deploy (with appengine-monkey) a very simple application on Google's App Engine, I'm thinking a little more about the support GAE could offer for geospatial apps. My keytree package works just fine on GAE (with xml.etree), so there's handy KML reading and writing. Simplejson, minus speed-ups, is good to go with GAE, and therefore so is geojson. Neither contain many files and so there's no harm in including them in each app (Pylons and its dependencies, on the other hand, eat up 80% of your file quota all by themselves). What's lacking is support for geometry operations and predicates, whether in the datastore API (ala GeoDjango/PostGIS), or in Python ala Shapely.

I can't believe that GEOS will ever be a part of GAE, but support for Java opens the door for JTS. I hope Martin Davis or somebody at OpenGeo grabbed one of the offered chances to try Java and will report on it. Would you have to use Jython to get at the JTS operations and predicates, or could you communicate via protocol buffers within your application? I do not know, being new to the platform, but would like to find out. Ideally, this kind of geospatial applications platform would allow you to create your own services for analyzing spatial features and share them with other applications if it wasn't going to provide them itself. Going over HTTP to a JTS-based geometry/topology service hosted in the same cloud would suck.

Comments

Re: Future for geospatial on App Engine?

Author: Brian Flood

you can upload both python and java to the same GAE application, so yes they can share the same datastore. the specific access models may differ, but the underlying entities are shared.

fwiw, calling intra-cloud over http is not as bad as you would expect.

cheers

brian

Re: Future for geospatial on App Engine?

Author: Martin Davis

This sounds very exciting! I was wondering when Google would add Java support to AppEngine - given their support for Java (eg GWT) it seemed a natural fit.

Not sure when I'll have the cycles to play around with JTS on AppEngine - but I'd be very interested in supporting/collorating/interacting with anyone trying it out.

Martin

Look what I found in KML

Look what I found in KML while looking for hyperlinks: a RESTful service description format. A neat find because we need more working examples of service descriptions.

KML's "NetworkLink" isn't the most well-named element. It's not so much of a link, really, but a slot for feature data that is filled in during a user session. The data is provided by a read-only web service described by the content of the network link's "Link" element. There are 3 service flavors. A link with nothing other than a "href" element tells the client to fetch the resource identified by link[@href]. A link with an "onStop" view refresh mode tells the client to append a bounding box parameter to the service's resource identifier. Lastly, a link may give the client a URI template of sorts via the "viewFormat" element.

This is the shape of RESTful service description. The server tells the client how to make requests for feature data in the context of a particular KML document. It might use a slightly or radically different description in a different context. It might evolve the description over time within the same KML context to meet changing client needs or engineering constraints.

KML was on the cutting edge here, but has diverged from the web. Using standard links with the same semantics of those in HTML and Atom instead of an "href" would be good. Standardizing on URI templates might be a nice feature for the future, as would better separation of concerns in "Link". Specification of on what event and how often the "network link" is refreshed is mixed up with the service description. Some of the elements seem to me to belong in the network link rather than the link.

I often gripe about the KML format ("Snippet"!), but there's much good in it – the RESTful service description pattern in particular.

Update (2009-05-13): all aboard the snippet train.

Comments

Re: Look what I found in KML

Author: Bent

<Icon> has the same semantics and refresh modes as <Link>

(And, <Link> in the context of <Model> is for the collada data type).

http://code.google.com/apis/kml/documentation/kmlreference.html#icon

More literally <NetworkLink> could perhaps have been <KmlLink>.

Personally I find onRegion the most interesting and most restful

mode of all...

New GIS and Python programming blog

I stumbled onto Solving GIS Problems today. It has an ESRI bent, but it's Python, and there's code – a rare thing in the geoblogosphere. And we agree on a bunch of things such as how logging wins and arcgisscripting cursors fail. Subscribed.

Comments

logging vs testing

Author: Jonathan Hartley

Hey there.

Is logging still relevant when you have unit and functional tests? I suspect not, but I'm asking because I genuinely don't know.

I used to use logging in previous projects. My current one is heavily tested, which seems to work really well for us, and we don't ever seem to need any logging, so we've stopped putting any in. Are the two factors correlated?

I think so, but I can't prove it. Clearly the two techniques don't do the same thing, but they do provide similar information (eg. what the code did on one particular run, verses what the code does under all the circumstances that the test writer could think of)

Anyone got any insight to add?

Re: New GIS and Python programming blog

Author: Simon Norris

Thanks for posting this. An ESRI bent can be useful!

Re: New GIS and Python programming blog

Author: sharkinsGIS

Thanks for posting your discovery. You may want to check out my latest post at http://foldingspace.wordpress.com/ as well.

Re: New GIS and Python programming blog

Author: Sean

Unexpected things happen even if your code is well tested: inaccessibility of network resources, topologically invalid user data, you name it. You'll want to log those events and move on when you're processing batches of data or providing a web service.

Philosopher's Stone

Jeff Harrison tries to flip the alchemist label back onto the web wonks:

Maybe what we should be thinking about is how to advance GeoWeb platforms NOT as proprietary implementations using an alchemy of REST, JSON, RSS, KML - but as Community GeoWeb Platforms that combine international standard geospatial web services with Web 2.0 technologies.

Nice try, but no. We agree about the proprietary flavor of some of the "GeoWebs" out there. Google, in particular, promotes a "GeoWeb" that is dominated by its proprietary browser and its proprietary search. Our role in this "GeoWeb" is to make KML files and expose them through sitemaps, increasing the value of Google's proprietary index in exchange for some return traffic through Google's proprietary browser. Sharecropping, in other words. Still, the OGC's SOA isn't appropriate for a "GeoWeb". There's no web there. Principles of web architecture, standard protocols like HTTP/1.1, and standard formats like JSON, Atom – and even KML – remain the way to geographically enrich the existing web.

Generic geometry library

The Generic Geometry Library (via Mateusz Loskot) looks interesting. Algorithms operating on coordinate arrays. Optional geometry classes. Shapely's new plugin framework was motivated by my interest in Python (or Java or .NET) implementations, but a plugin using a C or Cython interface to ggl could be neat.

Comments

Re: Generic geometry library

Author: Howard Butler

GGL doesn't really do squat right now. A CGAL or LEDA backend for Shapely would be much more interesting and immediately useful.

Re: Generic geometry library

Author: Sean

It doesn't? Too bad, because I liked the idea algorithms without a lot of class baggage.

Re: Generic geometry library

Author: Mateusz Loskot

I'm not sure what hobu means it doesn't do squat, probably due to lack of proper english understanding ;-)

But answering to the Python thing, I suppose that if GGL will get Python bindings it will be based on Boost Python as GGL is going to be submitted for review and if approved it will join the Boost Libraries.

GeoRSS media type?

As Randy George finds out, the notion of a "GeoRSS format" is confusing:

There seems to be some confusion about GeoRSS mime type - application/xml, or text/xml, or application/rss+xml, or even application/georss+xml show up in a brief google search? In the end I used a Virtual Earth api viewer to consume the GeoRSS results, which isn’t exactly known for caring about header content anyway. I worked for awhile trying to get the GeoRSS acceptable to OpenLayers.Layer.GeoRSS but never succeeded. It easily accepted static .xml end points, but I never was able to get a dynamic servlet endpoint to work. I probably didn’t find the correct mime type.

There is no GeoRSS media type. Atom's is application/atom+xml. RSS 2.0 uses application/rss+xml, and RSS 1.0 might use application/rdf+xml. You should use one of those depending on whether your GeoRSS elements appear in an Atom, RSS 2.0, or RSS 1.0 feed. Why? Atom has its own processing model. When you request a feed, the server writes application/atom+xml in the response's content-type header to inform you that you'd better use the Atom content processing rules if you want to make sense of the data.

Is there anything to be gained by having a special GeoRSS media type that overrode the Atom or RSS media types?

Comments

Re: GeoRSS media type?

Author: Peter Rushforth

Given that "GeoRSS" is like Kleenex, ie there are lots of formats which can be called that, would it not be helpful to have something like application/atom+gml, application/atom+geo etc to distinguish those brands?

Re: GeoRSS media type?

Author: Sean

Nevermind that +gml and +geo media types aren't standardized -- what would they mean to a processor that's not already said by one of the standard XML media types like text/xml, application/xml, and application/*+xml?

After a read of http://tools.ietf.org/html/rfc3023, I'm inclined to say that GML itself should be using something like application/gml+xml. GML is based on XML, yes, but rather opaque when displayed as text.