2010 (old posts, page 3)

Sensors, things, and the Web

My readers are probably aware of the OGC's Sensor Web initiatives, but there's another, different vision of a "Web of Things" using the architecture and infrastructure of the actual web we have now (URIs, HTTP, Atom, JSON, HTML, Javascript) that's well articulated in this SXSW presentation by Vlad Trifa and Dominique Guinard (and also in their blog, via This week in REST) and in an associated technical report (PDF).

Comments

Re: Sensors, things, and the Web

Author: Miguel Montesinos

I deeply agree with the "Web of Things" vision. I don't think that's the "different vision", but one of the most likely visions to happen.

I'm involved in several similar European R&D projects and initiatives, and none of them cares about OGC's SWE, apart from what we are proposing.

SWE is quite powerful, but really complicated for becoming a backbone of the Internet of Things.

Miguel

ISAW Visit

Last week I made my first 2010 trip to the Institute for the Study of the Ancient World at New York University for a workshop with researchers and programmers from the University of Heidelberg's Epigraphische Datenbank Heidelberg. The stuff I work on daily is only a fraction of ISAW's digital projects, which are in turn only a fraction of ISAW's business. I had a day before the workshop to catch up with what's going on in the ISAW library and exhibition groups.

But first I had to fly across the Atlantic in this A380, which flies as smoothly and quietly as advertised.

http://farm3.static.flickr.com/2772/4418841703_9600d1e379_d.jpg http://farm5.static.flickr.com/4040/4418841705_587a41402b_d.jpg

I shared a row with a retiree from Avignon. His US-based kids were flying him to NYC and then Aspen for his 60th birthday. We talked about the food and geography of France and the Southwest US – Mexican cuisine in particular, which I've been craving and he'd discovered on a previous trip to Colorado, Utah, and Arizona. I heard French on the street in New York, and among visiting scholars at ISAW, but that would be the last French I'd speak for a week.

Next is a crappy mobile phone photo of one of the fine banners ISAW put up on the Museum Mile (ISAW is just half a block east of 5th Ave on 84th Street) to advertise the Old Europe exhibit.

http://farm5.static.flickr.com/4055/4418841707_91de188dcd_d.jpg

In this context, "Old Europe" refers to a largely forgotten Neolithic and Copper Age culture established along the Danube River during a wave of emigration from Anatolia that also settled the Aegean islands and what are now Macedonia and Greece (see also Cucuteni-Trypillian culture). The objects in the exhibition are from museums in Bulgaria, Moldova, and Bulgaria, and are being shown in the US for the first time. Here's a nice Flickr photo set made by an exhibit visitor:

http://farm3.static.flickr.com/2793/4418996097_a2d03cd10e_d.jpg

The Metropolitan Museum of Art has nothing from this culture, but does have a collection of almost contemporary Early Cycladic objects.

The exhibit is very well done, widely reported and well reviewed in the local media, and well attended. Chapeau to Jennifer Chi and the exhibition team. There's a nice catalog book edited by David W. Anthony with chapters that dive deeper into the archaeology and history of the culture. The exhibits and digital projects groups have a bunch of ideas of how to improve the integration of physical, print, and web materials for upcoming exhibits as we roll out the new ISAW website. The catalog has some great maps by Brian Turner from UNC's Ancient World Mapping Center (where I worked previously) but I think a KML application could take the geography to another level. The exhibit runs through April 26 after which you'll have to travel to Southeastern Europe to see these objects.

I met ISAW's newest technical people, Michael Edgcumbe (who keeps the wheels on office computing) and Christopher Warner (lead on the new website), in person for the first time. The workshop went well, too. More about that later after I push new code up to our site.

While packing for the return trip, I heard that I'd be coming back to snow. Indeed: when we descended below the clouds I saw Montpellier and much of the Hérault department covered with snow. We got about 10 cm in the neighborhood, some of which remains to be seen in the photo below:

http://farm5.static.flickr.com/4014/4416631557_cb66eca519_d.jpg

Prunus dulcis

Normally, turkey vultures inform me of Spring's arrival; I'm looking around for other signs here in Montpellier. Returning yesterday from a week in the snowy Alpes, I saw trees in full bloom along the A9 near Nîmes. Perhaps we had slightly cooler weather here last week, because the almond trees in the neighborhood green space are just getting started:

http://farm5.static.flickr.com/4021/4394353543_76effcfcc7_d.jpg

PyCon interview with Sanjiv Singh

Here's an interview with Sanjiv Singh (TurboGears, GeoAlchemy) in The Bitsource:

Sanjiv Singh is a Python developer from New Delhi, India who contributes to the TurboGears and Toscawidgets projects. Singh recently gave a talk on “TurboGears2 Geopspatial Framework at PyCon 2010, which is a framework for developing Geographic Information System Applications, such as the Blind Audio Tactile System.

I only found two PyCon talks related to GIS: Sanjiv's TurboGears Geospatial Framework and Roy Hyunjin Han's How Python is guiding infrastructure construction in Africa.

Saving bandwidth and more using httplib2

Here's a comment that more properly belongs on Saving bandwidth using Python. The requirement to register was (to me) a blocker for leaving it there. Anne writes:

When Internet connection is a limited resource, a well-designed website doesn’t perform multiple times the same request. This little adjustment can significantly reduce the time required to load and refresh a page. First-world programmers should keep this in mind, or better come to South Africa and experience it in person ...

The solution involves a wrapper around urllib.urlretrieve that partially implements HTTP caching. A more robust solution might instead use the almost transparent Last-Modified or ETag validated caching that is built into httplib2. See also Mark Pilgrim's notes on httplib2 in Dive Into Python 3 (httplib2 works fine with Python 2.3+). Saves bandwidth, development time, and bug chasing.

Comments

Re: Saving bandwidth and more using httplib2

Author: Barry Rowlingson

Or set up a local, or institutional, caching proxy using squid and apache? Most programs these days can be configured to use them, either via setting http_proxy as an environment variable or custom config settings.

Then you get to set timeouts and E-tag magic and it can work for everyone on your PC/at your institution for all HTTP content. Big wins all round.

Re: Saving bandwidth and more using httplib2

Author: Sean

Better yet, I agree, for an enterprise. I assumed that the Linfinti work had a smaller scope.

Re: Saving bandwidth and more using httplib2

Author: Tim Sutton

Actually Anne left out a little bit of the story - we are writing a web mapping client that builds a legend using wms GetLegendGraphic requests to a third party service. So the client is on one (unknown) network where we cant impose squid etc, the server on another and the wms service on a 3rd. The problem is that each time the client requests a page that has a legend on it, they wait for time consuming getlegendgraphic requests. So our solution is to cache the legend graphic requests on our server so that we can give a known response time to the web client. In this context squid / proxying requests wouldnt be an option.

Regards

Tim

Saving bandwidth takes two

My earlier blog post on caching requires some follow up. In fact, the utility of httplib2 or any other HTTP cache hinges on whether or not the server hosting the wanted resources provides validating (ETag or Last-Modified) or freshness (Expires or Cache-Control) headers. In other words: the organization running the server has to want to help you save bandwidth. The original blog post says "That function caches the result of WMS requests for layer legends in a dedicated directory, assuming that the images are not changing over time."

WMS, eh? Do Web Map Services work with web caches? Not by design. How about in practice? I've plucked a WMS legend request from the USGS Framework WMS Viewer [1]:

http://frameworkwfs.usgs.gov/framework/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.2&LANGUAGE=en-US,en&REQUEST=GetLegend&LAYERS=COUNTY_OR_EQUIVAL%3AFramework,STATE_OR_TERRITOR%3AFramework,NHDPTHI%3AFramework,ROADS%3AFramework,NHDFLHI%3AFramework,NHDLIHI%3AFramework,NHDWBHI%3AFramework,NHDARHI%3AFramework&STYLES=,,,,,,,&SCALE=30000000&FORMAT=image%2Fpng%3B+PhotometricInterpretation%3DRGB&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&EXCEPTIONS=INIMAGE

If I read the Linfinti blog post right, this is the kind of image they'd like to avoid fetching unnecessarily. If you look closely at a HTTP request for the legend image (passing the -v option to curl, with [1] standing in for the request in the code below), you see:

krusty-2:~ seang$ curl -v [1] > /tmp/legend.png
* About to connect() to frameworkwfs.usgs.gov port 80 (#0)
*   Trying 76.10.128.162... connected
* Connected to frameworkwfs.usgs.gov (76.10.128.162) port 80 (#0)
> GET /framework/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.2... HTTP/1.1
> User-Agent: curl/7.19.5 (i386-apple-darwin9.8.0) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3
> Host: frameworkwfs.usgs.gov
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 18 Feb 2010 21:26:03 GMT
< Server: Apache/2.0.46 (Red Hat)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: image/png

No validator headers. No freshness headers. Neither httplib2 nor a HTTP caching proxy is going to be useful here. You're on your own. Hurrah for standards. Let a thousand ad-hoc caching schemes bloom.

Comments

Re: Saving bandwidth takes two

Author: geographika

The site you give as an example even causes the logo to be downloaded on each map request, so implementing caching on the WMS images was unlikely.

Surely there needs to be a thoudand different ways of setting up caching, as a WMS service can be used in many different ways - static background maps, weekly data updates, constantly changing data etc.

Re: Saving bandwidth takes two

Author: Sean

The web's own caching mechanisms serve all those use cases and more. A WxS can do real harm to developers in making them reinvent it.

Is really WxS to blame here ?

Author: Jérôme Andrieux

It seems to me you are blaming, all at once :

- the WS standard

- the WS implementation(s)

- the architecture of this particular site

Of course, OGC WxS standards could benefit a restful approach (I mean /a/unique/path/to/resources) when it comes to efficient web cache policy with frontal and external HTTP cache managers (like Varnish, nginx, ...).

However, in this particular case, isn't the WxS implementation to blame first ?

As someone said above, this particular doesn't pay to much attention to cache & compression policies anyway.

Re: Saving bandwidth takes two

Author: Sean

The WMS 1.3 spec has two sentences regarding HTTP headers. One suggests that a service should provide caching headers if possible, but there's no guidance at all for implementers. IMO, that is a failing of the spec, and as a consequence we see people needlessly inventing their own caching schemes.

Examples of WxS servers that do support web caching are very welcome.

Shapely 1.2b1

Update (2010-02-28): 1.2b3 is out. URLs are changed below.

Update (2010-02-19): 1.2b2 is out. URLs are changed below.

This is the first 1.2 release uploaded to PyPI: http://pypi.python.org/pypi/Shapely/. I hope the new README introduces the package more clearly. The manual lags: it will be the last thing before a 1.2 final.

A consequence for downstream developers is that a package that depends on Shapely in the setuptools sense without specifying a specific version will now begin to pull in 1.2b* releases instead of 1.0.14. 1.2b1 seems to be working with GEOS 3.0 and we didn't intentionally break backwards compatibility, but you should be cautious about this. If you're depending on GEOS 2.2.3 and haven't already pinned your package to Shapely==1.0.14 (updating as you go), you should.

To install and try it out (in a virtualenv):

$ pip install http://gispython.org/dist/Shapely-1.2b3.tar.gz

or

$ easy_install http://gispython.org/dist/Shapely-1.2b3.tar.gz

Geospatial Jython

I've fooled around with Jython, but this, via @erilem, is the first I've seen of it in real live GIS software:

>>> from geoserver import Catalog
>>> cat = Catalog('topp')
>>> print cat.stores()
[]
>>> st = cat['states_shapefile']
>>> print st.layers()
[]
>>> l = st['states']
>>> print l.count()
49
print l.bounds()
...

A little bit of its Java-ness leaks through, but at least one is spared the getters and setters. Cool, but then what's this smell at the end?

The following is the wish list based on feedback from teh community:

  • add a new output format
  • add a transaction listener
  • add a dispatcher callback
  • add a WPS process
  • add a datastore
  • a restful endpoint

Restful endpoint? Oh, for crying out loud. Head. Desk.

Comments

Re: Geospatial Jython

Author: Silas Toms

Congrats on your 1000th post. Here's to many more in the future.

Re: Geospatial Jython

Author: Sean

It'll take me a while yet to get to 2**10, but thank you :)

Re: Geospatial Jython

Author: Steven Citron-Pousty

So maybe my suggestion on stack overflow wasn't

totally

off base

Re: Geospatial Jython

Author: Justin Deoliveira

I am interested to hear thoughts on how to make the api less java like and more pythonic... but being a java hack old habits die hard. I guess I could start by reading over some of your older posts. I am not sure if http://geoscript.org/py has come across your radar but it is an attempt to wrap up geotools in python.

Re: Geospatial Jython

Author: Sean

Geoscript looks like it has a lot in common with Shapely and Geojson, which is good :)

But this "rest endpoint" thinking: what is up with that?

Re: Geospatial Jython

Author: Jachym

PyWPS works in Jython as well. We are running it as java-servlet.

Re: Geospatial Jython

Author: Sean

Steve0: Something like ogr2ogr could be a nice example app for geoscript.

Justin: read-only properties instead of methods would be nice, but maybe that's just me. The geoscript geometry constructors all looked just right.

Author: Justin Deoliveira

Sean: Thanks for the feedback. And yes much design inspiration was driven from Shapely so thanks for that. About the "rest endpoint" thing I would not read much into it. It was something mentioned in email and blindly copied into that wiki page. I fully agree the statement is meaningless in the context of that page. Python bindings for GeoServer/GeoTools are completely orthogonal to anything "REST" imo. But a blatant misuse of terminology on our part nonetheless. Thanks for the call out :)

Re: Geospatial Jython

Author: Sean

Orthogonal is right. Knee-jerk reaction on my part, which just distracts from how cool it is to see some geo-Jython action.

Re: Geospatial Jython

Author: Tim Schaub

And keep an eye out for GeoScript JS (http://geoscript.org/js/). End-to-end JavaScript mapping apps with GeoJSON as transport, that's my idea of slick :).

Future of Flash in GIS

Fuzzy, the blogger with the not-Flash banner animation, writes:

Every time I hear about devs (some of my co-workers included) choosing the Flex API for AGS I have to bite my lip. With the life span of web mapping sites they’re probably more than safe going with Flash – it’s slide will likely be long and slow. But I think that slide is becoming inevitable.

Last week I commented that one upside of the Apple-Adobe Flash brouhaha was that it reduces all the necessarily complicated, technical, wonky arguments against Flash (which clients know, somehow, that they "need") to: "it won't work on your iPad". But I'm resisting because I'm almost certain that Apple will eventually win this game of chicken, extract whatever concessions or guarantees or fees or pounds of flesh from Adobe, and relent. I'd like to be wrong.

Comments

Re: Future of Flash in GIS

Author: Dino Ravnić

We have built a vector mapping engine on top of Adobe Flash. It performs quite well. With it we can render directly almost any vector format. It is integrated into GIS Cloud project: http://www.giscloud.com/

Re: Future of Flash in GIS

Author: Sean

What brouhaha, eh Dino?

Re: Future of Flash in GIS

Author: Dino Ravnić

Yeah it is. Many of them are underestimating Flash today and its destiny. It is practically a standard for vector graphics in web browsers and it will take quite some time to HTML5 or whatever catches up with its performance and browser penetration.