Saving bandwidth takes two

My earlier blog post on caching requires some follow up. In fact, the utility of httplib2 or any other HTTP cache hinges on whether or not the server hosting the wanted resources provides validating (ETag or Last-Modified) or freshness (Expires or Cache-Control) headers. In other words: the organization running the server has to want to help you save bandwidth. The original blog post says "That function caches the result of WMS requests for layer legends in a dedicated directory, assuming that the images are not changing over time."

WMS, eh? Do Web Map Services work with web caches? Not by design. How about in practice? I've plucked a WMS legend request from the USGS Framework WMS Viewer [1]:

http://frameworkwfs.usgs.gov/framework/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.2&LANGUAGE=en-US,en&REQUEST=GetLegend&LAYERS=COUNTY_OR_EQUIVAL%3AFramework,STATE_OR_TERRITOR%3AFramework,NHDPTHI%3AFramework,ROADS%3AFramework,NHDFLHI%3AFramework,NHDLIHI%3AFramework,NHDWBHI%3AFramework,NHDARHI%3AFramework&STYLES=,,,,,,,&SCALE=30000000&FORMAT=image%2Fpng%3B+PhotometricInterpretation%3DRGB&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&EXCEPTIONS=INIMAGE

If I read the Linfinti blog post right, this is the kind of image they'd like to avoid fetching unnecessarily. If you look closely at a HTTP request for the legend image (passing the -v option to curl, with [1] standing in for the request in the code below), you see:

krusty-2:~ seang$ curl -v [1] > /tmp/legend.png
* About to connect() to frameworkwfs.usgs.gov port 80 (#0)
*   Trying 76.10.128.162... connected
* Connected to frameworkwfs.usgs.gov (76.10.128.162) port 80 (#0)
> GET /framework/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.2... HTTP/1.1
> User-Agent: curl/7.19.5 (i386-apple-darwin9.8.0) libcurl/7.19.5 OpenSSL/0.9.8k zlib/1.2.3
> Host: frameworkwfs.usgs.gov
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 18 Feb 2010 21:26:03 GMT
< Server: Apache/2.0.46 (Red Hat)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: image/png

No validator headers. No freshness headers. Neither httplib2 nor a HTTP caching proxy is going to be useful here. You're on your own. Hurrah for standards. Let a thousand ad-hoc caching schemes bloom.

Comments

Re: Saving bandwidth takes two

Author: geographika

The site you give as an example even causes the logo to be downloaded on each map request, so implementing caching on the WMS images was unlikely.

Surely there needs to be a thoudand different ways of setting up caching, as a WMS service can be used in many different ways - static background maps, weekly data updates, constantly changing data etc.

Re: Saving bandwidth takes two

Author: Sean

The web's own caching mechanisms serve all those use cases and more. A WxS can do real harm to developers in making them reinvent it.

Is really WxS to blame here ?

Author: Jérôme Andrieux

It seems to me you are blaming, all at once :

- the WS standard

- the WS implementation(s)

- the architecture of this particular site

Of course, OGC WxS standards could benefit a restful approach (I mean /a/unique/path/to/resources) when it comes to efficient web cache policy with frontal and external HTTP cache managers (like Varnish, nginx, ...).

However, in this particular case, isn't the WxS implementation to blame first ?

As someone said above, this particular doesn't pay to much attention to cache & compression policies anyway.

Re: Saving bandwidth takes two

Author: Sean

The WMS 1.3 spec has two sentences regarding HTTP headers. One suggests that a service should provide caching headers if possible, but there's no guidance at all for implementers. IMO, that is a failing of the spec, and as a consequence we see people needlessly inventing their own caching schemes.

Examples of WxS servers that do support web caching are very welcome.