Implications of WMTS for S3 tiles
Check out this interesting post by Randy George concerning S3 map tiles for DeepEarth:
The project also includes an example showing how to set up a local tile set. The example uses 256×256 tiles but not in the OSGeo TMS directory structure. Here is an example using this DeepEarth local example TileProvider DeepEarth BlueMarble. You can see the tiles spin in from a local directory store on the server. The resolution is not all that great, but a full resolution BlueMarble isn’t that hard to get from BitTorrent. The alternative selection “Blue Marble Web” tiles are full 500m resolution hosted on Amazon S3 courtesy of ModestMaps.org. The Amazon S3 bucket is a flat structure, in other words the buckets don’t have an internal directory tree, which is why the tiles are not stored in a TMS directory tree.
The DeepEarth local TileProvider was easily adapted to suit the TMS directory so I could then directly pull in my El Paso tiles and show them with a DeepEarth interface. However, if I wished to take advantage of the high availability low latency S3 storage, I would need to flatten the tile tree. In S3 subdirectory trees are hiddeen inside buckets as file names. In the S3 case the tile names include the zoom level as a prefix: http://s3.amazonaws.com/com.modestmaps.bluemarble/5-r8-c24.jpg The 5 on the 5-r8-c24 nomenclature is the zoom level while row is 8 and column 24 at that level. TMS would encode this in a tree like this: ../bluemarble/5/24/8.jpg. The zoom level= subdirectory, row = subdirectory name, and column = name of tile. The beauty of TileProvider class in DeepEarth is that minor modifications can adapt a new TileProvider class to either of these encoding approaches.
Performance and reliability is a lot nicer on an Amazon S3 delivery, especially with heavy use. Once in S3 a map could also be promoted to a CloudFront edge cache without much difficulty. I imagine that would only make sense for the upper heavily used zoom levels, say 1-5 for BlueMarble. Once down in the level 6 part of the pyramid the number of tiles start escalating dramatically and repetitive hit rates are less likely.
The OGC's proposed Web Map Tiling Service standard's strict tile resource URL specification (in effect, just another form of RPC) obstructs you from using S3 tiles directly. That's a problem.
Update (2009-03-20): It's pointed out in comments that I'm wrong about the obstruction, and that you can in fact simulate the proposed WMTS tile resource hierarchy in S3. I'm not wrong about the RPC-ness of the candidate spec, something that can yet be fixed.
Comments
Re: Implications of WMTS for S3 tiles
Author: Richard Marsden
I've also thought of using S3/Cloudfront for map tiles, and it has been asked about on the forums of my site.
It would be a relatively simple and cost effective way of creating a tile server that didn't involve the hassles of building & installing something like MapServer.
</p><p> The proposed URL standard could be a problem, but would this simply cause a pragmatic variation of the standard to develop on its own?</p>
Re: Implications of WMTS for S3 tiles
Author: Michal Migurski
I'm surprised they're not providing for arbitrary positions of the x/y/z bits. The S3 URL given as an example above looks the way it does because I didn't want to build a directory tree in the filesystem prior to uploading - there's no reason it can't look like the proposed standard if I understand it correctly, but the "(z)-r(y)-c(x).jpg" has been convenient: http://oakland-1877.s3.amazonaws.com/14-r6331-c2627.jpg, http://oakland-1967.s3.amazonaws.com/14-r6330-c2628.jpg, etc. It's worth remembering that the "/" is a perfectly valid character in S3 object IDs.
We've also used a more VEarth-like quadtree style: http://road.tiles.map.london2012.com/a/03/13/13/13/11/13/13/3.jpg
Re: Implications of WMTS for S3 tiles
Author: Sean
I didn't know about the slashes. That's interesting, but possibly misleading because buckets can't contain other buckets, and objects can't contain other objects. The object foo/bar/x isn't really contained in foo/bar.
The suggestion I've made to the WMTS authors would make it easier to link to your existing "(z)-r(y)-c(x).jpg" tiles.
Re: Implications of WMTS for S3 tiles
Author: Michal Migurski
Thanks for making that suggestion! As far as S3 is concerned, the semantics of the slashes is actually quite fluid. The fact that they give the appearance of containment is completely separate from the implementation behind the scenes. S3 provides ways to browse buckets with a client-provided delimiter, so you can pretend there's a full directory structure there. The more I work with S3, the more impressed I am with their design decisions.
Re: Implications of WMTS for S3 tiles
Author: Joan Maso
The use of a flat structure like
.../5-r8-c24.jpg
in a prerendered very large and detailed layer is not a good idea.
You could end with thousands of files in the same directory. Common
operating systems do not deal well with so many files a in single
directory.
Splitting scales and rows in different directories helps in these situations.
Re: Implications of WMTS for S3 tiles
Author: Michal Migurski
Joan: on S3, objects aren't files, necessarily - I don't actually know what they are, it doesn't seem to be a relevant detail. They aren't files in TileCache's ".../5/24/8.jpg" scheme, either. URL structure doesn't need to bear any resemblance to underlying implementation. In particular, I'm loving Sean's term "hierarchical URL fetishism" in the linked discussion: http://groups.google.com/group/geo-web-rest/browse_thread/thread/fa0633b1dd009c02?hl=en
Re: Implications of WMTS for S3 tiles
Author: Sean
Like Michal says: how tiles are stored is an implementation detail. It's completely orthogonal to REST concerns.