Sean Gillies (Posts about features)https://sgillies.net/tags/features.atom2023-12-31T01:26:25ZSean GilliesNikolaUnix style spatial ETL with fio cat, collect, and loadhttps://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html2014-10-05T00:00:00-06:002014-10-05T00:00:00-06:00Sean Gillies<p>In Fiona 1.4.0 I added a fio-cat command to the CLI which works much UNIX cat.
It opens one or more vector datastets, concatenating their features and
printing them to stdout as a sequence of GeoJSON features.</p>
<div class="code"><pre class="code console"><a id="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-1" name="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-1" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-1"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp <span class="p">|</span> head -n <span class="m">2</span>
<a id="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-2" name="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-2" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-2"></a><span class="go">{"geometry": {"coordinates": [...], "type": "Polygon"}, "id": "0", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}</span>
<a id="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-3" name="rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-3" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3f5eafa7c2fc46ac8f698b6aa97d7424-3"></a><span class="go">{"geometry": {"coordinates": [...], "type": "Polygon"}, "id": "1", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}</span>
</pre></div>
<p>I've replaced most of the coordinates with ellipses to save space in the code
block above, something I'll continue to do in examples below.</p>
<p>I said that fio-cat concatenates features of multiple files and you can see
this by using <code class="docutils literal">wc <span class="pre">-l</span></code>.</p>
<div class="code"><pre class="code console"><a id="rest_code_6c265843309349ea934188ccedb97125-1" name="rest_code_6c265843309349ea934188ccedb97125-1" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_6c265843309349ea934188ccedb97125-1"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp <span class="p">|</span> wc -l
<a id="rest_code_6c265843309349ea934188ccedb97125-2" name="rest_code_6c265843309349ea934188ccedb97125-2" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_6c265843309349ea934188ccedb97125-2"></a><span class="go"> 48</span>
<a id="rest_code_6c265843309349ea934188ccedb97125-3" name="rest_code_6c265843309349ea934188ccedb97125-3" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_6c265843309349ea934188ccedb97125-3"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp docs/data/test_uk.shp <span class="p">|</span> wc -l
<a id="rest_code_6c265843309349ea934188ccedb97125-4" name="rest_code_6c265843309349ea934188ccedb97125-4" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_6c265843309349ea934188ccedb97125-4"></a><span class="go"> 96</span>
</pre></div>
<p>If you look closely at the output, you'll see that every GeoJSON feature is a
standalone text and each is preceded by an ASCII RS (0x1E) control character.
These allow you to cat pretty-printed GeoJSON (using the <code class="docutils literal"><span class="pre">--indent</span></code> option)
containing newlines that can still be understood as a sequence of texts by
other programs. Software like Python's json module and Node's underscore-cli
will trip over unstripped RS, so you can disable the RS control characters and
emit LF delimited sequences of GeoJSON (with no option to pretty print, of
course) using <code class="docutils literal"><span class="pre">--x-json-seq-no-rs</span></code>.</p>
<p>To complement fio-cat I've written fio-load and fio-collect. They
read features from a sequence (RS or LF delimited) and respectively write them
to a formatted vector file (such as a Shapefile) or print them as a GeoJSON
feature collection.</p>
<p>Here's an example of using fio-cat and load together. You should tell fio-load
what coordinate reference system to use when writing the output file because
that information isn't carried in the GeoJSON features written by fio-cat.</p>
<div class="code"><pre class="code console"><a id="rest_code_faee96aa67a9455983e887f8335412ed-1" name="rest_code_faee96aa67a9455983e887f8335412ed-1" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-1"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp <span class="se">\</span>
<a id="rest_code_faee96aa67a9455983e887f8335412ed-2" name="rest_code_faee96aa67a9455983e887f8335412ed-2" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-2"></a><span class="p">|</span> fio load --driver Shapefile --dst_crs EPSG:4326 /tmp/test_uk.shp
<a id="rest_code_faee96aa67a9455983e887f8335412ed-3" name="rest_code_faee96aa67a9455983e887f8335412ed-3" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-3"></a><span class="gp">$ </span>ls -l /tmp/test_uk.*
<a id="rest_code_faee96aa67a9455983e887f8335412ed-4" name="rest_code_faee96aa67a9455983e887f8335412ed-4" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-4"></a><span class="go">-rw-r--r-- 1 seang wheel 10 Oct 5 10:09 /tmp/test_uk.cpg</span>
<a id="rest_code_faee96aa67a9455983e887f8335412ed-5" name="rest_code_faee96aa67a9455983e887f8335412ed-5" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-5"></a><span class="go">-rw-r--r-- 1 seang wheel 11377 Oct 5 10:09 /tmp/test_uk.dbf</span>
<a id="rest_code_faee96aa67a9455983e887f8335412ed-6" name="rest_code_faee96aa67a9455983e887f8335412ed-6" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-6"></a><span class="go">-rw-r--r-- 1 seang wheel 143 Oct 5 10:09 /tmp/test_uk.prj</span>
<a id="rest_code_faee96aa67a9455983e887f8335412ed-7" name="rest_code_faee96aa67a9455983e887f8335412ed-7" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-7"></a><span class="go">-rw-r--r-- 1 seang wheel 65156 Oct 5 10:09 /tmp/test_uk.shp</span>
<a id="rest_code_faee96aa67a9455983e887f8335412ed-8" name="rest_code_faee96aa67a9455983e887f8335412ed-8" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_faee96aa67a9455983e887f8335412ed-8"></a><span class="go">-rw-r--r-- 1 seang wheel 484 Oct 5 10:09 /tmp/test_uk.shx</span>
</pre></div>
<p>And here's one of fio-cat and collect.</p>
<div class="code"><pre class="code console"><a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-1" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-1" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-1"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp <span class="p">|</span> fio collect --indent <span class="m">4</span> <span class="p">|</span> head
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-2" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-2" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-2"></a><span class="go">{</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-3" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-3" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-3"></a><span class="go"> "features": [</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-4" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-4" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-4"></a><span class="go"> {</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-5" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-5" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-5"></a><span class="go"> "geometry": {</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-6" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-6" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-6"></a><span class="go"> "coordinates": [</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-7" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-7" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-7"></a><span class="go"> [</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-8" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-8" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-8"></a><span class="go"> [</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-9" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-9" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-9"></a><span class="go"> 0.899167,</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-10" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-10" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-10"></a><span class="go"> 51.357216</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-11" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-11" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-11"></a><span class="go"> ],</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-12" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-12" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-12"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp <span class="p">|</span> fio collect --indent <span class="m">4</span> <span class="p">|</span> tail
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-13" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-13" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-13"></a><span class="go"> "CAT": 232.0,</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-14" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-14" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-14"></a><span class="go"> "CNTRY_NAME": "United Kingdom",</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-15" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-15" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-15"></a><span class="go"> "FIPS_CNTRY": "UK",</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-16" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-16" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-16"></a><span class="go"> "POP_CNTRY": 60270708.0</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-17" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-17" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-17"></a><span class="go"> },</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-18" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-18" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-18"></a><span class="go"> "type": "Feature"</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-19" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-19" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-19"></a><span class="go"> }</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-20" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-20" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-20"></a><span class="go"> ],</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-21" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-21" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-21"></a><span class="go"> "type": "FeatureCollection"</span>
<a id="rest_code_3647e1f4b50642f6973cd31d9f7f401d-22" name="rest_code_3647e1f4b50642f6973cd31d9f7f401d-22" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_3647e1f4b50642f6973cd31d9f7f401d-22"></a><span class="go">}</span>
</pre></div>
<p>Does it look like I've simply reinvented ogr2ogr? The difference is that with
fio-cat and fio-load there's space in between for programs that process
features. The programs could be written in any language. They might use
Shapely, they might use Turf. The only requirement is that they read and write
sequences of GeoJSON features using stdin and stdout. A nice property of
programs like these is that you can sometimes parallelize them cheaply using
<a class="reference external" href="http://www.gnu.org/software/parallel/parallel_tutorial.html#pipe">GNU parallel</a>.</p>
<p>The fio-buffer program (<a class="reference external" href="https://github.com/Toblerity/Fiona/blob/sequence-processing/fiona/fio/ops.py">unreleased</a>)
in the example below uses Shapely to calculate a 100 km buffer around features
(in Web Mercator, I know!). Parallel doesn't help in this example because the
sequence of features from fio-cat is fairly small, but I want to show you how
to tell parallel to watch for RS as a record separator.</p>
<div class="code"><pre class="code console"><a id="rest_code_fae77df74f83476b8a9498dee728ae5e-1" name="rest_code_fae77df74f83476b8a9498dee728ae5e-1" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_fae77df74f83476b8a9498dee728ae5e-1"></a><span class="gp">$ </span>fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 <span class="se">\</span>
<a id="rest_code_fae77df74f83476b8a9498dee728ae5e-2" name="rest_code_fae77df74f83476b8a9498dee728ae5e-2" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_fae77df74f83476b8a9498dee728ae5e-2"></a><span class="gp">> </span><span class="p">|</span> parallel --pipe --recstart <span class="s1">'\x1E'</span> fio buffer 1E+5 <span class="se">\</span>
<a id="rest_code_fae77df74f83476b8a9498dee728ae5e-3" name="rest_code_fae77df74f83476b8a9498dee728ae5e-3" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_fae77df74f83476b8a9498dee728ae5e-3"></a><span class="gp">> </span><span class="p">|</span> fio collect --src_crs EPSG:3857 <span class="se">\</span>
<a id="rest_code_fae77df74f83476b8a9498dee728ae5e-4" name="rest_code_fae77df74f83476b8a9498dee728ae5e-4" href="https://sgillies.net/2014/10/05/unix-style-spatial-etl-with-fio-cat-collect-and-load.html#rest_code_fae77df74f83476b8a9498dee728ae5e-4"></a><span class="gp">> </span><span class="p">|</span> geojsonio
</pre></div>
<p>Here's the result. Unix pipelines, still awesome at the age of <a class="reference external" href="http://en.wikipedia.org/wiki/Pipeline_(Unix)#History">41</a>!</p>
<script src="https://gist.github.com/sgillies/25dbd9a22269446fa40a.js">
</script><p>The other point of this post is that, with the <a class="reference external" href="https://datatracker.ietf.org/doc/draft-ietf-json-text-sequence/">JSON Text Sequence</a> draft
apparently going to publication, sequences of GeoJSON features not collected
into a GeoJSON feature collection are very close to being a real thing that
developers should be supporting.</p>Rasterio 0.5https://sgillies.net/2014/01/23/rasterio-0-5.html2014-01-23T00:00:00-07:002014-01-23T00:00:00-07:00Sean Gillies<div class="code"><pre class="code console"><a id="rest_code_cea2df37553746b98220e422d6da0544-1" name="rest_code_cea2df37553746b98220e422d6da0544-1" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_cea2df37553746b98220e422d6da0544-1"></a><span class="gp">$ </span>pip install <span class="s1">'rasterio>=0.5'</span>
</pre></div>
<p>Which is to say, share and enjoy.</p>
<p>Here's a script that shows off everything new in rasterio 0.5: <a class="reference external" href="http://sgillies.net/blog/2014/01/22/rethinking-driver-management-in-fiona-and-rasterio.html">GDAL driver
environments</a>,
raster feature sieving, and a generator of raster feature shapes.</p>
<div class="code"><pre class="code python"><a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-1" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-1" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-1"></a><span class="ch">#!/usr/bin/env python</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-2" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-2" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-2"></a><span class="c1">#</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-3" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-3" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-3"></a><span class="c1"># sieve: demonstrate sieving and polygonizing of raster features.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-4" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-4" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-4"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-5" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-5" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-5"></a><span class="kn">import</span> <span class="nn">subprocess</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-6" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-6" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-6"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-7" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-7" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-7"></a><span class="kn">import</span> <span class="nn">numpy</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-8" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-8" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-8"></a><span class="kn">import</span> <span class="nn">rasterio</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-9" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-9" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-9"></a><span class="kn">from</span> <span class="nn">rasterio.features</span> <span class="kn">import</span> <span class="n">sieve</span><span class="p">,</span> <span class="n">shapes</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-10" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-10" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-10"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-11" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-11" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-11"></a><span class="c1"># Register GDAL and OGR drivers.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-12" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-12" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-12"></a><span class="k">with</span> <span class="n">rasterio</span><span class="o">.</span><span class="n">drivers</span><span class="p">():</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-13" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-13" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-13"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-14" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-14" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-14"></a> <span class="c1"># Read a raster to be sieved.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-15" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-15" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-15"></a> <span class="k">with</span> <span class="n">rasterio</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s1">'rasterio/tests/data/shade.tif'</span><span class="p">)</span> <span class="k">as</span> <span class="n">src</span><span class="p">:</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-16" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-16" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-16"></a> <span class="n">shade</span> <span class="o">=</span> <span class="n">src</span><span class="o">.</span><span class="n">read_band</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-17" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-17" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-17"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-18" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-18" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-18"></a> <span class="c1"># Print the number of shapes in the source raster.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-19" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-19" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-19"></a> <span class="nb">print</span> <span class="s2">"Slope shapes: </span><span class="si">%d</span><span class="s2">"</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">shapes</span><span class="p">(</span><span class="n">shade</span><span class="p">)))</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-20" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-20" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-20"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-21" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-21" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-21"></a> <span class="c1"># Sieve out features 13 pixels or smaller.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-22" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-22" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-22"></a> <span class="n">sieved</span> <span class="o">=</span> <span class="n">sieve</span><span class="p">(</span><span class="n">shade</span><span class="p">,</span> <span class="mi">13</span><span class="p">)</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-23" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-23" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-23"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-24" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-24" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-24"></a> <span class="c1"># Print the number of shapes in the sieved raster.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-25" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-25" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-25"></a> <span class="nb">print</span> <span class="s2">"Sieved (13) shapes: </span><span class="si">%d</span><span class="s2">"</span> <span class="o">%</span> <span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">shapes</span><span class="p">(</span><span class="n">sieved</span><span class="p">)))</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-26" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-26" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-26"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-27" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-27" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-27"></a> <span class="c1"># Write out the sieved raster.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-28" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-28" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-28"></a> <span class="k">with</span> <span class="n">rasterio</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s1">'example-sieved.tif'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="o">**</span><span class="n">src</span><span class="o">.</span><span class="n">meta</span><span class="p">)</span> <span class="k">as</span> <span class="n">dst</span><span class="p">:</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-29" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-29" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-29"></a> <span class="n">dst</span><span class="o">.</span><span class="n">write_band</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">sieved</span><span class="p">)</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-30" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-30" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-30"></a>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-31" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-31" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-31"></a><span class="c1"># Dump out gdalinfo's report card and open (or "eog") the TIFF.</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-32" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-32" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-32"></a><span class="nb">print</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-33" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-33" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-33"></a> <span class="p">[</span><span class="s1">'gdalinfo'</span><span class="p">,</span> <span class="s1">'-stats'</span><span class="p">,</span> <span class="s1">'example-sieved.tif'</span><span class="p">])</span>
<a id="rest_code_402e067e39e24a9e92f5e28a09b52b16-34" name="rest_code_402e067e39e24a9e92f5e28a09b52b16-34" href="https://sgillies.net/2014/01/23/rasterio-0-5.html#rest_code_402e067e39e24a9e92f5e28a09b52b16-34"></a><span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">([</span><span class="s1">'open'</span><span class="p">,</span> <span class="s1">'example-sieved.tif'</span><span class="p">])</span>
</pre></div>
<p>The images that you can operate on with the new functions in rasterio.features
don't have to be read from GeoTIFFs or other image files. They could be purely
numpy-based spatial models or slices of multidimensional image data arrays
produced with scipy.ndimage.</p>
<p>For fun and benchmarking purposes I've written a program that uses both Fiona and rasterio to emulate GDAL's
gdal_polygonize.py: <a class="reference external" href="https://github.com/sgillies/rasterio/blob/master/examples/rasterio_polygonize.py">rasterio_polygonize.py</a>. If you're interested in integrating these modules, it's a good starting point.</p>