Sean Gillies (Posts about shapefile)
https://sgillies.net/tags/shapefile.atom
2023-12-31T01:26:24Z
Sean Gillies
Nikola
Fiona and shapefile encoding
https://sgillies.net/2017/12/17/fiona-and-shapefile-encoding.html
2017-12-17T12:20:00-07:00
2017-12-17T12:20:00-07:00
Sean Gillies
<p>It's a common problem in GIS to have a shapefile that was encoded using
a character set other than the standard iso-8859-1 but lacks any record of what
that character set was. Some shapefiles lie about their encodings. You could
make an educated guess at the character set. Your decoding may fail. Worse, it
may not fail, but contain garbage when printed out. The Japanese term for this
is <a class="reference external" href="https://en.wikipedia.org/wiki/Mojibake">Mojibake</a>. Ned Batchelder
introduced me to the term in his <a class="reference external" href="https://nedbatchelder.com/text/unipain.html">Pragmatic Unicode</a> presentation. It's easy to
demonstrate in Python.</p>
<div class="code"><pre class="code pycon"><a id="rest_code_a454c71939444667ae34f9b5f52ea19a-1" name="rest_code_a454c71939444667ae34f9b5f52ea19a-1" href="https://sgillies.net/2017/12/17/fiona-and-shapefile-encoding.html#rest_code_a454c71939444667ae34f9b5f52ea19a-1"></a><span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="sa">u</span><span class="s1">'dérangé'</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'iso-8859-1'</span><span class="p">))</span>
<a id="rest_code_a454c71939444667ae34f9b5f52ea19a-2" name="rest_code_a454c71939444667ae34f9b5f52ea19a-2" href="https://sgillies.net/2017/12/17/fiona-and-shapefile-encoding.html#rest_code_a454c71939444667ae34f9b5f52ea19a-2"></a><span class="go">dérangé</span>
</pre></div>
<p>Fiona's <code class="docutils literal">open()</code> function has an <code class="docutils literal">encoding</code> keyword argument that is
intended to let developers override the missing or erronenous information for
a shapefile. There's been a regression in Fiona recently and users began to
report unexpected mojibake symptoms. They were using the encoding argument
property but seeing garbage displayed. This regression has been fixed in Fiona
1.7.11. Upgrade as soon as you can.</p>
<p>For a time I was in disbelief that users were reporting a real problem.
I chalked this up to dirty data, compilation of GDAL, DLL Hell, anything but
a regression. In the end, I think what got me unstuck was facing this note <a class="reference external" href="https://github.com/Toblerity/Fiona/commit/044490d4562216cda63ca5704c4fb9046c765232#diff-0b6c1416b1293274c4e2d11fb49e1795R11">I
recently added to Fiona's issue template</a>:</p>
<blockquote>
<p>You think you've found something? We believe you.</p>
</blockquote>
<p>Something I added to make the project more friendly to first-time contributors
ended up being a note to myself.</p>
Zipped Shapefiles on the Web
https://sgillies.net/2017/11/11/fiona-and-zipped-shapefiles-on-the-web.html
2017-11-11T23:31:40-07:00
2017-11-11T23:31:40-07:00
Sean Gillies
<p>Here's a preview of a <a class="reference external" href="https://github.com/Toblerity/Fiona/pull/426">new feature coming in Fiona 1.8</a>.</p>
<div class="code"><pre class="code console"><a id="rest_code_b0f7c48ad3ca4c91a6c946553204770c-1" name="rest_code_b0f7c48ad3ca4c91a6c946553204770c-1" href="https://sgillies.net/2017/11/11/fiona-and-zipped-shapefiles-on-the-web.html#rest_code_b0f7c48ad3ca4c91a6c946553204770c-1"></a><span class="go">fio cat zip+s3://mapbox/rasterio/coutwildrnp.zip | fio collect | geojsonio</span>
</pre></div>
<figure>
<img alt="https://c1.staticflickr.com/5/4585/38320664512_35027691c8_b.jpg" src="https://c1.staticflickr.com/5/4585/38320664512_35027691c8_b.jpg">
</figure>