Sean Gillies (Posts about design)https://sgillies.net/tags/design.atom2023-12-31T01:26:25ZSean GilliesNikolaPython at FOSS4G 2014https://sgillies.net/2014/09/15/python-at-foss4g-2014.html2014-09-15T00:00:00-06:002014-09-15T00:00:00-06:00Sean Gillies<p>There were plenty of other Python talks at FOSS4G and I plan to watch them when
the videos are online (<strong>update</strong>: talks are appearing now at
<a class="reference external" href="http://vimeo.com/foss4g">http://vimeo.com/foss4g</a>). I haven't been aware of <a class="reference external" href="https://github.com/sourcepole/ogrtools">ogrtools</a>, which is unlucky because there's
plenty of functional overlap between it and Fiona. The designs seem rather
different because Fiona doesn't emulate XML tool chains (GDAL's VRTs are not
unlike XSLT) and is more modular. For example, where ogrtools has
a file-to-file <code class="docutils literal">ogr translate</code> command, Fiona has a <code class="docutils literal">fio dump</code> and <code class="docutils literal">fio
load</code> pair connected by a stream of GeoJSON objects. The ogrtools talk is
right near the top of my list of talks to see.</p>
<p>I was very fortunate to go right after Mike Bostock's keynote. It got people
thinking about tools and design, and that's exactly the conversation that I'm
trying to engage developers in with Fiona and Rasterio, if with less insight
and perspective than Mike. I reminded attendees that the best features of our
day-to-day programming languages are sometimes disjoint and showed this diagram
(in which C is yellow, Javascript is magenta, and Python is blue. By "GC"
I mean garbage collection and by "{};" I mean extraneous syntax).</p>
<img alt="https://sgillies.github.io/foss4g-2014-fiona-rasterio/img/py-js-c.png" src="https://sgillies.github.io/foss4g-2014-fiona-rasterio/img/py-js-c.png" style="width: 600px;">
<p>D3 embraces browser standards and all they entail (a world wide knowledge
base and continuous performance improvements) and Fiona and Rasterio embrace
the good parts of Python. Written as C, like we usually see in GDAL/OGR
examples on the web, Python is quite slow. Idiomatic Python, including the
good parts like list comprehensions, generators, and iterators, is dramatically
faster. While Fiona and Rasterio don't do particular operations faster than the
older GDAL and OGR bindings (because it's the same C library underneath), they
are designed from the bottom up for a good fit with more efficient idiomatic
Python code.</p>
<p>I plugged <a class="reference external" href="http://click.pocoo.org/3/">Click</a> and <a class="reference external" href="http://cython.org">Cython</a> in my talk, too, and discussed them afterwards. I found
tons of interest in Python at FOSS4G and lots of good ideas about how to use
it.</p>
<p>I confess that I didn't pay a lot of attention to the talk schedule before the
conference. My summer was kind of nuts and I don't subscribe to any OSGeo
lists. When I did look closely I was surprised to find that many people were
giving two talks and some three. If any woman or first-timer didn't get
a chance to speak while some dude got three (and the multiple talkers were all
men and long time attendees as far as I can tell) – that's a bug in the talk
selection that needs to be fixed before the next edition.</p>
<p>Lastly, I think the views of Mount Hood you get when flying in and out of PDX to
destinations south and east are worth the airfare all by themselves.</p>
<a class="reference external image-reference" href="https://www.flickr.com/photos/by-sgillies/15249959145/"><img alt="https://farm6.staticflickr.com/5587/15249959145_91e47b3444_c_d.jpg" src="https://farm6.staticflickr.com/5587/15249959145_91e47b3444_c_d.jpg" style="width: 600px; height: 800px;"></a>Teaching Python GIS users to be more rationalhttps://sgillies.net/2013/12/17/teaching-python-gis-users-to-be-more-rational.html2013-12-17T00:00:00-07:002013-12-17T00:00:00-07:00Sean Gillies<p>We, developers and documenters of GIS software for Python, have been teaching
users to code irrationally. I'm certain that there's a cost to this and that we
need to try to undo the damage as much as we can. Irrational coding happens in
the real world. I'm largely self-taught as a programmer and have a long history
of using and misusing software that I don't fully understand, cargo-culting
code, etc. I've been there. This is part of the reason we write software, to
put power in the hands of less expert computer users, and it seems to
necessarily invite the problem of irrational programming. When software is
baffling rather than enlightening, it can make the problem worse. Software
ought to instead teach its users to think more rationally and reward them for
doing so. In theory, there ought to be a virtuous circle here for open source
software: rational users providing better bug reports and more effective
community support, and becoming code contributors at a higher rate. Teaching
users to be irrational, on the other hand, guarantees a steady and soul-sucking
stream of frequently asked low grade support questions.</p>
<p>I'm thinking about two things going on in Python GIS software in particular.
Exhibit A is the practice of teaching users to use <code class="docutils literal">del</code> to close or finalize
externalize resources like database connections and cursors. See
<a class="reference external" href="http://gis.stackexchange.com/search?q=%5Bpython%5D+%22+del+%22">http://gis.stackexchange.com/search?q=%5Bpython%5D+%22+del+%22</a>. Think hard
about this when you're developing, documenting, and supporting Python APIs. Do
you really want your users to have to know so much about the details of <code class="docutils literal">del</code>
or what happens to Python objects after they employ <code class="docutils literal">del</code>? And whether it's
dependent on their interpreter's garbage collection? Do you really want to
teach them to rely on side-effects of <code class="docutils literal">del</code>?</p>
<p>Python's <code class="docutils literal">del</code> statement is all about <a class="reference external" href="http://docs.python.org/2/reference/simple_stmts.html#the-del-statement">namespace and container management</a> and
has nothing directly to do with finalizing external resources:</p>
<blockquote>
<p>Deletion of a name removes the binding of that name from the local or
global namespace, depending on whether the name occurs in a global
statement in the same code block. If the name is unbound, a NameError
exception will be raised.</p>
<p>It is illegal to delete a name from the local namespace if it occurs as
a free variable in a nested block.</p>
<p>Deletion of attribute references, subscriptions and slicings is passed to
the primary object involved; deletion of a slicing is in general equivalent
to assignment of an empty slice of the right type (but even this is
determined by the sliced object).</p>
</blockquote>
<p>Think about this too: will <code class="docutils literal">del</code> work to close and finalize external stuff if
the name you're deleting is not in a local or global namespace, but is a list
or dict item? Or if you've bound two names to the same object do you have to
<code class="docutils literal">del</code> both names or just one?</p>
<p>Any closing of external resources is a side effect, and side effects in
software are almost always less than deterministic in nature. Teaching users to
rely on side effects is, in effect, teaching them to program irrationally.
Here's a user (not a GIS user, either) who has been particularly harmed, IMO:
<a class="reference external" href="http://stackoverflow.com/a/12626643/159235">http://stackoverflow.com/a/12626643/159235</a>.</p>
<p>The reason that <code class="docutils literal">del</code> works at all when it does is that the object to which
the name was bound has a __del__() method that calls another method that
finalizes. Just make that other method part of your API, document it well, and
you have users programming rationally and getting rational results like
external files and resources closed and freed precisely when they should be.</p>
<p>To their credit, the ArcPy developers have corrected this in recent versions,
but any visit to GIS StackExchange shows that the damage caused to programmers
continues.</p>
<p>Exhibit B is the practice of teaching users to rebind names to <code class="docutils literal">None</code> to
finalize, as in <a class="reference external" href="http://trac.osgeo.org/gdal/wiki/PythonGotchas#CertainobjectscontainaDestroymethodbutyoushouldneveruseit">GDAL/OGR Python bindings</a>.
Before you write an API like this or write a tutorial or book chapter that
advocates this, ask yourself what is so special about <code class="docutils literal">None</code> in this case?
Will rebinding that name to <code class="docutils literal">0</code>, or <code class="docutils literal">True</code>, or <code class="docutils literal">"abracadabra!"</code> work just
as well? Why or why not? A library developer or teacher or writer should know
the answer to this and be able to explain it.</p>
<p><code class="docutils literal">None</code> truly <a class="reference external" href="http://python-history.blogspot.com/2013/11/story-of-none-true-false.html">is special</a>
in Python (just compare the results of <code class="docutils literal">None = 1</code> and <code class="docutils literal">False = 1</code> at your
interpreter prompt to see), but is its special nature the thing that causes
finalization, or is it the binding, or what? What is actually going on in the
software and how do we teach users to use it rationally?</p>
<p>In fact, the reason rebinding to <code class="docutils literal">None</code> works when it does is the same as
above: some finalizing method is called by the interpreter when the object is
no longer referenced by anything else. Just expose that method to programmers
and you have a less baffling API.</p>
<p>In other words, <a class="reference external" href="http://www.python.org/dev/peps/pep-0020">explicit is better than implicit</a>. Teach users to close and finalize
external resources explicitly instead of teaching them to do something else
that has finalization as a side effect, and let's see if we all profit from more
rational Python GIS programmers.</p>