Sean Gillies (Posts about shapely)

Shapely 1.8.0

Sean Gillies — Tue, 26 Oct 2021 14:29:27 GMT

Shapely 1.8.0 is a transitional version. There are a few bug fixes and new features, but it largely exists to warn about the upcoming changes in 2.0.0. See the new migration guide for more details on how to update your code from Shapely 1.8 to 2.0.

One of the upcoming changes: the geometry objects of 2.0 will be immutable, simplifying their implementation, and making it possible to hash them. Here's an example of the warning you'll see when you set a new attribute on a version 1.8 geometry object.

>>> from shapely.geometry import Point
>>> pt = Point(0, 0)
>>> pt.id = 1
<stdin>:1: ShapelyDeprecationWarning: Setting custom attributes on geometry objects is deprecated, and will raise an AttributeError in Shapely 2.0

Shapely 2.0 roadmap

Sean Gillies — Sat, 02 May 2020 22:25:34 GMT

There's going to be a 2.0 version of shapely. Joris Van den Bossche has written an RFC for a roadmap.

Joris is a Pandas and GeoPandas developer. The GeoPandas team has identified some enhancements to shapely that would allow GeoPandas and shapely to reach new levels of performance, and we're going to implement them. A NumFOCUS grant will support some of this work.

I'm super late in blogging about this and commenting on the RFC, which has lead to a little confusion about whether I'm in favor of this. I am, 100 percent. Rewrites aren't easy, but this effort has a lot going for it: serious use cases, committed stakeholders, clever and pragmatic programmers, funding, and lots of goodwill. We're going to have a 2.0 version that removes cruft, adds killer new features, and isn't super difficult to migrate to.

Please read the RFC pull request and comment if you're into helping shape the future of Python's #1 open source GIS geometry package.

Shapely 1.7.0

Sean Gillies — Wed, 05 Feb 2020 02:20:15 GMT

At long last, Shapely 1.7.0 is done. We have wheels on PyPI, wheels that include GEOS 3.8.0, wheels for Manylinux1, OS X, and Windows. The changes since 1.6.4 are all listed in the change log.

There are now 96 people named in Shapely's credits. Special 1.7.0 thanks to Joris Van den Bossche and the GeoPandas project for pre-release testing and to Mike Taves for putting the finishing touches on the project's AppVeyor configuration and Windows wheel building infrastructure.

Stay tuned for more news.

Fix for Shapely's GEOS library loading bug in 1.7a2

Sean Gillies — Sun, 23 Jun 2019 19:54:00 GMT

Including GDAL and GEOS in binary wheels that go to the Python Package Index (PyPI) has been good for users of fiona, rasterio, and shapely, but has also exposed Python GIS programmers on OS X to a perplexing bug. Shapely's issue #553 has been weighing on me for a long time. It's been hard to understand, hard to explain, and bites very hard. The short program below will trigger the bug on any version of OS X with any version of fiona and any version of shapely before 1.7a2 which is installed using a wheel from PyPI.

# crash.py

import fiona
from shapely.geometry import Point
from shapely.ops import unary_union


print("Buffering points to create polygons")
SHAPES = [Point(i, i).buffer(1.5) for i in range(20)]

print("Computing union of polygons")
union = unary_union(SHAPES)

print("Union: {}".format(union))

For example:

$ python crash.py
Buffering points to create polygons
Computing union of polygons
Assertion failed: (!static_cast<bool>("should never be reached")), function itemsTree, file AbstractSTRtree.cpp, line 373.
Abort trap: 6

The unary_union function uses a GEOS STRtree and an assertion that should never be reached is in fact reached and the program aborts.

There has been a work-around – import shapely before importing other modules that are dynamically linked with GEOS, such as fiona and rasterio – but that's smelly and hard to ensure in a complex project.

The problem exists in a small niche: one operating system and one kind of program, one that that loads GEOS twice in different ways. Once when fiona's extension modules are loaded and cause the linked GEOS library to be loaded as with any compiled program, and once again when shapely is imported and it calls dlopen from python to load GEOS. I've found no diagnosis or solution of a similar problem on the web. Was the problem in GEOS? Was it in the library files that I built for the fiona, rasterio, and shapely wheels? Was the bug in delocate, the program I use to find dependencies and bundle them into the wheels? I asked GEOS developers and folks at work, to no avail. I was stumped for a long time.

I asked Even Rouault about the issue and he suggested a solution might be found in being more careful about using the correct mode with dlopen and he offered some example Python code that included this: handle = CDLL(None). My first thought was "can we use the ctypes CDLL class like this?" The docs say that the first argument is a path to a library file and that's how I've been using it in Shapely.

While looking for guidance on passing None to the CDLL constructor, I found some commented code in https://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi.

# CDLL(None) invokes dlopen(NULL), which loads the currently running
# process - in our case Python itself. Since Python is linked with
# libc, readdir_r will be found there.
# Alternatively, we can just explicitly load 'libc.so.6'.
lib = CDLL(None)

The OS X dlopen man page doesn't exactly say that,

dlopen() examines the mach-o file specified by path. If the file is compatible with the current process and has not already been loaded into the current process, it is loaded and linked. After being linked, if it contains any initializer functions, they are called, before dlopen() returns. dlopen() can load dynamic libraries and bundles. It returns a handle that can be used with dlsym() and dlclose(). A second call to dlopen() with the same path will return the same handle, but the internal reference count for the handle will be incremented. Therefore all dlopen() calls should be balanced with a dlclose() call.

If a null pointer is passed in path, dlopen() returns a handle equivalent to RTLD_DEFAULT.

but the POSIX spec is clear that dlopen(NULL) returns a handle to a global symbols object and that's what I observe on my MacBook. Here's an interpreter session where I import fiona and look in the global symbols object for a GDAL C API function.

>>> import fiona
>>> from ctypes import CDLL
>>> handle = CDLL(None)
>>> handle.GDALGetDriverCount
<_FuncPtr object at 0x111a3eb38>
>>> handle.GDALGetDriverCount()
187

When fiona is imported, Python calls dlopen with the path to fiona's extension (.so) modules, and as these are liked with libgdal, the GDAL library is loaded. Now, libgdal is linked with libgeos_c, so will we find GEOS C API functions in the global symbols object?

>>> handle.initGEOS
<_FuncPtr object at 0x111a3fb38>

Yes. This changes everything. The problem that has bedeviled shapely users is caused by using dlopen to load a library that's already been loaded. We can avoid the problem by detecting whether libgeos_c has already been loaded and skipping the troublesome dlopen call. I've done this for shapely and made a new 1.7 pre-release. Please considering running pip install -U --pre shapely to try it out.

The OS X man page for dlsym says that looking up functions in the global handle is the slowest possible approach, but ctypes does cache the addresses of functions it finds and my test programs don't run measurably slower, so this just might work out.

I'd like to understand why this problem doesn't also occur on Linux. Is it because of differences in the library loaders or the executable file formats themselves? Is there subtle platform-specific bug in GEOS? I'll write more if I get to the bottom of this.

Shapely 1.7a1

Sean Gillies — Mon, 30 Jul 2018 16:47:05 GMT

The first pre-release for Shapely 1.7 is up on PyPI now. I'd be grateful if you would pip install shapely==1.7a1 and try it out. Since 1.6 there have been a few significant changes. Still no Windows wheels, but we're working on it.

Shapely 1.6.3

Sean Gillies — Sun, 10 Dec 2017 15:19:53 GMT

Shapely 1.6.3 is on PyPI and conda-forge now. It fixes 3 bugs involving empty or invalid geometries and adds to the manual a note about potentially confounding effects of z coordinate values on topological validity of geometric objects.

GIS Stack Exchange at its best

Sean Gillies — Tue, 05 Dec 2017 17:20:11 GMT

A clear and intriguing question, no overzealous moderation, and an excellent answer. This is the GIS Stack Exchange at its best.

Trued Macosx Wheels

Sean Gillies — Tue, 31 Oct 2017 14:02:48 GMT

I've woken up from my Nightmare on C++ Street and have uploaded new Rasterio, Fiona, and Shapely macosx wheels to PyPI. Use the pip requirement specs below to get the latest and greatest compatible wheels.

rasterio==1.0a11
fiona==1.7.10.post1
shapely==1.6.2.post1

My Xcode command line tools were out of sync with my libraries, libc++abi.dylib in particular. In the Xcode preferences there's an option to select versions of the command line tools. On my home computer, I've got Xcode 7.2.1 and have selected version 7.2.1 of the command line tools.

On my work computer, where I've been building the wheels, I somehow ended up with Xcode 8.2.1 and version 9.0.0 of the command line tools. I've never even seen this preference before, but do remember accepting a command line tools upgrade from Apple's App Store a while back. Reverting to command line tools version 8.2.1 cleared up the segmentation faults reported and studied in https://github.com/sgillies/frs-wheel-builds/issues/20.

I hope this blog post finds the next person to trip over this Xcode and command line tools mis-configuration.

Now I'm going to go back into PyPI and remove the broken binary wheels for older versions, leaving only the source distributions.

Wobbly Fiona and Shapely Wheels

Sean Gillies — Fri, 27 Oct 2017 18:54:47 GMT

On 2017-06-05 I began to use GDAL 2.2.0 in the Fiona binary wheels I upload to the Cheeseshop. I also changed the MACOSX_DEPLOYMENT_TARGET environment variable from 10.6 to 10.9 in order to get GDAL 2.2.0 to compile. This seems to be the cause of Shapely and Fiona wheel compatibility issues. Have I mentioned that I'm not completely competent at C++ on OS X? I'm not.

My co-worker Matt Perry provided a script that exposed this issue clearly and I bisected in Fiona and Shapely version space until I found the root of the problem.

Until I get new wheels on PyPI, here are the three coping strategies.

Avoid the wheels entirely:

pip install -I --no-binary fiona,shapely fiona==1.7.10 shapely==1.6.2

This is what I do in my Dockerfiles at work. This strategy requires the GDAL and GEOS libraries and headers to be pre-installed on your system, and also Numpy and Cython.

Downgrade Fiona, using wheels:

pip install -I fiona==1.7.6

Downgrade Shapely, using wheels:

pip install -I shapely==1.6b4

The issue affects Rasterio, too. Rasterio version 1.0a9 wheels should be safe with any version of Shapely.

I'll have an announcement about new wheels soon.

Update (2017-10-31): the nightmare is over.

Current status

Sean Gillies — Wed, 30 Aug 2017 13:29:08 GMT

I've been neglecting my open source projects this summer due to time constraints. At Mapbox we've been onboarding a new President and COO and a raft of new employees, plus a manager for my team at Mapbox. These are hugely positive developments, but have also been a big lift. Moving back to Colorado and bringing our lives here out of suspension has similarly taken all of my personal time. I've been the blocker for new releases of Fiona, Rasterio, and Shapely all summer long and have been feeling rather guilty about it.

Things are looking up now. My kids are back in school and have seen their doctor and dentist. Their schedule of soccer practices and other activities for the season is getting settled quickly. I'm resuming weekly yoga and gym workouts along with my existing running schedule. I like having a weekly routine; it helps me stay relaxed and gives me time for personal projects like writing and computering.

Open source continues to be a big part of my job and as my share of the onboarding lift eases I've been able to increase my time on Fiona, Rasterio, and Shapely. I released the long overdue Shapely 1.6.0 and have supported the GeoPandas team on getting 0.3.0 out. Fiona 1.7.9 was the first bug fix release of that project since June and I'm happy to have that in user hands. This week I'm working on coding and writing about the upcoming Rasterio release. I feel like I'm doing a good job as an open source maintainer and mentor again and am excited about what I'll be able to do in the next few months.

None of these projects would be viable without the help of other developers. To my open source collaborators: thanks for hanging in there and being patient with me!