Sean Gillies (Posts about pixels)

Warping images with rasterio

Sean Gillies — Tue, 25 Feb 2014 07:00:00 GMT

Warping imagery from one spatial basis to another is a crucial function. I've been determined that rasterio be able to do this. Scipy provides a geometric_transform() function that I like, but I'd like rasterio to operate at a higher level and take advantage of GDAL's fast transformers. GDAL's gdalwarp program only operates on files, and I'm aiming lower than this (so rasterio can let us build better scripts). The ReprojectImage() function in osgeo.gdal, as pointed out in http://jgomezdans.github.io/gdal_notes/reprojection.html, only works on GDAL dataset objects. And the code to make them gets messy.

My requirements for a rasterio warper are:

Speed approaching gdalwarp.
Same results (same algorithms) as gdalwarp.
Sources and destinations can be ndarrays as well as bands in files.
Simplicity and clarity.

I think I've managed to satisfy these requirements. The warp module will be the key feature of the next rasterio release.

Here's an example within the interactive dataset inspector.

$ rasterio.insp rasterio/tests/data/RGB.byte.tif
Rasterio 0.7 Interactive Inspector (Python 2.7.5)
Type "src.name", "src.read_band(1)", or "help(src)" for more information.

The reproject() function takes source and destination ndarrays and an affine transformation vector and coordinate reference system for each. In this example, I'm warping the UTM source to a Web Mercator destination of slightly smaller extent. I computed the extents of the destination using pyplot, rasterio has yet no method to come up with an optimal destination extent (to do).

>>> source = src.read_band(1)
>>> dst_transform = [
...     -8789636.708,
...     300.0,
...     0.0,
...     2943560.235,
...     0.0,
...     -300.0]
>>> dst_crs = {'init': 'EPSG:3857'}
>>> destin = np.empty(src.shape, dtype=np.uint8)
>>> from rasterio import _warp
>>> _warp.reproject(
...     source, destin,
...     src.transform, src.crs,
...     dst_transform, dst_crs)
>>> import matplotlib.pyplot as plt
>>> plt.subplot(121)
<matplotlib.axes.AxesSubplot object at 0x1075362d0>
>>> plt.imshow(source)
<matplotlib.image.AxesImage object at 0x1078b1450>
>>> plt.gray()
>>> plt.subplot(122)
<matplotlib.axes.AxesSubplot object at 0x1075ffe50>
>>> plt.imshow(destin)
<matplotlib.image.AxesImage object at 0x1078b19d0>
>>> plt.gray()
>>> plt.savefig('/tmp/warp.png')

It's pretty much just three statements. One to get a source ndarray from a dataset. One to create a destination ndarray. One to call reproject().

I'm out of time to properly transform the axes in the plots to show coordinates in their proper CRS, but they're warped correctly.

Rasterio cookbook recipe number one

Sean Gillies — Fri, 14 Feb 2014 07:00:00 GMT

I got a nice message today from a developer who found rasterio to be the right tool for scaling the values of a 15 GB LiDAR DEM. Here's the code.

import numpy as np
import rasterio

"""
2014-02-13
Bryan Luman
Use it however you like at your own risk

Problem:
I have a huge DEM converted from LiDAR LAS points. I'd like to make
it slightly less massive for further processing.

All my z values are between 682 and 869 feet stored as float32. If
I convert them to centimeters and they will fit well within a 16 bit
unsigned integer.

This saves a significant amount of storage space and greatly reduces
any type of future analysis.

"""

CM_IN_FOOT = 30.48

with rasterio.drivers():
    with rasterio.open('massive_dem_as_float') as src:
        kwargs = src.meta
        kwargs.update(
            driver='GTiff',
            dtype=rasterio.uint16,
            count=1,
            compress='lzw',
            nodata=0,
            bigtiff='YES' # Output will be larger than 4GB
        )

        windows = src.block_windows(1)

        with rasterio.open(
                'less_massive_dem_with_as_int.tif',
                'w',
                **kwargs) as dst:
            for idx, window in windows:
                src_data = src.read_band(1, window=window)

                # Source nodata value is a very small negative number
                # Converting in to zero for the output raster
                np.putmask(src_data, src_data < 0, 0)

                dst_data = (src_data * CM_IN_FOOT).astype(rasterio.uint16)
                dst.write_band(1, dst_data, window=window)

This uses the windowed read and write and windows iterator features of rasterio that I worked on with Asger Skovbo Petersen.

Do you have any more short rasterio code examples? I'd love to start collecting these into a cookbook of sorts.

Rasterio 0.6

Sean Gillies — Mon, 10 Feb 2014 07:00:00 GMT

I uploaded Rasterio 0.6 to PyPI today. The new features are tags (aka GDAL metadata) and colormaps.

With just a couple lines of Python you can turn this simple greyscale image

into an eye-searing RGB color image.

Rasterio 0.5

Sean Gillies — Thu, 23 Jan 2014 07:00:00 GMT

$ pip install 'rasterio>=0.5'

Which is to say, share and enjoy.

Here's a script that shows off everything new in rasterio 0.5: GDAL driver environments, raster feature sieving, and a generator of raster feature shapes.

#!/usr/bin/env python
#
# sieve: demonstrate sieving and polygonizing of raster features.

import subprocess

import numpy
import rasterio
from rasterio.features import sieve, shapes

# Register GDAL and OGR drivers.
with rasterio.drivers():

    # Read a raster to be sieved.
    with rasterio.open('rasterio/tests/data/shade.tif') as src:
        shade = src.read_band(1)

    # Print the number of shapes in the source raster.
    print "Slope shapes: %d" % len(list(shapes(shade)))

    # Sieve out features 13 pixels or smaller.
    sieved = sieve(shade, 13)

    # Print the number of shapes in the sieved raster.
    print "Sieved (13) shapes: %d" % len(list(shapes(sieved)))

    # Write out the sieved raster.
    with rasterio.open('example-sieved.tif', 'w', **src.meta) as dst:
        dst.write_band(1, sieved)

# Dump out gdalinfo's report card and open (or "eog") the TIFF.
print subprocess.check_output(
    ['gdalinfo', '-stats', 'example-sieved.tif'])
subprocess.call(['open', 'example-sieved.tif'])

The images that you can operate on with the new functions in rasterio.features don't have to be read from GeoTIFFs or other image files. They could be purely numpy-based spatial models or slices of multidimensional image data arrays produced with scipy.ndimage.

For fun and benchmarking purposes I've written a program that uses both Fiona and rasterio to emulate GDAL's gdal_polygonize.py: rasterio_polygonize.py. If you're interested in integrating these modules, it's a good starting point.

Getting shapes of raster features with rasterio

Sean Gillies — Sat, 18 Jan 2014 07:00:00 GMT

I've just added a new function to rasterio that generates the shapes of all features in an image. The region labeling algorithm is provided by libgdal's GDALPolygonize() and I'm using GDAL and OGR in-memory raster and vector datastores to avoid disk I/O. Look, Ma, no temporary files.

In the interpreter session shown below, I made a 12 x 12 image with three different features: the background with value 1, a 4 x 4 square in the upper left corner with value 2, and a 2 x 2 square with value 5 on the left edge. Then I copied an affine transformation matrix from the rasterio test suite and passed it and the image to rasterio.features.shapes(). The result: the polygon boundaries of those three features in "world", not image, coordinates.

>>> import pprint
>>> import numpy
>>> from rasterio import features
>>> image = numpy.ones((12, 12), dtype=numpy.uint8)
>>> image[0:4,0:4] = 2
>>> image[4:6,0:2] = 5
>>> image
array([[2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1],
       [2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1],
       [5, 5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [5, 5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], dtype=uint8)
>>> t = [101985.0, 300.0379266750948, 0.0,
...      2826915.0, 0.0, -300.041782729805]
>>> for shape, value in features.shapes(image, transform=t):
...     print("Image value:")
...     print(value)
...     print("Geometry:")
...     pprint.pprint(shape)
...
Image value:
2
Geometry:
{'coordinates': [[(101985.0, 2826915.0),
                  (101985.0, 2825714.832869081),
                  (103185.15170670037, 2825714.832869081),
                  (103185.15170670037, 2826915.0),
                  (101985.0, 2826915.0)]],
 'type': 'Polygon'}
Image value:
5
Geometry:
{'coordinates': [[(101985.0, 2825714.832869081),
                  (101985.0, 2825114.7493036212),
                  (102585.0758533502, 2825114.7493036212),
                  (102585.0758533502, 2825714.832869081),
                  (101985.0, 2825714.832869081)]],
 'type': 'Polygon'}
Image value:
1
Geometry:
{'coordinates': [[(103185.15170670037, 2826915.0),
                  (103185.15170670037, 2825714.832869081),
                  (102885.11378002529, 2825714.832869081),
                  (102585.0758533502, 2825714.832869081),
                  (102585.0758533502, 2825114.7493036212),
                  (102285.0379266751, 2825114.7493036212),
                  (101985.0, 2825114.7493036212),
                  (101985.0, 2823314.4986072425),
                  (105285.41719342604, 2823314.4986072425),
                  (105585.45512010113, 2823314.4986072425),
                  (105585.45512010113, 2826915.0),
                  (103185.15170670037, 2826915.0)]],
 'type': 'Polygon'}

In its initial form, the function generated GeoJSON-ish feature dicts, but I decided these were too heavy and have switched to a pair of objects: one a GeoJSON style geometry dict and the other the labeled feature's characteristic image value.

This example doesn't read or write because I want to show how the new function doesn't require I/O. But of course you can always get the numpy array and affine transformation matrix from a raster on disk using rasterio.open() and read_band(), and the generated shapes can be written to a vector file on disk using fiona.open() and writerecords().

Rasterio windows and masks

Sean Gillies — Sat, 21 Dec 2013 07:00:00 GMT

Rasterio 0.4 is up on PyPI: https://pypi.python.org/pypi/rasterio/0.4. The new features since the last time I blogged about rasterio are windowed and masked reads and writes. Here's an example that combines both features and demonstrates a good pattern for reading raster data.

import numpy
import rasterio

with rasterio.open('rasterio/tests/data/RGB.byte.tif') as src:

    # The following asserts that all bands have the same block/stripe structure.
    assert len(set(src.block_shapes)) == 1

    # As they do, we can iterate over the first band's block windows
    # and apply to all other bands.
    for ji, window in src.block_windows(1):

        # Get data in this window for each band as a Numpy masked array.
        bands = [
            numpy.ma.masked_equal(
                src.read_band(src.indexes[i]),
                src.nodatavals[i])
            for i in range(src.count)]

        # Break out after the first block in this example.
        break

# Let's look at the first, blue band.
b = bands[0]
print(b)
print(b.mask)
print(b.fill_value)
print(b.min(), b.max(), b.mean())

The output:

[[-- -- -- ..., -- -- --]
 [-- -- -- ..., -- -- --]
 [-- -- -- ..., -- -- --]
 ...,
 [-- -- -- ..., -- -- --]
 [-- -- -- ..., -- -- --]
 [-- -- -- ..., -- -- --]]
[[ True  True  True ...,  True  True  True]
 [ True  True  True ...,  True  True  True]
 [ True  True  True ...,  True  True  True]
 ...,
 [ True  True  True ...,  True  True  True]
 [ True  True  True ...,  True  True  True]
 [ True  True  True ...,  True  True  True]]
0
(1, 255, 44.434478650699106)

You can see that all four corners of the test image are masked.

Is rasterio fast enough?

Sean Gillies — Fri, 13 Dec 2013 07:00:00 GMT

I've been working on a Python package named rasterio and have blogged about my goals: more Python idioms, no gotchas, and speed. I'm doing well on the first two, but now let's look at the last of these. Is rasterio fast enough?

Here is a raster band read to Numpy ndarray benchmark with a comparison to osgeo.gdal. The RGB.byte.tif file is a small (about 800 x 800 pixel) GeoTIFF file. I'm going to open the file, read the entirety of the first band to a ndarray, and then close the file. The gdal Dataset class has no explicit closing method, so I'm assigning the variable to None and indirectly freeing up the GeoTIFF file. In rasterio, you should call an opened file's close() method when done, or use a with statement.

import timeit

import rasterio
from osgeo import gdal

n = 100

# GDAL read as array
s = """
src = gdal.Open('rasterio/tests/data/RGB.byte.tif')
arr = src.GetRasterBand(1).ReadAsArray()
src = None
"""

t = timeit.timeit(s, setup='from osgeo import gdal', number=n)
print("GDAL:")
print("%f usec\n" % (t/n))

# Rasterio read band
s = """
with rasterio.open('rasterio/tests/data/RGB.byte.tif') as src:
    arr = src.read_band(1)
"""

t = timeit.timeit(s, setup='import rasterio', number=n)
print("Rasterio:")
print("%f usec\n" % (t/n))

Here are the results on my 1.7 GHz Intel Core i7 MacBook Air:

$ python benchmarks/ndarray.py
GDAL:
2.356141 ms

Rasterio:
2.655728 ms

I'm pleased that Rasterio is only about 15% slower for this benchmark. And the difference between the two will diminish as files grow larger. Rasterio does considerably more validation of input and logs quite a bit. Also, rasterio could be a little more lazy about getting and formatting a file's coordinate referencing system. Less code and fewer bugs is the main goal of rasterio, so getting close to gdal.py performance-wise is icing on the cake.

Introducing rasterio

Sean Gillies — Sun, 24 Nov 2013 07:00:00 GMT

Ever since I wrote Fiona, I've been asked if I have plans to do something similar for geospatial raster data. Having been out of the raster business for a few years, I always said "no serious plans, just blue-sky ideas." Today, I'm back in satellite image processing and very much want and need something like Fiona-for-rasters. Rasterio is my attempt to write such a Python package.

Whereas Fiona is about reading and writing GeoJSON-like objects from and to vector data files, rasterio exists to let Python programmers read and write Numpy arrays (or other array-like objects providing the PEP 3118 buffer interface) from and to raster data files. I intend rasterio to be fast and gotcha-free and let programmers do the things they do now with osgeo.gdal, but with familiar Python idioms and much less code.

I've just uploaded to PyPI rasterio 0.2. Share and enjoy.

Briefly, here's how you use rasterio to get the dimensions, number of bands, indexes of the bands, coordinate reference system, and coordinate transform matrix from a GeoTIFF.

>>> import rasterio
>>> with rasterio.open('rasterio/tests/data/RGB.byte.tif') as src:
...     print(src.width, src.height)
...     print(src.count)
...     print(src.indexes)
...     print(src.dtypes)
...     print(src.crs)
...     print(src.transform)
...
(791, 718)
3
[1, 2, 3]
[<type 'numpy.uint8'>, <type 'numpy.uint8'>, <type 'numpy.uint8'>]
{u'units': u'm', u'no_defs': True, u'ellps': u'WGS84', u'proj': u'utm', u'zone': 18}
[101985.0, 300.0379266750948, 0.0, 2826915.0, 0.0, -300.041782729805]

Rasterio subscribes to PROJ.4 style dicts for coordinate reference systems, as Fiona does. The indexes attribute contains the indexes of all image bands, and those are used to access bands and analyze them like so.

>>> with rasterio.open('rasterio/tests/data/RGB.byte.tif') as src:
...     npixels = src.width * src.height
...     for i in src.indexes:
...         band = src.read_band(i)
...         print(i, band.min(), band.max(), band.sum()/npixels)
...
(1, 0, 255, 29.94772668847656)
(2, 0, 255, 44.516147889382289)
(3, 0, 255, 48.113056354742945)

Rasterio is written using Cython, so it's fast and has Python 2/3 compatibility. I'm testing with Python 2.7 and Python 3.3 and have it deployed on Python 2.7 at work (where the benchmarks are very positive). I hope you'll find rasterio interesting and follow or fork.

Joining MapBox

Sean Gillies — Thu, 24 Oct 2013 06:00:00 GMT

As announced on the company blog, I'm now working at MapBox. I'm excited about this. I'm more than excited. My 8 year-old thinks she knows what excitement is, but dear child, this is something else. Working on bigger things with all the talented and thoughtful folks at MapBox is going to be fun.

Is the transition from academic to commerical work going to be hard? I've worked at fast-paced companies before – experience that definitely factored into my being hired by the AWMC – and I'm happy to be back in this environment. And because I'll be continuing to work on open source geospatial software, data, and protocols, I feel like my favorite academic communities win from having me on the mersh side of things.

What am I going to be working on? A bunch of different things, but my near future involves pixels. Gazillions of pixels. MapBox uses Shapely and Fiona already, so I'll get to stay involved with those projects.

Will I be moving to Washington, D.C. or San Francisco? No, I'm mainly going to work from Fort Collins, but I'll be at one of the offices regularly. I hope to persuade people that the Fort wouldn't be a terrible place to visit for sprints, either. Relatively cheap accomodations and food and plenty of nice beers to meet.

Thanks, everyone, for the overwhelming congratulations.