I added some control over rio-calc's memory usage yesterday and had fun doing it. In 1.0.22, rio-calc read all your sources into memory, preventing calculations on very large datasets. There are a number of ways to chunk the calculations and output of the results. GDAL's gdal_calc.py script divides the job into chunks that match the storage of the input files. I rewrote rio-calc to chunk the work more like GDAL's warper does: you specify a maximum amount of memory to be used by the program and rio-calc divides the job into chunks that need approximately less than this amount. You, the user, are free to trade memory for speed.
The rio-calc command uses Lisp-like expressions. Some people like them, some
people don't. Evaluation of the expressions is done with snuggs. The expression evaluator in gdal_calc.py
eval(). Snuggs does not use
eval() and uses a whitelist
of names and keywords. It is much safer. For example, you can't trick snuggs
into importing modules that give filesystem access. You
could use snuggs on your server to evaluate expressions written by untrusted
people on the internet. You would be asking for trouble if you did this with
gdal_calc.py. This is not to say that Snuggs is perfect and has no bugs. We
found some this week and fixed them. If you're a rio-calc user, make sure you
upgrade snuggs to 1.4.6 to get the latest and greatest version.
Thank you for using Rasterio and thank you for the ever-excellent bug reports. Clear writing and examples of failing code make it much easier for a busy programmer to switch context and dispatch bugs efficiently.