So that I don't forget, and in the hopes of picking up tips from more experienced users, I'm going to write down my notes on using git to work on OpenLayers without any commit access. OpenLayers is a big project, and enhancing it in a significant way takes time and care. It's worth doing this work within a revision control system. The project has SVN sandboxes, but you can't use them offline (which I've been quite a bit this summer), you have to apply for them (I'm impatient), and my OpenLayers guru is annoyed by them. For these reasons, I've gone with a DVCS, and chose git on account of its nice bridge for Subversion: git-svn.
To begin, clone the OpenLayers trunk. As of today, there have been 9623 revisions. Don't fetch all of them! In this example, I'm going to fetch a small revision range just short of the SVN head so that I can illustrate rebasing later.
$ git svn clone -r 9621:9622 http://svn.openlayers.org/trunk/openlayers
...
r9621 = 4f62382011af58e8dd8be3b81a817fff80b4f4c4 (git-svn)
Checked out HEAD:
http://svn.openlayers.org/trunk/openlayers r9621
Now I have cloned OpenLayers and begin by default in a branch named "master". I'll do my work around http://trac.openlayers.org/ticket/1366 in a new branch named "atom".
$ git checkout -b atom
Switched to a new branch 'atom'
I've already produced a patch using the methods I'm noting here, and will apply it to the new branch for the sake of this demo.
$ patch -p1 < atom-format-2.patch
patching file lib/OpenLayers.js
patching file lib/OpenLayers/Format/Atom.js
patching file tests/Format/Atom.html
patching file tests/list-tests.html
$ git commit -a
[atom 315def2] Apply patch from http://trac.openlayers.org/attachment/ticket/1366/atom-format-2.patch.
4 files changed, 1171 insertions(+), 0 deletions(-)
create mode 100644 lib/OpenLayers/Format/Atom.js
create mode 100644 tests/Format/Atom.html
If I were to run "git diff master" I'd recover that same patch. Now, there's a new attachment on my favorite OpenLayers ticket. A patch would have been nice, but a file is better than nothing. I'll create a new branch named "atom-pwr" based on "atom", copy the attached file and overwrite Format/Atom.js in the new branch.
$ git checkout -b atom-pwr
Switched to a new branch 'atom-pwr'
(copy atom.js)
Running "git diff atom" shows me the difference between atom-pwr and atom branches. I'll commit the changes to atom-pwr.
$ git commit -a
[atom-pwr 29ae464] Incorporate pwr's work from http://trac.openlayers.org/attachment/ticket/1366/atom.js.
1 files changed, 48 insertions(+), 7 deletions(-)
A patch to the OpenLayers trunk incorporating all these changes is generated with:
$ git diff master > atom-pwr-r9621.patch
Now, lib/OpenLayers/Control/GetFeature.js was added in r9623. To keep up with these changes, use git-svn rebase. First on the master branch:
$ git checkout master
$ git svn rebase
M lib/OpenLayers/Control/GetFeature.js
r9623 = 3e4282ccb05e9421d8f078893c5c495c55085621 (git-svn)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to refs/remotes/git-svn.
And then on the development branch:
$ git checkout atom
$ git svn rebase
First, rewinding head to replay your work on top of it...
Applying: Apply patch from http://trac.openlayers.org/attachment/ticket/1366/atom-format-2.patch.
Now a patch against the current head is generated by:
$ git diff master > atom-r9623.patch
As nothing was changed in the files I'm working on between r9621 and r9623, this patch is identical to the one I started with. Up the new patch goes to the OpenLayers issue tracker to make things as easy as possible for an actual committer.
Comments
Re: Where's the book?
Author: Matthew Perry
Instead of synthesizing all those disparate uses into a typical book, how about a collection of chapters on each, edited together to form a comprehensive overview of the python spatial ecosystem? This way you could have the best authors tackle their primary subject and still provide the big picture.
It could be called "Geospatial Python: Overlapping approaches to spatial programming" :-)
Re: Where's the book?
Author: Eric Wolf
Personally, I'd love to see a Python GDAL/OGR-oriented book that implemented basic geoprocessing. Essentially a "here's how to do geoprocessing without ESRI". It would also be easier to introduce more advanced programming concepts. I've found that GISers and Geographers tend to get stuck in the learning curve when using Python with the arcgisscripting module. The learn to do loops and manage simple variables but frequently have trouble seeing the need for objects and modules.
Ironically, I've found myself temporarily without an ESRI license for the first time in about 10 years. So the project I'm working on will be done in straight Python+OGR. It's actually a little annoying because I'm having to write a slew of computational geometry routines (I know, there are libraries like CGAL, but I like using my own objects).
I'd be game to help write a book...
Re: Where's the book?
Author: Sean
OGR? Here's an example of my perspective problem: I've developed enough loathing for the ogr.py API (since years) that I've written WorldMill and blogged a lot about using it. It prompted Howard to make ogr.py better, but I feel like my Shapely, WorldMill, Rtree stuff remains more usable.
Re: Where's the book?
Author: Sean
I'm on the loosely-coupled "Pylons" side of the divide, if it's not obvious.
Re: Eric
Author: Keith
Eric,
Have you tried Spatialite? It has the core spatial functions - intersect, union, distance, length, centroid, etc. It is so easy to use and has a nice GUI to view the spatial data. The commands are incorporated in SQL. The tutorial is good too.
Keith
Re: Where's the book?
Author: Sean
Keith: Spatialite is a neat data store, but only that. What you use to process data after you take it out of the store (and before you store the results) is the question.
Re: Where's the book?
Author: Peter
Use XSLT!
Re: Where's the book?
Author: Sean
Peter: I like declarative programming, and there are some neat options to combine and balance Python and XSLT in lxml's extension elements, but I imagine that the declarative style works best for geoprocessing in configuring complex pipelines of data processing modules that would be written in Python. Maybe like Paste Script wires up WSGI pipelines.
Re: Where's the book?
Author: Peter
Sean,
There's no doubt (in my mind) the heavy lifting in terms of geoprocessing needs to be done by an indexed datastore, hopefully accessible via spatial XQuery/XPath :). There are good current benefits in terms of simplicity, portability and ownership of your code, to thinking about low-volume, high-fidelity geoprocessing in pipelines, as you say. Some pipeline alternatives for such pipelines may in the future include
xmlsh, and
XProc. To provide sufficient performance, a geospatial implementation perspective needs to be baked in, something I would like to see.
Cheers,
Peter
Re: Where's the book?
Author: Keith
Sean, Spatialite just a datastore? I use pysqlite to script and create UserDefinedFunctions (UDF) to extend and reuse functionality. Sqlite is rock solid and fast, with a 2 terabyte database size limit on 32bit machines. All this functionality for free - Transactions, Virtual tables, Spatial indexes, attribute indexes, projections, overlay commands (clip,erase,intersect), aggregation, simplification, routing. For just a data store it sure sounds like a geospatial engine. The datastore aspect is compelling as well. You can store multiple geometry types in the same table. A shapefile will only hold a single geometry type. You can also have multiple geometry columns in a single Sqlite table - various projections, for example or you could store parcels as various geometries - rectangular lot, building footprint, property centroid. The business data is easier to manage in a database too. Gotta love it.
Re: Where's the book?
Author: Sean
Relational databases have a large, but not limitless set of uses. If you're going to process non-relational data such as documents or less structured data, you'll need a general purpose programming language and environment. For example, you're certainly using a general purpose language already (maybe even Python) to model data so that it can be loaded into your database. I think maybe you confuse me for a "NoSQL" partisan. I simply think an RDBMS is better suited to storing models than to doing all kinds of modeling and data processing.
Back to the topic of Python books: it wasn't my intention to throw "stop energy" at a book effort, but to point out that "Python and GIS" is a messy subject and potential readers are somewhat segregated in different camps. I'm hugely in favor of more, excellent narrative documentation.
Re: Where's the book?
Author: keith
I vote for starting a wiki instead of publishing a book. More people can access a wiki via the internet and it is easier/cheaper to maintain and update a wiki. A book might be out of date in a few years as software is updated. There is a huge variety of open source GIS tools that people use. I'd like to see work flow diagrams, recipes, code snippets, etc how real work is getting done. There is also IronPython (dotnet)and Jython (java) to consider along with Python.
Re: Where's the book?
Author: julia
A "Geospatial Python" wiki with evolving content from geospatial python experts that covers many of the various geospatial python implementations, including interacting with spatial data stores, would be a very welcome thing indeed to many people who are not geospatial python experts but need some good resources and examples - so we won't keep getting stuck in that learning curve....
Don't know that I'd characterize Open Source GIS as "the seedy side" of GISville. I think of it more as the "Company Town" side of GISVille being the snooty gated community where entry is controlled both by the size of one's bank roll and one's desire to pay a high entry fee for the "privilege" of feeling like one is part of some exclusive, superior society. The superiority thing is not true, but the "Company" really excels in marketing, so a lot of people believe the hype. The problem is that inside the gated community, the "Company Way" is the only allowed way to do things. Like an H.O.A. gone completely wrong :)
Many of us (who have been forced to live in the gated community for many years by our employers in the public sector) would like to start learning several different ways of doing similar tasks in GIS with python, and not limit ourselves to a single toolset. That same old "geoprocessing" hammer doesn't always do such a great job :) Knowing more than one way to solve the problem, or how to use more than one tool is a big help when deciding what method makes the most sense to use in any given situation. The aforementioned wiki would be very helpful to a lot of people as a learning aid and in helping to compare and contrast the benefits or drawbacks of employing different approaches.