Simple and reusable spatial queries

Quoting James Fee:

First we’ve got new functionality in the Google Maps Data API. First off you can now perform geospatial and attribution queries on data stored on Google’s MyMap. Now of course this isn’t paleo-type spatial queries, just simple stuff that solve 80% of all queries you’d need to complete. Simple web apps need not fancy complicated APIs and clearly Google is the master of this. So upload your data into Google’s My Maps and then query it to display on a Google Maps application. Simple and sweet.

This is also sweet from an engineering perspective. Yes, simple bounding box or radius search are "only" "80-20 rule" features, but at the same time are also features that power users are going to use almost 100 percent of the time for a lot of tasks. In other words: highly reusable features. Let's say you want to select features of a collection that are contained within another feature. You could build up a result set by iterating over each feature in the collection no matter where it is and evaluate containment with the filtering feature for every single one of them. Better yet would be to first eliminate candidate features that aren't even in the same ballpark as the filtering feature by using an indexed bounding box search. This is the default mode of operation for newer functions in PostGIS, for example, like ST_Contains:

This function call will automatically include a bounding box comparison that will make use of any indexes that are available on the geometries. To avoid index use, use the function _ST_Contains.

(By the way, nice new docs there, PostGIS people!) Simple indexed search is the initial boost to more interesting queries. Indexed operations are best provided by the map data collections themselves. Higher order processing, some of which can be done in parallel, can be left to other resources or services. This is, of course (except for the parallel considerations), the premise of the OGC's Web Processing Service, but taken to an extreme where a WFS provides only bounding box and attribute filtering and the WPS does everything else. A familiar pattern, but I'd be surprised if a future Google spatial data processing platform wasn't as different from OGC WPS as the maps data API is from WFS.

OpenLayers.Format.Atom

Right on the heels of this announcement by a company dipping its toe into queryable geographic feature services is word that my patch for an Atom format has been accepted into OpenLayers. OL features can be serialized to Atom + GeoRSS for posting to an AtomPub collection and fetched feeds with GeoRSS annotated entries can be deserialized into OL feature collections with a large degree of conformance to RFC 4287. Thanks, Tim.

It'll be interesting to see if this takes off and in which direction: position as extension or payload?

DDOS on climate science?

Ed Parsons has been beating the drum for open climate data. I like open data, but it's not not without its own problems. A potential problem for science, and scientific consensus, in a brave new world where we are all now climate scientists, is the ramping up of the social denial of service attacks identified by Steve Easterbrook:

But in reality, the denialists don’t care about the science at all; their aim is a PR campaign to sow doubt in the minds of the general public. In the process, they effect a denial-of-service attack on the scientists – the scientists can’t get on with doing their science because their time is taken up responding to frivolous queries (and criticisms) about specific features of the data. And their failure to respond to each and every such query will be trumpeted as an admission that an alleged error is indeed an error. In such an environment, is it perfectly rational not to release data and code – it’s better to pull up the drawbridge and get on with the drudgery of real science in private. That way the only attacks are complaints about lack of openness. Such complaints are bothersome, but much better than the alternative.

In this case, because the science is vitally important for all of us, it’s actually in the public interest that climate scientists be allowed to withhold their data. Which is really a tragic state of affairs. The forces of anti-science have a lot to answer for.

Joe Gregorio has this social denial of service thing nailed:

Let's go back to electronic denial-of-service attacks. They worked because of an inherent asymmetry between the attacker and the attacked. [i.e. from earlier in Gregorio's post: The attacker performs very little computation to send the packets, but the server has to accept them and perform some computation to determine if they are valid or bogus. In this way an attacker with the same or less computational power can overwhelm a bigger host.] The same is true of the social denial-of-service attack where arguments, responses, rebuttals and more importantly time has to be spent responding to the bad faith objections, which are easily written up and tossed onto the mailing list.

Denial of service on climate science was bad enough before the leaked emails, now scientists have to read the emails, parse them, and explain how they don't falsify the science in every public forum and every media outlet. Next, add to the mix climate data and models. What happens when some blog or cable TV gasbag complains that not only do the model results of scientists not match his interpretation of the data, but that he couldn't even get the model to run on his computer, no matter how hard he tried, and that the code itself might be fraudulent. That's not going to be a victory for transparency.

Perhaps we need to match open climate data and models with a change in the rules of our climate debate. Gregorio explains the rules used by the IETF:

Remember that one way to fight a denial of service attack is to raise the amount of computation required by the attacker. In the case of a Working Group the way to do that is by requiring disruptions to take more time and energy. This is where the call for "camera ready copy in the form of a Pace" comes from in the AtomPub WG. Camera ready copy is much more difficult to write than a one or two line objection tossed into a mailing list. Only if you are willing to put in the work to write up a Pace with reasonable text will it start to take up the time of the WG. Your willingness to put in the time and effort to create camera ready copy will distinguish your proposals and objections from those of an attacker.

Similarly, in the climate debate, we could demand that denialists publish their arguments and supporting evidence in peer-reviewed journals. (Note that I'm distinguishing denialists from the skeptics who already do publish in peer-reviewed journals.) Does it risk giving them unwarranted credibility? Maybe, but I think that it's balanced by increased cost. Even low-cost electronic journals completely stacked with friendly reviewers will help level the asymmetry that makes a DOS attack possible. Forcing the denialists to read and personally sign off on the work of others, or even just keeping them occupied correcting each other's grammar and spelling, would be a good start.

We could demand this, and by "we" I mostly mean our media, but that would require our media to transform itself into something that infotains us a little less and edifies us a little more, and that's probably too much to ask, yeah? I don't have an answer, but it's interesting to look at some aspects of the climate debate as a denial of service attack, and I didn't see that perspective come up in any of the many comments on Ed's blog. I also recommend Bryan Lawrence's post on this topic. He doesn't use the word "attack", but certainly expresses some frustration at the extra load put on climate scientists in these times.

Comments

Re: DDOS on climate science?

Author: Paul Bissett

Sean,

I agree w/ many of your arguments. However, it is not as easy as you suggest.

1) there is a difference between those who deny warming occurring since the onset of the industrial revolution and those who question the relative impact of anthropogenic activities on the background natural climate variability. These two groups often get lumped together, but they are very distinct. Propagandists tend to be in the first category (warming deniers), credible scientists tend to be in the other (anthropogenic questioners).

2) having worked in the field of predictive oceanic modeling for nearly 20 years, and published in peer-reviewed journals, and guested edited peer-reviewed journals, I can tell you the science of modeling depends heavily on the assumptions of the model, the mathematical equations used to approximate the physical environment, the tuning parameters of those equations, data input to the models, the validation data used to verify the models, and the computational horsepower to run those models. The only people qualified to run those models are those in climate research centers. These models require huge computers, large staffs, and millions of dollars of infrastructure support.

The only way to create a critical review of the models predicting anthropogenic impacts is to fund a separate effort to develop and tune the models differently to see if alternative theories could explain the observations. In practice this is almost never done, because peer-review research is subject to peer-reviewed funding. The bigger the project, the more group support you need to get your project funded. This tends to create positive feedback in the scientific community; a noted flaw, but like democracy is the best system compared to everything else.

Good observations eventually rule. Part of the current debate lost in the noise is that the last decade has been marked by "unexplained" cooling since 1998. This is unexplained only in terms of how the models were previously being forced, which just goes to say the models were not quite right, and they don't quite know why.

Modelers step in when observations are too sparse or limited to definitely make a case. Climate researcher do not have the equivalent of Large Hadron Collider (a true shame). If they did, the scientific debate would be a lot easier.

Re: DDOS on climate science?

Author: Kirk Kuykendall

So maybe Al Gore just didn't want to distract Dr. Maslowski from his work ...

http://www.timesonline.co.uk/tol/news/environment/copenhagen/article6956783.ece

Re: DDOS on climate science?

Author: Paul Bissett

Kirk,

that's the problem on both sides, rhetoric rather than facts. NASA measurements show a 7.8% increase in seasonal ice cover since the low stand in 2007.

http://www.nasa.gov/topics/earth/features/seaicemin09.html

Let the scientists do their jobs, fund the research adequately, and quit politicizing the science. The facts will surface, but they will take time and good measurements.

Re: DDOS on climate science?

Author: Kirk Kuykendall

Paul,

Yes, it will take time, but I don't think the Navy is waiting. I recall hearing about a classified doc produced by the Navy several years ago discussing where to move ports in response to rising sea levels. At the same time other parts of the gov't were saying there just isn't enough evidence to take action. Maybe the Navy's models are classified. (Malowski, incidentally, works for Naval Post grad school)

Re: DDOS on climate science?

Author: Jeff Thurston

Interesting points.

Having managed research in a University for a long time I offer the following.

1) Most research (esp. this type)is conducted by teams of people. The days of lone scientists completing work like this are few and far between. The denials of service would have to cover huge areas and large numbers of people.

2) Most universities expect scientists to do three things. a) teach, b)research and c) community work. The last item is the one that gets the shortest shift, yet it is the last one needing the most attention - for the situations you describe.

There needs to be more people explaining good science to everyday people in terms they understand, and to be doing it continually. Informed people can make better judgements.

What I find interesting is that few instititions actually sit with media to develop these forms of relationships.

Re: DDOS on climate science?

Author: Sean

Thanks for the comments. I'm very sympathetic to scientists who feel that global warming is over hyped and studies of it funded beyond reasonable levels. As an undergrad, I worked in a molecular biology lab under a professor who argued that the Human Genome Project was going to take more than its fair share of the pie and sideline other important work. I studied under some of the prominent skeptics as a atmospheric science grad student. I admire people who'll take an unpopular stand when necessary. I don't admire those who'll whip up the anti-intellectual segment of our societies into a DDOS on consensus.

Paul, I'm with you on observations, but there are some things I'm not willing to risk losing forever while we wait for absolute certainty. Personally, I'm a bit more concerned about direct damage to ecosystems and landscapes (over-fishing, deforestation, mountaintop removal) and the scientists studying these human impacts are just as vulnerable to consensus-jamming.

Re: DDOS on climate science?

Author: Kirk Kuykendall

I forgot to point out how the Navy is highly experienced in countering jamming efforts. (For some interesting history, read this story about the birth of spread spectrum.) In addition to rising sea levels, the Navy is also preparing for an ice free arctic. I expect renewed interest in Alfred Mahan.

Re: DDOS on climate science?

Author: Paul Bissett

Sean,

I think we're aligned in our concerns. Part of the reason for starting WeoGeo was an attempt to open up critical geospatial information that was locked in the silos of individual organizations. Kinda "Think globally, act locally" w/ respect to geo-content.

It has taken me away from direct science endeavors (which I miss), but I am hoping that by enabling easy access to quality measurements, our contributions will have an equally lasting impact.

Keep swinging at the blowhards. I got your back...

Re: DDOS on climate science?

Author: Bill

Sean, the points you raise have merit to be sure. But science is all about the examination of data. To deny the data to other scientists is to deny that science is being done. Denying access to the data is unforgivable in my view.

Second, you downplay the legitimate concerns of the people trying to get access. There is no question that there have been many serious errors in, for example, the hockey stick studies (data sets mislocated, data sets misrepresented, data sets used in the opposite sense of the original authors). We all make mistakes. But scientists go back and look at what they did, and at least do not repeat the same mistakes in the next paper. This is demonstrably not the case with Mann and his studies. That no one questioned follow on papers when these errors were publicly known is strong evidence of the corruption of the peer review process. The continued use of stripbark proxies when nearly everyone on the planet except these few agree that they are misleading is another example.

Third, while I agree that the world has warmed in the last ~130 years, the question of the rate of warming is a legitimate question at this point. As people begin to look at what parts of the record are available they are discovering that the adjustments are questionable. This has occurred in Australia, New Zealand, Siberia, Alaska, and Norway to mention just a few from the past couple of weeks. It may be the adjustments are appropriate, but in some cases it would be hard to see how this could be so (Darwin for example). And ultimately, it is the rate of warming that is most important. It is the rate(s) that will tell if we have serious climate issues or not. So it is vital we get it right.

Re: DDOS on climate science?

Author: Sean

Bill, you've got me wrong: I'm in favor of open data. As to the discoveries you say have been made in the past couple of weeks: let's get them written up and submitted to a peer-reviewed journal.

Montpellier street art

There is some inspired street art in Montpellier, the likes of which I've never seen in the uptight places I've previously called home. The works (by Mear One and Asylm, I learned today) on the exterior of the Gymnase Alain Achille caught my eye on my first tram ride to the centre-ville and today I finally took the time to hop off at the Stade Philippidès tram stop and take some photos. The first 4 are of the east-facing wall (click for higher-resolution images):

http://farm5.static.flickr.com/4044/4178208793_026a922b1a_d.jpg http://farm3.static.flickr.com/2773/4178210913_083a81c13b_d.jpg http://farm3.static.flickr.com/2789/4178971168_20e653c0e0_d.jpg http://farm3.static.flickr.com/2753/4178207797_8792b159f1_d.jpg

The north-facing wall has some interesting art too, not signed or having lost its signature:

http://farm3.static.flickr.com/2593/4178200955_bbe8d3b83c_d.jpg http://farm3.static.flickr.com/2738/4178203041_29fa140e0b_d.jpg http://farm3.static.flickr.com/2682/4178203949_41073d6459_d.jpg

A little research turned up documentation by Sophie. of this amazing work on a gymnase in Boutonnet. Wow! Must find this one. Mear One, again, I'm guessing, and also guessing that the humping deer weren't part of the original. The gym on Avenue Rimbaud direction Celleneuve has another good one (street view) that I'm hoping to be able to post photos of soon.

http://farm1.static.flickr.com/83/231824309_1c40630e8f_d.jpg

Sophie.'s Montpellier set also has some good shots of "Jonny Style" on the streets of Montpellier, like this on Rue du Père Soulas:

http://farm1.static.flickr.com/72/229510818_75d1124019_d.jpg

Judgement matters

Whuh?

I think judgment matters. If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place. If you really need that kind of privacy, the reality is that search engines -- including Google -- do retain this information for some time and it's important, for example, that we are all subject in the United States to the Patriot Act and it is possible that all that information could be made available to the authorities.

I think Google's hiding behind the Patriot Act here. A search engine company wants to hang on to data to analyze it for trends and develop predictive models that it can turn into revenue. There's a business need for the data that the company would find a way to justify, legislation or no.

The second sentence there, the one that Eric Schmidt is taking so much flak over, doesn't disturb me as much as it confuses me. Is he saying that we all need to simply drop the acts and quit keeping secrets from each other? Be more real? Let it all hang out, man? Is he using a provocative statement to check the internet community's pulse? Is it a "tell"? Is he just winding up the haters?

If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place.

How do we keep the tyranny implied in that statement out of the "GeoWeb"?

GeoWeb: utopia or dystopia?

Somedays I have very mixed feelings about where the "GeoWeb" is going:

Example 1: Intelligent Traffic Systems

Roughly translated: A million cars idling for 10 minutes will consume some 140,000 litres of gasoline. At the same time we have serious global problems with climate change and local problems with air pollution. Why should this be the case? The problem can be seen as one in which there is a lack of communication between the vehicles and the road.

I interpret this to mean that the traffic systems should regulate the highways such that this condition does not take place, or takes place much less frequently. One of the functions of Intelligent Traffic Systems would be to minimize the pollution generated by the use of the highway system. Of course, he does not say how that might entail regulation of an individual’s actions but one can easily imagine the vehicle being told it cannot enter a particular section of the highway, or cannot even be taken out of the drive way. What is key in Wen Jiabao’s remarks is that we can use technology to help us understand the consequences of individual actions, and the relationship between those actions and physical laws (”wisdom of the earth”). We can choose to let a million vehicles idle on the highway, but in doing so we cannot avoid the consequences for air pollution, and for damage to our health and to the planet. What an intelligent traffic system might do then, at the very least, is to make the linkage between actions and consequences visible to all of us, even if it does not yet constrain those actions.

I'm all for a "planetary nervous system", but the thought that it would sooner or later be hooked up to a state-operated planetary immune system that constrains our actions is a bit chilling, no? I'm probably to the left of many, if not most, of my readers, but I'm not ready to be of the body. I suspect it's going to be constant struggle to keep the "Wisdom of the Earth" from being rigged against civil liberties.

Comments

Re: GeoWeb: utopia or dystopia?

Author: Randy George

Come on Sean, surely you already knew about the "Intelligent Traffic System?"

"The shepherd cries

The hour of choosing has arrived

Here are your tools"

Al Gore

Do you still wonder who Obama will appoint as the next Poet Laureate?

Believe me, I didn't make this up! Osip Mandelstam, Not!

Al Gore Vanity Fair

Re: GeoWeb: utopia or dystopia?

Author: Kirk Kuykendall

Oh, the irony: Chinese teaching us the lessons of Adam Smith.

ITS will allow a market place to be built where we pay for the consequences of our actions, perhaps by combining congestion pricing with cap and trade. Clearly they want us to be more efficient so we don't fall behind on our payments.

Re: GeoWeb: utopia or dystopia?

Author: Sean

Al Gore is our shepherd? Good grief; you don't have to be a believer to cringe hard at that one.

I've been meaning to follow up on your post about Atom-formatted Microsoft data, Randy. Interesting stuff, I hadn't been following that application.

Re: GeoWeb: utopia or dystopia?

Author: Tom

In many places individual vehicles are already impractical or regulated out of feasible use and replaced by public transport - which is probably more constrained than the ITS, which would probably only be useful on congested commuter routes anyway.

There is a danger in using IT-based automation to turn economic and environmental levers wholesale into a pervasive "artificial gravity" though: for one thing it reeks of trying to solve a monolithic, too-hard problem wholesale with a mixture of theory and ideology. We know how that usually goes: some significant cost is

Re: GeoWeb: utopia or dystopia?

Author: Tom

*cough* ignored

Re: GeoWeb: utopia or dystopia?

Author: Sean

I just added "against civil liberties" to the tail of my blog post. I'd implied it from the start, but it's better made explicit.

Punk

David Rees and Get Your War On helped me weather the crazy bender the USA went on from 2001 to 2009. His short essay on Dennes Boon and the Minutemen from Christmas, 2005, is the best piece of rock and roll fan writing of all time:

The first time I heard the Minutemen--on a Saturday afternoon in 8th grade, when my friend lowered the stylus onto "Shit From An Old Notebook," and the song somersaulted out of his RadioShack speakers in an ecstasy of spasmodic guitar and drum fills--is the greatest "first time someone heard a band and their life changed for all time" of all time.

Comments

Re: Punk

Author: brad

Serious as a heart attack--that's a great essay about a great band. I was introduced through the 7 Inch Wonders of the World compilation from SST. Just listened to those songs again online, and yeah, it's all true.

Idiomatic programming

I've opined that GIS programmers should be taught to use Python idioms in the classroom. Less code, fewer bugs, efficiency, and to a smaller degree, socialization into the community of Python programmers are what I see as the happy results. I take it as a given that one wants to use and modify the code of others, share code with others, and see it all improve, which is why I value socialization. "Open Source" isn't just about the license, it's also about code as conversation. Because I've been learning Python (and computer programming in general) as I go, I've written as much clunky Python code as anyone else. Regretfully, I've even exposed some learners to clunky code in workshops. Looking back, I'd prefer to have been taught, and to have taught others, idiomatic Python.

I expect that opinions more informed than mine on the subject of language idioms are likely to appear in comments on Tim Bray's post on idiomatic Clojure.

More lessons of standardization

You can make a spec short and sweet, but there is no guarantee that it will be implemented correctly. Did my woofing about agile standardization anger the format gods, or is GeoJSON really the dribble cup of the "GeoWeb" after all? Check out this excerpt from the timeline of Bill Dollins, who writes from the countryside near Washington, D.C.:

{ ...,
  "id":5993294422,
  "geo":{"type":"Point","coordinates":[38.50745,-76.76051]},
  }

For all the bloggage, not a single geo pundit has noticed that there's a problem. Do you?

Update (2009-11-25): Chris Schmidt summarizes the GeoJSON coordinate order debate.

Update (2009-11-26): To be fixed in the next version of the Twitter API (via Andrew Larcombe and Bill Dollins).

Comments

Re: More lessons of standardization

Author: daryl

it has y x and not x y ?

Re: More lessons of standardization

Author: Christopher Schmidt

Either that, or he's mapping Antarctica.

Re: More lessons of standardization

Author: Brandon

No spatial reference? I know we can assume it, but it isn't explicitly stated.

Re: More lessons of standardization

Author: STH

Projection and Datum?

Re: More lessons of standardization

Author: cwhelm

"geo" should also be "geometry"

Re: More lessons of standardization

Author: cwhelm

and "id" belongs in the properties

Re: More lessons of standardization

Author: Sean

Coordinate order is the only problem. I don't think developers necessarily have to buy in to features and feature collections from the standard GIS model, but we all have to agree on coordinates.

Re: More lessons of standardization

Author: Paul Ramsey

Somewhere, Dale Lutz's head is exploding. Let your X be your X, your Y be your Y, anything else is the work of the Evil One.

Re: More lessons of standardization

Author: STH

Well, what is correct? X,Y or Y,X - in Norway we have the official Y X instead of the "international / US / UK" X Y.

Should all countries also ride the car on the right side of the road and not the left side?

Re: More lessons of standardization

Author: Sean

If all of our national roads were mapped into the same transit space, intertwined, without any borders or gates -- so that US-40 (for example) connected to France's A9 (the Languedocien), and also connected to almost any other major road in the world -- you'd certainly want to prevent systemic and massive accidents by forming a global consensus on traffic direction. This is the situation on the World Wide Web. Whether GeoJSON coordinates should be [easting, northing] or [northing, easting] (or [westing, southing]) is, to some degree, arbitrary. The benefits of agreement are not arbitrary. Global consensus allows global interoperability.