The simple availability of code and data isn’t enough – these need to be shared in such a way that the results can be verified. According to HARRY’s readme file he apparently had access to code and data from within the CRU and still wasn’t able to replicate results. The file also shows that layers and layers of complex data processing are involved in the work he was trying to replicate – all part of the scientific research, and important for understanding and evaluating the results. I haven’t questioned the scientific results, but secretiveness or obfuscation with regard to code and data makes it easier to believe computational results could possibly be in error, regardless of the field. Reproducibility of computational results is not a concern unique to climate science.

Reproducibility is being taken seriously in a growing number of fields, such as among seismic researchers and in the machine learning community. I don’t agree that the potential for public harassment is an argument for an exception to the scientific method. I’d argue instead that openness is especially important in the case of climate change because of its salience in public policy. I have a previous blog post on the release of government data arguing that the answer to bad speech is more speech (doesn’t necessarily have to come from the original researchers).

Sharing code and data from results that have not been prepared with the intention that they are to be reproducible is tough and I see your work as vital in this effort, for example your two recent papers in CiSE. I agree we self-correct in science, but this comes through openness and disclosure. Encouragingly, the view that science involves reproducibility of results appears to be resonating. Today The Times reported that The Met will conduct a review of it’s temperature data since it used work emanating from the CRU – and it will do this in a transparent fashion because it “wants to create a new and fully open method of analysing temperature data.” In yesterday’s Scientific American Michael Mann was quoted saying his most recent publication in Science includes the underlying code and data as a supplement – I’m hoping it was released in a reproducible fashion, and that we get to a place where our published results are routinely verifiable.

First, most climate code and data is freely available. The research results are reproduced widely throughout the field, and the comparison and validation of results is build into the IPCC process. Can you name any other field of science, in which 25 different research centres around the world are building large scale models of the same phenomena, and comparing the results in detail through controlled model inter-comparison projects (see CMIP5 for a taste: http://cmip-pcmdi.llnl.gov/cmip5/)

Second, climate phenomena are sufficiently complex that *exact* reproducibility is effectively impossible. This deserves a longer explanation (and I’m working on a paper on that), but for a taste, see: http://moregrumbinescience.blogspot.com/2009/11/data-set-reproducibility.html
So what is needed is a different approach to reproducibility, which is to independently arrive at the comparable results via different methods. Which is exactly what the climate science community has been doing for decades, and it is very effective (unfortunately, little has been written on this, but I’m working on it – it’s a fascinating discipline)

Finally, you’ve blown the Mann and Jones work out of all proportion. The field of dendro-chronology is a minor sideshow in climate science, and a very immature discipline. Mann’s early work on this had errors in it, but of course it did – that’s what science is! Subsequent work has improved the methodologies, without changing the results at all. But then the politicians and lawyers get involved, fail to understand the scientific process, and think that science should be a perfect process every step of the way. Can you honestly say that any of your publications could stand up to the kind of scrutiny at congressional levels that Mann’s has been subjected to? This is not how we do science.

The net result of politically inspired attacks on climate scientists has resulted in an understandable siege mentality, which is what you see in the CRU emails. I have characterized it as a kind of denial of service attack, and it’s not healthy for science in any way. He’s my take: http://www.easterbrook.ca/steve/?p=1001

Michael Mann is also under investigation by Penn, his employer, for possible research ethics violations:
http://blogs.sciencemag.org/scienceinsider/2009/11/climate-hack-sc.html