Climate research probe: science robust, communication lacking

Last week, I wrote an article about Michael Mann being completely exonerated by the inquiry at Penn State, and, at the time, I said that Phil Jones and the Climatic Research Unit (CRU) had been cleared by three inquiries. I got that wrong: at the time it was only two. Yesterday, the third inquiry reported back (pdf). And, yes, the science from the CRU has emerged unscathed. But— and its the sort of “but” that is both big and little—the reputation of both the CRU and the University of East Anglia (UEA) have taken a battering as the ugliness of character was there for all to see.

The latest inquiry was led by Sir Muir Russell, who had complete freedom to both set the terms of reference and recruit people to the inquiry. The members of the inquiry consisted of five people from both public and private sectors, all distinguished in their fields. The inquiry proceeded mostly by following document trails. In the initial stages, the inquiry determined that it would be as open as possible, but, in order to not cover the ground of the previous inquiries, would focus on the behavior of CRU and UAE staff. Here’s how the inquiry was laid out.

Given the nature of our remit, our concern is not with science, whether data has been validated or whether the hypotheses have survived testing, but with behaviour; whether attempts have been made to misrepresent, or “cherry pick” data with the intention of supporting a particular hypothesis, or to withhold data so that it cannot be independently validated, or to suppress other hypotheses to prevent them being put to the test.

In particular, the inquiry focused on whether researchers manipulated data or avoided Freedom of Information Act (FoIA) requests. To help focus the inquiry, they asked for—and received—written submissions, which, depending on the legality and confidentiality of the submission, were made available on the website. These submissions were then boiled down to a series of questions.

One thing that became clear is that the CRU has been a victim of changing circumstances. When the CRU began, climate science was a backwater as far as the public was concerned, and the CRU beavered away without anyone noticing. However, growing concerns, due to the findings of various researchers, pushed climate science onto center stage. When it turned out that the CRU was in the forefront of supplying data for the more public figures, it became a focus of attention for the denialist community.

In the meantime, digitization had hit, and suddenly it was possible to collect far more data than previously, and, not surprisingly, the folks at the CRU (and elsewhere—this is a problem that has hit many branches of science) didn’t really know how to handle it. So, as expected, they handled it badly. My impression is that data archiving practices have changed substantially in just the last year, so the CRU, along with other organizations, were aware that this was becoming a problem and took steps, but it really was too late by then. Add to that the fact that the Freedom of Information Act caught everyone unawares, suddenly making many a scientist’s records subject to potential scrutiny.

The third strike is something that I really hate to admit. The CRU simply didn’t want to engage with either the public or their “critics.” In some respects, I understand why the scientists would get sick and tired of dealing with the incompetent bumblings of the likes of Christopher Monckton. But Phil Jones in particular didn’t like blogs, or responding to blogs. And frankly, I am appalled by that attitude. I spend a good portion of my free time attempting to communicate science, and here we have a major contributor—who also happens to be pretty damn clever and in a great position to put forth his work—refusing to participate. The world has moved on, and if your science is front and center, you have take the microphone and sing.

Instrumental data

So with that background, let’s move on to specific allegations:

That CRU prevented access to raw data. 

That CRU adjusted the data without scientific justification or adequate explanation. Some allegations imply that this was done to fabricate evidence for recent warming. 

That CRU inappropriately withheld data and computer code, thus inhibiting others from checking the conclusions of the CRUTEM analysis.

Most of the submissions either implied this or made explicit statements. Most specific were the claims of McKitrick, claims not covering the availability of the raw data, but rather the lack of availability of the list of additional temperature stations used by the CRU. On the other side, there were several submissions supporting the scientific practices of the CRU, and noting that the raw data was publicly available.

I loved the way the inquiry dealt with this: “In order to test the principal allegations of withholding data and making inappropriate adjustments, the Review undertook its own trial analysis of land station temperature data. The goal was to determine whether it is possible for an independent researcher to (a) obtain primary data and (b) to analyse it in order to produce independent temperature trend results. This study was intended only to test the feasibility of conducting such a process, and not to generate scientific conclusions.” In other words, if we can do it, anyone can.

They found that the data was readily available at at least three different websites. They downloaded the data, selected every station that had an adequate amount of data and performed some smoothing and spatial averaging operations on them. In effect, they replicated the CRU’s main research results, producing nearly identical instrumental temperature records, in very little time. This is something that seemed to be beyond “expert critics” such as McIntyre and McKitrick. To quote the inquiry, “The test is therefore sufficient to demonstrate that, with respect to the declared method, the CRUTEM analysis does not contain either error or adjustments which are responsible for the shape of the resultant temperature trend. A researcher can evidently produce a study which would test the CRUTEM analysis quite precisely, without requiring any information from CRU to do so.”

So, in the very broadest terms, the CRU did not, and, in fact, could not prevent other researchers from accessing the instrumental data record. Further, the CRU has not manipulated the data to obtain a preordained result. And, at the most basic level, the code is simply not required. However, this is not replicating in every detail the CRU’s work, so the question of code and data sharing was examined in more detail.

The question of station identifiers was key to this, and is problematic on several levels. First, the provision of station identifiers under an FoIA request was subject to unwarranted bureaucratic delays from both the UEA and the CRU. It was eventually handed over, however. The more difficult task is taking the station identifiers and matching them to actual stations used for different temperature reconstructions: there is no coherent international naming scheme. In spite of this, using CRU records and records from one other agency was sufficient to identify 90 percent of stations uniquely. The CRU said that the remaining 10 percent can be identified using Japanese records.

The point is that the amount of work required to do this makes the whole process rather error-prone. As a result, the inquiry found that, although the information was publicly available, the CRU should have been helpful and supplied a list that uniquely identifies the different stations. In addition, they recommend the following:

It would benefit the global climate research community if a standardised way of defining station metadata and station data could be agreed, preferably through a standards body, or perhaps the WMO. As example an XML-based format which would make the interpretation, use, comparison, and exchange of data much more straightforward. 

Without such standardisation there will remain residual problems in issuing unambiguous lists, and assembling primary data from them. We feel it would be in the public interest if CRU and other such groups developed a process to capture and publish a snapshot of the data used for each important publication.

On the question of code sharing, the inquiry was entirely unsympathetic. It took them two days to reproduce the data extraction and statistical routines to reproduce the CRU’s work. The rest is the donkey work of checking each station’s record for discontinuities and malfunctions.

Finally, the inquiry reviewed claims that the CRU had published results based on fraudulent data from China. Here, I would say that the CRU got lucky. It turns out that the data was not what it purported to be, but, when that came to light, the CRU reinvestigated their findings using a different data set and found that it didn’t change the conclusions of their original paper. Once again, the CRU proved less than forthcoming. It is clear that they initiated their new investigation right after the claims came to light, but they refused to say anything of substance about it until they published the results of their new investigation.

Tree ring data

Another key claim that is constantly made by climate skeptics is that researchers manipulated tree ring data to make the historical climate look cooler. The inquiry investigated this claim as well. The first thing they did was note what would be required for deliberate manipulation to achieve a predetermined scientific outcome:

That it has been proven that series B is more representative over the whole period than series A. 

That the difference between series A and B is important in a temperature reconstruction in which it is included. 

That if A were replaced with B in each of the reconstructions in which A is included then the conclusions drawn in respect of the likely ranking of the past to the present would change significantly when considering all reconstructions together (including those in which it does not appear).

In other words: the CRU would have had to have known that a tree-ring-derived temperature series was unrepresentative of the actual temperature, and that the difference would significantly change a temperature reconstruction—a reconstruction is made up from temperature data derived from many different proxy data sets. And, finally, that using specific data series would lead to a preordained conclusion. There was no evidence for any of this occurring.

The inquiry also looked at how the uncertainty in the data was represented in the Intergovernmental Panel on Climate Change reports. Again, it found that all published reconstructions had been included in the report, and that the variance, both within and between reconstructions was clearly visible. However, they do note that the IPCC, in its phrasing, uses a rather simplified expression of uncertainty.

The problematic question of the difference between tree-ring-derived temperature data and the instrumental record was also addressed. Clearly, not all tree-ring data agrees with the instrumental data all the time. This is known to be true and has been published extensively. The inquiry found that in the published data, it was a standard caveat involved in interpreting temperature reconstructions. They also found that the reasons for divergence were the subject of active investigation, and had every sign that healthy science was in action.

The only blot was that the World Meteorological Organization had used a picture of a reconstruction on its front page without clearly identifying that it was spliced together from proxy and instrumental data. At the same time, it wasn’t referred to at all in the text. Even so, it would have been better to have clearly identified the data used to make the picture.

Finally, questions had been raised over EAU climatologist Keith Briffa’s technique, where he assumed that the tree ring width was correct and the density was not. He then used the width to scale the density and derived temperature data from the two. At the time that he did this, it was a guess, and the guess was described in the publications. Later on, work on tree growth showed that Briffa had, by intuition, done the right thing. Nothing was obfuscated at the time, nor subsequently. The inquiry felt it was unreasonable to keep questioning something that was both 20 years old and already had an explanation.

The upshot of all this? The inquiry found no evidence that any of these conditions had been met: no data manipulation, no cherry picking, and no misrepresentation of uncertainty. Yet again, the science of the CRU remains intact.

The CRU was found to have responded in the correct, but not helpful, fashion when tree-ring data was requested. In this case, the CRU used data that was not theirs, so when asked for it, could not supply it. They did direct the request to the owner of the data. However, the owner of the data did not archive it until 2009, making it difficult to access. And the CRU made no effort to get the data archived in a timely fashion.

Subverting peer review

Suppose I managed to publish a paper that the majority of my profession regarded as complete nonsense. What do you think the reaction of my peers would be? Outrage, directed at me, the editor that approved my paper, and the journal in general. That is science: people often manage to publish garbage, and smarter people take them to task for it. After a while, the bruises heal and everyone moves on with their life. (One other option might be to curl up in the fetal position and cry foul.) This is exactly what occurred when researchers from the CRU and others (including editors at the journal) responded to an article that they regarded as poor science, published in the journal Climate Research.

As academics, those leading the inquiry recognized exactly what was going on, winced in sympathy, and moved on to a more serious allegation raised by the editor of Energy and Environment: that Phil Jones tried to ruin her and her journal. Unfortunately, aside from what follows, she didn’t actually back her claims with any evidence. The whole of her complaint rests on an e-mail she sent, entitled: “Please take note of potetially [sic] serious scientific fraud by CRU and Met Office.” This message made its way to Jones, who complained to her head of department that the e-mail was “very malicious.”

The crux of the reply he received was “I’d want to protect another academic’s freedom to be contrary and critical.” To which Jones responded “I don’t think there is anything more you can do. I have vented my frustration and have had a considered reply from you.” Serious stuff, eh? As any reasonable person would conclude, there was nothing untoward in that.

Finally, Briffa was accused of using his role as editor at Holocene to reject papers from deniers. As it turns out, the e-mail presented as evidence was taken out of context. Here is the e-mail at the center of the storm:

I am really sorry but I have to nag about that review – Confidentially I now need a hard and if required an extensive case for rejecting – to support Dave Stahle’s and really as soon as you can.

Pretty damning stuff.

Now, let’s add some context. Briffa sent a paper out to Dr. Dave Stahle and another referee for review. Stahle thought the paper unworthy of publication, but noted that if substantial extra work were done, that would change. Briffa then sent a reminder to the second ref to get a move on. The referee responded saying that the paper was pretty weak, and could justifiably be rejected. Briffa then sent the e-mail in question, where he essentially asks the reviewer to be more substantive. Following this exchange, one of the authors asks for an update on the paper. Briffa apologized for the delay, returns the referee’s comments and offers fast-track publication if the referee’s comments could be dealt with.

Again, pretty normal academic stuff. The reviewer is not clear enough, and the editor tells him to get his act together and do his job properly. The reviewer has suggested the possibility of rejection, and the editor asks for a good reason. The fact that that e-mail was taken out of context demonstrates the dishonesty prevalent in the denialist movement.

To show that the inquiry’s overall conclusions—that the CRU did not have any undue influence on peer review—was correct, the inquiry got an editor of a medical journal to discuss the goings-on in peer review. Their conclusion? “The evidence from an editor of a journal in an often strongly contested area such as medicine (Appendix 5) suggests that such instances are common and that they do not in general threaten the integrity of peer review or publication.”

CRU: secret controllers of the IPCC, or not

The inquiry then turned its attention to the CRU’s role in writing and reviewing IPCC reports. Here, McKitrick and McIntyre had alleged that Jones and Briffa had conspired and broken rules to keep papers with contrary results out of the report. Except that they are in the report, and, in the case of one paper on the heat-island effect, the IPCC report goes on to tell the reader exactly what was not included in the paper, and how these need to be included to make the result reliable. Furthermore, in the inquiry’s interview with Jones, it was clear that, if the McKitrick and McIntye land temperature reconstruction were to be taken seriously, then we have to throw out instrumental records from satellites and oceans because they wouldn’t match up.

The inquiry noted that this is still part of an ongoing discussion in the literature and that it felt that the IPCC report reflected that.

The second paper reported the inability of McKitrick and McIntyre to replicate Mann’s temperature reconstruction. This paper was serious cause for concern; however, it soon turned out that McKitrick and McIntyre had not replicated Mann’s work accurately. Thus, there was a second paper published that reflected this. All of these papers are included in the report. The rebuttal paper was published rather late in the proceedings, however, meaning that when it was cited in the IPCC report, it had only been accepted for publication. The IPCC strongly prefers published literature, but no rule was broken in this process. The addendum to this story is that Briffa felt he needed expert advise and a more objective observer to help him review some of the comments. He sent the draft to one of the authors asking for their assistance. The skeptics cried foul over this, claiming it violated the rules of the IPCC.

Unfortunately, no such rule exists, and Briffa consulted with the appropriate section leaders on the appropriateness of his actions. Finally, I find it quite ironic that climate deniers have targeted Briffa, as he is one of the most cautious in his claims:

Peck, you have to consider that since the TAR, there has been a lot of argument re “hockey stick” and the real independence of the inputs to most subsequent analyses is minimal. True, there have been many different techniques used to aggregate and scale data – but the efficacy of these is still far from established. We should be careful not to push the conclusions beyond what we can securely justify – and this is not much other than a confirmation of the general conclusions of the TAR…..Just need to show the “most likely” course of temperatures over the last 1300 years – which we do well I think. Strong confirmation of TAR is a good result, given that we discuss uncertainty and base it on more data. Let us not try to over egg the pudding. For what it worth, the above comments are my (honestly, long considered) views – and I would not be happy to go further. Of course this discussion now needs to go to the wider Chapter authorship, but do not let Susan [Solomon – co-chair of IPCC WG1 for AR4] (or Mike) [Mann] push you (us) beyond where we know is right.”

That looks like a scientist in action to me.

Information: it wants to be free, man

Up until the second half of 2009, the CRU had to deal with a handful of FoIA requests, which it did in the least helpful manner that it possibly could. After that, it came under a deluge of FoIA requests of the following sort:

I hereby make a EIR/FOI request in respect to any confidentiality agreements) restricting transmission of CRUTEM data to non-academics involving the following countries: [insert 5 or so countries that are different from ones already requested] 

the date of any applicable confidentiality agreements; 

the parties to such confidentiality agreement, including the full name of any organization; 

a copy of the section of the confidentiality agreement that “prevents further transmission to non-academics”;

a copy of the entire confidentiality agreement.

Where, in this case, the sender has forgotten to fill in the brackets.

The key findings here are pretty bleak. Basically, the UEA logged FoIA requests, but that was about it. After that, everything was down to the individual researchers figuring out if the data had to be, or, indeed, should be released, and then figuring out how to release it properly. Essentially, the entire system was dysfunctional, and the CRU made no attempt to make life easier for anyone.

Jones certainly did delete a lot of e-mails. Luckily for him, these were only local copies and backups remained. He seems to have been unaware that should someone request that information, he would actually still have to provide it. Indeed, if he had deleted the backups, he would have been in serious breach of the FoIA legislation.

The point is that the entire university staff seems to have built up an obstructionist culture around FoIA requests, and that is not a good thing:

As a final comment we find that a fundamental lack of engagement by the CRU team with their obligations under FoIA/EIR, both prior to 2005 and subsequently, led to an overly defensive approach that set the stage for the subsequent mass of FoIA/EIR requests in July and August 2009. We recognise that there was deep suspicion within CRU, as to the motives of those making detailed requests. Nonetheless, the requirements of the legislation for release of information are clear and early action would likely have prevented much subsequent grief.

Summing it all up

The good

• The science was and still is robust.