How do scientists assess the skill of climate models?

Share and Enjoy:
  • Twitter
  • Facebook
  • Digg
  • del.icio.us
  • Google Bookmarks
Email Email Print Print

Pitcure from the {link:http://www.flickr.com/photos/pnnl/3659908105/}PNNL - Pacific Northwest National Laboratory's album in Flickr {/link}

Pitcure from the {link:http://www.flickr.com/photos/pnnl/3659908105/}PNNL - Pacific Northwest National Laboratory's album in Flickr {/link}

In our surveys among climate scientists, we have asked – among others questions – also how well the components of climate models would perform. Three surveys were run in 1998, 2003 and 2008. They sampled mostly North Americans, Britons and Germans (CLISCI – for further details, such as sampling, return rates and related issues, refer to Bray, 2010a,b). A fourth survey was conducted in 2010 among climate scientists dealing with climate, climate change and impact in the Baltic Sea region with a majority of Scandinavian and Baltic participants (this was done in the framework of BALTEX; details, see Bray 2010c). In the following we will refer to CLISCI 1998, CLISCI 2003, CLISCI 2008 and BALTEX 2010.

The surveys CLISCI 2008 and BALTEX 2010 allowed to broadly identifying “modellers”, and consequently “non-modellers”. While in CLISCI this was explicitly asked, we cavalierly assigned scientists “dealing with past and ongoing climate change” as well as “projections of climate change” to the modeller-category.

We address three questions on the confidence scientists have on climate models

  1. Has the confidence increased since the first survey CLISCI 1998?
  2. Is there a difference between the “global” (CLISCI)-group and the Baltic Sea group (BALTEX)?
  3. Is there a difference in confidence between “modellers” and “non-modellers”?

For brevity, we limit our discussion to two atmospheric components, namely hydrodynamics and clouds (cf. Washington and Parkinson, 2005). Among climate modellers the former is considered relatively uncontested, while serious problem are acknowledged with the latter (see also below). Respondents were asked to reply on a 1-7 scale, with 1 representing no confidence at all, while a 7 would go with absolute confidence. A value of 4 designates a position of somewhat indifference.

In brief the results are – the confidence, as logged by the answers of our respondents, has not only not increased but actually decreased since 1993. “Modellers” differentiate their confidence – they have reasonable confidence in the representation of hydrodynamics but little confidence in the representation of clouds in climate models – the “non modellers” have a more uniform confidence. Finally, the BALTEX group is considerably more optimistic than the CLISCI respondents.

The temporal development of the opinion of all respondents in the four surveys is shown in Figure 1; the means are listed in Table 1.

bray_tab_1_0
Table 1
click to enlarge

All differences are significantly (risk  5%) nonzero, apart of CLISCI 1998/2003 (hydrodynamics and clouds) and CLISCI 1998/BALTEX (hydrodynamics).

The confidence in the description of the hydrodynamics declined monotonously in the CLISCI samples from 1998 until 2008, and was in 2008 half point below the BALTEX 2010 level. This is surprising, first because half a point is a large difference, second because between CLISCI 2008 and BALTEX 2010 was the “crisis”, associated with “ClimateGate” and the failure of COP-15. The situation is similar with the clouds, with an even larger difference in the median.

bray_fig1
Figure 1: Confidence expressed by all respondents, as recorded in the CLISCI 1998, 2003 and 2008 surveys (in green) an in the BALTEX 2010 survey (blue).
Click to enlarge

Obviously the two surveys CLISCI and BALTEX have not sampled the same populations, even if a joint feature is the confidence in the functioning of the hydrodynamics, while there is clear scepticism with clouds. If the difference is mostly reflecting different cultural perceptions and trust in science in general, remains to be seen.

When selecting from the CLISCI 2008-sample only the Northern European (almost all Germans or Dutch) respondents, the hydrodynamics mean went up to 4.43, which is however still significantly less than the 4.81 of BALTEX 2010. In CLISCI there were hardly any Scandinavian, Polish, Russian and Baltic state participants, while in BALTEX two thirds of the surveyed scientists were from these countries. A remarkable detail is the high percentage of about 15% of “7″ in case of BALTEX/hydrodynamics. In CLISCI hardly ever a response rate of 10% for 7 occurred.

Finally, we examined if there would be a significant difference between “modellers” and “non-modellers”; the distributions are shown in Figure 2, the means are listed in this Table 2.

bray_tab_2_0
Table 2
click to enlarge

Here significance, i.e., inconsistency with a true zero difference, is established for the differences within CLISCI 2008, and between CLISCI 2008 and BALTEX, but not within BALTEX (modellers vs. non-modellers). The latter is certainly due to the considerably smaller sample of BALTEX.
Not really surprisingly, “non-modellers” discriminate less between the two components – the difference “hydrodynamics – clouds” in CLISCI 2008 is 0.63, and in BALTEX 0.46 – compared to the “modellers”, which consistently gave the representation of hydrodynamics an assessment larger than 4, and that of clouds less than 4. For the modellers, who know better, the differences were much larger, namely 1.84 (CLISCI 2008) and 1.67 (BALTEX 2010). In both cases, the non-modellers vote for numbers closer to the indifferent value of 4 than the modellers.

bray_fig2
Figure 2. Confidence in the ability of contemporary climate models to describe properly atmospheric hydrodynamics and clouds among “modellers” and “non-modellers”. Top two diagrams: CLISCI 2008, bottom two diagrams. BALTEX 2010; left diagrams: hydrodynamics, right diagrams: clouds.
Click to enlarge

Interestingly, in all cases, modellers and non-modellers, hydrodynamics and clouds, the BALTEX 2010 sample is more confident than the CLISCI 2008, underscoring the difference between the two considered populations.

Thus, the three questions raised, may be answered in this way:
1.The confidence in the model has not been increased, at least not in the CLISCI sample, covering mostly North America, UK and Germany.
2.The BALTEX-scientists have generally a more confident view of the climate models.
3.”Non-modellers” do not understand the different quality of representing such different subsystems as hydrodynamics and clouds in climate models. “Modellers” are mostly well aware of these differences, which is illustrated by the fact that Working Group I of the Fifth Assessment report (AR5) of IPCC will have an extra chapter on clouds and aerosols.

References:
  • Bray, D., 2010a: Consensus among climate scientists revisited. Env. Sci. Policy, in press.
  • Bray, D., and H. von Storch, 2009: ‘Prediction’ or ‘Projection’? The nomenclature of climate science. Sci. Comm. 30, 534-543, doi:10.1177/1075547009333698
  • Bray, D., 2010c: Baltic Climate Scientists Assessment of Climate Change and Climate Science in the Baltic Sea Basin. BALTEX report, in press
  • Washington, W.M. and C.L. Parkinson, 2005: An Introduction to Three-Dimensional Climate Modelling. 2nd edition, University Science Books, Sausalito, California, 354 pp. (1st edition, 1986, 422 pp)
Share and Enjoy:
  • Twitter
  • Facebook
  • Digg
  • del.icio.us
  • Google Bookmarks
Email Email Print Print
published August 18th, 2010
Category: Articles

Leave a Reply