On a subsequent visit you ask him about his confidence and he says: "I have very high confidence that you have late stage colon cancer". You are now convinced that you are going to die.
On a third visit you decide to ask him a different question: "How many people are you wrong about? How many people to whom you say that it's "very likely" or that you have "very high confidence" that they have cancer don't have cancer?" And he replies: 1 in 10.
So, despite the prognosis offered by this eminent scientist you have a 1 in 10 chance that he's wrong. 1 in 10 people walk out of his office thinking they are going to die, and they are not going to.
The difficult question is how should the specialist turn his probability into everyday language. What does "very likely" mean to you? How about "virtually certain"? And how would you map those onto probabilities.
The same problem has to be addressed by the Intergovernmental Panel on Climate Change. For example, in 2007 the IPCC said that it was "very likely" that man was responsible for warming since 1750. Prior to that they had said that it was "likely".
It turns out that the IPCC has gone to lengths to carefully define guidelines for the use of language to describe probabilities. In the snappily titled Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties (yes, I lead an exciting life!) published in July 2005 the IPCC published the following guidelines:
|Terminology||Degree of confidence|
|Very High Confidence||> 90%|
|High Confidence||About 80%|
|Medium Confidence||About 50%|
|Low Confidence||About 20%|
|Very Low Confidence||< 10%|
And they published another scale for likelihood.
|Virtually Certain||> 99%|
|Very Likely||> 90%|
|About as likely as not||> 33% and < 66%|
|Very unlikely||< 10%|
|Exceptionally unlikely||< 1%|
Interestingly in 2000 a report published by the IPCC called UNCERTAINTIES IN THE IPCC TAR: Recommendations To Lead Authors For More Consistent Assessment and Reporting had a slightly different definition of "very high confidence". It has greater than 95%. So, it would appear that the IPCC has widened the range (i.e. loosened the definition) of "very high confidence".
I asked one of the authors of the 2000 paper about the change and he said:
It was negotiated by several rounds of email peer review, and the guidance paper was no more than that--guidance in the TAR. Some working groups evolved their own tweaks to it in the TAR, and in AR 4 a formal group got together and redid it again so the margins evolved. Still evolving.
Returning to the most recent report from the IPCC: what's the chance that the IPCC is wrong? It's gone from "likely" (wrong between 1/3 and 1/10 of the time) to "very likely" (wrong 1/10 to 1/100 of the time). So, at this point the consensus is that the chance that warming is not caused by man is somewhere between 1/10 and 1/100.