Science and Consensus
17 February 2011Sometimes I've simplistically said that invocation of consensus is not a scientific method. A more accurate claim would be that its use is a way of approximating the results of more rigorous methods — a way of approximation that should never be mistaken for the more rigorous methods, and that is often unacceptable as science.
Calling upon consensus is a generalization of calling upon an expert. Using an expert can be analogous to using an electronic calculator. In some sense, using a calculator could be said to be scientific; there are sound empirical reasons for trusting a calculator to give one the right answer — at least for some classes of problems.
But note that, while possibly scientific, the use of the calculator is, itself, not scientifically expert in answering the question actually asked of the calculator (though some scientific expertise may have gone into answering the questions of whether to use a calculator, and of which calculator to use). Likewise, calling upon opinion from a human expert is not itself scientifically expert in answering the question actually asked. That distinction might not matter much, if ultimately scientific expertise from someone (or from some thing) ultimately went into the answer.
The generalization of invoking consensus proceeds in at least one direction, and perhaps in two. First, using consensus generalizes from using one expert to using n experts. But, second, invoking consensus often generalizes from invoking the views of experts to invoking the views of those who are less expert, or even not expert at all.
Individual human experts, like individual electronic calculators, may not be perfectly reliable for answers to some sorts of questions. One response to this problem is the generalization of getting an answer from more than one, and, using a sort of probabilistic reasoning, going with the answer given by a majority of the respondents, or with some weighted sum of the answers. However, this approach goes astray when a common error prevails amongst most of the experts. If one returns to the analogy of digital calculators, various limitations and defects are typical, but not universal; a minority of calculators will answer some questions correctly, even as the majority agree on an incorrect answer. Likewise with human experts. That's not to say that being in the minority somehow proves a calculator or a human being to be correct, but it does indicate that one should be careful in how one responds to minority views as such. (In particular, mocking an answer for being unpopular amongst experts is like mocking an answer for being unpopular amongst calculators.) Counting the votes is a poor substitute for doing the math.
A hugely important special case of the problem of common design flaws obtains when most specialists form their opinions by reference to the opinions of other specialists. In this case, the expert opinion is not itself scientifically expert. Its foundation might be in perfectly sound work by some scientists, or it might be in unsound work, in misreading, in intuïtion and in guess-work, or in wishful thinking; but, in any case, what is taken to be the scientifically expert opinion of n experts proves instead to be that of some smaller number, or of none at all! In such cases, consensus may be little better, or nothing other, than a leap-of-faith. It isn't made more scientific by being a consensus.
In a world in which expert opinion were always scientifically expert, broadening the pool to include those less expert would typically be seeking the center of opinion in less reliable opinion. However, as noted above, a field of expertise isn't necessarily dominated by scientific experts, in which case, people less expert but more scientific may move the center of opinion to a better approximation of a scientific opinion.
Additionally, for an outsider in seeking the opinion of experts, there is the problem of identifying who counts as an expert. The relevant knowledge and the relevant focus do not necessarily reside in the same people. As well as experts failing to behave like scientists, there are often people instead focussed on other matters who yet have as much relevant knowledge as any of those focussed on the subject in question.
So a case can be made for sometimes looking at the opinions of more than those most specialized around the questions. None-the-less, as the pool is broadened, the ultimate tendency is for the consensus to be ever less reliable as an approximation of scientific opinion. One should become wary of a consensus of broadly defined groups, and one should especially be wary if evidence can be shown of consensus shopping, where different pools were examined until a pool was found that gave an optimal threshold of conviction for whatever proposition is being advocated.
What I've really been trying to convey when I've said that invocation of consensus is not a scientific method is that a scientist, acting as a scientist, would never treat invocation of consensus — not even the consensus of bona fide experts — within his or her own area of expertise as scientific method, and that everyone else needs to see consensus for no more than what it is: a second-hand approximation that can fail grotesquely, sometimes even by design.
Tags: consensus, pseudo-science, science, scientific method
More simplistically, a calculator is not science, it is a tool. Replace scientist with mechanic, and experts as a crew, then even a dummy like me gets what you're after here. Concensous cannot diagnose an engine problem, nor should it.