If one wanted to know the solution to particular mathematical problem, and found that different groups gave different answers, then it might be interesting to hear or to read what each group said about the motives of rival groups, but one really ought to chose which answer or answers were correct based upon principles of mathematics, rather than based upon which groups seemed most noble. If one lacked the competence to decide the issue based upon principles of mathematics, then it would probably be best to resist coming to any decision if at all possible.

Likewise, if one wanted to know the solution to a particular problem of the natural sciences, but found that different groups gave different answers, then it might be interesting to hear or to read what each group said about the motives of rival groups, but one really ought to chose which answer or answers were correct based upon principles of science, rather than based upon which group seemed most noble. If one lacked the competence to decide the issue based upon principles of science, then it would probably be best to resist coming to any decision if at all possible.

And if one wanted to know what sort of social policy ought to be applied to some case, but found that different groups gave one different answers, then it might be interesting to hear or to read what each group said about the motives of rival groups, but one really ought to chose which answer or answers were correct based upon principles of science in combination with rational criteria for evaluating ethical philosophies (if, indeed, those criteria are not themselves scientific). And if one lacked the competence to decide the issue based upon such principles, then it would probably be best to resist coming to any decision if at all possible.

Now, all of that ought to be obvious; but consider how much pundits and the major media focus on personalities and theories of motive when it comes both to policy and to science applicable to policy, and how little real science and how little careful dissection of philosophical case is presented. If one party wants one thing, and another wants something different, then we are given some tale of the nobility or at least the level-headedness of one group, and of the knavery or foolishness of the other; accompanying this narrative will be cartoon physics, cartoon biology, or cartoon economics. If ethics are relevant, then one might get cartoon philosophy of ethics, or some ethical philosophy might be implicitly imposed, as if no rival philosophy were conceivable. (If something is treated as good, there generally ought to be an explanation somewhere of what makes it good. If something is treated as bad, there likewise ought to be an explanation of what makes it bad.)

This practice is so prevalent because so many listeners and readers unthinkingly accept it. And I'm not just talking about low-brow or middle-brow people. The self-supposed high-brow folk, more educated and ostensibly more thoughtful, accept this practice. Most of the people who would, if they read them, say that the previous four paragraphs were trivially obvious accept this practice. I don't simply mean that they don't cancel subscriptions or write angry letters to the editor; I mean that they allow their own beliefs to be shaped by some group engaging in the practice. They fall into attending to one narration of this sort, and let it guide them until and unless some crisis causes them to turn their backs on it, at which point they almost always begin to be guided by a narration using the same basic practice to advance some different set of policies.

Sometimes, one must make a decision, with nothing upon which to go except the discernible motives of conflicting parties. In those cases, one should bear in mind that, except to the extent that they are reporting brute fact (rather than interpretation), one typically learns more about the narrators themselves from what they say (and avoid saying) of their opponents, than one learns about their opponents. (And one should not allow the emotional appeal of a narrative to lead one to pretend that one must make a decision that one can in fact defer.)

Sometimes I've simplistically said that invocation of consensus is not a scientific method. A more accurate claim would be that its use is a way of approximating the results of more rigorous methods — a way of approximation that should never be mistaken for the more rigorous methods, and that is often unacceptable as science.

Calling upon consensus is a generalization of calling upon an expert. Using an expert can be analogous to using an electronic calculator. In some sense, using a calculator could be said to be scientific; there are sound empirical reasons for trusting a calculator to give one the right answer — at least for some classes of problems.

But note that, while possibly scientific, the use of the calculator is, itself, not scientifically expert in answering the question actually asked of the calculator (though some scientific expertise may have gone into answering the questions of whether to use a calculator, and of which calculator to use). Likewise, calling upon opinion from a human expert is not itself scientifically expert in answering the question actually asked. That distinction might not matter much, if ultimately scientific expertise from someone (or from some thing) ultimately went into the answer.

The generalization of invoking consensus proceeds in at least one direction, and perhaps in two. First, using consensus generalizes from using one expert to using n experts. But, second, invoking consensus often generalizes from invoking the views of experts to invoking the views of those who are less expert, or even not expert at all.

Individual human experts, like individual electronic calculators, may not be perfectly reliable for answers to some sorts of questions. One response to this problem is the generalization of getting an answer from more than one, and, using a sort of probabilistic reasoning, going with the answer given by a majority of the respondents, or with some weighted sum of the answers. However, this approach goes astray when a common error prevails amongst most of the experts. If one returns to the analogy of digital calculators, various limitations and defects are typical, but not universal; a minority of calculators will answer some questions correctly, even as the majority agree on an incorrect answer. Likewise with human experts. That's not to say that being in the minority somehow proves a calculator or a human being to be correct, but it does indicate that one should be careful in how one responds to minority views as such. (In particular, mocking an answer for being unpopular amongst experts is like mocking an answer for being unpopular amongst calculators.) Counting the votes is a poor substitute for doing the math.

A hugely important special case of the problem of common design flaws obtains when most specialists form their opinions by reference to the opinions of other specialists. In this case, the expert opinion is not itself scientifically expert. Its foundation might be in perfectly sound work by some scientists, or it might be in unsound work, in misreading, in intuïtion and in guess-work, or in wishful thinking; but, in any case, what is taken to be the scientifically expert opinion of n experts proves instead to be that of some smaller number, or of none at all! In such cases, consensus may be little better, or nothing other, than a leap-of-faith. It isn't made more scientific by being a consensus.

In a world in which expert opinion were always scientifically expert, broadening the pool to include those less expert would typically be seeking the center of opinion in less reliable opinion. However, as noted above, a field of expertise isn't necessarily dominated by scientific experts, in which case, people less expert but more scientific may move the center of opinion to a better approximation of a scientific opinion.

Additionally, for an outsider in seeking the opinion of experts, there is the problem of identifying who counts as an expert. The relevant knowledge and the relevant focus do not necessarily reside in the same people. As well as experts failing to behave like scientists, there are often people instead focussed on other matters who yet have as much relevant knowledge as any of those focussed on the subject in question.

So a case can be made for sometimes looking at the opinions of more than those most specialized around the questions. None-the-less, as the pool is broadened, the ultimate tendency is for the consensus to be ever less reliable as an approximation of scientific opinion. One should become wary of a consensus of broadly defined groups, and one should especially be wary if evidence can be shown of consensus shopping, where different pools were examined until a pool was found that gave an optimal threshold of conviction for whatever proposition is being advocated.

What I've really been trying to convey when I've said that invocation of consensus is not a scientific method is that a scientist, acting as a scientist, would never treat invocation of consensus — not even the consensus of bona fide experts — within his or her own area of expertise as scientific method, and that everyone else needs to see consensus for no more than what it is: a second-hand approximation that can fail grotesquely, sometimes even by design.