This plot is from chapter 4 of Michael Shermer’s book “Why People Believe Weird Things”, and it’s quite amusing.
The section of the book this is from concerns ESP experiments. Participants in these experiments have to predict one of five symbols on an unseen card, and the plot is meant to show the chance of a participant getting x of the answers right when being asked 25 times. Predictably, getting 5 right out of 25 is the most likely, but higher numbers of hits are entirely expected, and their occurence alone does not mean much. Shermer stops with the negative statement – high performance doesn’t mean ESP – and doesn’t spell out what a correct analysis would look like – not looking at a single result, but comparing a distribution of results to that predicted from chance – but this unfortunate focus on gleeful debunking rather than the education of the reader isn’t my concern. It’s the plot itself. Look at it.
Its x axis runs from -2.5 to 12.5. Bars of unit width indicate the probabilities of getting a certain number of answers right, and a Gaussian curve (Shermer calls it a Bell curve in the legend, and a normal distribution in the chapter text) to fit the bar data. Pretty much all of this is awful. First of all, fractions of x make no sense. The number of correct answers can only be a whole number, therefore probabilities should be plotted for whole numbers alone. If you want to use bars, make them discontinuous to emphasize their discrete character. Additionally, both the Gaussian and the leftmost bar extend into negative values, which also do not make sense for x. Negative values shouldn’t be on this plot at all. All this could be simply due to sloppy graphical design, of course, but the Gaussian extending into the negative is in fact a hint of the biggest error here: this distribution is not a normal distribution at all. To show a Gaussian here is a blatant statistical and conceptual error.
Random answering in a task like this in fact follows a binomial distribution. Interestingly, the numbers on the bars are from the correct binomial (except for 0.0238, which should have read 0.0236), meaning that whoever prepared the data knew what they were doing. But Shermer, fitting a Bell curve to them, clearly does not. You can even see it’s a bad fit. The bars don’t look symmetrical at all. Furthermore, the claim in the legend that “in a group of 25 several scores [above 7] will always occur purely by chance” is also quite false. He himself provides the probability for scoring below 8, which is 89.1%. Thus, in a group of 25, the chance of everybody scoring below 8 is 89.9% to the 25th power, or 5.56%. In other words, roughly one in eighteen such groups will have nobody scoring 8 or higher. There are lies, damn lies, and statistics done by fools.
Claims of the paranormal are emotionally and spiritually appealing, and it is important to counter them both on their own grounds by providing equally appealing stories about the world, and also by clearly showing where their purported proofs are untenable. Michael Shermer is the founder of the Skeptics Society, who claims this as their task, and he adopts a smug tone of intellectual superiority over the misled and uneducated foolish masses throughout this book. Yet apparently he doesn’t know what he’s talking about himself. He even told me so at the end of his foreword: “why should [you] believe anything [I] say? […] You shouldn’t.”
In the end, the whole thing is probably just a clever lesson in scepticisim. Or is it?
Deprecated: Function link_pages is deprecated since version 2.1.0! Use wp_link_pages() instead. in /mnt/web224/a3/17/5110917/htdocs/engine/wp-includes/functions.php on line 6170