Most professions have a tendency to use jargon that is impenetrable to outsiders. Statisticians are no exception to this general rule. If you look in a statistics textbook, you’ll probably be unlucky enough to find words like ‘kurtosis’ and ‘heteroscedasticity’ (unless it’s an American book, when they’ll spell it ‘heteroskedasticity’). Don’t worry, I’m not going to explain what these mean; but statisticians display another kind of jargon use that can be even more confusing. We use everyday words, but give them special meanings that differ from the meaning in everyday use. Sometimes the difference in meaning is small, sometimes it is large. A full list of such words would be rather long. It would include: bias, block, bootstrap, censored, contrast, deviance, deviation, distribution, error, expected, hazard, improper, influence, information, jack-knife, kernel, leverage, likelihood, mode, model, moment, moral, normal, pie, regression, scree, stress, tail, variance; and many others.

Actually, this is not as confusing as you might think. With some of the words, the technical statistical meaning is so close to the everyday meaning that no important confusion is likely to arise. Others are generally used in a technical context, so that it is clear that they are not being used in their everyday sense.

However, this is not always the case. I want to describe two words, each of which has a well-understood everyday meaning, and a technical statistical meaning that is rather different. Both of these words are used in contexts where it may not be clear whether they have the everyday meaning or the technical meaning. The two words are ‘significant’ and ‘reliable’.

First, ‘significant’. Its statistical meaning is a little complicated. Suppose I’ve invented a new pill that is supposed to cure headaches. I’ve tried giving it to a few people with headaches and most of them got better, but I know that headaches often go away on their own, and I know that there are already some pretty effective headache cures around. So I decide to do an experiment. I get a group of volunteers who all have headaches. I choose half of them at random and give them my new pill. I give the other half a standard dose of aspirin. After an hour I ask all of them whether their headache has gone away, and I record the results. I find that more of the people who took my new pill got better than did the people who took aspirin.

Does that mean my pill works? Well, it might, or it might not. Perhaps, by chance, the group to whom I gave the new pill happened to include more people whose headache would have got better anyway, whatever they had taken. But I can do a calculation that will throw light on this possibility. I can calculate what’s known as a P value, which is a kind of probability. The smaller this P value is, the less likely it is that my results are simply due to chance. (The connection between the P value and the likelihood that my results are due to chance is a little complicated, but let’s ignore that detail.) If my P value is small enough, I conclude that my new pill probably does work better than aspirin. In the jargon, I would say that the difference between the two groups is statistically significant.

There’s already a potential cause for confusion here, in that it’s a small P value that gives a significant result (in the statistical sense), not a big P value as you might intuitively expect. Most statistics students get confused over this at some point.

But, the important thing to remember is this: if I say a difference is statistically significant, all I mean is that ‘we can pretty well rule out the possibility that the result is due to chance alone.’

    next > Page 1 of 2