79% of Statistics Quoted Harm Puppies

You know about statistics, right? About how to judge whether the news you’re hearing or reading is actually reflecting the numbers behind it? Because you really, really should. (This post is inspired by the stats quoted at the end of this AdAge piece about the recent KFC grilled-chicken promotion fiasco I didn’t hear about till today.)

One of my biggest pet peeves: Election poll coverage. News agencies seem to love ignoring a little thing called the margin of error. Since elections are over in my world right now, I’ll switch to entertainment as an example. The principles are the same.

Statistics are compiled on a sample of the population in question because in most cases that population is too big to be polled in its entirety. So lets say a network wants to know what youngish women think of the host of their talent show. They can’t ask every woman, so they hire a polling company to compile some statistics for them. The polling company will randomly pick a fairly large number of people who belong to the population they’re interested in (say, female Canadians between the ages of 25-34), and they’ll ask them the question(s) the network is interested in. Depending on the size of the sample (this is where the math comes in), a margin of error is calculated to account for any errors involved in polling the smaller sample. These errors might result in discrepancies against what the entire population’s results would be.

This means the statistic quoted (for example [and I made this up], 52% of Canadian women between the ages of 25-34 like Ben Mulroney as host of Canadian Idol) includes a range. It could really be that 49% of all women of that age like the host (that’s the quoted 52% minus 3% to account for possible error), or that 55% agree (that’s 52% plus 3%). If this were an election poll, perhaps you can see that the difference in impact of 49% vs. 55% could be huge.

Now consider if the statistic quoted compares results from people polled this month versus the same demographic polled with the same methodology last month. Say 51% of women 25-34 liked Ben Mulroney’s hosting of Canadian Idol last month, and this month their support has dropped to 49%. This might make entertainment-news headlines (“Mulroney popularity plummeting!”), but it shouldn’t. A 2% change like that falls within the margin of error, which means that change could just as easily be due to chance.

It’s not interesting news to report that “Opinions have reached an all-time constant!” It’s up to us, consumers of news on all topics, to judge statistics ourselves and not to blindly trust headlines that are often simply aimed at selling papers and magazines or getting eyeballs via Twitter.

Be smart, people!