This morning you read about the commissioned study from CreditCards.com and hopefully you went to read the article from Yahoo! Finance.  If you didn’t then you should now.  We aren’t going to analyze the results this time and instead analyze the analysis.  Surely you’ve read about how the media manipulates statistics to sensationalize results.  This article is another great example and we’re going to talk about it because I want to show you how to read between the lines.

First of all, at the bottom of the story under “Poll Methodology” they mention how the survey was conducted.  They polled 1,000 people.  1,000!?  I’ve got more people within a block of me than that.  1,000 is no where close to a representation of the population.  You need to poll at least 100,000 people in order to get some type of representation.  1,000 is too few and cannot be considered a portrayal of the general population.

Another problem.  Sadly, the study was conducted by random phone calls.  The problem with random phone calls is they aren’t so random.  In order to catch people at home, they’ll usually call during dinner time.  Many people don’t answer the phone while they’re eating.  I don’t, my parents don’t either.  As a general policy, many people with caller ID (my parents do not) won’t answer the phone if they don’t recognize the number.  Again I’m in this group because somehow my cell-phone number was sold off to a marketing group and I’m getting telemarketing calls now.  If you aren’t in my phonebook, you can leave a message.  I don’t think the random poll folks left a message.

Along this same string of thought is the demographic of people that would answer the phone during dinner.  You aren’t going to reach my parents since they don’t answer the phone, and they wouldn’t have a problem at all talking about credit card debt, because they don’t have any.  If you answer the phone while eating you generally hear from telemarketers more often, making you more likely to listen to their sales pitch.  This makes you more likely to fall for it, and buy into their crap.  What are they always selling?  Credit cards!  My point is the people that answered the call might be more likely to have more credit cards and debt meaning the bell curve of the study is skewed.  You can’t get accurate results this way even if you phoned 100,000 people.  You’re still only hitting the demographic of the people that are likely to answer the phone during dinner.  Do I have better suggestions on how to poll the population?  Sure, but I’m not a company that specializes in market research.

Finally, the study was commissioned by CreditCards.com.  CreditCards.com is more or less a broker of the little pieces of plastic.  There’s nothing wrong with that, in fact, I used them at one point to compare credit cards.  However they shouldn’t be paying an agency to do studies.  This was simply a marketing gimmick.  As a matter of fact, CreditCards.com is a partner with Yahoo! Finance.  In other words, they pay Yahoo! to be allowed to write articles on the topic.  Not exactly an impartial observer of the industry.

I revisited the article to point out a few things.  This type of study can easily end up on Suze Orman’s show (another Yahoo! Finance partner) thus giving it relevance and validity and becoming cited all over the place as fact.  The conversation could go something like:

Caller:  I’m just scared to confront my husband about these credit card debts.

Suze:  You’re not alone doll, in fact I just read about a study that said 80% of Americans are embarrassed to talk about credit card debt.

Should a phone study of 1,000 people funded by a credit card broker be cited as fact?  No, but it will be.  I’ve seen on CSPAN these bogus polls being cited by Congress.  Please readers don’t buy into “studies” and “statistics” because this is how many of them become mainstream.  This doesn’t mean the poll was meaningless, but it only warrants the attention of one small blog for 5 minutes.  If I hear of this study being cited somewhere in the media I’m going to get “Stabby” (to quote a recent commenter).

Related posts:

1. You’re Embarrassed by Your Credit Card Debt
2. Irony: Credit Card Company Desperate For Credit
3. Give me some credit, or not

Related posts brought to you by Yet Another Related Posts Plugin.

Liked this post? Hated this post? Hate me? Email me and let me know: Philip@weakonomics.com