Let’s continue to explore this PPM issue.

The primary problems that Arbitron is having with the PPM results are (1) that overall radio listening is down, and (2) that declines seem worst among stations serving African-Americans.

So, are the PPM results a perfect measure of reality and the diary has simply been wrong for 40 years, or somewhere in the PPM system is there either (1) a sample size problem, or (2) a response bias problem, or (3) both?

My belief is, it’s both.

Regarding sample size, I believe that there’s a huge flaw in the PPM system, at least as it’s been presented to us. Arbitron is putting PPMs in the hands of far fewer people than they used to put diaries in the hands of, arguing that the PPM is giving them 13 weeks’ of data rather than one, so they’ve got a lot more data to work with.

But, but, but… This is wrong!

Let me try to explain:

Suppose we used to survey 10 Chicagoans every week to find out what percentage are Chicago Blackhawks fans. That means over the course of a calendar quarter, we’ll talk to 130 people. That’s probably a decent sample size to answer a question like this. (I could do the math on standard deviation and margin of error, but let’s save that for another day.)

The analogy for Arbitron’s new approach is to survey maybe 30 Chicagoans — but survey them every week. “Hey, that’s 390 data points every quarter! Better than the old system’s 130 data points!”

No, it isn’t.

In week #1, 30 people is too small a sample size for the question — we could get a result that’s way off of reality. And surveying the same people every week for 12 more weeks is going to give us, basically, the same wrong answer every week.

Yes, the results* will be more stable than under the old, 130-person system, but that doesn’t mean it’s a better approach.

More to follow.

Comment

1. Kurt…
   Thank goodness someone has finally pointed to what I consider to be THE flaw in PPM. Thank-you for that.
   If you’ve grabbed 18 “wrong” respondents in your group of 30, they remain wrong for the entire 13-week panel period. At least with the old methodology those 18 were only a problem for one of the 13 weeks.
   I did not see very much discussion of the paneling approach until after the ridiculous cell sizes made headlines. No wonder; I feel this is a real biggie.

   Bob Harper

   — Bob Harper · Nov 30, 12:49 PM · #

2. Marbles in a swimming pool – eh Kurt.
   Welcome to my nightmare

   — AEV · Nov 30, 02:31 PM · #

3. How would you like to own a restaurant and not know for a month or so how you did? Your place looks busy…but you won’t know if you’re doing the right thing for a month. It just sucks now… I hope PPM will get it right for us on air staff.

   — JoBo · Dec 2, 01:07 PM · #

4. Maybe the reason the sample size is smaller is that to print and mail diaries is cheaper than manufacturing a piece of equipment that may or may not work. Also, how many stations truly serve the African American community anymore?

   — AC · Dec 3, 10:51 AM · #

Other blog entries
Kurt's summary of the Internet radio royalty dispute
iPhone's radio apps are a canary in a coal mine
Part 2: DI's Ari Shohat on how to grow audience
Digitially Imported's Ari Shohat reveals how he built a huge global audience
Emmis's Jeff Smulyan responds re: FM in cell phones
Ramsey: "Broadcasters don't understand the radio 'experience'"
Honolulu's Brock Whaley: "I have heard the future in my car"
Newspaper seems intent on driving print subscribers away
CBS Radio gets it; Mason and Goodman appear in Chicago
Radio Heard Here is a misguided campaign