Juliet Eilperin drops this vial of nitroglycerin into her latest Washington Post piece:
… according to several people familiar with his private remarks at the home of clean-tech entrepreneur Vinod Khosla, Obama expressed concerns about the political pain involved, saying that “dial testing” of his State of the Union speech showed that the favorability ratings “plummeted” when he vowed to act on climate change if Congress refused to do so.
Not exactly “profiles in courage.” Not exactly “the Environmental President.”
This may not come as a big surprise given how Obama’s once soaring rhetoric on the moral urgency of climate action has recently crash landed.
But what makes this particularly feckless is that dial testing is all but meaningless. Compared to using polls to determine political positions — a common if widely criticized practice — using dial tests to do so is like consulting your horoscope.
For those who aren’t political junkies, I recommend this introduction, “What Are Those Squiggly Lines on CNN Telling You?”
Dial-testing relies on hand-held dials that can be turned to register positive and negative reactions in real time. Participants in the focus group — 30 is a typical size — sit together and are instructed to continually adjust the dial to reflect how they react to a word, phrase, or sentence.
Here is a typical expert view of the value of dial group information:
Cliff Zukin, director of the public-policy program at Rutgers University and former head of the American Association for Public Opinion Research, argues that dial-testing is unhelpful and misleading. He points to the fact that the sample of voters is far smaller than even the tiniest poll.
“It has no scientific validity — it’s not a sample of anything that has generalized validity,” he says. What’s more, he argues, it introduces inaccurate numbers that assume a power of their own. “The problem with bad numbers is that people tend to believe their eyes.”
So the President is basing his climate policy decisions on something that has no scientific validity. Awesome. Perhaps next time it’ll be Tarot cards — or denier blogs, which are much the same thing.
Even worse, it’s entirely possible that respondents give a negative dial reaction for something that in fact works.
CNN’s focus group is run by Rita Kirk, who concedes: