Survey-ing

The latest survey is recent indeed: It is a reading of the voters’ mood taken by the Social Weather Stations only two days ago, through a universe of survey respondents provided with free cell phones. The results should be pleasing to Grace Poe and her campaign; they show that almost two-thirds or 66 percent of voting-age Filipinos support the March 8 Supreme Court decision allowing the senator to run for president.

Or do they? That detail about free cell phones arouses suspicion, even as it encourages a healthy respect for the innovative approach SWS took with its partners, including TV5. The survey organization called its “Bilang Pilipino-SWS Mobile Survey” a “pioneering” poll, because it incorporates a second stage of survey work into the first.

The SWS press statement read:

“A few days earlier, on March 8-11, SWS had done face-to-face interviews of 1,200 validated voters, drawn nationwide by means of standard SWS random sampling procedures. These voters were invited, and had then agreed, to be members of a panel of respondents to receive and reply to a few survey questions each day using mobile phones provided to them for free by Starmobile.”

In other words, a random sample of 1,200 voting-age Filipinos were asked to take part until May in a daily mini-survey conducted through the free cell phones. “There is no cost to them to receive and reply to the questions, due to zero-rated Internet access provided for the survey and selected websites by Smart Communications,” SWS said.

The first question suggests itself readily: Does a random sample chosen at the start of a weeks-long project remain a random sample at the end of the project? The methodology used in this approach is akin to that adopted by TV ratings firms, but the question needs an answer nevertheless, precisely because of the innovative nature of the survey.

The “few survey questions” that were asked on March 14 were answered only by 71 percent of the original 1,200 respondents invited to take part in the daily mini-survey. Second question: What does this imply for the polling project itself? Statisticians and survey experts agree that polls are best understood as snapshots of public opinion at a given time; what happens when the definition of “given time” both narrows to a daily culling and expands to include a weeks-long period with daily snapshots?

Again, a lengthy and detailed explanation would be welcome.

Third question: The use of cell phones had previously been seen as problematic, especially in the determining of survey respondents in the Philippine setting. If a random sample is determined without the benefit of cell phones, would the lack of face-to-face interviews in the second round of the survey (when respondents answer a few survey questions using only their cell phones) undermine the credibility of the poll itself?

In the same way that more and more journalists have accepted the challenge of explaining the news cycle to news-consuming audiences, survey organizations like SWS and Pulse Asia should welcome a similar challenge —that of explaining how surveys work—in a similar way, as a regular or even daily duty.

(It is apropos to mention that news organizations should refrain, at least during the campaign period, from reporting mock polls, precisely because the polls themselves are self-selecting and therefore unscientific.)

We agree that reputable survey organizations like Pulse Asia and SWS have an enviable track record in predicting the winners in national elections—but that is because the so-called bandwagon effect does not really affect the results of the surveys. In other words, we can trust these scientific surveys because, in fact, very few people vote according to the surveys.

But at the same time, surveys influence campaign financing and political strategy: Donors decide to withdraw funds, or allies decide to shift allegiance, when a candidate consistently lags behind in the surveys. Or donors open their wallets even wider and allies stick together when a candidate jumps up in the polls or in a new tweaking of the polls. For these reasons, understanding exactly how surveys work, including innovative ones like the new Bilang Pilipino-SWS Mobile Survey, is crucial.