Survey Research and Cell Phones: Is There a Problem?

Transcription

1 Survey Research and Cell Phones: Is There a Problem? Prepared for Harvard University conference on New Technologies and Survey Research May 9, 2008 Scott Keeter Director of Survey Research Pew Research Center Washington, DC

2 Growth in the Cell-only Population All Hispanic Ages % 30% 25% 20% 29.1% 18.0% 15% 10% 12.6% 5% 4.4% 0% Source: National Health Interview Survey

3 The Cell Phone Problem for RDD Surveys One-in-eight U.S. adults is cell-only according to June 2007 NCHS data The cell-only population is demographically different from the landline population As a result, landline surveys have experienced a sharp decline in the percentage of younger respondents interviewed in their samples

4 Percent Ages Parameter Source: Pew Research Center surveys

5 Percent Ages Parameter Unweighted survey statistic Source: Pew Research Center surveys

6 Practical Considerations Is it Feasible to Survey People on Their Cell Phones?

7 Yes, But It s Expensive Cell interviews roughly 3x more expensive Manual dialing Reimbursement ($10 vs. $20) Lower eligibility rate Many (42%) under age 18 Higher incidence of non-english speakers Cell-only interviews roughly 4-5x more expensive than landline About 35% in cell sample are cell-only

8 Interviewing Rates Landline sample Cell phone sample Contact rate (2) 84% 83% Eligibility rate 86% 45% Cooperation rate (3) 27% 28% Response rate (3) 23% 23% Break-off rate 12% 10% Source: October 2007 survey (results for three other surveys were highly similar)

9 What We Did 4 dual-frame surveys in 2006, 4 in 2007, 1 in 2008 Two surveys on politics and the campaign One on gadgets/internet One on economics One of the Hispanic population Oct 07 GP Dec 07 GP Oct-Nov 07 Hispanic Oct-Dec 07 GP Jan-Feb Landline N 1,507 1, Cell phone N Total 2,007 1,430 2,000 2,054 2, GP

10 Respondent Characteristics Landline sample versus Cell-only adults

11 Characteristics of Landline Sample and Cell-only Respondents* Standard landline sample Cell-only Age % Income <$30K *Figures based on unweighted data 0% 20% 40% 60% 80%

12 Characteristics of Landline Sample and Cell-only Respondents* Standard landline sample Cell-only Age % 46% Income <$30K *Figures based on unweighted data 0% 20% 40% 60% 80%

13 Characteristics of Landline Sample and Cell-only Respondents* Standard landline sample Cell-only Age % 46% Male 48% 61% Black 11% 19% Hispanic 6% 13% Married 26% 57% Income <$30K 21% 41% *Figures based on unweighted data 0% 20% 40% 60% 80%

14 How are estimates affected?

15 Impact on Estimates from Including Cell Phones Standard landline sample Combined landline + cell sample Republican/Lean Republican 37% Approve of President Bush Republican/Lean Republican Conservative 0% 10% 20% 30% 40%

16 Impact on Estimates from Including Cell Phones Standard landline sample Combined landline + cell sample Republican/Lean Republican 37% 36% Approve of President Bush Republican/Lean Republican Conservative 0% 10% 20% 30% 40%

17 Impact on Estimates from Including Cell Phones Standard landline sample Combined landline + cell sample Republican/Lean Republican Conservative 37% 36% 35% 35% Approve of President Bush Satisfied with state of nation 30% 30% 27% 28% 0% 10% 20% 30% 40%

18 Impact on Estimates from Including Cell Phones Standard landline sample Combined landline + cell sample McCain 21% 22% Giuliani 19% 20% Huckabee 18% 17% Romney Thompson Paul Hunter 4% 4% 1% 1% 11% 12% 8% 9% Based on Republican & Rep-leaning RVs (December 2007) 0% 10% 20% 30% 40% 50%

19 Impact on Estimates from Including Cell Phones Standard landline sample Combined landline + cell sample Clinton Obama Edwards Kucinich Richardson Biden Dodd 3% 3% 4% 3% 2% 2% 0% 0% 14% 14% 27% 26% 44% 46% Based on Democrat & Dem-leaning RVs (December 2007) 0% 10% 20% 30% 40% 50%

20 Media Consumption Estimates Standard landline sample Cell-only Newspaper (yesterday) 40% Local TV news 54% Network news 28% Web: newsp yesterday 5% Web: local newspaper 8% Web: Natl newspaper 8% 0% 20% 40% 60% 80%

21 Media Consumption Estimates, 2006 Standard landline sample Cell-only Newspaper (yesterday) 27% 40% Local TV news 43% 54% Network news 28% 21% Web: newsp yesterday Web: local newspaper Web: Natl newspaper 5% 9% 8% 12% 8% 11% 0% 20% 40% 60% 80%

22 Media Consumption Estimates, 2006 Standard landline sample Combined landline + cell-only Newspaper (yesterday) 40% 40% Local TV news 54% 53% Network news 28% 27% Web: newsp yesterday Web: local newspaper Web: Natl newspaper 5% 6% 8% 9% 8% 8% 0% 20% 40% 60% 80%

23 Under Age 30: Landline Sample and Cell-only Respondents* Age Male Black Hispanic Married Income <$30K Landline sample Cell-only 16% 12% 32% 32% 48% 55% *Figures based on unweighted data 0% 20% 40% 60% 80% 100%

24 Under Age 30: Landline Sample and Cell-only Respondents* Age Landline sample Cell-only 55% 70% Male 48% 62% Black Hispanic Married 16% 15% 12% 15% 15% 32% Income <$30K 32% 48% *Figures based on unweighted data 0% 20% 40% 60% 80% 100%

25 Under Age 30: Impact on Estimates from Including Cell Phones Standard landline sample estimate for ages Combined landline + cell sample estimate for ages Iraq: keep troops in 35% 38% Approve of President Bush 27% 28% Republican/Lean Republican 35% 34% Conservative 25% 27% 0% 10% 20% 30% 40%

26 What s the Potential for Bias? Example: % Favoring Iraq Withdrawal Is the combined estimate less biased? Under current conditions If cell-only adults uniformly favored withdrawal If few cell-only adults favored withdrawal

27 What s the Potential for Bias?

28 What s the Potential for Bias?

29 What s the Potential for Bias?

30 What s the Potential for Bias?

31 Quality of Responses landline versus cell phone interviews

32 Quality of Responses Are cell phone respondents more distracted? Source: October survey Landline sample Cell phone sample Very distracted 1 2 Somewhat Not too Not at all % 100% (n=1,507) (n=500)

33 Quality of Responses Are cell phone respondents less cooperative? Respondent s cooperation was Landline sample Cell phone sample Very good Good Fair 6 5 Poor/Very poor % 100% Source: October survey (n=1,507) (n=500)

34 Sample Sizes: Will a combined design yield more interviews with groups relying mostly on cell phones?

35 Not if the cost is fixed Expected n landline sample Standard Design ($100,000) Expected n cell sample Total sample 2, , yr olds Blacks

36 Not if the cost is fixed Expected n landline sample Standard Design ($100,000) Expected n cell sample Total sample 2, , yr olds Blacks Combined Design ($100,000) Total sample 1, , yr olds Blacks

37 Conclusions The risk of error from exclusion of cell-only adults is increasing No evidence of error yet for overall estimates Comparable response rates for cell / landline Cell interviewing is about 3x more expensive Some evidence for gains in estimates for groups relying mostly on cell phones Budget permitting, a cell sample may be prudent & boost credibility of findings