Metodološki zvezki, Vol. 1, No. 1, 2004, 21-32 Response Trends in a National Random Digit Dial Survey Robert D. Tortora1 Abstract This paper reviews response trends over 24 consecutive quarters of a National Random Digit Dial telephone survey. Trends for response rates and refusal rates are studied as well as the components of response rate, namely, contact, cooperation and completion rates. In addition other rates, including answering machine, busy and no answer are studied. While refusal rates declined over the six year period, contact and cooperation rates significantly declined causing response rates to decline. Answering machine rates and busy rates also showed a significant increase over time. Finally, correlation’s among the variables of interest are presented. The response rate is negatively correlated with the busy rate, the answering machine rate and the no answer rate. Implications of the above trends are discussed. 1 Introduction This paper examines response trends in a stratified national Random Digit Dial (RDD) survey over 24 consecutive quarters. The key variables studied include the refusal rate, the contact rate, the cooperation rate, the completion rate, and the response rate. After a brief review of nonresponse in household surveys the paper describes the survey design, the trends by stratum and overall over the 24 quarters and summarizes the key issues associated with this RDD survey. 2 Nonresponse in telephone surveys Groves and Couper (1998) indicated that there was a decline in response rates in telephone surveys and attributed this decline to decreases in the contact rate. They 1 Bob Tortora, Chief Methodologist, The Gallup Organization Europe s.a. 1000 Bruxelles, av Michel Angel 70; bob_tortora@gallup-europe.be 22 Robert D. Tortora also indicated that refusal rates were increasing. DeHeer (1999) and Djerf (1999) identified that the increase in noncontacts is an important contributor to the decrease in response rates. Steeh et. al. (2001) studied response patterns in a nationwide RDD survey as well as a State level RDD survey. They studied nonresponse trends for the Survey of Consumer Attitudes (SCA) during the period 1980-1999 and found an increase in refusals during the 1960’s and 70’s but a recent abatement during the years 1995 to 1999. They also examined the Georgia State Poll. Here refusal rates declined but the noncontact rate increased. In addition the difference between the two are the largest when examining data from the Atlanta area, a large metropolitan area. They attributed this phenomena to a growing number of attempts that result in a “no answer” or a busy” outcome. Tuckel and O’Neill (2001) noted the inaccessibility of potential respondents due to such devices as caller-id and the use of lines for Internet access and fax. They also noted that potential respondents indicated an unwillingness to participate in telephone surveys. This was found among several subgroups of the telephone population including those 60 years of age and over (42.6%), Hispanics (43.8%), the most affluent (47.6%), one-adult households (43.5%), residents of large cities (43.4%) and their surrounding suburbs (45.5%), and those in the New England (50.5%), East South Central (45.9%), Mountain (44.2%), and Pacific (48.9%) regions. Finally they found that a higher proportion of those with unlisted versus listed telephone numbers tend to be hostile towards participation. Thus these authors have identified serious issues that are affecting the quality of telephone surveys. The intent of this paper is to study response trends for one quarterly RDD survey that has a constant design over nearly six years of data collection. 3 The survey design This survey, conducted for a client of The Gallup Organization, has two main objectives. First, to screen for adult Internet users (those individuals that have accessed the World Wide Web in the last 30 days for activities other than the use of email). Those adults that qualify are asked a short series of demographic questions and then asked to go to a Gallup website to complete a detailed survey on their Internet activities including web-graphics and web sites visited yesterday or in the last 30 days. They are also asked a wide variety of questions covering such topics as business and vacation travel, entertainment behavior, credit ownership and use and make and model of primary and secondary vehicle ownership or lease. The second objective is to collect data for a sample of nonInternet users. The first objective requires a 6 - 8 minute telephone interview followed up by the web data collection task that has a median time of 40 minutes to complete. The second objective requires an 18 minutes, on average, telephone interview. In the last two years the final sample size of web completes was reduced Response Trends in a National Random Digit Dial Survey 23 form 10,00 to 9,000 per quarter and the number of non-internet completes was reduced from 2,000 to 1,000 per quarter. A list-assisted stratified RDD sample design is used for the telephone part of this survey. Telephone area code - exchange (telephone numbers in the US are standardized to a three digit area code, a three digit exchange and four following digits) can be mapped to geographic areas and hence mapped to census tracts. Thus each area code-exchange combination can be assigned a household income amount using (modeled) census tract data. This survey uses three strata: those area code-exchanges with household income less than $50,000US, those with household income more than $50,000US but less than $75,000US and those with household income of $75,00US or more. The survey used a stratified design for two reasons. First, in some initial studies prior to the start of this data series income – these sampling strata - was used to target those more likely to be using the Internet. Second, Tortora and Russo (2001) found that the use of these sampling strata were critical to minimizing the cost of a pre-paid cash incentive designed to increase the number of qualified adults that completed the Internet survey. The survey uses tracking interviewers2, is conducted in English only and uses a five-call design. More than 15,000 adults are interviewed on the telephone each quarter. The data used in the paper covers the data collection period September 1997 through September 20033. We study the refusal, contact, cooperation and completion rates. The response rate4 is defined as the product of the contact, cooperation and completion rates. In addition, we examine other components of response. They are deafness/language barrier, busy, answering machine and no answer rates. The deafness/language barrier rate is examined since Hispanics form the fastest growing segments of the US population and is now the largest minority group. The busy rate is studied as a proxy for households using their phone lines for fax and internet communications. The answering machine rate will indicate the growth of these devices over the nearly six years of the data. Finally the no answer rate may be an indicator of a busier population that is harder to reach since they are unavailable more often. Appendix A contains the definitions of these rates. The next section presents the trends for these variables over the 24 quarters. 2 A team of interviewers that always work on a particular survey. 3 Interviewing on this surveys started in October 1996 and a tracking team was assigned in the summer of 1997. Data collection was stopped after October 2003. 4 The response rate used here is different than any of the AAPOR response rates. It is the “traditional” Gallup response rate this is the product of the contact, cooperation and completion rate. Gallup also calculates the CASRO response rate. This latter rate was not used in this paper in order to avoid having an estimate of the presumed eligible. However, it is noted that the CASRO response rate is typically between one and two per cent less than the Gallup response rate over the time period studied in this paper. 24 Robert D. Tortora 4 Trends The data used in this analysis comes from the response reports Gallup produces at the end of each CATI data collection effort. The main objective of this section is to analyze the variables listed in the previous section for significant trends over the time period October 1997 through Sept 2003. Table 1 gives the overall or combined refusal, contact, cooperation, completion and response rates over the time period studied. Table 1: Overall for Refusals, Contact, Cooperation, Completion and Response Rates by Quarter. Quarter - Ending Refusal Rates Contact Rates Cooperation Rates Completion Rates Response Rates 1 – Dec 1997 30.8 77.2 49.0 97.8 37.0 2 – Mar 1998 36.0 77.5 47.3 96.2 35.3 3 - June 1998 41.6 66.6 38.4 90.0 23.0 4 – Sept 1998 37.7 71.7 44.1 95.1 30.1 5 – Dec 1998 36.5 69.0 45.0 96.3 30.9 6 – Mar 1999 35.2 71.9 44.9 98.9 32.0 7 – June 1999 30.7 67.0 43.6 98.6 28.8 8 – Sept 1999 33.8 68.5 43.6 98.3 29.4 9 – Dec 1999 34.9 70.3 43.0 98.2 29.7 10 – Mar 2000 34.4 67.0 43.2 97.8 28.3 11 – June 2000 33.8 68.0 41.1 97.5 27.5 12 – Sept 2000 32.7 68.3 43.3 97.5 28.8 13 – Dec 2000 28.0 64.9 45.8 97.9 29.1 14 – Mar 2001 32.9 68.5 44.9 98.5 30.3 15 – June 2001 32.8 64.7 41.9 96.9 26.3 16 – Sept 2001 33.5 66.2 43.6 97.9 28.3 17 – Dec 2001 31.9 65.7 42.3 97.9 27.2 18 – Mar 2002 31.4 64.9 43.4 99.1 28.9 19 – June 2002 37.0 62.8 41.4 98.7 25.6 20 – Sept 2002 32.4 56.1 36.6 98.8 20.3 21 – Dec 2002 31.5 64.2 41.4 99.0 26.2 22 – Mar 2003 30.0 63.8 41.8 99.2 26.5 23 – June 2003 29.2 62.1 39.6 98.7 24.6 24 – Sept 2003 27.7 56.8 36.9 98.8 20.4 Mean 33.2 66.8 42.8 97.6 28.1 Max 41.6 77.5 49.0 99.2 37.0 Min 27.7 56.1 36.6 90.0 20.3 Response Trends in a National Random Digit Dial Survey 25 Refusal Contact Cooperation Response Completion Fiqure 1: Main rates. Table 2: Overall for Deafness/Language Barrier, Busy, Answering Machine and No Answer rates by Quarter. Quarter Deafness/Language Barrier Busy Answering Machine No Answer 1 – Dec 1997 1.8 1.1 3.5 18.3 2 – Mar 1998 2.0 1.4 2.8 18.3 3 - June 1998 2.0 1.8 9.6 21.9 4 – Sept 1998 2.5 2.3 6.0 20.0 5 – Dec 1998 2.2 2.2 7.2 21.7 6 – Mar 1999 2.3 2.0 6.2 19.8 7 – June 1999 2.3 2.1 8.3 22.6 8 – Sept 1999 2.3 2.2 7.1 22.2 9 – Dec 1999 2.2 2.3 6.0 21.4 10 – Mar 2000 2.3 2.5 7.7 22.8 11 – June 2000 2.4 2.0 8.0 21.9 12 – Sept 2000 2.8 1.9 7.8 21.9 13 – Dec 2000 2.7 2.3 8.7 24.0 14 – Mar 2001 3.0 2.4 7.4 21.8 15 – June 2001 3.2 2.3 10.2 22.9 16 – Sept 2001 3.3 2.2 9.7 21.9 17 – Dec 2001 3.3 2.5 10.7 21.0 18 – Mar 2002 3.5 2.7 11.8 20.6 19 – June 2002 3.3 2.5 13.6 21.2 20 – Sept 2002 2.2 3.2 17.2 23.5 21 – Dec 2002 3.0 2.7 12.3 20.8 22 – Mar 2003 3.1 2.9 13.0 20.3 23 – June 2003 3.3 2.6 14.3 20.2 24 – Sept 2003 2.7 3.0 18.4 22.5 Mean 2.6 2.3 9.5 21.4 Max 3.5 3.2 18.4 24.0 Min 1.8 1.1 2.8 18.3 120 100 80 60 40 20 0 Quarter 26 Robert D. Tortora Refusal rates average 33.9% with a high of 41.6% in Quarter 3 and a low of 28% in Quarter 13. Contact rates appear to decline with a maximum of 77.5% in Quarter 2 and a minimum of 56.1% in Quarter 20. Cooperation rates average 43% with a high of 49% in the first Quarter and a low of 36.6% in Quarter 20. Completion rates are very high averaging 97.6%. Finally response rates average 28.4% with a maximum of 37% in Quarter 1 and a low of 20.3% in Quarter 20. Figure 1 displays these rates over the 24 quarters. Table 2 reports overall rates for deafness/language barrier, busy, answering machine and no answer. The mean deafness/language barrier rate is 2.6% with a low of 1.8% in the first quarter and a high of 3.5% in Quarter 18. The busy rate has a mean of 2.3% with a high of 3.2% in Quarter 20 and a low of 1.1% in Quarter 1. The answering machine rate averages 9.5% with a maximum of 18.4% in Quarter 24 and a minimum of 2.8% in Quarter 2. The no answer rate averages 21.4% with a high of 24% in Quarter 13 and a low of 18.3% in Quarters 1 and 2. Figure 2 graphically displays these rates. Figure 2: Other rates. Looking at Tables 1 and 2 notice that for several of these variables the “best” rates are early quarters of the survey and there appears to be a degradation in the rates over time. The one exception is the refusal rate which appears to improve over time. Therefore we use the Haan (1997) t-test for linear trend in a time series to test hypotheses about trends in these variables. The test considers the linear regression of the independent variable time on the variable of interest. The test statistic is given by: t = r (n - 2)1/2 /(1 - r2)1/2 with n –2 degrees of freedom, Response Trends in a National Random Digit Dial Survey 27 where r is the Pearson correlation coefficient. The test statistic is compared to (Student) t values at the a = .05 level. The critical t value is 2.07 with 22 degrees of freedom. Table 3 summarizes the results of the testing. Table 3: Haan t-test for significant trend. Rate r t Refusal -0.59 -3.39* Contact -0.84 -7.30* Cooperation -0.62 -3.75* Completion 0.56 3.18* Response -0.70 -4.56* Deafness/Language Barrier 0.77 5.58* Busy 0.83 6.99* Answering Machine 0.88 8.60* No Answer 0.29 1.45 * a = 0.05 From Table 3 only No Answer shows no trend. Refusal, contact, cooperation and response rates are significantly declining. The completion rate is significantly increasing as is deafness/language barrier, the busy rate and answering machine rate. Table 4: Average of First Four Quarters, Average of Last Four Quarters for Those Variables with Significant Trend and Relative Change, Overall All Strata. Rate First Four Quarters (%) Last Four Quarters (%) Relative Change (%) Refusal 36.5 30.8 15.6 Contact 73.3 61.6 -16.0 Cooperation 44.7 39.9 -10.7 Completion 94.8 98.9 4.3 Response 31.4 24.4 -22.3 Deafness/Language Barrier 2.1 3.0 42.8 Busy 1.7 2.8 64.7 Answering Machine 5.5 14.5 164.6 No Answer 19.6 20.9 6.6 The direction of the trends are further illustrated by comparing the average of the first four quarters of each significant variable with the average of the last four quarters of data. Table 4 shows these averages for the data overall as well as the absolute relative change from the first four quarters to the last four quarters. Relative change is defined as ((FirstFourQuarterAverage–LastFourQuarterAverage)/FirstFourQuarterAverage)* 100 28 Robert D. Tortora Table 5: Correlation’s, r, among the rates of interest with the significance of the two tailed test (Sig.) with n = 24 observations. 0.048 0.822 0.85** 0 1 Refusal 1 0.392 0.137 -0.717** -0.364 -0.365 -0.403 -0.166 SIG 0.058 0.522 0 0.08 0.079 0.051 0.439 Contact 1 0.93** -0.272 -0.486* 0.869** -0.961** -0.629** SIG 0 0.198 0.016 0 0 0.001 Cooperation 0.971** 0.062 -0.23 -0.687** -0.848** -0.458* SIG 0 0.774 0.279 0 0 0.024 Response 1 0.021 -0.336 -0.756** -0.898** -0.587** SIG 0.922 0.109 0 0 0.003 Completion 1 0.402 0.438* 0.293 0.024 SIG 0.052 0.032 0.165 0.912 Deafness/ Language 1 0.533** 0.517** 0.101 SIG 0.007 0.01 0.639 Busy 1 0.835** 0.467* SIG 0 0.022 Answering Machine 1 0.398 SIG 0.054 No Answer 1 SIG . ** Correlation is significant a t the 0.0 1 level (2-tailed). * Correlation is significant at the 0.05 level (2-tailed). The largest relative change of 164.6% occurs for answering machines, increasing from an average of 5.5% in the first four quarters to 14.5% in the last four quarters. The next largest relative change is 64.7% for busy. But the magnitude of the actual change is small, increasing from an average of 1.7% in the first four quarters to 2.8% in the last four quarters. The third largest relative change is 42.8% for deafness/language barrier. Here again the magnitude is small increasing from an average of 2.1% in the first four quarters to 3.0% in the last four quarters. Looking at the response rate and its’ components we see that the contact rate has a relative change of -16%, decreasing from 73.3% to 61.6%. The cooperation rate has a relative change of -10.7%, decreasing from 44.7% to 39.9%. The completion rate increases from 94.8% to 98.9%, a relative change of 4.3%. These last three changes result in the response rate decreasing from 31.4% to 24.4%, a relative change of -22.3%. Finally the refusal rates drops from 36.5% to 30.8%, a relative change of 15.6%. Response Trends in a National Random Digit Dial Survey 29 Because the sample size seriously limits the number of independent variables one could include in a regression analysis we conclude this section with Table 5, the correlation’s, r, among the variables of interest. For significant correlation’s at the 0.01 level note that the refusal rate is only significantly (negatively) correlated with one variable, completion rate (-0.717). As expected response rate is significantly (positively) correlated with cooperation (0.97) and contact rate (0.93) and negatively correlated with the busy rate (-0.76), answering machine rate (-.90) and the no answer rate (-0.59). The deafness/language barrier rate is positively correlate with the busy rate (0.53) and the answering machine rate (0.52) and the busy rate is correlated with answering machine rate (0.84). At the 0.05 level contact rate is negatively correlated with the deafness/language barrier rate (0.49) and the completion rate is positively correlated with the busy rate (0.44). The next section summarizes the above findings and draws some conclusions with respect to response for RDD surveys. 5 Summary and conclusions The time series analysis of response trends suggests that there is good and bad news for RRD surveys. The good news is that refusal rates significantly declined over the 24 quarters and completion rates significantly increased. The bad news is that for all other response outcomes analyzed, except for “no answer” the situation is deteriorating. The significant decrease in contact and cooperation rates off set the increase in completion rates resulting in the significant decrease in response rates. Zeplin (2003) attributes the decrease in refusal rates to better and more consistent interviewer training. One important change that Gallup made was to be sure interviewers did not code answers like “I am busy now” or “I am just sitting down to eat” as a refusal. The small increase in completion rates is a positive sign. This increase may be attributed to interviewer training and that fact that a “tracking team” is used for the survey. However the completion rate started out at a relatively high level of 94.8% (first four quarters average) and now stands at 98.9% so it has little room to affect the response rates. The contact and cooperation rates are decreasing at a faster relative rate than the completion rate is increasing. It appears that the increasing presence of answering machines (from 5.5% in the first four quarters to 14.2% in the last four quarters) is associated with the decrease in contact rate. The use of other devices like caller id or number blocking devices may also be attributing to the decrease in contact rate. Tuckel and O’Niell (2001) mention that survey organization may have to develop their own brand to offset this problem. The “Gallup Poll” is arguably a well-know brand, perhaps the best-known survey brand in the world. Gallup has attempted to improve telephone efficiency by leaving a message on caller id devices that either the “Gallup Poll” or “Gallup” wording on the caller id device. 30 Robert D. Tortora Because of telephone company technology this could not be done experimentally but rather was done on a pre-post basis comparing pre message productivity data with the data from the time period when the messages were left on caller id. The evaluation was implemented in six call centers, three using each message. One RDD track and four listed tracks were studied. Ludwig (2003), in an analysis of the effect of this procedure found that a) the message “Gallup Poll” displayed on called id devices produces productivity improvements in terms of completes per hour and in terms of reduced capacity (projected production hours/ actual production hours. In addition, for one listed study the response rate increased for the “Gallup Poll” message. Because of the limitation on the design of this study caution should be used in generalizing the results. The increase in the busy rate may be attributed to multiple uses of a household telephone line for such things as accessing the Internet or using fax machines. The increase in the deafness/language barrier rate is most likely attributed to the increase of racial and ethnic minorities in the US. The Hispanic population is now the largest minority in the US. The Asian population is also increasing. Many recent immigrants may not speak English or not speak it well enough to feel comfortable with responding to a survey over the telephone. This survey use English speaking interviewers. With this increase in the Deafness/Language Barrier outcome survey organizations may have to consider using interviewers that can conduct Spanish language interviews or interviews in some of the major Asian languages and dialects. What are some of the impacts of the decreasing response rates? First, the decreasing response rates may require a heroic assumption about ignorable unit nonresponse. If the nonresponse is nonignorable then applying some method of nonresponse adjustment will be necessary. For surveys (like this one) that have a subset of the population (adults that have used the Internet in the last 30 days) as the target and where demographic data is not available for the target for nonresponse adjustment another nonresponse adjustment models may have to be used. Another alternative is to use a double sampling method where for a sub-sample of nonrespondents intensive methods are used to obtain cooperation. Both of these alternatives have cost implications. Improved call scheduling and longer (than the 5-call design used for this survey) call designs may help. The former requires an up-front investment that may or may not pay off. The latter has continuing cost implications as longer call-design keep interviewers in the field longer. In addition, the use of Hispanic or other language interviewers will also increase cost. Other options that may have to be used include sending advance letters to numbers in telephone directories, the use of incentives, and introducing multiple modes of data collection when possible. In short, if the results of this study can be generalized then the future of RDD surveys is bleak. Falling response rates have the potential to decrease data quality Response Trends in a National Random Digit Dial Survey 31 and it appears that using methods that at least attempt to stem the falling tide will drive cost increases which may in turn force clients to other methods. Acknowledgement The author would like to thank the referees for their valuable comments. References [1] DeHeer, W. (1999): International response trends: results of an international survey. JOS, 15, 129-142. [2] Groves, R.M. and Couper, M.P. (1998): Nonresponse in Household Interview Surveys. New York: John Wiley & Sons. [3] Haan, C.T. (1977): Statistical Methods in Hydrology. Ames, Iowa: Iowa State University Press. [4] Ludwig, J. (2003): Evaluating the impact of caller ID listing on interviewing operations. Internal Gallup presentation, Omaha, Nebraska. [5] Tortora, R. and Russo, S. (2001): Incentives in an Internet survey. Presented at the Annual Meeting of the American Statistical Association, Dallas, Texas. [6] Tuckel, P.S. and O'Neill, H.W. (2001): The vanishing respondent in telephone surveys. Presented at the annual conference of the American Association for Public Opinion Research, Montreal. [7] Steeh, C., Kirgis, N., Cannon, B., and DeWitt, J. (2000): Are they as bad as they seem? Nonresponse rates at the end of the twentieth century. Journal of Official Statistics (forthcoming). [8] Zeplin, J. (2003): Declining Refusals Rates. Personal Communication. 32 Robert D. Tortora Appendix Definitions This appendix contains the definitions of the rates studied in the paper. The first definition is for working numbers because it is used in many calculations. WorkingNumbers = Total Used - NonResidential/Business -NonWorking/Disconnected Contact rate = Contacted / Working Cooperation rate – Cooperated / Contacted Completion rate = Completed / Cooperated Response rate = Contact rate X Cooperation rate X Completion rate Refusal rate = Refusals / Contacted Deafness-Language Barrier rate = (Deafness-Language Barrier) / Working Busy rate = Busy / Working Answer Machine rate = Answering Machine / Working No Answer rate = No Answer / Working