COSTS AND BENEFITS OF AN AUTOMATED RESEARCH INTERVIEW: THE HEALTH ATTITUDES AND PRACTICES (HAP) SURVEY

Transcription

1 i3 COSTS AND BENEFITS OF AN AUTOMATED RESEARCH INTERVIEW: THE HEALTH ATTITUDES AND PRACTICES (HAP) SURVEY A.R. Meyers, Boston University, Boston, MA, USA M.E. Fortini, Vermont Alcohol Research Center, Burlington, VT, USA W.M. Arce-Quinones, University of Vermont, Burlington, VT, USA M.W. Perrine, Vermont Alcohol Research Center, Burlington, VT, USA Summary: The Vermont Alcohol Research Center's HAP survey is a comprehensive research instrument which incorporates several well-known assessment scales and includes extensive information about respondents' demographics, personal and family histories of alcohol use and alcohol-related problems, criminal histories, and drug use. This paper examines the costs and benefits of an automated version of this survey for a large research project with multiple remote sites. In addition to differences in the tangible costs between the automeated and printed versions, there are intangible costs and benefits, as well as suggestions for improvements which could lead to the wider use of the automated HAP. INTRODUCTION Personal computers (PCs) are said to be able revolutionize social science data collection, particularly when the data are obtained from long and detailed interviews of large samples using standardized protocols in a wide variety of field settings. Though this is an attractive hypothesis, there is little evidence that suggest the benefits of using PCs to collect data exceed the costs. Indeed, there is little serious documentation of relevant benefits and costs. The Vermont Alcohol Research Center (VARC) has developed a PC-based survey instrument, which is currently used at multiple research sites in three states (California, Maryland and Vermont). Some form of this instrument will ultimately be employed in a number of studies of large and heterogeneous samples from many other sites. This paper summarizes VARC researchers' calculations of major costs and benefits associated with this effort. The report begins with a description of the original HAP, followed by a discussion of the circumstances which led to the decision to use PCs. There is a detailed analysis of current and projected costs and benefits of the automated version, a critique of the automation process, and recommendations for others who would automate similar tasks. BACKGROUND: THE HEALTH ATTITUDES AND PRACTICES QUESTIONNAIRE The Vermont Alcohol Research Center's Health Attitudes and Practices (HAP) survey stems from earlier studies of drunken driving (Perrine, Waller, & Harris, 1971). The HAP is a comprehensive research instrument which incorporates such widely used and validated instruments as the Mortimer-Filkins Test (Mortimer, Filkins, Kerlan, & Lower, 1976), the Alcohol Dependence Scale (Skinner & Allen, 1982), and Donovan's General Beliefs, Attitudes, and Behavior and Driving Opinions and Behavior Scales (Donovan & Marlatt, 1982a, 1982b; Donovan, Marlatt, & Salzberg, 1983). The HAP also includes extensive 745

2 information about respondents' demographic characteristics; personal and family histories of alcohol use and alcohol-related problems; health status, attitudes, and behavior; attitudes toward alcohol use; knowledge and attitudes about drinking and driving; alcohol and drug-related behaviors, including, but not limited to drinking and driving; driving histories and behaviors, and arrests and criminal histories for a wide variety of offenses, including but not limited to alcohol-traffic offenses. There is also a long section which asks about the use of 12 different types of drugs. Each time the respondent says that he or she has used a particular drug, there is a series of nested questions about the use of the drug and driving, the combined use of alcohol and the drug, and driving after combining the drug and alcohol. A printed HAP protocol has 108 pagesand 463 questions, including substantial numbers of contingent questions and complicated branching logic. Completed interviews average about 90 minutes. Ultimately, VARC planned to complete 10,000 HAP interviews in a variety of settings, including offices, alcohol treatment facilities, jails, and respondents' homes. Upon completion of an interview, interviewers reviewed their completed questionnaires, noting responses that needed clarification. Completed HAP protocols were checked by site supervisors for completeness, logical consistency of contingent questions, and coding of open-ended questions. There was also a detailed review for internal consistency of each completed questionnaire. Data were entered into PCs as word processor files, cleaned in the word processor format, converted to standard (ASCII) format, and transferred to mainframe computer files for analysis. Thirty percent of the completed interviews were selected at random for quality control. THE DECISION TO AUTOMATE The experience of about 1,200 HAP interviews in Vermont and Virginia convinced VARC staff to change HAP data collection procedures. This decision was motivated mainly by concerns about data security, data management, and cost. The costs of transporting the first 1,200 completed protocols to Vermont, insuring them, and storing them in Vermont, with all necessary supplemental paperwork, added about $2.00 to the cost of each completed questionnaire. The corresponding costs for the next 12,000 questionnaires, which would have come from as far away as California, would have exceeded the total projected budget for data collection. Moreover, insurance would have covered only the replacement cost of the printed protocols (-$2.00 per copy), not the replacement costs of conducting the interviews, if any were lost or damaged in transit. Staff reviewed and rejected a number of options, before they decided to develop a PC-based automated HAP. This, they felt, could both expedite data collection and assure safe transfer of large numbers of completed HAP questionnaires from distant field sites to VARC headquarters in Burlington, Vermont. Following this decision, staff reviewed existing database and computeraided interview (CAI) software. Review criteria included purchase price, adaptability, ease of use ("friendliness"), and the ability to operate in a PC (rather than a mini or mainframe computer) environment, preferably, though not necessarily PC/MS-DOS. Due to the HAP questionnaire's size and complexity and the researchers' desire to conform as closely as possible to the original paper protocol, few commercial software packages were adequate. Satisfactory 746

3 software was either very expensive (i.e., purchase prices in excess of $15,000 and costly annual maintenance costs), or required extensive (and costly) training and continued reliance upon the vendor, or required that vendors rather than users retain copyrights. Therefore, VARC staff decided to develop their own HAP software, over which they could retain full copyright and which could be administered in a manner comparable to the original manual HAP. THE AUTOMATED HAP After abortive discussions with several software developers, and on the basis of a completely fortuitous contact, VARC signed a contract with The Oakland Group, Inc., of Cambridge, MA. The Oakland Group had extensive experience with custom data base software, but had never before developed applications for social science research. Working closely with VARC staff, Oakland developed an automated version of the HAP, for use in the MS/PC-DOS environment. Written in "C" language, the program has complicated branching logic features, screen prompts, full text editing, range checks, and an audible signal to mark out-of-range entries. It automatically saves each response as it is entered and has utilities to convert alpha-numeric data files to SPSS format. Another utility counts out-of-range responses, to monitor interviewers' accuracy. Another extracts responses to open-ended questions in two ways: (1) all respondents' responses to a single question, and (2) an individual's responses to all open-ended questions. The HAP program requires 640 Kilobytes of random access memory (RAM) and is written to operate with MS/PC-DOS 3.0 or higher and an 8088 or higher central processing unit. Each completed HAP interview occupies about 275 Kilobytes of storage memory, most of which contains the interview text. Data files are recorded on the same drive as the HAP program. Each questionnaire's abstracted data, in a form ready for analysis by SPSS on a mainframe computer, occupies about 2 Kilobytes of storage space. Though the HAP software can be used with any computer in the MS/PC-DOS environment, VARC interviewers use two kinds of hardware: Zenith (Z-181) laptop computers (equipped with 2, 3-1/2-inch [720 Kilobyte] disk drives) for field interviews, and desktop computers, with both floppy and hard drives for office use. A 3-1/2 inch flexible diskette (720 Kilobytes) can accommodate the program and 25 completed interviews. VARC staff developed a new training manual for interviewers and site supervisors who use the automated HAP. Interviewer training for the automated version requires about two hours more than the training for the manual version. Field supervisors' protocols for in-house reviewing, auditing, and transmitting completed interviews to VARC headquarters, were modified slightly when the HAP was automated. Completed interviews are sent from field sites to VARC headquarters on 3-1/2" diskettes in lots of 25 completed interviews. Copies of each diskette are kept on site until the data have been checked and approved by VARC staff. Once approval has been given, site supervisors clear the duplicate diskettes and re-use them. COSTS AND BENEFITS Table 1 indicates that the total cost of the development and implementation of the automated HAP protocol was approximately $31,000, or for the first 1000 automated HAPs, about $31.00 per interview, excluding inter 747

4 viewer wages. The cost per interview decreases with additional interviews. Given the projected target of 10,000, and an inflation rate of 5% per year, the total cost per interview will be approximately $4.00. Table 2 shows the comparable total costs for the first 1,000 HAP interviews, using paper protocols. This figure represents a unit cost of approximately $ After completing the estimated 10,000 interviews and adjusting for inflation, the unit cost of the automated HAP ($4.02) is less than half that of the manual HAP ($8.19). The automation of the HAP has addressed the original concerns about data security, data storage, and data transfer. Automation also has virtually eliminated the costs of coding, data entry, and storage, and drastically reduced quality control and editing costs, resulting in a largely unanticipated saving of approximately $7,125 for the first 1,000 interviews. Automation has had other benefits more difficult to quantify. Logical errors of omission and commission have been substantially reduced, because the software has branching logic and will not permit the recording of logically inconsistent responses. This property, in turn, has significantly reduced interviewer errors, particularly in the "drug section" of the HAP, where there several nested loops of contingent questions. PROBLEMS AND RECOMMENDATIONS The automation of the HAP has not been completely without problems. For example, the program is so large and complicated that it originally contained a subtle "bug," which appeared only after several entry errors. Apparently, after some unknown number of attempted data entry errors, the next error resulted in the computer keyboard locking and sounding a constant error message. In order to continue the interview, it was necessary to "warm boot" the computer and return to the beginning of the interview. Fortunately, this happened very infrequently and most often with inexperienced interviewers. Moreover, since the HAP software saved responses after each question, the interviewer could return to the point of interruption and then continue with no risk of data loss. However, such interruptions were annoying and disruptive, both for interviewers and subjects, and often resulted in the loss of rapport, especially with new interviewers and reluctant subjects. Although automation is clearly seen as a positive change by VARC members, there was a transition period in which many lessons were learned. Using the manual protocol, interviewers often wrote useful information about respondents and the interview process in the margins: for example, hesitations, changes of answers, and questions for the supervisors. This could be done unobtrusively, because comments were written in the HAP questionnaire itself. Once automated, there were no margins. Therefore, about two months after the automated version was implemented, staff developed a "Special Notes Sheet" on which interviewers could record such information. However, this method is more obtrusive and, although informative, has not been as useful as margin notes had been. A software "notepad window" might prove less obtrusive, but has not yet been added to the automated HAP. An additional modification would be a change in the default drive. The present version of the HAP saves interviews on the same drive as the interview program. It would save supervisor time if the completed interviews were 748

5 automatically saved to another disk drive or to a hard disk, thereby eliminating the need to clean and recycle diskettes continuously. During the transition from manual to automated protocol, other similar problems were encountered, which resulted in some loss of data. Approximately four months were needed to identify the need and to develop new policies. The automated version of the HAP has now been used for over two years with the new policies (approximately 3,200 interviews to date). Although hindsight provides clearer vision, more time should have been spent identifying the changes required by automation. In summary, automation of the HAP has been beneficial in terms of the original concerns about data security, storage, and transfer. Substantial savings in management costs and data accuracy have been achieved. The savings in costs and data accuracy lend support to the wider use of the automated HAP, and to automation of other research protocols that are large, complex, and administered at multiple, distant sites. ACKNOWLEDGEMENT Preparation of this paper was supported by PHS Reserach Grants R01-AA06926 and R01-AA06774 from the National Institute on Alcohol Abuse and Alcoholism to M.W. Perrine, Principal Investigator REFERENCES Donovan, D.M., & Marlatt, G.A. (1982a). Personality subtypes among drivingwhile-intoxicated offenders: Relationship to drinking behavior and driving risk. Journal of Consulting and Clinical Psychology, 50, Donovan, D.M., & Marlatt, G.A. (1982b). Brief report: Reasons for drinking among DUI arrestees. Addictive Behaviors, ]_, Donovan, D.M., Marlatt, G.A., & Salzberg, P.M. (1983). Drinking behavior, personality factors, and high risk driving: A review and theoretical formulation. Journal of Studies of Alcohol, 44, Mortimer, R.G., Filkins, L.D., Kerlan, M.W., & Lower, J.S. (1976). Psychometric identification of problem drinkers. Quarterly Journal of Studies of Alcohol, 83, Perrine, M.W., WWaller, J.A., & Harris, L.S. (1971). Alcohol and highway safety: Behavioral and medical aspects (Technical Report, DOT HS ). Washington, D.C.: National Highway Traffic Safety Administration. Skinner, H.A., & Allen, B.A. (1982). Alcohol dependence syndrome: Measurement and validation. Journal of Abnormal Psychology, 91,

6 Table 1. Total costs for development of the automated HAP. Category interviews) Personnel Development Costs Travel Software Product review Equipment Computers Cost Assumptions and information (first $4,429 2 Research 5% effort for 1 year (includes 25% fringe) Responsibilities included: researching existing software packages, interviewing Oakland Group, meetings with Oakland Group, development of training procedures and manual trips from Vermont to Boston for meetings with Oakland Group 7,800 Oakland Group staff members totaling 48 $9.50/hour Responsibilities included: conducting mock interviews to examine all possible branching, entry errors, etc. 16, Zenith laptop $1,624 (includes accessories) Quality control and editing Data Transmittal Shipping Storage Total Expenses 200 1,008 $30,813 2 hours per 100 $10.00 per hour. $28 per month (Federal Express) per site (3) for 12 months. The "per interview" rate for shipping decreases as the number of interviews per month increases. At the present time, over 3,000 interviews have been conducted using the automated HAP. All data disks fit in the drawer of one file cabinet. Since the only continuing costs of the automated HAP are quality control, editing, and shipping, the basic cost per interview (excluding interviewer wages) will decrease as the number of interviews increases. Cost per interview, including a 5% per year inflation rate, is presented below: interviews $30.81 Total costs from above ($30,813) interviews Includes one additional year of cost ($32,354) interviews 6.96 Includes another 1 1/2 years of costs ($34,780) interviews 4.99 Includes another 1 1/2 years of costs ($37,389) 10,000 interviews 4.02 ncludes another 1 1/2 years of costs (40,193) 750

7 Table 2. Total costs for use of manual HAP protocol. Category interviews) Printing Coding Costs $2,450 1,500 Assumptions and information (first $2.45 (1986 prices) 4 interviews $6.00/hour Included: coding all open-ended questions, checking all responses to insure valid response categories Data Entry 3, interviews $6.00/hour Quality Control 2,000 3 interviews $6.00/hour Editing $6.00/hour Data Transmittal Shipping 933 interviews. Purolator courier. Cost for shipment of 1000 (1987 prices) Includes insurance for replacement of paper protocols. Storage 1,140 Rental of 8 ' x 10' storage space for 1 year. TOTAL EXPENSES $11,716 Assuming 5% inflation per year, and that the storage space rented is large enough to hold all the completed interviews, the costs per interview change only slightly. 1,000 interviews $11.72 Total costs from above ($11,716) 2,500 interviews 9.84 Includes one additional year of costs 5,500 interviews 8.82 Includes another 1 1/2 years of costs 7,500 interviews 8.40 Includes another 1 1/2 years of costs 10,000 interviews 8.19 Includes another 1 1/2 years of costs ($24,596) ($44,075) ($62,968) ($81,915) 751