Psychology of Addictive Behaviors Copyright 2002 by the American Psychological Association, Inc.

Transcription

1 Page 1 of 15 Psychology of Addictive Behaviors Copyright 2002 by the American Psychological Association, Inc. Volume 16(1) March 2002 p Test Retest Reliability of Alcohol Measures: Is There a Difference Between Internet-Based Assessment and Traditional Methods? [Articles] Miller, Elizabeth T. 1,4 ; Neal, Dan J. 1 ; Roberts, Lisa J. 1 ; Baer, John S. 2,3 ; Cressler, Sally O. 1 ; Metrik, Jane 1 ; Marlatt, G Alan 1 1 Addictive Behaviors Research Center, Department of Psychology, University of Washington 2 Department of Psychology, University of Washington 3 Veterans Affairs Puget Sound Health Care System Center of Excellence in Substance Abuse Treatment and Education, Seattle, Washington. 4 Correspondence concerning this article should be addressed to Elizabeth T. Miller, DatStat Inc., 8835 Paisley Place NE, 2nd Floor, Seattle, Washington lizza@datstat.com. Dan J. Neal is now at the Department of Psychology, Syracuse University, and the Interdisciplinary Program of Applied Statistics, Syracuse University. Lisa J. Roberts is now at the Veterans Affairs Puget Sound Health Care System Mental Illness Research, Education and Clinical Center, Seattle, Washington. Sally O. Cressler is now at the Department of Counseling Psychology, University of Wisconsin Madison. Jane Metrik is now at University of California, San Diego/San Diego State University, Joint Doctoral Program in Clinical Psychology. This research was supported by Grant AA05591 from the National Institute on Alcohol Abuse and Alcoholism, awarded to G. Alan Marlatt. We gratefully acknowledge the support and assistance of George Dittmeier, Sally Weatherford, Mary Larimer, Leah Era, Jill Carlsen, and Nina Mauritzen; article review by Dan Kivlahan, Thad Leffingwell, and Aaron Turner; and editing by Jessica Cronce. Received Date: August 23, 1999; Revised Date: July 22, 2001; Accepted Date: August 9, 2001 Outline Abstract Method Results Participants Assessment Format and Incentives Measures P&P condition. Web condition. Web-I condition. Demographics. Screening for hazardous alcohol use. Assessment of alcohol dependence. Alcohol-related problems. Drinking rates. Readiness to change. Assessment format preferences. Preliminary Analyses Analytic Approach Test Retest Reliability Validity

2 Page 2 of 15 Subjective Convenience and Preferences Discussion References Graphics Table 1 Table 2 Table 3 Abstract This study compared Web-based assessment techniques with traditional paper-based methods of commonly used measures of alcohol use. Test retest reliabilities were obtained, and tests of validity were conducted. A total of 255 participants were randomly assigned to 1 of 3 conditions: paper-based (P&P), Web-based (Web), or Web-based with interruption (Web-I). Follow-up assessments 1 week later indicated reliabilities ranging from.59 to.93 within all measures and across all assessment methods. Significantly high test retest reliability coefficients support the use of these measures for research and clinical applications. Furthermore, no significant differences were found between assessment techniques, suggesting that Web-based methods are a suitable alternative to more traditional methods. This cost-efficient alternative has the advantage of minimizing data collection and entry errors while increasing survey accessibility. The use of emergent Web-based computer technology offers an unprecedented opportunity to conduct cross-sectional and longitudinal research studies in a cost-efficient manner while increasing survey accessibility to study participants and providing a more accurate data collection alternative to researchers (Miller, 1997). Costs associated with traditional assessment methods, such as publishing and distributing paper surveys, mailing materials to and telephoning study participant reminders, and data collection and entry are eliminated. The estimated costs to develop, publish, and maintain Web-based surveys are significantly lower (Schmidt, 1997). In addition, the data retrieved from Web-based surveys are potentially more accurate and more complete, and they provide essentially clean data moments after the survey is completed. The Web may prove superior to paper, because it potentially provides increased accessibility; capability for dynamic and interactive forms, which eliminate the viewing of irrelevant questions; and customized feedback tailored to the content of the responses. Given all of the aforementioned benefits of Web-based research, there are concomitant concerns about the reliability and validity of the data collected by means of this technology. Previous research indicates that the psychometric properties of computerized psychological assessments are not compromised (Skinner & Pakula, 1986) and that disclosure of high-risk sexual behaviors, HIV infection, and alcohol and tobacco misuse may in fact be enhanced (Gerbert et al., 1999; Turner et al., 1998). Web-based assessments of personality constructs do not appear to compromise the psychometric properties of the measures used (Buchanan & Smith, 1999; Pasveer & Ellard, 1998; Smith & Leigh, 1997). It is unfortunate that within the field of alcohol research there are limited data available on the

3 Page 3 of 15 psychometric properties of measures of rates of alcohol use, symptoms, and problems (Del Boca & Brown, 1996) and a troubling omission of test retest reliability data for alcohol screening measures (Allen, Litten, Fertig, & Babor, 1997). For example, previous reports provide psychometric data regarding internal consistency of a commonly used measure of alcohol problems, the Alcohol Dependence Scale (ADS; Skinner & Allen, 1982), and a commonly used alcohol problem screen, the Alcohol Use Disorders Identification Test (AUDIT; Allen et al., 1997). However, to our knowledge there is no current published information on test retest reliabilities for the AUDIT or the ADS. Neither are we aware of test retest reliabilities for other frequently used measures, such as the Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989), a problem screen for adolescents, or measures of stages of change (i.e., the University of Rhode Island Change Assessment [URICA]; Prochaska & DiClemente, 1986). Test retest reliability data are available for measures of alcohol consumption (quantity: r =.93, frequency: r =.87, and peak: r =.94) with a 30-day interval between assessments (L. C. Sobell, Sobell, Leo, & Cancilla, 1988; M. B. Sobell, Sobell, Klajner, Pavan, & Basian, 1986). Virtually no data exist comparing the psychometric properties of established measures based on means of administration: Web or paper and pencil. Our primary goal was to compare traditional test administration methods (paper-based) with the use of innovative Web-based assessment techniques. A secondary benefit of the study was to provide reliability data for measures commonly used in research on alcohol use. We chose to study college students not only because of national concerns about the risks of heavy drinking on college campuses (Johnston, O'Malley, & Bachman, 1996) but also because college students present a range of drinking habits (from abstinence to heavy problem use) and a range of experience with the Web (from those who never use the Internet to those who use the Internet on a regular daily basis). Method Participants The participants were 255 undergraduate students (aged 18 29) at a large West coast university recruited by means of on-campus newspaper advertisements and flyers. Members of the fraternity and sorority system and graduate students were excluded from the study because of potential conflict with other ongoing research projects. The average age of the sample was 20.9 years (SD = 1.95), with 25 individuals (10%) between the ages of 24 and 29. Participants were primarily female (64%), with a range of ethnic groups represented (60% Caucasian, 26% Asian/Pacific Islander, 4% Hispanic, 3% African American, and 7% Other). The majority of participants (62%) lived off campus; 26% lived on campus, 11% lived at home with family, and 1% indicated other living arrangements. This sample included a range of self-reported levels of alcohol use: light users (45%), moderate users (37%), heavy users (5%), abstainers (11%), and those who had never tried alcohol (3%). There were no significant differences in drinking rates as a function of any demographic variables. Regardless of drinking status, the median number of drinks consumed per week by all students was 2. Wechsler, Molnar, Davenport, and Baer (1999) similarly reported a median of 1.5 drinks per week among a representative national sample of college students. The first 300 participants who returned a signed consent form were randomly assigned to one of three conditions: paper-based (P&P, N = 100), Web-based (Web, N = 100), or Web-based with interruption (Web-I, N = 100). We included the final condition to determine whether taking a break (for a minimum of 1 hr and a maximum of 48 hr) would affect the psychometric properties of the data. Having an interruption provided a proxy of real world interruptions (e.g., participant fatigue, lack of time to initiate or complete the survey) that may be common with

4 Page 4 of 15 self-paced and home-based assessments and may lower a test's reliability (Babor, 1996). A total of 280 participants (93% of those randomized) completed the assessment at Time 1 (P&P, n = 94; Web, n = 93; Web-I, n = 93), resulting in an attrition rate of 7% for the Time 1 assessment. At Time 2, 255 of the participants completed the assessment (P&P, n = 88; Web, n = 83; Web-I, n = 84), resulting in an overall attrition rate of 15% (P&P: 12%, Web: 17%, Web-I: 16%), [chi] 2 (3, N = 280) = 0.165, p =.92. Comparisons of demographic variables and drinking measures assessed at baseline made with analysis of variance (ANOVA) and chi-square procedures between participants with complete data and those lost to attrition at Time 2 revealed no significant differences in age, sex, ethnicity, residence, drinking rates (quantity and frequency), alcohol-related problems (RAPI scores), or alcohol dependence (AUDIT and ADS scores). Assessment Format and Incentives Participants were randomly assigned to one of three conditions (see following sections). Communication was conducted primarily through electronic mail ( ), although a telephone number was also provided in the event of problems or questions. reminders were sent out before each assessment along with instructions on how to access and complete the Web-based assessments for participants assigned to Web conditions and where to pick up and return the paper-and-pencil packet. The measures were identical in terms of questions and possible responses. The only difference between the P&P and Web conditions was the method of data collection. Participants completed Web-based assessments by accessing a designated secure Web site.1 Participants received $15 payment for each completed assessment. Payments were mailed to all participants on the last day of each of the data collection periods. P&P condition. Participants (N = 88) picked up each of the Time 1 and Time 2 packets of paper-based selfreport measures at the Addictive Behaviors Research Center (ABRC). On completion, at each time period, they deposited their completed packet in a secure drop box at the ABRC. Web condition. For the purpose of security and data integrity, participants (N = 83) were instructed to access a secure Web site for the study and enter a personal identification number composed of the student identification number and birthdate. Internet or survey-related problems were addressed via e- mail and telephone. Participants who did not have access to the Internet (N = 1) were provided access to computers at the ABRC. Participants were also reminded that some of the departmental locations would be public, thus possibly minimizing privacy. On completion of the survey, participants were prompted to submit their data. On submission, the data were automatically entered into a tab-delimited format file and were no longer available to participants. Web-I condition. The Web-I condition was identical to the Web condition, with one exception: Participants (N = 84) were asked to take a break from their survey by quitting the browser at any point during the survey and reconnecting to the secure study Web site when they were ready to resume. Once logged back into the survey, participants automatically returned to the page, which they had bookmarked, where previously entered data were saved. The interruption period was not predetermined and could range from a minimum of 1 hr to a maximum of 48 hr. This experimental condition was included to test for the reliability of results with a break during the

5 Page 5 of 15 assessment procedure. Measures Measures assessed in this study included screening for hazardous use of alcohol, dependence, alcohol-related negative consequences, and measures of consumption (quantity, frequency, and peak). Motivation to change and stage of change were also assessed. All participants were assessed on two separate occasions during the same timeframe. Data collection was conducted over two 48-hr periods 1 week apart. Demographics. Demographic information included age, sex, ethnicity, height, weight, and resident status. Weight was included in these analyses for purposes of estimating blood alcohol level. Screening for hazardous alcohol use. The AUDIT (Babor et al., 1992; Saunders, Aasland, Babor, de la Fuente, & Grant, 1993) is a measure used to identify individuals at risk for developing alcohol use disorders. It is a 10-item questionnaire related to dependence criteria. Assessment of alcohol dependence. Participants also completed the ADS (Skinner & Allen, 1982), a widely used assessment of severity of physical dependence symptoms. Alcohol-related problems. Participants completed the RAPI (White & Labouvie, 1989), which asks respondents to rate the frequency of occurrence of 23 items reflecting alcohol's impact on social and health functioning over the past 6 months. Sample items include not able to work or study for a test, caused shame or embarrassment, was told by a friend or neighbor to stop or cut down on drinking. Drinking rates. We assessed drinking rates using three different measures of alcohol use at each assessment. Participants reported their typical drinking quantity, frequency, and the single greatest amount of alcohol consumption (peak consumption) over the past month. For the assessment of typical drinking quantity and most recent peak consumption, response options ranged from 0 to 15 or more drinks. Participants also reported their average drinking quantity and peak consumption for each day of a typical week (Collins, Parks, & Marlatt, 1985). Response options and associated labels for the assessment of number of drinks ranged from 0 to 15 or more drinks. To assess the number of hours over which the drinks were consumed, response options and associated labels ranged from 0 to 1 hr (0) to 10 or more hr (10). We used this to compute a weekly average of alcohol consumption. These quantity frequency peak indexes have been effective in documenting reductions in drinking in previous studies with college student drinkers (Baer, 1993; Kivlahan, Marlatt, Fromme, Coppel, & Williams, 1990; Marlatt, Baer, & Larimer, 1995). Readiness to change.

6 Page 6 of 15 We used a modified version of the URICA (Prochaska & DiClemente, 1986) to measure participants' increases in precontemplation, contemplation, action, and maintenance scores as well as readiness-to-change behavior. We adapted the URICA to reflect stages of change for alcohol use, rather than problem smoking, and shortened it to include only items relevant for college-age students, a 20-item version. Sample items include As far as I'm concerned, my drinking does not need changing, Sometimes I think I should cut down on my drinking, I have a problem with alcohol and I really think I should work on it. Assessment format preferences. We measured assessment format preferences by comparing three self-report items: perceived accuracy of responses, convenience of assessment method, and future format preference. The accuracy of responses was captured in the single item How accurate were your responses to this survey? with response options ranging from 0%, completely inaccurate(0) to 100%, completely accurate(100) in 10% increments. The convenience-of-assessment-method item read How convenient was it to complete this survey on paper/web? (depending on format). Response options included not at all (0), slightly (1), moderately (2), very (3), or extremely (4). We assessed future format preference with the following item: In the future, how would you prefer to complete a survey? Response options included on the Web (0), on paper (1), or either way (2). Results Preliminary Analyses We first compared assessment format groups with respect to demographic variables, including age, gender, ethnicity, residence, and level of alcohol use. No significant differences were observed. Next, with a series of one-way ANOVAs we examined mean differences in responses by assessment format at Time 1 and 1 week later, at Time 2 (see Table 1). Overall there were no significant mean differences among the three assessment groups on any measures of alcohol use at Time 1 or Time 2. It should be noted that, given our sample size, we had power of.80 to detect only moderate effect sizes (i.e., F = 0.22). However, given such power one would expect significant results for approximately one quarter of the tests at random, which was not the case. In addition, a multivariate analysis of variance of repeated measures over time revealed no significant patterns of change as a function of method of assessment, multivariate F(30, 392) = 1.32, p =.12, ns.

7 Page 7 of 15 Table 1 Means, Standard Deviations, and Significance by Group and Measure at Time 1 and Time 2 Of the Web and Web-I participants, 78 (47%) reported completing the Web-based survey at a campus computer cluster, 60 (36%) reported completing it at home, 14 (9%) said they completed it at work, 6 (4%) completed it at a friend's home, 3 (2%) completed it at a parent's home, and 3 (2%) completed it at another university. On average, participants in the Web-I condition took close to a 3-hr break (M = 2.89, SD = 6.31), with 9 participants extending the break beyond a 12- hr period. Analytic Approach Because of the heavily skewed distributions of our data, and the need to provide confidence intervals for tests of differences between reliability estimates, we adopted a bootstrap approach to test for differences in reliabilities among groups (Efron & Tibshirani, 1993). Bootstrapping is a nonparametric technique that involves repeatedly resampling with replacement from the data set to approximate the distribution function of the statistic. Unlike the Fisher R-to-Z method, which is a traditional approach to comparing independent correlation coefficients, bootstrapping assumes not that the underlying distribution of the data is bivariate normal but only that the empirical distribution is representative of the population. Confidence intervals constructed by means of a bootstrapping method are more likely, probabilistically speaking, to contain the more accurate parameter estimates as compared with the confidence intervals constructed by means of traditional normal-distribution based formulas. Using bootstrap analyses, we made pairwise comparisons between the P&P versus Web groups, the P&P versus Web-I groups, and the Web versus Web-I groups, for each of 16 measures. Therefore, a total of 48 comparisons were made. For each correlation coefficient we created 1,000 bootstrap samples, yielding 1,000 estimates for the parameter. We then estimated means and standard errors for each group. We then applied a two-sample Z test using the means and standard errors from the bootstrap replications. Test Retest Reliability

8 Page 8 of 15 We assessed in two ways the overall test retest reliabilities for all the measures, collapsed across groups: Pearson's product moment correlation coefficients and intraclass correlations. The intraclass correlation coefficient, which measures agreement, is a more stringent assessment of test retest reliability than Pearson's r, which measures association (Cicchetti, 1994). According to the guidelines described by Cicchetti (1994), when the reliability is below.70, the level of clinical significance is unacceptable; when it is between.70 and.79, it is fair; when it is between.80 and.89, it is good; and when it is.90 or above it is excellent. Table 2 shows Pearson's reliability coefficients and the intraclass correlation coefficients for the test and retest ratings of each of the alcohol measures with their associated subscales when appropriate. The test retest reliabilities ranged from.59 to.93. All of the correlation coefficients observed were significant at the.01 level (two-tailed). Thus, these measures have sufficient reliability for both scientific research and clinical applications. Table 2 Pearson Product Moment and Intraclass Correlation (ICC) Test Retest Reliabilities With the bootstrap technique, only 3 significant differences out of 48 comparisons emerged when group comparisons were conducted with alpha set at.05. On the AUDIT Quantity Frequency subscale, there were significant differences between the P&P and Web groups (z = 2.22, p =.03) and the Web and Web-I groups (z = 2.35, p =.02). On the AUDIT Dependence subscale there was a significant difference between the Web and Web-I groups (z = 2.04, p =.04). As shown in Table 2, when alpha is relaxed to.10 and.25, 5 and 12 significant differences emerge, respectively. Given the large number of analyses needed to compare each measure between each group and the alpha used in the tests, it is not unrealistic to expect to find statistically significant results; in fact, it would be more surprising if there had been no statistically significant differences. The problem arises from an overinflated simultaneous error rate. The more tests that are run, the more likely it becomes to find a significant result even when the null hypothesis is true (cf. Moore & McCabe, 1993). Given that at each level of significance the percentage of significant tests is approximately equal to alpha, we interpreted these significant results as simple random deviations that could occur in probability testing. This conclusion is further strengthened by the fact that the pattern of significant effects appears to be random; if small effect sizes were present, one would expect that the pattern of results would at least partially indicate that one

9 Page 9 of 15 condition is showing higher test retest reliabilities compared with the other two conditions. Validity An examination of all pairwise correlations across three groups was prohibitively large for bootstrap analyses. We used two techniques to test issues of validity across experimental conditions: Box's M test (Box, 1949) and comparison of a subset of correlations between variables. Box's M test is a test of the equality of covariance matrices. Box's M was computed on the covariance matrices of the three groups, and each covariance matrix included both the Time 1 and Time 2 total scores for each measure (i.e., each matrix included the ADS; AUDIT; RAPI 1 month, 6 months, and 1 year; URICA; peak quantity, peak blood alcohol concentration, and average quantity at Time 1 and Time 2. (To simplify analyses, subscales were not included.) Box's M test indicated significant differences among the three covariance matrices, M = 721.6, F (342, 50390) = 1.74, p <.001. We examined validity by comparing selected correlations across groups. First, we chose three measures to reduce the overall number of tests conducted and therefore reduce the probability of Type I errors. We used the three measures of average quantity/week, ADS, and AUDIT, because they best represent standard measures of alcohol-related use, abuse, and negative consequences. The intercorrelations for each group were calculated (i.e., for average quantity and ADS, average quantity and AUDIT, and ADS and AUDIT we calculated three correlation coefficients for every assessment method: P&P, Web, and Web-I) for both assessment periods and are presented in Table 3. Table 3 Intercorrelations of Measures by Group at Time 1 and Time 2 Next, we made a series of pairwise comparisons of the intercorrelations. We again computed standard errors for the intercorrelations using bootstrap estimates with 1,000 replications. For each group, pairwise comparisons were made against the other two groups for each correlation between the same measures, leading to three significance tests (P&P vs. Web, P&P vs. Web-I, and Web vs. Web-I) at each assessment for each pair of measures. We computed a total of 18 comparisons. At the.05 level, there were no significant differences in any of the intercorrelations at either assessment period; there were two significant differences when alpha was relaxed to.10. The results of these analyses, and the confidence intervals for the intercorrelations, are presented in Table 3. Thus, although Box's M indicates some significant differences among the covariance matrices of the three groups, few differences were found with inspection. Box's M could also be inflated on the basis of the non-normality of the variables that were selected for the analyses. In either

10 Page 10 of 15 case, the magnitude of the Box's M is quite small, suggesting that differences in validity across experimental conditions, if replicable, are likely small. Our selective examination of pairwise correlations suggests possible differences in the relationship between drinking quantity and ADS scores based on methods of administration (p <.10), although patterns of differences were not consistent when examined at Time 1 and Time 2. Subjective Convenience and Preferences We conducted analyses to test for differences among subjective ratings of accuracy, convenience, and assessment format preference between participants completing Web-based versus paper-based assessments. An ANOVA revealed no significant differences between selfreported accuracy of response reporting, F(2, 250) = 1.64, p =.20, ns. Highly significant differences were detected among groups in terms of convenience of use, with 26% of P&P participants reporting survey completion as being slightly to not at all convenient, compared with only 7% of Web and 7% of Web-I participants, whereas more than 80% of Web and 80% of the Web-I participants reported that completing the Web survey was very to extremely convenient, compared with only 56% of the P&P participants, F(2, 207) = 13.42, p <.001. Significant differences were also found among groups in terms of assessment preference in the event of a future survey. More than 40% of P&P participants indicated a preference to complete a Web survey, and 63% of Web and 55% of Web-I participants indicated a preference for the same method. Only 16% of the P&P participants said they would rather complete another paper survey, and close to 40% reported no preference. Moreover, only 6% of Web and 2% of Web-I participants reported a preference for a paper compared with a Web survey, and slightly more than 30% reported no preference, F(2, 250) = 12.79, p <.001. Discussion In this study we compared Web-based assessment techniques with traditional paper-based methods and obtained test retest reliabilities of measures commonly used in research on alcohol use. Our results provide evidence for the test retest reliability of the total scores for the ADS, AUDIT, RAPI, URICA, and quantity frequency items for research and clinical applications. However, we caution against reliance on the AUDIT Dependence subscales and the URICA subscales. Our data generally demonstrate that completing a survey on the Web did not result in moderate to large differences in response sets of participants compared with those of participants who completed a paper survey. No significant differences were found between assessment techniques on test retest reliability, suggesting that Web-based modes of data collection do not compromise the integrity of the data and are a suitable alternative to more traditional methods. Our data, as currently analyzed, appear encouraging yet equivocal with respect to differences in validity of scales as a result of method of test administration. Some differences likely exist, although they appear small. We found that allowing breaks during a lengthy Web-based assessment battery did not compromise the reliability or validity of the measures. Students preferred the Web-based assessment to the paper-and-pencil assessment. It should also be noted that although we failed to find many significant differences, this could be a function of our analytic technique. Because of the non-normality inherent in the type of data we collected, we chose to use a bootstrapping method that does not rely on any assumptions regarding the marginal or joint distributions of the data. A different approach (i.e., Fisher's r to

11 Page 11 of 15 Z), which requires an assumption of bivariate normality, produces some significant differences between the paper-and-pencil and Web-interrupted versions in the intercorrelations between weekly quantity and other measures.2 We believe, however, that in this case the bootstrapped standard errors do yield results that are more accurate with regard to the variability of the estimates of reliability and intercorrelation. With regard to the analyses reported here, it should be noted further that the power to detect small differences was relatively poor. For example, with our ANOVAs, although the power to detect medium to large effects was quite adequate (for medium F = 0.25, power =.95, and large F = 0.4, power =.99), for quite small (F = 0.10) effect sizes, the corresponding power estimate was only.28. Thus, it is ultimately impossible to determine whether the nonsignificant differences observed herein were a result of a true null hypothesis or a lack of statistical power. Yet, given the results reported, it is also reasonable to conclude that if group differences existed that were in fact different from zero, they would be very small. As previously noted, the lack of significant findings beyond what would be expected by chance provides further support for the null hypothesis. The testing of differences in validity among many measures posed additional challenges. Our analyses suggest that some small differences are likely as a function of assessment method, yet we do not have more specific clues for which among many intercorrelations are most affected. It is possible, of course, that we simply did not select variables where differences exist. Studies with larger samples with greater power might detect differences not observed herein. Most participants (80%) found the Web-based survey very convenient to use, and only 8% indicated a preference to use a paper-based survey in the future, if given a choice. Because participants were randomly assigned to conditions, and thus required to complete the survey on the Web (regardless of previous Internet experience), they were obliged to learn how to access the Internet for purposes other than . If study participants had been given the option to complete the survey on the Web or on paper, these results may have been different. Anecdotal evidence, including open-ended survey comment sections, suggests that this opportunity increased some participants' sense of empowerment in their general use of the Internet and increased the likelihood of future Internet use for coursework and other activities. Previous Webbased assessment studies suggest that students with no previous Internet experience are willing and able to successfully complete a Web-based assessment (e.g., Miller, 2000). Unlike Gerbert et al. (1999) and Turner et al. (1998), who found that technologically advanced assessment methods (audio, computer, and video) produced higher rates of risk disclosure, our participants' responses did not differ with more advanced technology. Our findings suggest that Web-based data collection does not statistically enhance or diminish the consistency of responses. Moreover, our results address concerns about the impact of computerization on the psychometric properties of instruments raised by Skinner and Pakula (1986). Given the findings from this study, the application of these Web-based assessment measures offers advantages to both researchers and study participants without compromising the reliability of the results drawn from the data. Using the Internet for data collection is a cost-efficient alternative to traditional techniques and has the potential to minimize data collection and entry errors while increasing accessibility. This study had several limitations related to the use of a primarily computer literate non-highrisk drinking college student sample. Because the measures tested in this study are frequently used as outcome markers in prevention effectiveness trials which, by their very nature, include participants with varied levels of use we chose to cast a broad net and not limit the sample to high-risk users. An additional important question not investigated in this study was whether

12 Page 12 of 15 high-risk students, those frequently targeted for prevention and treatment research, are likely to differ with respect to computer literacy and access. It is unclear whether high-risk populations are likely to differ from this sample with respect to accessibility to, literacy regarding, and comfort with computers. The limited research on this topic suggests there are no significant differences and in fact suggests that there are benefits associated with the use of a computer when questions of a sensitive nature are asked, including a sense of privacy, cost efficiency, and the application of skip patterns (Gerbert et al., 1999; Turner et al., 1998). Finally, the perceived convenience of assessment format may have been affected in both conditions by unfamiliar procedures. Web participants who were inexperienced with the Internet were obliged to learn how to access the Web-based survey, and the P&P participants were required to pick up and deliver their paper-based surveys to a specific location on campus or complete them on site. Although returning a paper-based survey by mail is typically experienced as inconvenient, the additional physical requirements for pickup and delivery or on-site completion may have lowered the perceived convenience for the paper-and-pencil format. Additional studies thus should be conducted with different samples in particular, adult samples and clinical populations to determine whether significant differences exist between both the method of data collection as well as the reliability and validity of responses. Furthermore, although Web-based assessment may be a cost-effective alternative and offer increased accessibility to researchers and participants, conducting Web-based data collection may not be a practical option for every population because of limited economic resources (i.e., access) and physical or mental cognitive impairments. It is important that the practical and technical considerations, such as Internet accessibility, computer literacy of participants, validity of responses, multiple submissions, security and data integrity violations, browser incompatibility, and modem speed be addressed prior to the initiation of a Web-based project. Furthermore, confidentiality of participants and participant responses is an important issue and not unique to Web-based assessment techniques. We obviated these obstacles by using a Webbased research consulting service provider (see footnote 1). We did not ask participants how private or not private the environment in which they completed their surveys was; however, this may be an important variable and should be included in future Web-based research. To our knowledge, no studies have examined the effect of private versus nonprivate settings on Webbased assessment item response. The questions explored in this research study were of a nature such that we tried to support, as opposed to reject, the null hypothesis. Yet our statistical techniques were designed to measure differences, not similarities. With the more conservative bootstrap estimates, we observed only 3 statistically significant results over 48 tests when testing reliability estimates and only 4 statistically significant results over 80 tests when testing reliability and validity estimates, which does not rise above what one would expect on the basis of chance alone. In our case, increasing alpha increased our power to detect significant differences but revealed a number of differences consistent with increased Type I error. A better approach is to increase sample size in repeated studies. Nevertheless, it is with caution that we conclude that there are no significant differences in method of assessment of the aforementioned measures. We may lack power to detect all possible effects, and we may not have measured all domains where differences could exist. Although the heavily skewed distributions of our data were unexpected, this fact was not surprising given our population. In fact, the nature of high-risk user populations suggests that the data will be skewed, and therefore reliable analytic techniques should be addressed with this fact in mind. Repeated studies of Web-based assessment techniques with various populations and divergent topics over time are highly recommended in order to fully understand whether these findings are

13 Page 13 of 15 a true representation of a lack of differences in assessment technique or response sets. It is our hope that researchers will continue to keep pace with technology by conducting these comparative studies in parallel with increased use of maximizing the benefits that Web-based data collection offers. References Allen, J. P., Litten, R. Z., Fertig, J. B., & Babor, T. (1997). A review of research on the Alcohol Use Disorders Identification Test (AUDIT). Alcoholism: Clinical and Experimental Research, 21, [Medline Link] [BIOSIS Previews Link] [Context Link] Babor, T. F. (1996). Reliability of the Ethanol Dependence Syndrome Scale. Psychology of Addictive Behaviors, 10, [Fulltext Link] [Context Link] Babor, T. F., Hofmann, M., Del Boca, F. K., Hesselbrock, V., Meyer, R. E., Dolinsky, Z. S., & Rounsaville, B. (1992). Types of alcoholics: I. Evidence for an empirically derived typology based on indicators of vulnerability and severity. Archives of General Psychiatry, 49, [Medline Link] [BIOSIS Previews Link] [Context Link] Baer, J. S. (1993). Etiology and secondary prevention of alcohol problems with young adults. In J. S. Baer, G. A. Marlatt, & R. J. McMahon (Eds.), Addictive behaviors across the life span: Prevention, treatment, and policy issues (pp ). Newbury Park, CA: Sage. [Context Link] Box, G. E. P. (1949). A general distribution theory for a class of likelihood criteria. Biometrika, 36, [Context Link] Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, [Medline Link] [Context Link] Cicchetti, D. V. (1994). Guidelines, criteria and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, [Fulltext Link] [Context Link] Collins, R. L., Parks, G. A., & Marlatt, G. A. (1985). Social determinants of alcohol consumption: The effects of social interaction and model status on the self-administration of alcohol. Journal of Consulting and Clinical Psychology, 53, [Context Link] Del Boca, F. K., & Brown, J. M. (1996). Issues in the development of reliable measures in addictions research: Introduction to Project MATCH assessment strategies. Psychology of Addictive Behaviors, 10, [Fulltext Link] [Context Link] Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. London: Chapman and Hall. [Context Link] Gerbert, B., Bronstone, A., Pantilat, S., McPhee, S., Allerton, M., & Moe, J. (1999). When asked, patients tell: Disclosure of sensitive health-risk behaviors. Medical Care, 37, [Medline Link] [CINAHL Link] [Context Link] Johnston, L. D., O'Malley, P. M., & Bachman, J. G. (1996). National survey results on drug use from the Monitoring the Future Study, : Vol. 2. College students and young adults. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health. [Context Link] Kivlahan, D. R., Marlatt, G. A., Fromme, K., Coppel, D. B., & Williams, E. (1990). Secondary prevention with college drinkers: Evaluation of an alcohol skills training program. Journal of Consulting and Clinical Psychology, 58, [Context Link] Marlatt, G. A., Baer, J. S., & Larimer, M. E. (1995). Preventing alcohol abuse in college students: A harm-reduction approach. In G. Boyd, J. Howard, & R. Zucker (Eds.), Alcohol problems among adolescents: Current directions in prevention research (pp ). Hillsdale, NJ: Erlbaum. [Context Link] Miller, E. T. (1997). Predicting successful self -initiated health-related behavior change in the context of New Year's

14 Page 14 of 15 resolutions: Utilizing the Internet for survey research. Unpublished master's thesis, University of Washington. [Context Link] Miller, E. T. (2000). Preventing alcohol abuse and alcohol-related negative consequences among freshmen college students: Using emerging computer technology to deliver and evaluate the effectiveness of brief intervention efforts (Doctoral dissertation, University of Washington, 2000). Dissertation Abstracts International, 61, 8. [Context Link] Moore, D. S., & McCabe, G. P. (1993). Introduction to the practice of statistics (2nd ed.). New York: W. H. Freeman. [Context Link] Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, & Computers, 30, [Context Link] Prochaska, J. O., & DiClemente, C. C. (1986). Toward a comprehensive model of change. In W. R. Miller & N. Heather (Eds.), Treating addictive behaviors: Processes of change. Applied clinical psychology (pp. 3 27). New York: Plenum. [Context Link] Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption: II. Addiction, 88, [Medline Link] [BIOSIS Previews Link] [Context Link] Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments and Computers, 29, [Context Link] Skinner, H. A., & Allen, B. A. (1982). Alcohol dependence syndrome: Measurement and validation. Journal of Abnormal Psychology, 91, [Medline Link] [Context Link] Skinner, H. A., & Pakula, A. (1986). Challenge of computers in psychological assessment. Professional Psychology: Research and Practice, 17, [Context Link] Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, and Computers, 29, [Context Link] Sobell, L. C., Sobell, M. B., Leo, G. I., & Cancilla, A. (1988). Reliability of a timeline method: Assessing normal drinkers' reports of recent drinking and a comparative evaluation across several populations. British Journal of Addiction, 83, [Medline Link] [BIOSIS Previews Link] [Context Link] Sobell, M. B., Sobell, L. C., Klajner, F., Pavan, D., & Basian, E. (1986). The reliability of a timeline method for assessing normal drinker college students' recent drinking history: Utility for alcohol research. Addictive Behaviors, 11, [Medline Link] [BIOSIS Previews Link] [Context Link] Turner, C. F., Ku, L., Rogers, S. M., Lindberg, L. D., Pleck, J. H., & Sonenstein, F. L. (1998, May 8). Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science, 280, [Context Link] Wechsler, H., Molnar, B. E., Davenport, A. E., & Baer, J. S. (1999). College alcohol use: A full or empty glass? Journal of American College Health, 47, [Medline Link] [CINAHL Link] [Context Link] White, H. R., & Labouvie, E. W. (1989). Toward the assessment of adolescent problem drinking. Journal of Studies on Alcohol, 50, [Medline Link] [BIOSIS Previews Link] [Context Link] 1 Web-based data collection and management services for this project were provided by DatStat Inc. [Context Link] 2 At Time 1, differences between P&P and Web-I were evident between weekly quantity and ADS (z = 3.10) and weekly quantity and AUDIT (z = 2.93). At Time 2, differences between P&P and Web-I were evident between weekly quantity and ADS (z = 2.01). [Context Link]

15 Page 15 of 15 Accession Number: Copyright (c) Ovid Technologies, Inc. Version: rel6.1.0, SourceID