Special Section QUANTITATIVE METHODS IN THE STUDY OF DRUG USE. Edited by George J. Huba ABSTRACT

Transcription

1 J. DRUG EDUCATION, Vol. 14(2), 1984 Special Section QUANTITATIVE METHODS IN THE STUDY OF DRUG USE Edited by George J. Huba RELIABILITY OF SELF-REPORT MEASURES OF DRUG USE IN PREVENTION RESEARCH: Evaluation of the Project SMART Questionnaire via the Test-Retest Reliability Matrix* JOHN W. GRAHAM BRIAN R. FLAY C. ANDERSON JOHNSON WILLIAM B. HANSEN LINDA GROSSMAN JUDITH L. SOBEL Health Behavior Research Institute University of Southern California, Los Angeles ABSTRACT The present article describes an evaluation of a self-report questionnaire administered to whole classrooms of 7th graders. Using the test-retest reliability matrix (based on concepts of Cronbach [ 1] and Campbell and Fiske [2]), eight of nine drug-use indices appeared to have acceptable to good reliability. The three measures included in the test-retest reliability matrix provide stronger evidence for good reliability than could any single measure. *This research was supported by NIDA Grant# 1 R18 DA l. 1984, Baywood Publishing Co., Inc. 175

2 176 I GRAHAM ET AL. The value of research in drug abuse is limited by the degree to which drug use and abuse can be measured reliably and validly. Ideally, drug use would be measured directly, for example, via biological measures, so that reliability and validity would be virtually guaranteed. However, while there are many good, direct measures of drug use and abuse, most of these require a high level of cooperation from the person, and/or a high level of control over the person (e.g., police requiring that a drunk driver take a breath test). Few good biological measures exist that are appropriate for use in classroom-based drug-abuse prevention research involving adolescent students. Some success has been achieved with classroom-appropriate biological measures of cigarette smoking [3,4]. However, these measures are not perfectly valid, and have other limitations. Valid, feasible, non-reactive, classroom-appropriate biological measures of alcohol, marijuana, and other drug use are not yet well-developed. Other methods for collecting high-quality drug-use information include individual structured interviewing and self-monitoring [5,6]. However, while these methods are promising for examining drug use in close detail, they are less applicable for larger-scale, classroom-based intervention studies. In sum, despite recent advancements in biological measurement of drug use, and improvements in interview techniques, self-report measures remain the best method in classroom-based studies for collecting multidimensional drug-use information, and the only feasible method in such studies for collecting information about drugs other than tobacco. Thus, it is important in classroom-based drug-abuse prevention research that methods be developed: 1) for improving the validity and reliability of self-report measures, and 2) for assessing the degree to which self-reports are valid and reliable. Much attention has been given to improving and to assessing the validity of self-reports. One, often-used method designed to improve the validity of selfreports, described as the bogus pipline [7-9] is based on the collection of biological data. When such data are collected and can be used to determine the level of drug use (actually or ostensibly), people will presumably want to be more honest in their self-reports of drug use. Additionally, one of the major methods for assessing the validity of self-reports involves the collection of biological information about the same drugs [3,4,10]. Relatively little attention, on the other hand, has been given to improving, or to assessing the reliability of self-reports. A cursory review of existing drug abuse prevention literature reveals no specific data concerning the reliability of drug-use scales, and no discussion of the issues of assessing reliability in prevention research. What follows is 1) a brief discussion of some of the issues surrounding the assessment of reliability of drug-use instruments, especially those used in large-scale, classroom-based prevention projects, and 2) some reliability data for self-reports of drug use from Project SMART [11-13].

3 SELF-REPORT MEASURES IN DRUG RESEARCH I 177 RELIABILITY ISSUES IN LARGE-SCALE PREVENTION RESEARCH Reliability is an important issue in large-scale multivariate research. In such research, experimental control of all relevant variables is impossible, so statistical control (e.g., analysis of covariance) is the only form of control possible. In large-scale studies it is necessary to rule out the relationship between some constructs in order to explain more clearly the obtained relationships between others. It is well documented that the validity of a given measure, as well as the strength of relationships between any two measures are limited by the reliability of the measures. If one finds a relationship, one may assume (provided certain assumptions are met) that the relationship does exist. However, failing to find a relationship may be due to a true lack of relationship or to poor measurement of one or both constructs. In order to rule out the latter possibility, one should have independent evidence that the measures are sufficiently reliable. Psychometricians [ 1,14-16] have outlined several methods for assessing the reliability of scales, including 1) coefficient alpha, a measure of internal consistency, 2) alternate forms reliability, which measures the degree to which a constellation of items is related to a second constellation of items presumably measuring that same construct, and 3) test-retest methods, which measure the degree to which responses to an item, or to a constellation of items are similar at two different times. Each of these three types of reliability has its own strengths and weaknesses, and each provides unique information about reliability. Coefficient Alpha Coefficient alpha is widely used as a measure of internal consistency, and is what Cronbach [ 1] defined as a "coefficient of equivalence." [ 1] For many applications it provides a good estimate of true reliability, and is relatively easy to obtain. Also, since it is based on inter-item correlations at one point in time, it is often the only reasonable method for assessing reliability. Unfortunately, for many scales, coefficient alpha is not an appropriate measure of reliability. The use of coefficient alpha for estimating internaliconsistency requires that several assumptions be met. First, every person must have responded to every item. However, in many questionnaires about drug use, respondents are first asked a branching question (e.g., "Have you ever smoked even part of cigarette?"). If they answer "yes", they are instructed to go on and answer other items about smoking. If they answer "no", they are instructed to skip all further questions about smoking, and are quite appropriately assumed to have the value "0" for other cigarette questions such as "How many cigarettes have you smoked in the past week?" However, these people have not actually responded to the other items, and their assumed values cannot be used in assessing the internal consistency of those items. For such items, one can estimate reliability

4 178 I GRAHAM ET AL. r based only on people who have ever used the particular drug. This limitation affects the interpretation of the obtained reliability index. A second assumption of coefficient alpha is that the items all come from the same domain, i.e., that they measure the same thing. This may or may not be the case in measuring drug use. For example, it could be argued that twenty-fourhour recall and one-week recall are both measures of recent use. On the other hand, it is less clear that measures of recent use, and measures of typical use meet this assumption. An unacceptably low alpha may be the result of violating this assumption. A third assumption of coefficient alpha is that the error variance from the different items is random. If all of the items come from the same questionnaire, this assumption is probably never met. If this assumption is violated, correlations among items will be spuriously high in that they contain common error variance as well as true covariance components. The problem is that one cannot know how much of the covariance is true and how much is due to common error variance. When this assumption is violated, it is essential to obtain another measure of reliability that is less susceptible to this particular problem. Alternate Forms Reliability It is often suggested that alternate forms reliability be calculated along with coefficient alpha. This suggestion is based on the assumption that the construct would be measured with a scale of several individual items. However, each druguse construct is typically measured with a single item, and even closely related drug-use constructs may have no more than two or three measures. Still, a kind of alternate-forms reliability may be possible if two or more similar terms are measured at different times. This type of reliability assessment would be obtained by examining the relationship between one item at time 1 and a second item at time 2, and is consistent with Cronbach's coefficient of stability and equivalence [ 1]. Such a measure is also similar to that suggested by Lehnen in using only a pair of items to assess alternate "forms" reliability [17]. The strength of such a measure is that common error variance between the two measures is less likely to exist since the two measures are taken at different times. Weaknesses of this measure are 1) that only a few items would be involved, thereby producing a relatively conservative estimate of reliability, 2) that the different items may stem from subtly different domains, leading to a conservative estimate of reliability, and 3) the difference in time may lead to a real difference in the construct, again producing a relatively conservative estimate of reliability. Test-Retest Methods There are three problems with standard test-retest methods: 1) respondents may recall responses from one administration to the next, and may be motivated

5 SELF-REPORT MEASURES IN DRUG RESEARCH / 179 to be consistent, 2) respondents may have a response style, especially in the case of guessing, which would be consistent across administrations of the test, and 3) it is possible that the constructs being measured would undergo real change between the first and second administration of the test. This third problem led Cronbach to define test-retest correlation as a coefficient of stability. To the extent the first two problems exist, test-retest correlations will be artificially high. To the extent the third problem exists, test-retest correlations will be artificially low. Strengths of the test-retest method are 1) that it may represent the only feasible measure of reliability for individual items, 2) it may be the only feasible measure of reliability for scales in which the individual items are not drawn from the same domain, and 3) it is less susceptible to the kind of correlated measurement error that would be expected when items are answered in the same questionnaire at the same time. TEST-RETEST RELIABILITY MATRIX None of the three measures of reliability discussed above is entirely satisfactory for use in drug-abuse prevention research. Coefficient alpha may be artificially high due to correlated error. Alternate forms reliability may be inappropriate due to the limited number of individual items available for each drug-use construct. Test-retest correlations may either be artificially high due to consistent response patterns, or artificially low due to real changes in the measured constructs. Nunnally suggests that both coefficient alpha and alternate forms reliability be assessed so that the potential flaws of one method could be offset by the strengths of the other [ 15]. Lehnen's [ 17] application of Campbell and Fiske's [2] multitrait-multimethod matrix (MTMM) is a practical example of this suggestion. Cronbach suggested that all three types of reliability are different and should be reported [ 1]. We agree with Cronbach's point and suggest that for major drug-abuse prevention research in which relatively long questionnaires are administered, the MTMM idea should be expanded to include test-retest correlations, using the different administrations of the questionnaires as the "methods." If test-retest data are collected, it is possible to obtain coefficient alpha at both administrations of the questionnaire, as well as test-retest correlations. Additionally, it is possible to obtain a kind of alternate forms reliability by correlating items at time 1 with other items from approximately the same domain at time 2. All three types of reliability assessment can be obtained conveniently from a single correlation matrix. Table 1 shows the test-retest reliability matrix for three related drug-use items measured on two occasions. The triangle of correlations at the upper half of the diagonal, labeled "A1," is the basis for coefficient alpha at time 1. The triangle of correlations at the lower half of the diagonal, labeled "A2," is the basis for coefficient alpha at time 2. The elements labeled "B" in Table 1 are the

6 Table 1. The Test-Retest Reliability Matrix Lifetime One-Month General Lifetime One-Month General Smoking 1 Recall 1 Smoking 1 Smoking 2 Recall 2 Smoking 2 (X) 0 Lifetime Smoking One-Month Recall 1 A General Smoking 1 A1 A Lifetime Smoking 2 B c c 1.00 One-Month Recall 2 c B c A General Smoking 2 c c B A2 A Note: A 1 elements form basis of alpha at time 1. A2 elements form basis of alpha at time 2. B elements are test-retest correlations. C elements can be viewed as a kind of alternate forms reliability.

7 SELF-REPORT MEASURES IN DRUG RESEARCH I 181 test-retest correlations for the individual items. The off-diagonal elements in the two triangles labeled "C" in Table 1 (across-time correlations) are a kind of alternate forms correlations. The average of the six alpha correlations could be used to summarize the internal consistency of the items within times. The average of the six across-time correlations could be used to summarize the alternate forms reliability. The three test-retest correlations could be used to examine the test-retest reliability (stability) of the individual items. Test-retest reliability of the entire scale is best assessed by standardizing and summing the items at each time and obtaining the scale test-retest correlation. Campbell and Fiske's MTMM involved constructs and focussed on validity, while the present matrix involves individual items and focusses on reliability. Campbell and Fiske described four criteria that could be used to evaluate the convergent and discriminant validity of the constructs. These criteria, with their analogs in the reliability matrix, are: I) the validity diagonals (test-retest correlations here) should be substantially different from 0; 2) the elements in the validity diagonals should be higher than elements in the same row or column in the corresponding heterotrait-heteromethod triangles (test-retest correlations should be higher than across-time correlations); 3) the validity diagonals should be higher than the monomethod-heterotrait triangles (test-retest correlations higher than within time correlations); and 4) the same pattern of interrelationship among the traits should exist in all hetero-trait triangles (patterns within and between times should be similar). For the test-retest reliability matrix, criteria 1, 2, and 4 are desirable, but are somewhat different for this application. Criterion 1 should be expanded: All elements of the matrix should be substantially different from 0. Criterion 2, in this case, should be that the across-time correlations should be no greater than the test-retest correlations. While these across-time correlations are expected to be somewhat lower than test-retest correlations, a large difference is neither expected, nor desirable. Similarly, criterion 4, while desirable, is somewhat artificial if all the items are drawn from the same general domain. One might expect that the pattern of correlations might change for highly intercorrelated items. Finally, since the various items are expected to be within the same general domain, criterion 3 does not apply. That is, it may well happen that within-time correlations are higher than test-retest correlations. While some of the logic of Campbell and Fiske's MTMM carries over to the present application, the test-retest reliability matrix is better conceived as a magnification of the reliability diagonal in the MTMM. In fact, it may often be useful to expand the test-retest reliability matrix to include all items for all constructs, thus yielding the more familiar MTMM form. In the following sections, the test-retest reliability matrix is used to assess the reliability of nine indices of tobacco, alcohol, and marijuana use measured in the context of a classroom-based drug-use prevention project.

8 182 / GRAHAM ET AL. METHOD Project SMART The base study was Project SMART, a 5-year smoking and drug-abuse prevention project [ 11,12]. The study was designed to test the effectiveness of two peer-pressure resistance training interventions at three different grade levels. In the first year of the project, approximately 7500 sixth and seventh graders in the Los Angeles Unified School District received the program or served as usualtreatment controls. The present reliability study involved the four junior high schools serving as controls for the seventh-grade intervention in the spring of The SMART questionnaire was administered on two occasions approximately three weeks apart to the seventh graders in these four schools. In total, 396 seventh graders were present at both administrations of the questionnaire, and their responses form the basis of the reliability study. Questionnaire The Project SMART questionnaire included measures of cigarette smoking, alcohol use, marijuana use, and intentions to use tobacco, alcohol, and marijuana, as well as measures of numerous psychosocial constructs. The drug use and intentions items used in the questionnaire appear in Appendix A. The reliability study was conducted on nine drug-use indices made up of these individual items. The selection of these nine indices was based on 1) previous theoretical work on adolescent drug use [e.g., 18,19], 2) previous research on related questionnaire items [e.g., 20], 3) factor analyses of items in a previous version of the SMART questionnaire, and 4) factor analyses on this version of the questionnaire based on a sample different from the 396 used in this reliability study. The nine druguse indices examined were as follows: 1) smoking behavior (lifetime use, onemonth recall, generalized smoking); 2) smoking intentions (intentions to smoke again, intentions to smoke daily, intentions to smoke monthly, intentions to smoke if offered by best friend); 3) lifetime drinking (lifetime use, usual number of drinks); 4) recent drinking behavior (one-month recall of amount, one-week recall, one-month recall of days); 5) drunkenness (times drunk in lifetime, times drunk in previous month, intentions to get drunk again); 6) drinking intentions (intentions to drink again, intentions to drink daily, intentions to drink monthly); 7) marijuana use behavior (lifetime use, one-month recall, one-week recall); 8) marijuana use intentions (intentions to use again, intentions to use daily, intentions to use monthly, intentions to use if offered by best friend); and 9) multiple-substance use (tobacco and alcohol, tobacco and marijuana, alcohol and marijuana). The item numbers of the specific items in each of the nine scales are indicated in Table 2.

9

10 184 I GRAHAM ET AL. T Reliability Assessment A correlation matrix was obtained for the items in each index for each administration of the questionnaire (see Table 1 for the general format of the correlation matrix). Reliability was assessed for the items in each of the nine indices in four ways. First, a measure of within-administration internal consistency was obtained by summing the within-administration elements (see Table 1, elements labeled "Al" and A2"). Second, a kind of alternate-forms reliability was obtained by summing the across-time correlations (elements labeled "C" in Table 1 ). Since the standardized alpha is a more familiar metric for reliability, standardized alpha was also calculated based on both within-time and across-time correlations. Third, test-retest correlations were obtained for each of the individual items making up the index (elements labeled "B" in Table 1 ). Finally, the items were standardized and averaged, and the scale testretest correlations were obtained. As a means of placing the reliability estimates in perspective, estimates of discriminant validity for the nine scales were also obtained. These estimates were obtained 1) by averaging the correlation for each pair of indices at time 1 and the corresponding correlation at time 2, and 2) by averaging the two across-time correlations for each pair of indices. When these average correlations are compared with the reliability estimates (shown in Table 2), they correspond to Campbell and Fiske's [2] tests of criteria 2 and 3 for discriminant validity. RESULTS Table 2 presents a summary of the results of the reliability study. The average alpha shows what scale reliability would normally appear to be in a study of this sort. The alpha coefficients for each of the two administrations have simply been averaged here to give a more reliable estimate of alpha. For eight of the nine scales, the standardized alpha coefficient is greater than.70 (usually considered to indicate reasonable reliability). For six of the nine scales, the average alpha is greater than.80. The average within-time correlations provide the same information as alpha, but in a different metric. Since the item scores within each scale are likely to have common method variance, the within-time correlations and alpha coefficients may be somewhat inflated. However, the average across-time correlations (excluding test-retest correlations) are less likely to be inflated due to this common method variance, and in some ways may provide a more realistic estimate of reliability than would the within-time correlations. In order to compare the two methods in a more familiar metric, standardized alpha coefficients were also calculated based on the across-time correlations. The two correlations obtained for each pair of items (Al with B2, and Bl with A2) were averaged and used as the single correlation between the two items in calculating the standardized alpha. The number of items (k) was thus the same as that used for within-time alphas.

11 L0 co (') CO Ol (') 0 Ol (') "<t co (') 0 CO M CO M ~ N 0 LO N M M "<t M Ol "<t "<t "<t LO M M "<t "<t "<t ~ "<t V> QJ u "0 c <0 N M oo <0 en Ln Ln M Ol "<t 0 0 N CO CO M "<t N M M LO N ~ LO CO 0 QJ c z... 0 '+co >... c co c E ;:: u.~ 0 cv) QJ ::0 co f- ~ c:: c:: -'< "' c:: ;:, '- Cl "<t "<t N ~ "<t CO CO N CO N 0 Ol Ol Ol CO LO "<t M M Ln <0 ld ~ M M M i g "'" ~~~~~CD~C'\1~ E~LOr'-MMMlDMC.OM Cl)~ (') Ol "<t N LO <;!" OMCO"<t~ CONCIJ CO 0 L0 MOCOCO~ CO CO CO "<t "<t LO M "<t "<t "<t "<t -. 0 ~ Q) 0 ~E Q) Q) > ~ "' ~ ~ 8 ~ Q) ~ -~ "' ';' ;1.~ c.<: Q) ~ E ~ Q) Q) Q) Cl -"' ~ c Q) g~ "' Q) ~ ~ -~co "' c ~g, CO ~ii 0 c..c ro : E 0 Q) C.c Q) ~ ~ ~ ~ ~ Q).~.0 0 ~ a.~ ~ E - Q) ~w 0. u) c c 0 0.e ;:;.;::; 0~~ z ~ ~ a o u u 185

12 ,! 186 I GRAHAM ET AL. Not surprisingly, the across-time alpha coefficient for each of the nine scales was lower than the within-time alpha. Still, for five of the nine scales (smoking behavior, smoking intentions, lifetime alcohol use, marijuana use, and marijuanause intentions), the across-time alpha coefficients were greater than.70. For three other scales (recent alcohol use, drunkenness, and alcohol-use intentions) the across-time alphas were greater than.66. For only one scale (multiple-drug use) was the across-time alpha coefficient less than.60. Five of the nine scales (smoking behavior, smoking intentions, lifetime alcohol use, drunkenness, and alcohol use intentions) have scale test-retest correlations greater than r =.70, and two others (marijuana use, and marijuana use intentions) have test-retest correlations greater than r =.66. The lowest testretest correlation (r =.528) was obtained for the recent alcohol use scale. Half of the individual items had test-retest correlations greater than r =.60. Three-fourths had test-retest correlations greater than r =.50. The average testretest correlation for the twenty-eight individual items was r =.601. Seven of the individual items (3,13,14,15,18,27,28) measured recall of drug use over the previous month or week, and were expected to be particularly susceptible to real shifts in the construct being measured. The average test-retest correlation for these seven items was r =.498, while the average test-retest correlation for the remaining twenty-one items was r =.635. Table 3 presents the results of the test for discriminant validity. The elements of the main diagonal of Table 3 are the average of the three reliability estimates from Table 2 (within-time alpha, across-time alpha, and scale test-retest correlations). The elements below the main diagonal are the average within-time correlations, and those above the diagonal are the average across-time correlations. For eight of the nine indices, the main diagonal element was greater than all elements in the corresponding row and column. The reliability estimate for the multiple-drug use index was greater than all across-time correlations and all but one within-time correlation. The bottom row of Table 3 shows, for each index, the average of the eight within-time correlations with the other indices. The rightmost column shows, for each index, the average of the eight across-time correlations with the other indices. For all nine indices, these summary values are substantially lower than the corresponding reliability values shown in the main diagonal. DISCUSSION Coefficient alpha is the most common measure of scale reliability, and is especially useful when other measures, such as test-retest reliability, are not available. The conclusions in the present study, based solely on coefficient alpha, would have been that six of the nine drug-use scales had good reliability (.80 or higher), that two others had acceptable reliability (.70 or higher), and that only one scale (multiple-drug use) had unacceptable reliability (.60), although even

13 SELF REPORT MEASURES IN DRUG RESEARCH I 187 this scale may have been judged to have marginally acceptable reliability. The problem with using coefficient alpha alone in this type of research is that it is inflated to an unknown degree by correlated measurement error. Across-time consistency, as measured by across-time correlations and acrosstime alpha, should be less susceptible to correlated measurement error. Using this more conservative measure, five of the nine scales appeared to have acceptable reliability, while another three appeared to have marginally acceptable reliability. Again, only the multiple-drug use scale was unacceptably unreliable. Based on the third measure of scale reliability (scale test-retest correlations), seven scales were either acceptably reliable, or had marginally acceptable reliability. For this measure, the multiple-drug-use scale again appeared to have questionable reli@ility. The recent alcohol use scale also had questionable testretest reliability. Overall, seven of the nine drug-use scales in the Project SMART questionnaire had acceptable or good reliability on two of the three measures, and at least marginally acceptable reliability on the third. The smoking behavior and smoking intentions scales showed the best overall reliability of all scales, which is not surprising since the items in these scales have the longest history in drugabuse prevention research. The alcohol scales also showed good overall reliability. The lifetime alcoholuse scale had good or acceptable reliability on all three measures while the drunkenness and alcohol-use intentions scales each had acceptable ~ithin-time consistency, acceptable test-retest reliability, and marginally acceptable acrosstinle consistency. The recent alcohol use scale had good within-time consistency, but only marginal across-time consistency, and less than marginal test-retest reliability. The two marijuana-use scales appeared to have reasonable overall reliability. Both scales had good within-time consistency, acceptable across-time consistency, and marginally acceptable test-retest reliability. The multiple-drug-use scale appears to have unacceptable reliability regardless of which measure is used. However, coefficient alpha may not be appropriate for this scale. Although use of any two drugs in combination should be correlated with the use of two other drugs in combination, it is not fair to say that these two measures are tapping the same construct. Thus, one might expect alpha to be suppressed substantially for this scale. Still, the test-retest correlation for this scale was also lower than that for the other scales. For this scale, the conclusion must be that further work is needed. The recent alcohol use scale may have good reliability, despite its low testretest correlation. Because it was made up entirely of recent recall items, responses to these items would reasonably be expected to change even over a period as short as three weeks. The pattern of test-retest correlations for the individual items supports this interpretation. The one-week recall measure

14 J 188 I GRAHAM ET AL. (item 14) should be less stable than one-month recall items (items 13 and 15), and the test-retest correlation for item 14 is substantially lower than that for items 13 and ls.ln fact, for smoking, alcohol use, drunkenness, and marijuana use, lifetime use measures should have been least susceptible to real changes and had the highest test-retest correlations, one-month recall items should have been intermediate in susceptibility to real change and had intermediate test-retest correlations, and one-week recall measures should have been the most susceptible to real change and had the lowest test-retest correlations. These findings argue strongly for the idea that for measures of recent drug use, testretest correlations (and across-time correlations to some extent) are attenuated by instability. On the other hand, although high test-retest correlations for such measures may indicate high stability, such stability implies good reliability, and if other assumptions are met, should be usable as a lower bound for reliability. The results of the brief test of discriminant validity give added meaning to the reliability results. With the possible exception of the multiple-drug use index, all indices appeared to have good discriminant validity. These results argue against the possibility that good scale reliability was obtained simply because all variables were spuriously interrelated. More extensive examination of the validity of these scales will be the focus of later research. The present paper provided good evidence that self-reports of drug use can be reliable when obtained as part of a lengthy questionnaire administered to whole classrooms of seventh graders. The paper also provides evidence that eight of the nine drug-use and intentions scales employed in Project SMART have acceptable to good scale reliability for this sample. The method used here, described as the test-retest reliability matrix, magnifies the reliability diagonal of Campbell and Fiske's multitrait-multimethod matrix, and accomplishes in convenient fashion the goal of providing three different estimates of reliability, which correspond to Cronbach's coefficients of equivalence, stability and equivalence, and stability [ 1]. Since these three types of scale reliability have somewhat non-overlapping weaknesses, researchers can have more confidence in these reliability estimates than if any one of the methods had been used alone. APPENDIX A Project SMART Questionnaire Drug-Use Items SMOKING BEHAVIOR QUESTIONS 1. Have you ever had even one puff on a cigarette? 0) yes 1) no (Students were instructed to complete the following questions only if they answered "yes" to the above question.) 2. How many cigarettes have you smoked in your lifetime? 1) I have only had one puff 4) 5 to 20 cigarettes 2) Part of one cigarette 5) 1 to 5 packs 3) 2 to 4 cigarettes 6) more than 5 packs

15 SELF-REPORT MEASURES IN DRUG RESEARCH I How many cigarettes have you smoked in the last month (30 days)? 0) None 3) ~ 2 to 4 cigarettes 1) I have only had one puff 4) ~ 5 to 20 cigarettes 2) Part of one cigarette 5) more than one pack 4. How much do you smoke now? 0) I used to smoke but now I don't 4) About a pack a week 1) ~ I've only tried a few puffs 5) About half a pack a day 2) A few cigarettes a month or less 6) A pack a day or more 3) Less than a pack a week 5. When you smoke cigarettes do you ever drink alcohol? 0) yes 1) no 6. When you smoke cigarettes do you ever smoke marijuana? 0) yes 1) no SMOKING INTENTIONS ITEMS (All students responded to the smoking intentions items. For students who had never smoked, item 7 was phrased: "Honestly, do you think you would like to try smoking a cigarette?") 7. Do you think you will ever have another cigarette? 0) yes 1) maybe 2) ~maybe not 3) no 8. Some people smoke every day, some smoke every week, some don't smoke. Do you think you will ever smoke about every day? 0) no 1) maybe 2) yes 9. Do you think you will ever smoke every month? 0) no 1) maybe 2) yes I 0. If your best friend offered you a cigarette, what would you do? 0) smoke it 3) maybe refuse it 1) maybe smoke it 4) refuse it 2) I don't know what I'd do ALCOHOL USE BEHAVIOR ITEMS I I. Have you ever had even a sip of alcohol? (I can of beer= I glass of wine = I mixed drink) 0) yes I) no (Students were instructed to complete the following behavior items if they responded "yes" to the above question.) I 2. How many alcoholic drinks have you had in your whole life? I) ~only sips 5) 11 to 20 2) part or all of one 6) 21 to 100 3) 4) 2 to 4 5 to I 0 7) more than 100

16 190 I GRAHAM ET AL. 13. How many alcoholic drinks have you had in the last month (30 days)? O) none 4) 5 to 10 1) only sips 5) 11 to 20 2) part or all of one 6) more than 20 3) 2 to How many alcoholic drinks have you had in the last week? 0) none 1) V2 drink or less 2) 3) 2 to 4 4) 5 to How many days in the last month (30 days) have you had alcohol to drink? 0) none 1) 2) 2 or 3 3) 4 to 7 4) 8 to 14 5) 15 to When you drink alcohol, how many drinks do you usually have? 1) 1 or less 2) 2 3) 3 4) 4 5) 5 or more 17. Have you ever been drunk? O) no 1) only once 2) 2 to 4 times 3) 4) 5) 5 to 10 times 11 to 20 times more than 20 times 18. In the last month (30 days) how many times were you drunk? O) none 3) 4 to 6 times 1) once 4) 7 to 10 times 2) 2or3times 5) morethanlotimes 19. When you drink alcohol, do you ever smoke cigarettes? 0) yes 1) no 20. When you drink alcohol, do you ever smoke marijuana? 0) yes 1) no ALCOHOL USE INTENTION ITEMS (All students responded to alcohol use intention items except for item 22. Only students responding "yes" to item 10 responded to item 22. The word "again" in item 21 did not appear in the version asked of students who had never used alcohol.) 21. Do you think you will have a drink again in the next couple of months? O) yes 1) maybe 2) maybe not 3) no 22. Do you think you will get drunk in the next couple months? O) yes 1) maybe 2) maybe not 3) no 2 3. Some people drink every day, some every week. Do you think you will ever drink every day? 0) no 1) maybe 2) yes 24. Do you think you will ever drink every month? O) no 1) maybe 2) yes

17 SELF-REPORT MEASURES IN DRUG RESEARCH I 191 MARIJUANA-USE BEHAVIOR QUESTIONS 25. Have you ever had any marijuana? 0) _yes 1) no (Students were instructed to complete the following questions only if they answered "yes" to the above question.) 26. How many times have you used marijuana in your lifetime? 1) once 4) 11 to 20 times 2) 2 to 4 times 5) 21 to 100 times 3) 5 to 10 times 6) more than 100 times 27. How many times have you used marijuana in the past month (30 days)? 0) none 3) 5to10times 1) only once 4) 11 to 20 times 2) 2 to 4 times 5) more than 20 times 28. How many times have you used marijuana in the last week (7 days)? O) none 1) only once 2) 2 to 4 times 3) 5 to 10 times 4) more than 10 times 29. When you smoke marijuana do you ever some cigarettes? 0) yes 1) no 30. When you smoke marijuana do you ever drink alcohol? 0) yes I) no MARIJUANA-USE INTENTIONS ITEMS (All students responded to the marijuana-use intentions items. For students who had never used marijuana, item 31 was phrased: "Honestly, do you think you would like to try smoking marijuana?") 3!. Do you think you will ever smoke marijuana again? 0) yes I) maybe 2) maybe not 3) no 32. Do you think you will ever smoke marijuana every day? 0) no 1) maybe 2) yes 33. Do you think you will ever smoke marijuana every month? 0) no 1) maybe 2) yes 34. If your best friend offered you marijuana, what would you do? 0) smoke it 3) maybe refuse it I) maybe smoke it 4) refuse it 2) I don't know what I'd do

18 192 I GRAHAM ET AL. REFERENCES T 1. L. J. Cronbach, Test "Reliability": Its Meaning and Determination, Psychornetrika, 12, pp. 1-16, D. T. Campbell and D. W. Fiske, Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix, Psychological Bulletin, 56, pp , R. V. Luepker, T. F. Pechacek, D. M. Murray, C. A. Johnson, F. Hund, and D. R. Jacobs, Saliva Thiocyanate: A Chemical Indicator of Cigarette Smoking in Adolescents, American Journal of Public Health, 71, pp , T. F. Pechacek, D. M. Murray, R. V. Luepker, M. B. Mittelmark, C. A. Johnson, and B. A. Shultz, Measurement of Adolescent Smoking Behavior: Rationale and Methods, Journal of Behavioral Medicine, in press. 5. A. Biglan, H. Severson, J. Bavry, and S. McConnell, Social Influence and Adolescent Smoking: A First Look Behind the Barn, Health Education, in press. 6. A. Biglan, H. Severson, H. Hops, C. Faller, D. Ary, L. S. Friedman, C. L. Nautel, and W. Weissman, Methodological Considerations in Studying the Smoking Acquisition Process, paper presented at the Annual Convention of the American Psychological Association, Anaheim, CA, August E. E. Jones and H. Sigal!, The Bogus Pipeline: A New Paradigm for Measuring Affect and Attitude, Psychological Bulletin, 76, pp , R.I. Evans, W. B. Hansen, and M. B. Mittelmark, Increasing the Validity of Self-Reports of Smoking Behavior in Children, Journal of Applied Psychology, 62, pp , K. E. Bauman and C. W. Dent, Influence of an Objective Measure on Self Reports of Behavior, Journal of Applied Psychology, 6 7, pp , A. Biglan, C. L. Nautel, D. V. Ary, and R. Thompson, Self-Reported Marijuana Smoking: Relationship to Carbon Monoxide and Thiocyanate Measures, paper presented at the Annual Convention of the American Psychological Association, Anaheim, CA, August L. M. Grossman, C. A. Johnson, B. R. Flay, and W. B. Hansen, Theoretical and Methodological Factors in Substance Abuse Prevention, paper presented at the Annual Convention of the American Psychological Association, Anaheim, C. A. Johnson, J. W. Graham, W. B. Hansen, and B. R. Flay, Categorical Drug Use as a Risk Factor for Other Drug Use, paper presented at the Annual Convention of the American Psychological Association, J. W. Graham, C. A. Johnson, B. R. Flay, W. B. Hansen, and J. L. Sobel, The Synchronous Correlates of Drug Use: A Preliminary Correlational Analysis of Data from Project SMART, paper presented at the Annual Convention of the American Psychological Association, Anaheim, CA, August 1983.

19 SELF-REPORT MEASURES IN DRUG RESEARCH I F. M. Lord and M. R. Novick, Statistical Theories of Mental Tests, Addison-Wesley, Reading, Mass., J. C. Nunnally, Psychometric Theory, McGraw-Hill, New York, N. Cliff, Elementary Multivan ate Analysis, Academic Press, New York, in press. 17. R. G. Lehnen, Assessing Reliability in Sample Surveys, Public Opinion Quarterly, 4, pp , D. B. Kandel, Stages in Adolescent Involvement in Drug Use, Science, 190, pp , G. J. Hub a and P. M. Bentler, A Developmental Theory of Drug Use: Derivation and Assessment of a Causal Modeling Approach, Life-Span Development and Behavior, 4, pp , W. B. Hansen, C. K. Malotte, L. M. Collins, and J. A. Fielding, Dimensions and Psychosocial Correlates of Adolescent Alcohol Use, Journal of Alcohol and Drug Education, in press. Direct reprint requests to: John W. Graham, Ph.D. Health Behavior Research Institute University of Southern California 1985 Zonal Avenue Los Angeles, CA 90033