Appendix A Coding Manual

Transcription

1 Appendix A Coding Manual page 1

2 SOCIAL INFORMATION PROCESSING PROGRAMS CODING MANUAL ELIGIBILITY CRITERIA A. Interventions To be eligible, interventions must meet the three criteria that define social information processing programs. Although other treatment components may be included (e.g., behavioral social skills training, parenting skills training), the social information processing component must be the clear focus of the program. The definitional criteria are as follows: 1. Programs involve training in one or more of the social information processing steps: (1) encoding situational and internal cues, (2) interpretation of cues, (3) selecting or clarifying a goal, (4) generating or accessing possible responses, (5) choosing a response, (6) and behavioral enactment. 2. Programs emphasize cognitive skills or thinking processes rather than specific behavioral skills. By teaching generic thinking skills, the programs aim to improve information processing in myriad social situations. 3. Programs involve the use of structured tasks and activities through which the cognitive skills are learned and applied to actual social situations. B. Settings and Subjects The interventions must be delivered to school-aged children (K-12 or equivalent in international settings) in regular school settings during school hours. Special education classrooms, private, parochial, and alternative schools are eligible school settings, although classrooms in residential facilities (e.g., psychiatric hospitals) are not. After-school programs are not eligible. Any qualifying school in any region or country is eligible. C. Outcomes Quantitative data must be reported for at least one outcome variable for which a numeric mean-difference (i.e., treatment vs. control) effect size can be calculated. At least one outcome variable should represent aggressive behavior, broadly defined to include violence, aggression, fighting, person crimes, disruptive behavior problems, acting out, conduct disorder, externalizing problems, and so forth. Once an aggressive behavior outcome is found, all other outcomes will be coded as well. D. Research Design Only experimental and quasi-experimental designs are eligible. The intervention and control groups can be randomly or nonrandomly assigned but, if nonrandom, must be matched or provide evidence of initial equivalence on key demographic variables and/or pretests. Control groups can represent placebo, waitlist, no treatment, or treatment as usual conditions. E. Date of Publication The date of publication or reporting of the study must be 1950 or later even though the research itself might have been conducted prior to [NOTE: It is unlikely that any programs prior to the 1970s will be identified because the social information processing models and accompanying social problem solving intervention techniques were not developed until the 1970s]. Eligibility Criteria page 1

3 FILEMAKER INFORMATION There are six different FileMaker files that we use to code studies for meta-analysis. All of these files are linked together so that you can navigate between them by clicking the appropriate buttons. These files are defined below. Note that you may not need to use all of these files in your coding. I. Bibliographic Database (bib.fp5): We use this database to maintain the bibliography of potentially eligible reports. This database includes bibliographic information about each report, library location information, and tracking data about how far along each report is in the retrieval and coding process. You might use this database to search for related reports or to indicate when an article has been coded. II. Eligibility Database (eligible.fp5): This database, generally accessed through the Bibliographic Database, contains information about the eligibility of each identified study; each study may be represented by one or more reports in the Bibliographic Database. II. Header Database (header.fp5): This database includes the general information pertinent to a study. A study is defined as an investigation involving one independent group of subjects. For example, in a quasiexperiment with a treatment and control group, the study includes both the treatment and control subjects. There will be one record in this database for each study that is coded (there may be multiple reports per study or multiple studies per report). Variables coded at this level include information about the author(s), methodology, treatment groups, initial group equivalence, etc. III. Treatment Groups Database (groups.fp5): This database includes information about the various treatments delivered as part of a study. There will be one record in this database for each aggregate treatment OR comparison group for which there is sufficient data. Thus, in a quasi-experimental study with one treatment group and one comparison group, you will have two records in this database. Variables coded at this level include information about the treatment and subjects under study. IV. Dependent Variables Database (dvinfo.fp5): Within a given study, the researcher might evaluate the effectiveness of the intervention using multiple outcomes (or dependent variables). For example, the researcher might assess the number of aggressive acts per subject during an observation period and the number of school suspensions per subject. Thus, for this study, you would have two records in the dependent variables database, one for each outcome measure that you are coding. If the same outcome measure is used multiple times in a study, as in a pretest, a posttest, and a follow-up, you will only have one record for the outcome, but you will have multiple records for the effect sizes (see VI below). This database includes a description of each dependent variable, and some basic methodological information about the variable, such as reliability. V. Breakouts (breakout.fp5): Breakouts are comparisons for subgroups of any treatment or control group, e.g., a treatment group compared with a comparison group using only the males in the sample. Each variable (e.g., gender, age) by which a group or groups are crossed constitutes one breakout; each value of that variable defines one subgroup; e.g., a male vs. female stratification is one breakout with two subgroups, one male and one female. If only the male subgroup is reported, there is still one breakout, but only one subgroup. Note that a simple report of the number of males and females in the treatment and control groups does not constitute a breakout (though it is relevant to group equivalence issues). To be a breakout, outcome data must be reported for the treatment-control comparison for at least one subgroup of the breakout variable. Breakouts are usually presented because the authors think that subgroups (e.g., males and females) are sufficiently different to warrant separate presentation of results (because, for example, males may exhibit more antisocial behavior than females). VI. Effect Sizes (es.fp5): The effect size database tracks the actual statistical results of the study being coded. Because different researchers may present their results using different statistics (e.g., with a t-test or using means and standard deviations), we need to convert the statistics from each study into a common metric for our own analyses. The metric we use for this purpose is called an effect size. You may use the Effect Size Determination Program (Excel Toolkit), or, in some cases, the Shadish ES Calculator, to do the calculations. FileMaker Information page 2

4 Header Variables CODING Step 1. Study Identifiers, Study Context, and Group Identification & Selection STUDY IDENTIFIERS The unit you will code here consists of a study, i.e., one research investigation of a defined subject sample or subsamples compared to each other, and the treatments, measures, and statistical analyses applied to them. Sometimes there are several different reports (e.g., journal articles) about a single study. In such cases, the coding should be done from the full set of relevant reports, using whichever report is best for each item to be coded; BE SURE YOU HAVE THE FULL SET OF RELEVANT REPORTS BEFORE BEGINNING TO CODE. Sometimes a single report describes more than one study, e.g., one journal article could describe a series of similar studies done at different sites. In these cases, each study should be coded separately as if each had been described in a separate report. Each study has its own study identification number, or StudyID (e.g., 619). Each report also has an identification number (e.g., ), which you will find printed on the folder holding the report. The ReportID has two parts; the part before the decimal is the StudyID, and the part after the decimal is used to distinguish the reports within a study. (These two types of ID numbers, along with bibliographic information, are assigned and tracked using the bibliographic database.) When coding, use the study ID (e.g., 619) to refer to the study as a whole, and use the appropriate report ID (e.g., ) when referring to an individual report. While reading reports for coding, be alert to any references to other early intervention studies that may be appropriate to include in this meta-analysis. If you find appropriate-looking references that are not currently entered into the Bibliography, the references may need to be entered. NOTE: SPSS variable names are shown in brackets. [StudyID] Study identification number of the study you are coding, e.g., [Coder] Coder's initials (select from menu) [SH21] Year of publication (four digits): If more than one report, choose earliest date. [CodeDate] Date you began coding this study (will be inserted automatically) STUDY CONTEXT [SH5] Senior author's discipline. In cases where the senior (first listed) author may have several disciplines (such as medicine and public health), select the primary discipline of the author. If you cannot tell which discipline is primary, select the first one (e.g., in the case where the author s title is John Smith, MD, MPH, choose medicine). 01 psychology 02 sociology 03 education 04 criminal justice; criminology 05 social work 06 psychiatry; medicine 07 political science 08 anthropology 09 public health 10 nursing 11 evaluator 99 cannot tell Header Variables Coding Sheet page 3

5 other: [SH6] Country in which study was conducted. 1 USA 2 Canada 3 Britain 4 other Commonwealth/English speaking: 5 continental Europe: 6 other: 9 cannot tell [SH20] Type of publication. If you are using more than one type of publication to code your study, choose the publication that supplies most of the data. 1 Book 2 Journal article 3 Book chapter (in an edited book) 4 Thesis or dissertation 5 Technical report 6 Conference paper 7 Other 9 Cannot tell GROUP IDENTIFICATION AND SELECTION At this stage, you will need to identify the aggregate treatment and/or comparison groups used in the study for which effect size statistics can be computed. To do this, you will need to distinguish aggregate groups, which you will code here, from subgroups, which you will code later: (1) Aggregate treatment and/or comparison groups. The largest subject groupings on which contrasts between experimental conditions can be made. Note that the designations comparison group and control group refer to any group with which the treatment of interest is compared that is presumed to represent conditions in the absence of that treatment, whether a true random control or not. Often there is only one aggregate treatment group and one aggregate control group, but it is possible to have a design with numerous treatment variations (e.g., different levels) and control variations (e.g., placebos) all compared (e.g., in ANOVA format). (2) Subgroups. Sometimes researchers will present data for some subset(s) of the subjects from an aggregate group; e.g., for an aggregate group composed of males and females, the researchers may present some results for the males and females separately. These are called breakouts. You will code information about breakouts later. Identifying the Aggregate Groups Type in the name or identifier for each aggregate treatment group and each aggregate comparison group described in the study, whether you believe the group is eligible for coding or not. Group labels used by researchers do not necessarily conform to the definitions of group types used in this project. In some cases, for example, researchers may compare one treatment with another treatment, and may call this other treatment a comparison or control group. For our purposes, if this other treatment group can realistically be expected to be effective, list it as a treatment group below; if it is a minimal or placebo treatment, not expected to produce an effect, list it as a comparison group. Header Variables Coding Sheet page 4

6 Write in Name Treatment Groups [SH8a-d] Write in Name Comparison Groups [SH10a-d] [SH9] Total number of treatment groups [SH11] Total number of control groups Selecting Aggregate Groups for Treatment-Control Comparison For an experimental or quasi-experimental study in which at least one treatment group is compared with at least one control group intended to represent conditions when the treatment of interest is not present, ONE treatment and ONE control group should be selected. If there is only one treatment group and one control group, then they are the groups coded. If there is more than one treatment or control group, select the group to be coded as follows: If each treatment group is paired with its own distinct control group, separate studies are constituted, and each treatment-control pair should be coded as a separate study. Remember that each separate study needs its own StudyID; see the bibliography coordinator for more information. If you are sure that you have a single study with more than one treatment or control group, select the groups to be used in the treatment-control comparison according to the following rules: 1. More than one aggregate treatment group. First, determine how many of the various treatments are social information processing programs. If there is only one, select that group for coding. 2. More than one social information processing program [NOTE: this will be rare]. Next, determine if the multiple eligible treatments are sufficiently similar to combine. This requires that treatment be virtually the same, at least by generic label, for each group, e.g., groups with the same treatment but implemented at different sites or stratified into subgroups that can be recombined into a sensible whole. In such cases, combine the treatment groups into a composite whole if appropriate statistics are available (e.g., main effect in an ANOVA design). If statistics for combination are unavailable or if the treatments cannot be combined into a relatively uniform composite, select the one treatment group to code as follows: (a) If one treatment is clearly the focal concern of the study, select the focal treatment. (b) If no one treatment is clearly the focal concern of the study, and if the treatments are parametric variations, e.g., class curriculum with and without role playing, then select the most complete or extensive treatment, e.g., the class curriculum with role playing. Extensive refers primarily to breadth of services, not number of hours of service. (c) If the treatments are of equal interest to the study, and of equal completeness (not more or fewer components), then select the one with the largest N. If equal N, select the one that is least unusual and if equal in that regard, make a random choice (coin toss). Header Variables Coding Sheet page 5

7 2. More than one aggregate control group. Reminder: If a report contains multiple control groups, and each is paired with its own treatment group, separate studies are constituted. When multiple controls and/or multiple treatments are compared all together in a single multi-group study, select the best control group available to code from the rank order listing below: (a) treatment as usual control (controls get usual handling instead of special treatment, e.g., regular class time while treatment group gets special curriculum) (b) no treatment (control group receives nothing). See below under item [G2] for information on the difference between treatment as usual and no treatment. (c) placebo control (controls get some attention or sham treatment) (d) straw man or alternate treatment control not expected to be effective but used as contrast for treatment group of primary interest (note: if the alternate treatment is not minimal and could realistically be expected to be effective, it is not a control condition and should not be used here). Write in: [SH12] Treatment group for aggregate treatment-control comparison [SH13] Control group for aggregate treatment-control comparison Step 2. Group Equivalence For The Selected Treatment-Control Comparison ASSIGNMENT OF SUBJECTS [SH26] Unit of group assignment. The unit on which assignment to groups was based. 1 individual (i.e., some children assigned to treatment group, some to comparison group) 2 group (i.e., whole classrooms, schools, therapy groups, sites, residential facilities assigned to treatment and comparison groups) 3 program area, regions, school districts, counties, etc. (i.e., region assigned as an intact unit) 9 cannot tell [SH27] Method of group assignment. How subjects/units were assigned to treatment vs. control groups. Random or near-random: 01 randomly after matching, yoking, stratification, blocking, etc. The entire sample is matched or blocked first, then assigned to treatment and comparison groups within pairs or blocks. This does not refer to blocking after treatment for the data analysis. 02 randomly without matching, etc. This also includes cases when every other person goes to the control group 03 regression discontinuity design: quantitative cutting point defines groups on some continuum (this is rare) 04 wait list control or other quasi-random procedure presumed to produce comparable groups (no obvious differences). This applies to groups which have individuals apparently randomly assigned by some naturally occurring process, e.g. first person to walk in the door. The key here is that the procedure used to select groups doesn t involve individual characteristics of persons so that the groups generated should be essentially equivalent. Header Variables Coding Sheet page 6

8 Nonrandom, but control group selected to match treatment group: Matching refers to the process by which individuals or groups of individuals assigned to treatment and control groups are equated based on various characteristics. Comparing the characteristics of two experimental or other groups AFTER they have been assigned to treatment and control groups does NOT constitute matching. When the matching is done based on individual characteristics, it is called matching. When the matching is done based on group characteristics, it is called equated groupwise. 05 matched ONLY on pretest measures of some or all variables used later as outcome measures (individual level) 06 matched ONLY on personal characteristics, previous violent behavior, introversion-level, selfesteem, etc. other than dependent variables used later as outcome measures (as above) or demographic variables (see below) 07 matched ONLY on demographics: big sociological variables like age, sex, ethnicity, SES, (individual level) 08 matched on all three above variables 09 matched on pretest and personal characteristics 10 matched on pretest and demographics 11 matched on personal characteristics and demographics 12 equated groupwise; e.g., picking intact classroom or school of similar characteristics to treatment classroom or school, e.g. mean ages of groups are similar. Nonrandom, no matching prior to treatment but descriptive data, etc. regarding the nature of the group differences: 13 convenience comparison groupwise: other available group such as a classroom taken w/o matching or equating (like individual selection but done groupwise) 14 purposive selection: subjects are chosen for the treatment group because they are at-risk, are selected by teachers for behavior problems, etc. NOTE that this category applies only when the subjects in ONE group are selected for intervention while the other group is comprised of individuals selected but not matched to intervention group (e.g., whoever is left after at-risk kids are selected for treatment). Cases where at-risk kids are chosen and THEN divided into treatment and control groups do not fit in this category they fit into one of the above categories based on how the kids were assigned to the two groups. 15 random assignment by group Cannot tell: 99 cannot tell [SH28] Confidence in assignment ratings. Rate your overall confidence of your judgment on how subjects were assigned: 1 Very Low (Little Basis) 2 Low (Best Estimate) 3 Moderate (Weak Inference) 4 High (Strong Inference) 5 Very High (Explicitly Stated) Header Variables Coding Sheet page 7

9 COMPARISONS BETWEEN GROUPS [SH30] Number of variables on which treatment and comparison group differences were statistically compared prior to the intervention. A statistical comparison is one in which a statistical test was performed by the authors, whether they provide data or not (e.g., no statistically significant differences were found ). [SH33] Results of statistical comparisons. 1 no comparisons made 2 no statistically significant differences 3 significant differences judged unimportant by coder. See note below regarding importance judgment. 4 significant differences, judged of uncertain importance by coder 5 significant differences, judged important by coder [SH31] Number of variables on which treatment and comparison group differences were or can be descriptively compared prior to the intervention. A descriptive comparison is any comparison across treatment and control groups that does not involve a statistical test (e.g., the actual number of males and females in each group or a statement by the author(s) about group similarity). [SH29] Results of descriptive comparisons. 1 no comparisons made or available 2 negligible differences, judged unimportant by coder. See note below regarding importance judgment. 3 some differences, judged of uncertain importance by coder 4 some differences, judged important by coder Note: An important difference means a difference on several variables relevant to the outcome variables, or on a major variable, or large differences; major variables are those likely to be related to antisocial behavior, e.g., history of aggression or other antisocial behavior, antisocial risk or prediction, sex, age, ethnicity, SES, family circumstances, or temperament. GROUP EQUIVALENCE EFFECT SIZE CODING At this point, you should go to the Effect Size Database to code group equivalence effect sizes and descriptive information about initial group differences for this treatment-control comparison. See the Effect Size Coding Sheet section of this manual for more information on effect size calculation. For each measure you can identify on which the treatment and control group were compared (other than dependent variables) or on which you can tell equivalence (e.g. if all males then code it here), determine which group is favored and if possible, calculate an effect size (ES, standardized difference between means). Do not include here any comparisons on pretest variables, that is, measures of a dependent variable taken prior to treatment, e.g., prior number of suspensions when subsequent number of suspensions is used as an outcome measure. In such cases the pretreatment ES is coded later as pretest information, not here as group equivalence information. [StudyID] Indicate the Study ID for the study you are coding. [ReportID] Enter the Report ID for the report in which you found the information on group equivalence. Use the complete Report ID, e.g [pagenum] Enter the page number on which you found the information on group equivalence. Header Variables Coding Sheet page 8

10 [ES24] Type of Effect Size. Select group equivalence here. 1 Pre-Test 2 Post-Test 3 Follow-up 4 Pretest-Posttest 5 Group Equivalence [ES15] Variable on which comparison is made. 1 Sex 2 Age 3 Ethnicity 4 Prior antisocial behavior (e.g., behavioral problems, school disciplinary incidents, aggression, delinquency, etc.) 5 Risk status (groups are compared on risk factors, e. g. school performance, family characteristics). The list of risk factors for antisocial behavior in the Eligibility Criteria can serve as a guide for the kinds of things that are viewed as risk factors. 6 Any other variable (describe below) [ES16] If 4, 5, or 6 above, describe comparison variable. [ES17] Which group is favored? Whichever group has more of the characteristic that presumably makes them better off or more amenable to treatment (e.g., less violent, higher SES, smarter, etc.) is considered favored. NOTE: You should code this item even for cases in which you are unable to calculate a numeric effect size but have information about which group is favored. 1 Treatment (fewer males, younger, fewer minorities, less antisocial, less risk) 2 Control (fewer males, younger, fewer minorities, less antisocial, less risk) 3 Neither, exactly equal 9 Cannot tell, no report [ES42] Write a brief description of the group differences or similarities. For example, the treatment group is older than the control group or the control group has more special education kids than the treatment group. Data Fields: Fill in the data fields using the relevant statistical information provided in the report(s). You do not need to fill in all the fields; fill in only the information necessary to calculate an effect size. Thus, if the report provides sample sizes, means, standard deviations, and t-test scores, you need only enter the sample sizes, means, and standard deviations. ONCE YOU HAVE FINISHED CODING THE GROUP EQUIVALENCE EFFECT SIZE INFORMATION, YOU SHOULD RETURN TO THE HEADER FILE TO COMPLETE THE CODING OF HEADER VARIABLES. TREATMENT-CONTROL OVERALL SIMILARITY [SH52] Rating of similarity of treatment and control groups. Using all the available information, rate the overall similarity of the treatment group and the comparison group, prior to treatment, on factors likely to have to do with antisocial behavior or responsiveness to treatment (ignore differences on any irrelevant factors). Note: Greatest equivalence from clean randomization with prior blocking on relevant characteristics and no subsequent attrition/degradation; least equivalence with some differential selection of one type of individual vs. another on some variable likely to be relevant to antisocial behavior, e.g., disruptive kids compared with normal sample. Header Variables Coding Sheet page 9

11 Guidelines: Use ratings in the 1-3 range for good randomizations and matchings, e.g., 1=clean random, 2=nice matched. Use ratings in the 5-7 range for selection with no matching or randomization or instances where it has been seriously degraded, e.g., by attrition before posttest. Within this bracket, the question is whether the selection bias is pertinent to the outcomes being examined. Were subjects selected explicitly or implicitly on a variable that might make a big difference in antisocial behavior? The middle three points are for sloppy matching designs, degradations, bad wait list designs, and the like. If the data indicate equivalence but the assignment procedure was not random give it a 4 or thereabouts since not all possible variables were measured for equivalence between groups. 1 Very similar, equivalent Very different, not equivalent [SH53] Overall confidence on rating of group similarity: 1 Very Low (Little Basis) 2 Low (Best Estimate) 3 Moderate (Weak Inference) 4 High (Strong Inference) 5 Very High (Explicitly Stated) [SH1] Click here to record any problems you encountered while coding this header. Header Variables Coding Sheet page 10

12 TREATMENT GROUPS CODING Create one record in this database for each of the aggregate treatment and/or control groups that you selected earlier for coding. Studies with a single treatment group will have one record, studies with a treatment group and a control group will have two records, etc. Group Identification And General Nature Of Treatment [StudyID] Type in the StudyID for the study you are coding if it does not appear automatically. [GroupID] Number each group consecutively within a study, starting with 1. [G2] What general type of treatment does this group receive? Intervention Condition 1 Focal program or treatment. Control Condition 2 Straw man alternate program or treatment, diluted version, less extensive program, etc., not expected to be effective but used as contrast for treatment group of primary interest. If the alternate treatment is not minimal and could realistically be expected to be effective, it is not a control condition and should be classified as a focal treatment instead. 3 Placebo (or attention) treatment. Group gets some attention or sham treatment (e.g., watching Wild Kingdom videos while treatment group gets therapy) 4 Treatment as usual. Group gets usual handling instead of some special treatment. 5 No treatment. Group gets no treatment at all. Note: The difference between no treatment and treatment as usual hinges on whether or not the treatment and control groups in this study have an institutional framework or experience in common. For example, consider a case in which treatment subjects are selected from normal school classes and the intervention is delivered in the classroom. The comparison group is in a separate class and the subjects receive their usual curriculum. This circumstance should be coded as treatment as usual because the treatment and comparison group children are in the same institutional setting and the subjects in the comparison group are doing what they would normally be doing. But, say we have a program in which children are pulled out of class for treatment because they have behavior problems. The comparison group children also have behavior problems but they stay in their regular classrooms. This is a no treatment situation because the two groups are not in the same institutional setting; the comparison group children are not receiving the usual pull-out services for behavior problem children so this is not a treatment as usual situation. [G8] Program name. Write in program or treatment label for this group (e.g., I Can Problem Solve, waiting list control, etc.). REMEMBER: YOU MUST CREATE A PROGRAM LABEL FOR CONTROL GROUPS AS WELL AS TREATMENT GROUPS. [G9] Program description. Write in a brief description of the treatment this group receives. Please try to keep the description short by focusing on the key elements of treatment, but make sure you include ALL treatment elements in your description. As much as possible, quote or give a close paraphrase of the relevant descriptive text in the study report. REMEMBER: YOU MUST CREATE A DESCRIPTION FOR CONTROL GROUPS AS WELL AS TREATMENT GROUPS. Treatment Groups Coding Sheet page 11

13 [G10] Overall confidence in description of program: 1 Very Low (Little Basis) 2 Low (Best Estimate) 3 Moderate (Weak Inference) 4 High (Strong Inference) 5 Very High (Explicitly Stated) [Prog1 Prog44] Service coding. Identify all the treatment components, elements, activities, experiences, etc. reported as part of the intervention. Note that to qualify, a component should be something the treatment group receives that the control group does not receive. Use the following rating scale for each reported component. At least one component must be rated for every intervention but as many components can be rated as needed to describe every distinct element reported. Initially you should assume that each such component will receive a rating of "1," as if the 1 was a checkmark to check off every item present. However, if there is any indication in the study report(s) that one or more components are of lesser scope or importance than others, then those secondary items should be coded as "2." A component might be identified as secondary in this sense because: (a) it is clearly a subcomponent of something else or there is a broad program type to be coded "1" and the component is only one aspect of that; (b) it is provided to only a subset of subjects or only occasionally in contrast to other components provided to all subjects or on all occasions; (c) some other distinction is made that shows that the component is not of equal importance, stature, or scope as others that are coded as "1." Behavioral-type interventions [1] Behavioral contracting, contingency management [2] Token economy [3] Shaping of specific behaviors, reinforcement for desired behavior (e.g., timeout) Behavioral Affect-Control techniques [4] Behavior modification, desensitization, flooding [5] Relaxation training (e.g., breathing, imaging of peaceful scenes) Cognitive Skills/Cognitive Restructuring [6] Cognitive restructuring, interpersonal (monitoring automatic thoughts, identifying triggers, identifying and correcting distortions e.g., attribution retraining) [7] Cognitive restructuring, not interpersonal (e.g., cognitive impulse control) [8] Cognitive anger management (identify triggers, identify physical cues, self-statements and/or nonhostile explanations for others behavior, using reducers to minimize anger, self evaluation) [9] Cognitive social problem solving (decision making, problem solving, alternative solutions, consequential thinking) [10] Cognitive skills for stress or failure (e.g., dealing with being left out, dealing with divorce) [11] Perspective taking, feeling identification Interpersonal skills [12] Behavioral social skills training, interpersonal skills building (communication skills) [13] Resisting group pressure, assertiveness training Instructional [14] Alcohol, drug education (factual, instructional) [15] Violence prevention education (factual, instructional) [40] Life skills training Treatment Groups Coding Sheet page 12

14 Educational [16] Receiving tutoring assistance [17] Academic training (e.g., spec. ed) [18] School readiness instruction Counseling/Therapy (not behavioral, cognitive, or cognitive-behavioral) [19] Individual counseling [20] Group counseling [21] Family counseling [22] Mentoring (having a mentor) Treatments Delivered to Others [23] Parenting skills training [24] Teacher capacity building, in-service [25] Therapy, counseling, support for parents [26] Drug treatment for parents Peer Mediation Programs [27] Peer mediation practice, service as mediator [28] Peer mediation program, availability of mediation services [29] Peer mediation participant, received mediation services from peers Recreational Programs [30] Martial arts training [31] Therapeutic day camp [32] Recreational activities (e.g., sports, shop) Service Programs [33] Serving as peer counselor [35] Community, school service [36] Serving as peer tutor Other [34] Pharmaceutical treatment [37] Case mgt, service brokerage [38] School-wide systemic, physical plant, etc. [39] Wilderness program [41] Participates in work program [42] Job skills training Characteristics Of Focal Treatment NOTE: If this aggregate group was identified above as receiving a focal program or treatment, an alternate treatment, or a placebo, complete the section below; if the group is a treatment as usual or no treatment condition, skip to the Characteristics of Juveniles section of this coding sheet. [G3] Program/treatment age at time of research. This item refers to the age and stage of development of the treatment protocol IN THE CURRENT SETTING. Thus, rate the age of the program based on when it was implemented and evaluated in the setting described in this study, even if the program is no longer in existence. 1 relatively new; less than two years old, or relatively few cohorts of participants 2 established program; in place two years or more, or many cohorts of participants 3 defunct program, evaluated post hoc 9 cannot tell Treatment Groups Coding Sheet page 13

15 [G4] Role of the evaluator(s)/author(s)/research team or staff in the program. This item focuses on the role of the research team working on the evaluation, regardless of whether they are all listed as authors. 1 evaluator delivered therapy/treatment 2 evaluator involved in planning or controlling treatment or is designer of program 3 evaluator influential in service setting but no direct role in delivering, controlling, or supervision 4 evaluator independent of service setting and treatment; research role only 9 cannot tell [G6] Routine practice or program vs. research project. Indicate the appropriate level for the treatment you are coding: at one end of the continuum are research projects (option 1), in which a researcher decides to implement and evaluate a particular program for research purposes; in many cases, the program may require the cooperation of a service agency (school, clinic, etc.), but the intervention is delivered primarily so the researcher can conduct research. At the other end of the continuum are evaluations of real-world or routine programs (option 3): a service agency implements a program on its own, and also decides to conduct an evaluation of the program; the evaluation may or may not be conducted by outside researchers. In the middle of the continuum are demonstration projects (option 2), which are conducted primarily for research purposes, but generally have more elements of real world practice than typical research projects as defined under option 1. Demonstration projects generally involve a special or new program that is being tested for effectiveness. If a researcher is also a school principal and is conducting the evaluation as part of his/her dissertation project, the decision depends on the extent of the program. If the program is small-scale and implemented in, say, a classroom or two, and supervised by the researcher/principal, code it as a research project. If the program is a broader school-wide program that the researcher/principal happens to be evaluating, code it as either a demonstration or routine program, depending on whether the program is a special program being tested (demonstration) or something that the school does on a routine basis (routine practice). 1 research project: The intervention would not have been implemented without the interest of the researcher(s). The intervention is delivered by the research staff or by service providers (regular agency personnel, teachers, etc.) trained by the researchers. 2 demonstration project: A research project that involves a new or special program being tested, rather than a routine program. Although generally implemented by researchers for research purposes, a demonstration project has more elements of actual practice than a research project. 3 evaluation of a real-world or routine program: A service agency implemented the program using routine personnel and the typical clients for that program; there may be outside researchers who conduct the evaluation, but the program they are evaluating was already in place in the agency before the research began and is presumed to continue after the research has ended. [G12] Treatment Site. Nature of the site on which treatment generally delivered: [select all that apply] G12_1 Regular Class Time (this includes interventions delivered during regularly scheduled classes AND in the children s regular classroom) G12_2 Special Class (e.g., children in treatment are in a classroom-type setting that is different from a typical classroom, although it may be the subjects usual classroom includes such settings as special education classrooms, schools-within-schools, alternative schools, etc.) G12_3 Resource Room, School Counselor's Office, or other similar setting that is NOT the children s regular classroom; the idea here is that children are removed from class for treatment G12_9 Other: [G53] Does the entire school receive the intervention? 1 Yes 2 No Treatment Groups Coding Sheet page 14

16 [G13] Who delivers or provides the treatment? This is the person (or persons) who interacts with the youth (and/or their teachers and families) in the treatment transaction. Teachers 01 Regular teachers 02 Specially trained teachers (teachers who are trained specifically to deliver the intervention AND who are not the students regular teacher) Other persons 03 Laypersons (e.g., aides, interns, volunteers) 04 Mental health personnel (e.g., psychiatrists, psychologists) 05 Non-mental health counselors or caseworkers (e.g., social worker, caseworker, school counselors, vocational counselors, etc.) 06 Researcher or research team (including graduate students supervised by the researcher) 07 Police, probation, juvenile justice or security personnel 08 Parents or caregivers 09 Peers 10 Mixed or multiple, some combination of above individuals (write in below) 11 School administrator Cannot tell 99 Cannot tell [G54] If multiple treatment delivery personnel, please write in the different personnel here. [G14] Did treatment personnel receive special training in this specific program, intervention, or therapy? If the treatment is delivered by the researcher, use yes below, unless the report indicates otherwise. 1 yes 2 no 9 cannot tell [G15] If yes, amount of training of personnel for providing this treatment. Code number of hours of training provided: [G17] Intervention or program protocol. This has to do with the availability of a manual describing how treatment should be implemented. 1 manualized; program has written manual or protocol 2 written guidelines, principles, policies, but no specified treatment protocol 3 no mention of protocol, guidelines, etc., and program is not branded [G18] Primary format of treatment sessions. The primary emphasis of this question is on who was present with the juvenile during treatment sessions. If format of treatment sessions is mixed and one predominates, select the predominant format. 01 subject alone (self-administered treatment, e.g., bibliotherapy; nobody else is present but the subject) 02 subject and provider, one on one 03 group of subjects with provider, not a classroom setting (e.g., group therapy session) 04 student group, classroom setting 05 child and parent(s) together with provider 06 treatment professionals, teachers, school staff only; child not present 07 mixed, no single format predominates; see [G60] below. If one format is predominant, select that format instead. 08 group of subjects, their families, and provider(s) 99 cannot tell Treatment Groups Coding Sheet page 15

17 [G60] If primary format is mixed, list ALL distinct treatment session formats. Use language from [G18] when possible. Focal Treatment Implementation/Length/Integrity NOTE: If this aggregate group was identified above as receiving a focal program or treatment, an alternate treatment, or a placebo, complete the section below; if the group is a treatment-as-usual or no-treatment condition, skip to the Characteristics of Juveniles section of this coding sheet. When answering questions on treatment duration, you will also be asked to indicate whether the answer you provide is based on "facts," "format," or "other." Only answer "facts" when the author reports having measured the dosage dimension for the sample and provides actual data (means, proportions, etc.) regarding the frequency, duration, number of hours, etc. (These cases are rare.) All other cases are either "format" or "other." Statements about "a 20-week program" are exactly what is meant by "format" this is the standard format but each of the participating individuals may or may not actually have received the whole 20 weeks. If format information is not reported, make an informed estimate about the amount and frequency of contact whenever possible and report the estimate as "other." Assume that a counseling session and a school period are probably each an hour long unless otherwise specified. [G20] Duration of treatment. Approximate (or exact) number of weeks that juveniles received treatment, from first treatment event to last excluding follow-ups designated as such. If individuals other than juveniles were the primary recipients of treatment, indicate the number of weeks the juveniles were exposed to the others who received treatment. Divide days by 7; multiply months by 4.3. Code 999 if cannot tell. Estimate for this item if necessary and if you can come up with a reasonable order of magnitude number. [G21] Determined by: 1 facts (data about how long clients in treatment, e.g., average client attended 7.3 weeks) 2 format (standard package or plan without information on actual participation, e.g., a ten-week program) 3 other estimate [G22] Approximate (or exact) frequency of contact between juvenile and provider or treatment activity. This refers only to the element of treatment that is different from what the control group receives. 2 daily contact (not 24 hours of contact per day but some treatment during each day, perhaps excluding weekends) times a week times a week 5 less than weekly 9 cannot tell [G23] Determined by: 1 facts (data) 2 format (standard package/plan). Code continuous treatments here unless you have specific information about discrepancies from the prescribed format. 3 other estimate [G24] Approximate (or exact) mean hours actual contact time between juvenile and provider or treatment activity per week if reported or calculable. Code 9999 if not available. Treatment Groups Coding Sheet page 16

18 [G25] Determined by: 1 facts (data) 2 format (standard package/plan). 3 other estimate [G26] Approximate (or exact) mean number of hours total contact between juvenile and provider or treatment activity over full duration of treatment per juvenile if reported or calculable. Code 9999 if not available. [G27] Determined by: 1 facts (data) 2 format (standard package/plan) 3 other estimate [G28] Overall confidence in estimates of treatment duration, frequency, and contact. 1 Very Low (Little Basis) 2 Low (Best Estimate) 3 Moderate (Weak Inference) 4 High (Strong Inference) 5 Very High (Explicitly Stated) [G61] Did others receive part or all of the treatment? 1 Children/adolescents were the sole recipients of treatment 2 Both children/adolescents and others received elements of the treatment 3 Others were the sole recipients of treatment. Describe: [G62] If others received part or all of the treatment, enter mean HOURS of TOTAL CONTACT between treatment provider and others. Code 9999 if cannot tell. Treatment Groups Coding Sheet page 17

19 [G29] Monitored treatment implementation. Was the implementation of the program monitored by the author/researcher or program personnel to assess whether it was delivered as intended? 1 Yes. Do not infer that monitoring happened. Select yes ONLY if the report specifically indicates that implementation was monitored. 0 No 9 Cannot Tell [G30] Based on evidence or author acknowledgment, was there any uncontrolled variation or degradation in implementation or delivery of treatment, e.g., high dropouts, erratic attendance, treatment not delivered as intended, wide differences between settings or individual providers, etc.? Assume that there is no problem if one is not specified. This question has to do with variation in treatment delivery, not research contact. That is, there is no dropout if all juveniles complete treatment, even if some fail to complete the outcome measures. 1 yes (describe below) 2 possible (describe below) 3 no, apparently implemented as intended [imp problems] Implementation Problems. Click to describe implementation problems, if any. Characteristics Of Juveniles In This Aggregate Group (Treatment Or Control) [G31] Source of subjects. This question asks who took the initiative in identifying or choosing the subjects who ended up in this aggregate group. Thus, if the researcher initiated the referral process by asking teachers to nominate children for treatment, you would choose solicited by researcher. If, on the other hand, teachers initiated the process by identifying at-risk children, and then asked a local researcher to implement an intervention for them, you would select teachers. If this is a control group that did not actually receive treatment, respond to this item in terms of the nature of the referral or selection that made them candidates for that control condition. 02 Sought Treatment Voluntarily: The individual(s) receiving treatment initiated the treatment, e.g., self-referrals or walk-ins. 03 Parents: The parents of the individual in treatment 04 Peers, friends: The friends of the individual in treatment; peer nominations 05 Teachers: The teachers of the individual in treatment 13 Other school personnel (e.g., principals, school counselors, etc.) 06 Treatment professional; Voluntary: Specifically mental health, social worker, and the like. Assume that referral from this source is voluntary unless explicitly stated otherwise. 07 Treatment professional; Mandatory: Specifically mental health, social worker, and the like. Only use this category when mandatory is explicitly stated; otherwise, use the voluntary category above. 08 Non-Treatment professional; Mandatory: E.g., law enforcement, court system, etc. Assume that referral from this source is mandatory unless explicitly stated otherwise. 09 Non-Treatment professional; Voluntary: E.g., law enforcement, court system, etc. Only use this category when voluntary is explicitly stated; otherwise, use the mandatory category above. 10 Referred from Multiple Sources. Select this option only when you cannot tell which source is dominant; if you can determine which source is dominant, code by dominant source. 11 Solicited by Researcher; Investigator arranged subject recruitment. Use this option both when the researcher chooses who receives treatment, AND when the researcher asks teachers or other individuals to select kids for treatment. 12 Teachers and peers; e.g., as when both teachers and classmates nominate kids who are aggressive. 99 Cannot Tell Other: Treatment Groups Coding Sheet page 18

20 [G32] Predominant level of risk of juveniles in this group at onset of the study. 1 Universal: Normal children, general population, school-wide samples, etc. Selected: Selected populations are those exhibiting a risk factor for aggression, violence, or related antisocial behaviors. 2 Selected based on neighborhood, environment, or group characteristics (e.g., inner city, low SES area) 3 Selected based on individual characteristics (e.g., low reading ability, temperament) 4 Indicated: Indicated samples are those chosen for intervention because they are displaying aggression, violence, or related antisocial behaviors. 5 Mixed, mostly low end of range from above 6 Mixed, mostly moderate to high end of range from above 7 Mixed, full range of kids from above 9 Cannot tell [G7] If juveniles are selected or indicated, write in the risk factor(s) here. BE BRIEF. For example, if kids are selected because they live in an impoverished neighborhood, write in poor neighborhood; or, if kids are indicated based on teacher ratings, write in teacher ratings of aggression. [G33] Does the history of the juveniles in this group include aggression, violence, fighting, bullying, assaults, or similar person-directed antisocial behavior, whether officially recorded or not? 1 no. Select this option only if the report(s) clearly indicate that the group has no such history; do not make assumptions. 2 yes, some juveniles (<50%) 3 yes, most juveniles (= or >50%) 4 yes, all juveniles (>95%) 5 some, but cannot estimate percent 9 cannot tell [G34] Sex of juveniles in this group. 1 no males (<5%) 2 some males (<50%) 3 50% to 60% male 4 mostly males (>60%) 5 all males (>95%) 6 estimated 50-50% split. You should ONLY use this option when no other information about gender is available and ONLY for intact groups, such as classrooms or schools where you might expect the gender distribution to be approximately equal. If you have a group in which individuals are selected for treatment for ANY reason (e.g., low reading ability, aggressive behavior, poverty, etc.) you should NOT use this option. However, if an INTACT GROUP is selected for treatment for some reason (say, a school in a bad neighborhood), then this option would be appropriate, provided you have no other gender information. 9 cannot tell [G35] Enter the total number of males in this group. Leave this field blank if you can t tell. [G36] Enter the total number of females in this group. Leave this field blank if you can t tell. [G37] How was gender information obtained? 1 Given 2 Estimated from information given Treatment Groups Coding Sheet page 19

21 [G38] Predominant ethnicity of the subjects in this group. 1 Anglo 2 Black 3 Hispanic 4 Other Minority: 5 Mixed, Cannot Estimate Percent 9 Cannot Tell [G39] Percent of predominant ethnicity. Enter the approximate percent of the predominant ethnicity for this group (use 75, not.75). [G40] Enter the total number of Anglos in this group. Leave blank if you cannot tell. [G41] Enter the total number of non-anglos in this group. Leave blank if you cannot tell. [G42] How was race information derived? 1 Given 2 Estimated from information given [G43] Enter the average age of the sample at the beginning of the study using number of years. [G44] Enter the average grade level of the sample at the beginning of the study. [G45] Indicate the type of average reported above. 1 mean 2 median 3 mode 4 midpoint [G46] Enter the lowest age using number of years at the beginning of the study. [G47] Enter the highest age using number of years at the beginning of the study. OR [G48] Enter the lowest grade level at the beginning of the study. [G49] Enter the highest grade level at the beginning of the study. [G50] Type in a brief description of the socioeconomic composition of the sample. This might include information on the percentage of children eligible for free lunches, the income level of the children s parents, or a description of poverty in the community. Quote or closely paraphrase the relevant descriptive information in the report. [G51] Type in a brief description of any other personal characteristics that are mentioned in the report. This might include information about the educational or health status of the subjects, e.g., special education status, IQ, etc. [G52] Please describe any problems you encountered while coding this record. Treatment Groups Coding Sheet page 20

22 DEPENDENT VARIABLES CODING Step 1. Study And DV Identification Create one record in this database for each dependent variable that you will be coding. If the study measures aggressive behavior and the number of school suspensions, you will have two dependent variable records. This is different from the number of times a dependent variable is measured in a study. For example, if the study measures aggressive behavior before and after treatment, you will have only one record here for the aggressive behavior measure you may, however, have more than one record in the effect size database (described later), e.g., one for the pretest and one for the posttest. [StudyID] Identification number of study [VarNo] Variable number. This number is an identification number for the dependent variable you are coding. Each dependent variable is numbered consecutively, within the study you are coding, so that each dependent variable for a study has a unique VarNo. If there is only one dependent measure for this study, you will create only one record in this worksheet, and the variable number will be 1. If there are three dependent measures, they will be numbered 1, 2, and 3. [DV Description] Write in a brief description of the dependent measure you are coding. This should include the authors label for this variable (e.g., disruptive behavior), the instrument (e.g., the Child Behavior Checklist), the direction of scoring (e.g., lower scores are better), and information about what is being measured (e.g., fighting, acting out, spitting at the teacher, etc.). Quote or closely paraphrase the description that is provided in the original report. Sometimes it may be difficult to determine the direction of scoring, because some studies use surveys or paper-and-pencil measures in which it is unclear whether a high score or a low score is better. In these situations, a thorough reading of the text from the results and discussions sections, in combination with data from figures or tables, can usually bring to light the direction of scoring for a particular measure. In addition, the Measures database in FileMaker may provide information about the measure in question. Note that if you cannot determine the direction of scoring for a particular DV, you will not be able to calculate numeric effect sizes for that DV. (You will still be able to create effect size records for the DV just not numeric effect sizes.) Remember that every study must produce at least one numeric effect size to be eligible for coding; if you find that you cannot determine the direction of scoring for any of the DVs in a study, the study is not eligible. [DV1], [DV2] Select the general construct grouping using the list of general construct groups in [DV1]. Then, from the list of specific constructs for that group [DV2], select the specific construct that best represents the dependent variable you are coding. [DV1] Construct Group 01 Antisocial Behavior 02 Individual Characteristics 03 Family Characteristics 04 Peer, Social Relations 05 Education, School Issues 06 Community, SES, Employment 07 Substance Use/Abuse 08 Knowledge, Attitudes 09 Adjustment to Treatment 10 Adjustment to Institution 99 Other [DV2] Specific Construct. Following is a list of specific constructs for each construct group. Dependent Variables Coding Sheet page 21

23 Antisocial Behavior 10 Aggression, mixed or general: Aggressive and disruptive behaviors; any combination of verbal and physical aggression, or aggression with other behaviors such as talking back, disruptiveness, etc., e.g., CBCL Aggressive Subscale, Eron's Aggressive measure, etc. 9 Aggression, object: Violence against objects: breaks things, throws things 8 Aggression, verbal: Verbal aggression: Yelling, cursing, shouting 34 Bullying 35 Carried Weapon at school: Brought weapon to school, possession of weapon at school or outside of school 3 Conduct disorder: Clinical diagnosis of conduct disorder (CD) or oppositional defiant disorder (ODD) 64 Conflicts: Has conflicts with friends, siblings, or others; no specific information about whether conflicts are physical, verbal, or both. 18 Crime, Index/Serious/Felony: Select this construct if the measure focuses on the FBI Part I Index Crimes: Criminal homicide, Forcible rape, Robbery, Aggravated assault, Burglary--breaking or entering, Motor vehicle theft, and Arson. Generally reported as "index crimes" with a subset of the above. 16 Crimes against Persons: Includes robbery, and crimes of violence 19 Crimes, Mixed: Any measure of general (unspecified) delinquent or criminal activity 39 Expulsions: Number of times expelled from school 5 Externalizing Behavior: Unspecified acting out, disruptive behavior, overly expressive 7 Fights: Initiates or participates in fights 37 Gang activity: Any gang activity that is not contained in any of the other antisocial behavior categories 31 Hitting, beating, punching, etc. 72 Police or law enforcement contact: Police or law enforcement contact 4 Problem Behavior: Age inappropriate behavior and generally inappropriate behaviors and attitudes; Follows Jessor's notion of problem behavior 14 Property Crime: Nonspecific crimes against objects: e.g., theft, vandalism, shoplifting, burglary 20 Severity of Mixed Crimes: Any rank ordering in terms of severity of measures of general (unspecified) delinquent or criminal activity 17 Severity of Person Crime: This requires a measure that ranks the severity of person crimes so that the scale has low severity person crimes at one end of a continuum and higher severity person crimes at the other end. For example "robbery" "assault" "murder" 15 Severity of Property Crime: This requires a measure that ranks the severity of property crimes so that the scale has low severity property crimes at one end of a continuum and higher severity property crimes at the other end. For example "no theft" "theft to $50" "theft over $50" 25 Sexual Offenses: Rape, exposure, attempted rape 220 Sold drugs: Subject sold drugs 13 Status Offenses: Offense by reason of age: e.g., running away, curfew violation 40 Suspensions: Number of times suspended from school 47 Temper tantrums: Primarily a child's measure, excessive display of emotion over not getting one's way 32 Threats (to hit, commit violence, etc.) 36 Used Weapon at school: Used weapon at school or outside of school 22 Violence, Domestic: Spousal / partner violence, Violence in the home 24 Violence, Mixed: The sample is primarily a violent one with a few non-violent folks mixed in: e.g., a measure comprised of murder, assault, rape and robbery 21 Violence, Physical: Violence is that behavior of an individual which is not tolerated by society and which causes another person to experience physical pain, injury, or death as a direct, nonaccidental consequence of that behavior, does not include robbery, threat of violence, or violence to property. Individual Characteristics 68 Ability, aptitude, IQ: Measures of ability, aptitude, performance, intelligence, spatial ability, etc. IQ. 58 Activity level, ADD, ADHD, hyperactivity: Hyperactivity, Hyperactivity with Attention Deficit Disorder, measures of hyperactivity that include restlessness, short attention span, distractibility, etc. 110 Adjustment: General measures of adjustment, including personal and social adjustment, mixed. Social adjustment should go in "peers" section -- use this only when you can't find a better adjustment choice. 78 Age appropriate development: Age appropriate development or lack thereof 80 Anger, angry: Angry, anger ratings, anger expression indices, defiance, etc. 42 Antiestablishment: Anti-establishment attitudes: e.g., doesn't trust police, low social conformity; Rebelliousness, follows/doesn't follow rules 69 Anxious, anxiety: Anxious, fearful, distressed, worried, anxiety, etc. 83 Competitiveness: Competitiveness: measures of how competitive kids are; overly competitive 102 Coping skills: Measures ability or inability to cope with stress in healthy ways 55 Decision making skills: Measures of subjects' abilities to make decisions appropriately 65 Depression: Measures of depression, depressive symptoms, e.g., Child Depression Inventory, Beck's Depression Inventory Dependent Variables Coding Sheet page 22

24 71 Early sexual behavior: Early sexual behavior; sexual activity out of wedlock; adolescent sexual activity. 67 Empathy: Measures of empathy; ability to understand others' feelings. 6 Enuresis, encopresis: Bed wetting and other discharge difficulties. 60 High dependency: Excessive reliance on others for attention, approval or affection 45 Impulsiveness: Impulsive or lacks judgment; lacking stability, low impulse control 44 Internal conflicts: Internal conflicts (e.g., a psychologist's report that the child is conflicted or confused) 81 Internalizing and externalizing problems: Combined internalizing and externalizing measures (use only if you can't split the measure into two dvs). 62 Internalizing behavior: Shy, withdrawn, quiet, etc. 43 Locus of control: Measures of locus of control; an external locus of control is thought to be less healthy than an internal locus of control. 28 Obsessive compulsive disorder: Clinical diagnosis of obsessive compulsive disorder (OCD) or symptoms of the disorder. 63 Odd habitual behavior: Behavior that is unusual, but not antisocial 61 Perspective taking skills: Measures of the subject's ability to view a situation from another person's point of view; being able to understand someone else's viewpoint. Code "hostile attributions" here. 70 Pregnancy out of wedlock: out-of-wedlock pregnancy; adolescent pregnancy 46 Psychopathology: Psychiatric exam, psychiatric hospitalization, neurosis, psychosis; mania, schizophrenia, etc. 59 Risk taking/sensation seeking: Risk taking, reckless, engages in dangerous behaviors 11 Self-control: Measures of self-control, including control over emotions and actions; frustration tolerance; patience 23 Self-efficacy: Self-efficacy is the belief that one can interact successfully in his/her environment; belief in one's abilities. 1 Self-esteem, self-concept: Any measure of selfesteem, self-concept; how the person feels about him/herself 26 Sleep disturbances: Sleep disturbances or problems; problems falling asleep, sleeping through the night, etc. 48 Socialization: Not well socialized, doesn't believe in mainstream values. This is an intrapsychic construct. Does not include antisocial behaviors, and is separate from low social adjustment (which is concerned with how the person interacts with others). 104 Stress level of target child: Exposure of target child to stressful life events (e.g., Life Events Scale), feelings of stress or pressure, etc. 105 Suicide risk: Measures of suicide risk, including suicidal thoughts, threats, past suicide attempts, etc. 66 Task Orientation: Off-task behavior; on-task behavior; attention (this is less serious than ADD or ADHD but is related to these constructs). 84 Temperament: A mixed bag of temperament issues, direction of relationship towards outcomes not necessarily straightforward; be sure to provide adequate description to allow the option of recoding later Family Characteristics 109 Adjustment to divorce: Child or parent measures of adjustment to divorce (e.g., Children's Divorce Adjustment Scale) 113 Attitude toward child: dislikes child; markedly critical of child; rejects child 108 Compliance with parent: Compliance with parental commands, follows parents rules, etc. 29 Discipline, appropriate or consistent: Appropriate or consistent discipline strategies and techniques used by parents. 118 Discipline, Mixed: Little use of rational/social control discipline; Inconsistent discipline; inconsistency between parents use of discipline 119 Discipline, Punitive: Use of corporal (physical) punishment 126 Family functioning: Measures of family functioning 122 Parent aggression: parent antagonistic, argumentative, difficult 112 Parent attitudes, beliefs about child: Measures of parents beliefs about children, child rearing, conflict with children, etc. 123 Parent criminal: Parental record, criminality-- make sure that the age of the index child at time of parental assessment can be determined 116 Parent involvement: Parent not involved with child, includes parental illness, travel: other activities that remove the parent from the child; not the same thing as low supervision or low interest in child 111 Parent knowledge about child rearing: Measures of parental knowledge of parenting skills, child rearing practices, appropriate/inappropriate discipline, etc. 124 Parent psychopathology: Mental disorder; depression, mania, schizophrenia, etc.; Hospitalized for mental / emotional disorder. 56 Parent stress: Measures of parents' stress levels, feelings of support as parents, parent stress about raising/disciplining children, etc. 121 Parent supervision: Caretaker fails to actively monitor the child; doesn't know whereabouts of child; supervises child appropriately 125 Parent violent: Parent physically violent; engages in violent behavior 117 Parent warmth: low caring, withdrawal of love, parent is indifferent to child; OR warm, loving parenting Dependent Variables Coding Sheet page 23

25 115 Parental practices: A general catch-all category for generalized parental activities, primarily used for multi-item factors dealing with parental competence or child rearing skills; be sure to include thick description of the measure used to capture this construct. 114 Parent-child relations: Poor relations with parents; conflict with parents; low attachment or identification between parent and child 120 Severity in training child: Frequency or severity of discipline; Strict parental rules and enforcement Peer/Social Relations 90 Antisocial peers: Peers engage in any variety of deviant activities 95 Attachment to antisocial peers: Antisocial attitudes or opinions of peers; perceived approval or disapproval of antisocial / drug using behavior; be sure to check the direction of coding on the approve/disapprove measures 27 Conflict resolution, interpersonal problem solving skills: Measures of conflict resolution techniques or skills; measures of social or interpersonal problem solving skills 94 Drug attitude, peers: Peers approve of (fail to disprove of) substance use; Friends tolerant of substance use. Do not code peer drug usage here, (code under Exposure). 33 Negotiation skills: Measures of negotiation steps, ability to negotiate solutions to social problems, etc. 91 Peer criminality: Delinquent or criminal behavior by friends and peers 92 Peer normlessness: Peer anomie, lack of direction, lack of social investment 74 Peer pressure resistance: Measures of peer pressure resistance 130 Peer relations: Relations with peers 93 Peer violence: Peers engage in physical violence 88 Popularity (unpopularity; peer rejection): Number of friends, considered popular (or not) among peer group; also not popular, rejected by peers. 106 Prosocial/antisocial influences: Perceived social support from families, teachers, etc., antisocial 'others' (e.g., parents, teachers, friends). NOTE: if antisocial peers ONLY, code under 'antisocial peers'. 89 Social activities: Participation in social activities 87 Social adjustment: Adjustment to social or peer group. Avoids others, not bonded to the dominant / majority / mainstream culture; Alienation 85 Social, interpersonal skills: Subject possesses (or doesn't possess) SKILLS related to social interaction, peer and other relationships. Measures of assertiveness would fit into this category. 96 Victimized: Has been a victim, been bullied, attacked, etc Education, School Issues 75 Attachment to school: Attachment to school, integration at school 142 Classroom behavior: School discipline problems; doesn't follow school rules; code aggressive classroom behavior in the antisocial family of constructs 137 Dropped out (or graduated) from school: Quit school without obtaining High School degree 138 GPA, grades: Grade Point Average (GPA) or grades 140 Interest in education: Low level of commitment to school and education; how important is school?; Little value on academic achievement 49 Other School Disciplinary: School disciplinary actions other than expulsions and suspensions 73 Retention in grade, held back: Held back in grade level, retained, "flunked" (or, number of kids NOT held back) 141 School achievement: Academic Achievement; Success in school; not just grades, but grades with extracurricular 76 School safety, fear at school: Measures of school safety, fear at school, avoidance of places at school because of fear 79 Special education placement: Referral or placement in special education for learning disabilities or emotional disturbances 145 Tardiness, lateness: Being tardy for class or school. 143 Teacher, education personnel attitude toward child: Teacher's or other education personnel's attitude toward child 30 Truancy, attendance: Absent without leave Substance Use 86 Drug, alcohol availability: Perceived or actual availability of controlled substances, including drugs, alcohol, cigarettes, etc. 191 Alcohol Abuse: Abuse of alcohol including alcohol, beer, wine, and hard liquor. The alcohol content of each kind of drink is equivalent. 162 Alcohol Use: Alcohol use includes all types of alcohol, beer, wine, and hard liquor. The alcohol content of each kind of drink is equivalent. 195 Amphetamine Abuse: Abuse. Does not include cocaine or crack cocaine. Includes Preludin, Ritalin. AKA speed, crank, white crosses, methamphetamine, meth, crystal 166 Amphetamine Use: Does not include cocaine or crack cocaine. Includes Preludin, Ritalin. AKA speed, crank, white crosses, methamphetamine, meth, crystal 193 Cocaine Abuse: Cocaine abuse. In the class of stimulants. Code crack use separately. Includes cocaine, coke (AKA flake, snow) 164 Cocaine Use: Use of cocaine. In the class of stimulants. Code crack use separately. Includes cocaine, coke (AKA flake, snow) 199 Crack Abuse: Crack cocaine 170 Crack Use: Crack cocaine Dependent Variables Coding Sheet page 24

26 198 Depressant Abuse: includes barbiturates, (Phenobarbital, Tuinal Amytal, Nembutal, Seconal, Lotusate), Benzodiazepines (ativan, azene, clonopin, dalmane diazepam librium, xanax, serax tranxene, valium, verstran, Jalcion, Paxipam, restroil), Methaqualone (Quaalude), also equanil, Miltown, noludar, placidyl, valmid 169 Depressant Use: includes barbiturates, (Phenobarbital, Tuinal Amytal, Nembutal, Seconal, Lotusate), Benzodiazepines (ativan, azene, clonopin, dalmane diazepam librium, xanax, serax tranxene, valum, verstran, Jalcion, Paxipam, restroil), Methaqualone (Quaalude), also equanil, Miltown, noludar, placidyl, valmid 211 Drinking and Driving: Drunk driving, Driving while intoxicated (DWI), Driving under the influence (DUI). 196 Hallucinogen Abuse: Abuse. Includes LSD (AKA acid microdot), mescaline and peyote (AKA mesc, buttons, cactus), Phencyclidine, Phencyclidine analogs (AKA PCP, Angel dust), and Psilocybin 167 Hallucinogen Use: Includes LSD (AKA acid microdot), mescaline and peyote (AKA mesc, buttons, cactus), Phencyclidine, Phencyclidine analogs (AKA PCP, Angel dust), and Psilocybin 197 Inhalant Abuse: Includes glue, aerosols, sniffing, whiff 168 Inhalant Use: Includes glue, aerosols, sniffing, whiff 192 Marijuana Abuse: Abuse of marijuana includes Cannabis, Hash, Hashish, (AKA pot, grass, reefer, Joints, sinsemilla, Thai, THC). Active ingredient is delta-9-tetrahydrocannabinol. 163 Marijuana Use: Includes Cannabis, Hash, Hashish, (AKA pot, grass, reefer, Joints, sinsemilla, Thai, THC). Active ingredient is delta-9-tetrahydrocannabinol. 181 Minor Drug Use: Mixed drug use through marijuana (i.e., any combination of tobacco, alcohol, and marijuana) 194 Other Narcotics Abuse: Abuse of narcotics. Includes heroin (AKA horse, smack) morphine, (AKA pectoral syrup) hydromorphone (AKA dilaudid) codeine, opium, (AKA Dover's Powder, Paregoric, Parepectolin) methadone, LAAM, percodan, darvon, talwin 165 Other Narcotics Use: Use of narcotics. Includes heroin (AKA horse, smack) morphine, (AKA pectoral syrup) hydromorphone (AKA dilaudid) codeine, opium, (AKA Dover's Powder, Paregoric, Parepectolin) methadone, LAAM, percodan, darvon, talwin 182 Serious Drug Use: Mixed drug use beyond marijuana; combinations of hard drug use 180 Substance Abuse, Mixed: Use of mixed drugs or drug use not broken out by available codes; be sure to include thick description to aid in later construct development. 160 Substance Use, Mixed: Includes "any substance use," or substance use without a description 190 Tobacco Abuse: Abuse of Cigarettes, Pipe, Cigar, Chew, Chaw, Snuff, Chewing tobacco, etc. Be sure to include type of use in the description of the measure. 161 Tobacco Use: Cigarettes, Pipe, Cigar, Chew, Chaw, Snuff, Chewing tobacco, etc. Be sure to include type of use in the description of the measure. Knowledge, Attitudes 54 Attitudes about community: Subject's attitudes about his/her neighborhood or community 51 Attitudes about conflict: Attitudes about interpersonal conflict and conflict management 2 Attitudes towards others, peers: Attitudes towards peers and other people, e.g., how the child feels about people who are different, how the child feels (in general) about his/her peers, etc. 50 Attitudes about violence: Subject's attitudes about violence and violent behavior 205 Belief drugs enhance mental experience: Subject believes that drugs enhance psychological well-being 204 Belief drugs increase social desirability: Subject believes that drugs enhance (doesn t harm) social standing or social well-being; Not concerned about others' reactions to drug use 203 Believe drugs don't harm health: Subject believes that drugs are not harmful to health 202 Drug attitudes: Subject expresses positive or negative attitude towards drugs, doesn't feel they are harmful, see themselves as likely to use, thinks drugs are bad for you, etc. 107 Knowledge about conflict & conflict resolution: Knowledge of conflict resolution steps, how to resolve conflicts peacefully, etc. 82 Knowledge about drugs, alcohol: Measures of subjects' knowledge about drugs, alcohol, drug use/abuse, alcohol use/abuse; NOTE: code knowledge about health effects under construct # Knowledge about sex, sexual behavior, STDs: Knowledge about sexual behavior, sex, pregnancy, etc. 38 Knowledge re: violence, crime, morals: Subject's knowledge of violence, aggression, crime, delinquency, legality of crime, morals Dependent Variables Coding Sheet page 25

27 [DV3] Confidence in identification of construct type 1 Very Low (little basis) 2 Low (best estimate) 3 Moderate (weak inference) 4 High (strong inference) 5 Very High (explicitly stated) [DV4] Fit of construct to measure. This item asks you to rate how well the construct you have selected from the list FITS the measure that the study uses. Does the construct you have chosen exactly capture the meaning of the dependent variable as intended by the authors of the study or is it more approximate? 1 good fit 2 moderate fit 3 poor fit [DV15] Describe any problems you encountered in coding this record. Step 2. Measurement Properties [DV5] Source of information. Who provided the original information for this dependent variable? 01 Subjects 02 Parents 03 Peers 04 Teachers 05 Principal 06 Therapist/Service Provider (treatment agent) 07 School Records 08 Police Records 09 Probation Records 10 Court Records 11 Custodial Institution 12 Crime Statistics 13 Observers. This must involve natural observation in which observers, researchers, teachers, etc. observe the subjects in natural or quasi-natural (e.g., playroom in a laboratory) settings; i.e., the children s behavior is presumably unsolicited. 15 Researcher or interviewer 16 Involved other (not treatment agent, not researcher), e.g., bus driver, school counselor. 14 Multiple sources, cannot tell which is dominant [DV6] Type of Measure. How is this measure operationalized? 01 Survey or Multi-Item Measure (e.g., Child Behavior Checklist, California Achievement Test), other than 05 below 03 Behavioral Observation: any observation-based measure, whether the setting is natural or not 05 Ratings by Treatment Professional (general ratings only, e.g., how well is the subject doing? ) 06 Archival Report 07 Projective Test: measures in which ambiguous stimuli (e.g., ink-blot tests and open-ended scenario-based items) are presented to subjects, and subjects responses are coded or rated for content, quality, etc. 99 Cannot Tell Other: Dependent Variables Coding Sheet page 26

28 [DV7] Origin of Measure. 01 Pre-existing measure or subscale of pre-existing measure (intact), other than 04 below 02 Modification or portion of existing measure 03 Developed by author/researcher for this study 04 Archival measure, from records 09 Cannot tell [DV8] How was measure administered? 1 Paper and Pencil 2 Personal Interview (interviewer reads questions out loud and/or records answers) 3 Telephone Interview 4 Archival Report 5 Observational Data 6 Physical Test (e.g., urinalysis) [DV9] Social desirability bias. Rate the extent to which this measure seems susceptible to a social desirability response bias, that is, the extent to which the respondents are (a) able to recognize what response looks good, (b) may be motivated to look good and (c) are able to exaggerate the response in the direction of looking good. Note that you are not to rate how much social desirability bias you think actually occurred, only how susceptible you think the measure might be. At one end of the continuum would be measures based on objective procedures administered by impartial other, e.g., random surprise urinalysis for drug testing. At the other end of the continuum would be the child s own reports made to someone with authority over him or her (e.g., a teacher) on sensitive issues (e.g., drug use) in open-ended fashion without expectation of verification. This combines format or structure of the measure, demand characteristics of the situation in which the measure is taken, and the ego involvement of the provider of the measure. This is not a measure of the extent to which one s behavior is changeable but the changeability of the report of that behavior. Objective measures should rate in the 1-3 range with arrest records for violent crimes = 1 and those for status offenses = 2. Self-report or a rating by those who are ego involved in some way would be in the 6-7 range. In descending order of ego involvement are: the target juveniles, parents, therapists, teachers, non-blind researchers, criminal justice or school administrative personnel. In descending order of response format sensitivity to bias are: self-report, rating, objective count, and independent cross-checking or review. 1 Very Low Moderate Very High [DV10] Treatment-test overlap. Rate the extent to which the treatment content overlaps or resembles the content of this measure, e.g., as in teaching the test. At one end of the continuum are measures that are virtual duplicates of the treatment, e.g., a behavioral treatment that reinforces a specific list of behaviors and an outcome measure that counts how often those same behaviors are performed. At the other end of the continuum are measures that have virtually no content similarity to the treatment, e.g., a treatment of insight-oriented counseling about family relations and an outcome measure of math grades in school. This is not a question about the extent to which the treatment caused the dependent variable. The question concerns the content of the treatment (and the dependent measure), not the plausibility of the hypothesized causal relationship. The topic area of the treatment in relation to the topic area of the measure determines the general category. Use the 1-3 range for treatments and measures of generally different content and involving different activities; use 3-5 for those situations like general counseling and violence measures where discussion of violent behaviors may well have been part of the treatment content, giving topic overlap, but the activities of treatment (talking about violence) are different from Dependent Variables Coding Sheet page 27

29 those in the measure (committing violence). Use the 5-7 range for fairly clear overlap in both topic area and activity, e.g., substance abuse treatment involving role playing resistance to peer pressure and actual substance abuse incidents as an outcome measure. Within these ranges, adjust for the degree of overlap according to the specifics of the individual case. 1 Very Low Moderate Very High [DV11] Is the data collector blind to the group assignment? This question deals with whether the person collecting the data (the reporter) knows whether the subjects are assigned to treatment or control condition. If this is a one-group study, it is presumed that the condition is known. 1 Data collector knows assignment/condition 2 Data collector blind; does not know assignment/condition 9 Cannot tell [DV12] Reliability Proxy. Use the available information to assess the approximate reliability of the measure. 1 single item measure (or one observer) 2 multiple item measure with 5 or fewer items (or two observers) 3 multiple item measure with more than 5 items (or more than two observers) Other [DV13] Enter Reliability Coefficient (if available). Use two digits and a decimal point, e.g.,.96. You may use any type of reliability coefficient (test-retest, Cronbach s alpha, etc.) and any sample. That is, if the researchers provide a reliability coefficient from another study, you may use it here. [DV14] If you entered a reliability coefficient, indicate the type of coefficient you entered. If the study reports more than one type of coefficient, select only one in order of priority from 1 to 4, according to the list below. 1 internal consistency (e.g., split half, Cronbach s alpha or alpha-reliability, Kuder-Richardson reliability, etc.) 2 test-retest reliability (e.g., test-retest reliability, coefficient of stability) 3 inter-rater reliability (e.g., interrater reliability, percent agreement, Kappa coefficient) 4 alternate form reliability (e.g., coefficient of equivalence) 5 other [DV16] Source of the reliability coefficient. Indicate whether the reliability coefficient you entered above was derived from the current sample or some other group of individuals (e.g., sometimes author(s) will provide reliability coefficients given by the developers of the instrument). 1 all or part of the sample of individuals from the study you are coding 2 the developer of the instrument (e.g., test manual, other studies by the test developer); this implies that the sample of individuals upon which the reliability was determined is NOT the sample of individuals from the study you are coding 3 studies by other researchers (but not the test developer); this implies that the sample of individuals upon which the reliability was determined is NOT the sample of individuals from the study you are coding 9 cannot tell [DV17] Write in any comments, notes, or other information about the reliability of this measure. Dependent Variables Coding Sheet page 28

30 BREAKOUT/SUBGROUP CODING Breakouts are comparisons involving subgroups of an aggregate treatment and/or control group. For example, the males in a treatment group might be compared with the males in a comparison group, or pretest-posttest results might be presented for males and females separately. Each variable (e.g., gender, age) by which the aggregate group(s) are subdivided constitutes one breakout, and each value of that variable defines one subgroup; i.e., a males vs. females stratification is one breakout (gender) with two subgroups, one male and one female. If only the male subgroup is reported, there is still one breakout, but only one subgroup. Note that a simple report of the number of males and females in the treatment and control groups does not constitute a breakout (though it is relevant to group equivalence issues). To be a breakout, outcome data must be reported for the treatment-control or pretest-posttest comparison for at least one subgroup of the breakout variable. Breakouts are usually presented because the authors think that subgroups (e.g., males and females) are sufficiently different to warrant separate presentation of results (because, for example, males may exhibit more aggressive behaviors than females). NOTE: Only certain breakout variables are eligible for coding. These include gender, age, ethnicity, and prior antisocial or aggressive behavior. If you encounter another breakout variable that may be relevant to antisocial or aggressive behavior, please check with Sandra. Create a new record for each subgroup that you will be coding for this study. [StudyID] Study ID for the study you are coding. [Break] Subgroup number. Assign a number to the subgroup such that the first subgroup you code is numbered 1, the second is numbered 2, and so on. These numbers are used within a study, so when you code subgroups from another study, you would start over with 1 again. [Labels:B2] Write in descriptor for the subgroup you are coding, e.g., males, 8 year olds, whites, etc. Breakout/Subgroup Coding Sheet page 29

31 EFFECT SIZE CODING Although this is the final section of coding, it is a good idea to identify at least one codable effect size before you start coding a study, because studies that appear eligible frequently end up presenting data that cannot be coded into an effect size. This portion of coding requires familiarity with some basic statistics, including means, standard deviations, proportions, t-tests, chi-squares, ANOVA (or F-tests), and the like. Step 1. General Information [Study ID] Type in the appropriate Study ID if it does not appear automatically. [reportid] Report ID for this effect size. Indicate the report number (e.g., ) for the report in which you found the information for this effect size. This is important so that we can find the effect sizes later on, if necessary, and is especially important for studies with multiple reports. [ESID] Effect size ID. Use this field to number the effect sizes for THIS study. Thus, a study with 10 effect sizes would have the numbers 1 through 10. Start over with 1 for each new study that you are coding. [pagenum] Page number for this effect size. Indicate the page number of the report identified above on which you found the effect size data. If you used data from two different pages, you can type in both, but use a comma or dash between the page numbers. There are 4 types of effect sizes that can be coded: pretest, posttest, follow-up, and group equivalence (or pretreatment similarity) effect sizes. They are defined as follows: Pretest effect size. This effect size measures the difference between a treatment and comparison group before treatment (or at the beginning of treatment) on the same variable used as an outcome measure, e.g., aggressive behaviors measured before the treatment begins are used as a pretest for aggressive behaviors measured after the treatment ends. Posttest effect size. This effect size measures the difference between a treatment and comparison group after treatment on some outcome variable. A posttest can occur right after treatment ends or after some delay, but it is distinguished from a follow-up (see below) because it is the first measure taken after treatment ends, regardless of the time period between the end of treatment and posttest measurement. Follow-up effect size. Follow-up effect sizes measure the differences between a treatment and comparison group after treatment (as with the posttest effect sizes above), but they involve later measurement waves. That is, some studies may measure the differences between treatment and comparison groups directly after treatment and then 6 months later. The measurement taken at 6 months would be coded as a follow-up effect size. Group equivalence effect size. Group equivalence effect sizes are used to code the equivalence of a treatment and comparison groups prior to treatment delivery on variables that might be related to outcome, such as gender, age, ethnicity, and the like. A pretest that is used later in the study as a posttest would not be coded here you would code it as a pretest effect size. You will ordinarily calculate group equivalence effect sizes as part of the process for the header coding sheet, rather than as part of the process for the effect size coding sheet. This is very important!!!! These four types of effect sizes are different from the multiple breakouts and multiple dependent variables that you might have in a study. For example, you might have a study that measures the treatment and comparison groups at pretest, posttest, and at 6 months after treatment on 3 different dependent variables. The results might be presented for the entire sample and broken down by gender. In this case you would have 9 treatment-control effect sizes for the entire sample three for the Effect Size Coding Sheet page 30

32 pretest, 3 for the posttest, and 3 for the follow-up. In addition to these 9 aggregate effect sizes, you will have 9 more for the girls (the same as for the aggregate groups but just for the subgroup of girls) and 9 for the boys (also the same as for the aggregate groups but just for the subgroup of boys). [ES24] Type of effect size: 1 Pretest (for treatment-control comparison on a dependent variable) 2 Posttest (for treatment-control comparison on a dependent variable) 3 Follow-up (for treatment-control comparison on a dependent variable) 5 Group Equivalence (for pretest treatment-control comparisons on variables other than the dependent variables) [ES19] Follow-up number. This item applies only to follow-up effect sizes and pretest-to-follow-up effect sizes. If your effect size does not involve a follow-up measurement, leave this field blank. Some studies involve more than one follow-up measurement, and we need to be able to distinguish one from another. Give the first follow-up after the posttest a 1, the second a 2, and so on. [ES47] Timing of measurement. Approximate (or exact) number of weeks after the END of treatment when measure was taken. Divide days by 7; multiply months by 4.3. Code 999 if cannot tell, but try to make an estimate if possible. Code 0 if pretest. Code 1 if posttest is taken right after treatment. NOTE: sometimes information is reported on the time from the beginning of the intervention. In this case, subtract the estimated duration of treatment when calculating the number of weeks after treatment when the measure was taken. [ES43] Estimate of timing of measurement based on: 1 Given 2 Estimated 3 Best Guess [ES52] Interval from pretest measurement. Approximate (or exact) number of weeks since the pretest when this measure was taken. Divide days by 7; multiply months by 4.3. Code 999 if cannot tell, but try to make an estimate if possible. Code 0 if pretest. [ES53] Estimate of interval from pretest based on what information? 1 Given 2 Estimated 3 Best Guess Step 2. Dependent Variable Selection [VarNo] Select the dependent variable for this effect size. A list of the dependent variables you identified using the Dependent Variable Coding Sheet will appear; type in the number that corresponds to the DV for this effect size. Step 3. Breakouts [Break] Subgroup number (if applicable). A list of the subgroups you identified using the Breakout Coding Sheet will appear. If the effect size you are coding involves a subgroup of the groups you selected in Step 2 above, type in the number of the appropriate subgroup here. Effect Size Coding Sheet page 31

33 Step 5. Effect Size Calculation and Data Entry It is now time to identify the data you will use to calculate the effect size, and to calculate the effect size yourself if necessary (see below). Effect sizes can be calculated ONLY from data based on the number of subjects, e.g., mean number of aggressive acts per subject (and the corresponding standard deviation) or proportion of subjects who acted aggressively during a given time period. Effect sizes can NOT be calculated from data based solely on the incidence of events, e.g., total number of aggressive acts per group. Effect sizes can be calculated from subject-based data in a variety of forms; to determine which data you should use for effect size calculation, please refer to the following guidelines: 1. Compute ES from descriptive statistics if possible (means, sds, frequencies, proportions). 2. If adequate descriptive statistics are unavailable, compute ES from significant test statistics if possible (t, F, Chi square, etc.). 3. If significance tests statistics are unavailable or unusable but p value and degrees of freedom (df) are available, determine corresponding t value and compute ES as if t-test had been used. 4. If only complex statistics are available (e.g., factorial ANOVA, multiple regressions, etc.) refer the study to Sandra. Note that if the authors present both adjusted and unadjusted data, you should use the adjusted data in your calculations. See [ES50] below for more information. NOTE: If your study reports results in the form of average percents, remember that you need to treat the data as means, not as simple percents. [ES17] Which group is favored? For treatment-control comparisons, the treatment group is favored when it does better than the control group. The control group is favored when it does better than the treatment group. Remember that you cannot rely on simple numerical values to determine which group is better off. For example, a researcher might assess the amount of violent behavior, and report this violent behavior in terms of the number of violent acts per subject per day. Less violent behavior is better than more, so in this case a lower number, rather than a higher one, indicates a more favorable outcome. Sometimes it may be difficult to tell which group is better off, because some studies use surveys or paperand-pencil measures in which it is unclear whether a high score or a low score is more favorable. In these situations, a thorough reading of the text from the results and discussions sections usually can bring to light the direction of effect e.g., the authors will often state verbally which group did better on the measure you are coding, even when its not clear in the data table. In addition, the Measures database in FileMaker may provide information about the measure in question. Note that if you cannot determine which group has done better, you will not be able to calculate a numeric effect size. (You will still be able to create an effect size record just not a numeric effect size.) Remember that every study must produce at least one numeric effect size to be eligible for coding; if you find that you cannot determine which group has done better for any of the potential effect sizes in a study, the study is not eligible. Select the group that has done better : 1 Treatment 2 Control 3 Neither, Exactly Equal 4 Cannot tell Effect Size Coding Sheet page 32

34 [ES23] Effect size derived from what type of statistics? 1 N successful (frequencies) 2 Proportion successful (percentage successful or not successful) 3 Multi-category (polychotomous) frequency or % 4 Means and SDs, means and variances, means and standard errors 5 Independent T-test 6 Dependent T-test 7 Probability With N/degrees of freedom 8 One-way ANOVA (2 groups, 1 degree of freedom) 9 One-way ANOVA (>2 groups, >1 degree of freedom) 10 Factorial Design (Repeated measures ANOVA, 2x2 ANOVA, MANOVA, etc.) 11 Covariance Adjusted (ANCOVA) 12 Chi-square statistic (1 degree of freedom) 13 Chi-square (> 2x2 table) 14 Nonparametric statistics (Mann Whitney, etc.) 15 Correlation coefficient (zero-order) 16 Multiple regression 17 Effect sizes [ES50] For this effect size, did you use adjusted data (e.g., covariate adjusted means) or unadjusted data? If both unadjusted and adjusted data are presented, you should use the adjusted data. Adjusted data are most frequently presented as part of an analysis of covariance (ANCOVA). The covariate is often either the pretest or some personal characteristic such as socioeconomic status. If you encounter data that is adjusted using something other than a covariate, please see Sandra. 1 Unadjusted data 2 Pretest adjusted data 3 Data adjusted on some variable other than the pretest (e.g., socioeconomic status, IQ) 4 Data adjusted on pretest and other variables [ES22] Confidence in effect size calculation 1 High Estimate (e.g., have N and crude p values only, e.g., p<.10, and must reconstruct via rough t- test equivalence) 2 Moderately Estimated (e.g., have complex but relatively complete statistics, e.g., multiple regression, LISREL, multifactor ANOVA, etc. as basis for estimation) 3 Some Estimation (e.g., have unconventional statistics and must convert to equivalent t-values or have conventional statistics but incomplete, such as p values only) 4 Slight Estimation (e.g., must use significance testing statistics rather than descriptive statistics, but have complete statistics of the conventional sort, such as a t-value or F-value) 5 No Estimate (e.g., have descriptive data: means, sds, frequencies, proportions, etc.; can calculate an ES directly.) [ES44] Significance information for this comparison. For treatment-control comparisons: Did the authors make any comment about the statistical significance of the difference between the values (e.g., mean test scores) for the two groups you selected, with regard to the dependent variable you have selected, at the time point you have selected for this comparison? Sometimes authors will state that a particular comparison was not significant, but not provide any calculable effect size data. In these cases, you should select 5 for this item. The effect size field should remain blank. In other cases, authors will state that a particular comparison was significant, but not provide any calculable effect size data. In these cases, you should select 4 for this item. Again, the effect size field should remain blank. NOTE: the last three options (4, 5, and 6) are for cases for which you have direction (i.e., you know which group is favored) for no effect size information. Effect Size Coding Sheet page 33

35 1 Significant result, ES data below 2 Non-significant result, ES data below 3 Significance not reported, ES data below 4 Significant result, no ES data 5 Non-significant result, no ES data 6 Significance not reported, no ES data [ES18] Variance control techniques. Type of statistical test done for this comparison, if any. The issue here is whether the author(s) used a variance-control technique when analyzing the comparison for which you are calculating an effect size. 1 No Test 2 No Report. Use this option when you have significance info, but don t know the kind of test used. 3 No variance control techniques (e.g., t-test, oneway ANOVA, z-test, Π 2, non-parametric, raw means, etc.) 4 Variance control techniques used (e.g., ANCOVA, multiple regression, repeated measures ANOVA, adjusted means, etc.) DATA ENTRY FIELDS FOR EFFECT SIZE CALCULATION You should not fill in all the data entry fields; enter only the data for Assigned and Observed N, the remaining data used to calculate the effect size, and the effect size itself if necessary. Assigned and Observed N [ES36], [ES37], [ES1], [ES2] Assigned N, Observed N. These fields refer to the number of subjects who were originally assigned to the two groups (Assigned N) and to the number of subjects who were actually observed or measured (Observed N). If you cannot tell how many subjects were originally assigned to a group, take a look at the number of subjects (Observed N) at pretest; you can frequently use pretest sample sizes for assigned N. However, in cases where the authors have removed the subjects who do not have both pretest and posttest measures (such that the pretest N and the posttest N are the same), do not assume that the number of subjects at pretest is the correct number for Assigned N, and leave this field blank. In cases where there is no attrition, the Assigned N is the same as the Observed N. Only use the same numbers for Assigned N and Observed N when you are SURE that there is no attrition. [ES36] Assigned N for the treatment group (or pretest, if this is a pretest-posttest effect size). [ES37] Assigned N for the comparison group (or posttest, if this is a pretest-posttest effect size; if this is a pretest-posttest effect size, this value should be the same as the assigned N for the pretest). [ES1] Observed N for the treatment group (or pretest, if this is a pretest-posttest effect size). [ES2] Observed N for the comparison group (or posttest, if this is a pretest-posttest effect size). Other Effect Size Data Fields [ES9] Mean for treatment group [ES10] Mean for comparison group [ES12] Variance for treatment group [ES13] Variance for comparison group [ES25] Standard deviation for treatment group [ES26] Standard deviation for comparison group [ES27] Standard error for treatment group [ES28] Standard error for comparison group [ES29] Proportion successful for treatment group [ES30] Proportion successful for comparison group [ES31] N successful for treatment group Effect Size Coding Sheet page 34

36 [ES32] N successful for comparison group [ES33] Independent t-value [ES46] Dependent t-value [ES34] F-value (df=1) [ES35] χ 2 (df=1) Final Effect Size Determination If you have entered means and standard deviations above, FileMaker will calculate the effect size value for you in a separate field. Otherwise, you will need to enter the effect size you calculated into this field. [ES21] Effect size value FileMaker will take either the effect size you have entered or the effect size as calculated by FileMaker, and combine it with the information you provided in [ES17] ( Which group is favored? ) in order to produce a final effect size with the correct sign (+ or -). You will need to confirm that FileMaker has in fact given the correct sign to your effect size. This can get complicated, so please think about it carefully when you are coding. Remember, once again, that you can not rely on simple numerical values to determine which group has done better. For treatment-control comparisons, a positive effect size indicates that the treatment group did better on some measure than the comparison group, while a negative effect size indicates that the comparison group did better than the treatment group, and a zero effect size means that the two groups are exactly equal on the measure. You must make sure that the sign of the effect size in [ES21] matches the way we think about direction, such that the effect size is positive when the treatment group is better off and negative when the comparison group is better. Effect sizes can range anywhere from around 3 to +3. However, you will most commonly see effect sizes in the 1 to +1 range. [ES39] Any problems coding this effect size? Effect Size Coding Sheet page 35

37 Appendix B Sample Coding Screens I. Sample Header Coding Screen page 36

38 II. Sample Groups Coding Screen page 37

39 III. Sample DV Coding Screen page 38

40 IV. Sample ES Coding Screen page 39