A SUMMARY ASSESSMENT OF USES OF META-ANALYSIS IN CRIMINAL JUSTICE RESEARCH

Transcription

1 A SUMMARY ASSESSMENT OF USES OF META-ANALYSIS IN CRIMINAL JUSTICE RESEARCH L. Edward Wells Dept. of Criminal Justice Science Illinois State University This paper was presented at the 2004 annual meetings of the American Society of Criminology. November Nashville, Tennessee.

2 A Summary Assessment of Uses of Meta-Analysis in Criminal Justice Research The Issue: The methodology of meta-analysis seems ideally suited for criminal justice-related research, because it directly addresses several of the most pervasive and persistent problems afflicting research in this field. (Admittedly, these are fundamental problems in the social and applied sciences more generally, so the criminal justice sciences are not unique in this respect.) These difficulties include: (a) the problem of proliferation without accumulation, (b) the problem of weak and inconsistent findings (from which no clear conclusions seem possible), and (c) the inappropriate reliance on statistical significance tests to determine substantive findings in research. The first of these problems, well-known and shared with most social and applied science fields, occurs because research studies on a topic tend to multiply into a hodge-podge of redundant but incommensurable findings. Each study is unique, similar to but not fully comparable to the others. Inevitably they yield a variety of different outcomes with different interpretations--i.e., a clutter of findings that yields no clear conclusion. The conventional response to such embarrassments of riches is a narrative review in which the scholar surveys the available studies (or at least the important ones), critiques and summarizes each one, and interpretively synthesizes their findings. Such reviews are periodically done in important research but without much success or real resolution. The weaknesses of narrative reviews have been detailed by. Meta-analysis allows us to move beyond the limitations of traditional research reviews by providing a methodical, objective, quantitative set of analytical procedures for rigorously combining and contrasting the results of multiple studies (with no limit on the number of studies that can be included in the synthesis--indeed, the more the better). It does so by explicitly coding and quantitatively analyzing the differences among the studies, as well as their similarities, so that we may effectively triangulate to a rigorous estimate of their common patterns, along with their divergences. The second difficulty is a ubiquitous problem of weak effects, where research findings are suggestive but not strong enough, given limited samples and measurements, to draw strong conclusions. This is reflected in modest values of correlation and regression coefficients, low percentages of variance explained, and a persisting reliance on confidence levels computed with very large samples. The studies inevitably conclude with a call for more research to validate the particular pattern of effects found; additional research invariably produces different patterns which neither confirm or disconfirm the original pattern (see difficulty #1 above). Periodic introductions of more rigorous statistical procedures produce more elegant estimates of effects, but not more robust findings due to limitations of the samples, measurements, and other peculiarities of real-world research. The lack of strong findings also may be reflective of the complexity and subtlety of the phenomena being studied--a complexity that simply cannot be neatly and simply indexed with the simple, static, linear models we use to analyze them. The third stumbling block to accumulating a body of robust findings is an unfortunate reliance on statistical significance tests by which to decide if an observed pattern of results is meaningful or substantial. One weakness is that the logic of conventional significance tests almost never matches the kinds analytical decisions for which we use them, noting that null-

3 hypothesis-rejection model was developed in the early 20th century for agricultural experiments and reflected the prevailing positivist philosophy of that time (which has long since been abandoned as unworkable). It survives as statistical ritual endowed with quasi-religious sacredness but without meaningful validity. Apart from the lack of a clear logical rationale, an additional difficulty is that our data never meet, or even approximate, the assumptions upon which the probability theory of statistical tests are based. Real research almost never (if ever) employs simple random sampling (or comes anywhere close to it with the vast bulk of research using non-probability convenience samples); measurements of social phenomena do not really yield interval-level variables; and the distributions of many important social outcomes do not approximate normality. While many testing procedures are relatively robust against such deviations, they still mean that the null models to which the tests apply are implausible fictions of unknown utility. While statistical testing procedures compute error probabilities as if each test were a single independent trial, each real-world study computes dozens of correlated statistical tests (e.g., a t-test for each coefficient, an F-test for each model or contrast, a Chi-Square test for each cross-tab). This results in overall study-wide statistical error rates of essentially 1.0, and guarantees that every study will contain significant findings that are really Type-I errors; we just don t know how many. And lastly, conventional significance tests are used without any consideration of their statistical power (i.e., ability to detect effects that are trivially small), which is especially problematic when very large samples are used in many criminal justice studies. Meta-analysis provides us with a way to get beyond each of these nagging shortcomings. By combining many different studies using different samples and different procedures we may systematically triangulate to a more confident estimate of the common patterns, as well as the differences, contained among the studies. By analyzing multiple studies as conceptual replications we can directly estimate the sampling variation of our results, rather than relying on hypothetical, artificial, and untenable assumptions about the likely sampling errors in our data. I will leave the specific details of exactly how this occurs to the other papers on this panel, with introductory referrals to Lipsey and Wilson (2001) and Wilson (2001). This allows us to avoid the inherent limitations of statistical testing by empirically measuring, rather than assuming or hypothetically predicting, the repeatability of our findings. In sum, meta-analysis offers a very promising correction for the stumbling blocks noted above and seems a natural choice for advancing the output of criminal justice research. Metaanalysis was officially invented in 1976 by Glass who coined the term meta-analysis and laid out the basic logic of the strategy, although it was effectively used informally before that. It has been available for use in criminal justice research for almost 30 years, but there does not seem to have been a stampede of adoptions or a methodological revolution. What exactly has happened? How much is meta-analysis used in criminological or criminal justice? The Research Questions: The last two sentences in the previous paragraph indicate the motivating interest for this paper. Its aim is description, rather than explanation or evaluation--merely to have an empirical picture of how much and where meta-analysis is being used in criminology and criminal justice research. In this, no explicit theory of methodological diffusion or revolution for research in criminal justice is being assumed. It is directed merely by a lot of questions about what actually is being done in research loosely identified as criminal justice research (that is, having something

4 to do with crime, delinquency, anti-social behaviors, correction of criminal offenders, policing and law enforcement, justice and legal processes). The basic orienting question for this analysis is simply: How much and in what ways is meta-analysis used in criminological and criminal justice-related research? This rather broad question may be more usefully parsed into three more specifically focused sub-issues: First, what has been the trend or temporal pattern for use of meta-analysis in criminal justice research? Was the adoption of this strategy gradual (and linear) or abrupt (and discontinuous)? Has its use been increasing (suggesting the diffusion of an enduring methodological innovation)? Or did use of meta-analysis peak at some point (suggesting that its use may be a fad)? Second, what different kinds of meta-analysis research are being done on criminal justice topics? Is this largely pragmatic applied research involving evaluations of particular programs and policies? Or is it substantially basic theoretical research aimed in the development and refinement of substantive knowledge about crime, delinquency, and other anti-social behaviors? Is the use of meta-analysis in criminal justice mostly synthesizing research aimed at combining together the results from many disparate and limited studies into one grand, cumulative, comprehensive finding? Is it mainly moderating research aimed at identifying the important contingencies within the overall effect--i.e., particular factors or conditions that strongly modify when or how the effect may occur? Third, how does the adoption and use of meta-analysis in criminal justice compare with its use in other social or applied sciences? Does criminal justice research seem out of step with the development of other behavioral and social sciences at this point in the 21st century? Are we behind the growth curve for adopting new and better methods for knowledge production (as often seems to be the case)? The Research Study: This study is an extension of update of an earlier review, presented in a paper for the 1991 ACJS meetings. At that point, meta-analysis had been used only a little, but very suggestively. Mark Lipsey s comprehensive review of juvenile correctional programs had been presented but not yet published (Lipsey, 1992). The lively exchange of meta-analyses between Whitehead and Lab (1989) questioning the effectiveness of juvenile correctional treatment and Gendreau and Andrews (1990) affirming the conditional effectiveness of correctional treatment had just begun. At that point I found only 13 published meta-analysis on criminal justice-related topics, plus several other quantitative reviews that were near-meta-analyses (i.e., what might be called box score reviews in which quantitative findings are summarized in cross-tabular form but not standardized or computationally analyzed and combined). At that time, the majority of the meta-analysis were pragmatic evaluations of the effectiveness of correctional treatment programs; only a few meta-analyses had addressed theoretical issues about crime, delinquency, justice, or legal processes. That parallels the early development of meta-analysis in psychology (the discipline from which criminal justice received this methodology), in that the first significant attempts at systematic quantitative syntheses there (beginning with Glass in 1976) all dealt with questions about the effectiveness of psychotherapy. Since that 1991 review, the statistical technology for doing meta-analysis has been greatly elaborated and made available to a wider audience of researchers, including criminal justice scholars/practitioners. The initial wave of methodological exegeses instructing how and when to

5 do meta-analyses appeared mostly in the psychological literature in the 1980s (e.g., Glass, McCaw, and Smith, 1981; Hedges and Olkin, 1985; Hunter, Schmidt, and Jackson, 1982; Mullen, 1989; Mullen and Rosenthal, 1985; Rosenthal, 1984; Wolf, 1985), followed by a wave of practical and substantive applications to a variety of psychological fields, with gradual diffusion from psychology to other social science disciplines, including education, sociology, economics, and criminology. It was also adopted by the biological and health-related sciences, who have utilized the methodology with an enthusiasm that dwarfs that of the behavioral and social sciences. For example, while only 50 articles referred to meta-analysis between 1980 and 1984 in the Medline bibliographic database, that category had grown to almost 7000 references in the period. The task is the current review is to attempt a meta-analysis of the use of meta-analysis in criminal justice research. To do that, I searched a number of online databases in which criminal justice-related might plausibly appear for any articles that appeared between 1980 and 2004 (the most recent search being done in early October of 2004). The search looked for any mention in the title, the abstract, or the keyword list of the word meta-analysis plus any of the following terms: crime, delinquency, corrections. law, justice, legal decisions, police, law enforcement, anti-social, drug abuse, violence. The specific databases used were: Criminal Justice Abstracts, Sociological Abstracts, PsychINFO, Social Science Abstracts, and Current Contents. The specific lists of references retrieved from each of the databases were scanned to identify that those that might have used meta-analysis procedures (versus those that merely mentioned the term in their abstract or those that were methodological or pedagogical discussions of the metaanalysis). The abstracts of likely references were reviewed to confirm that they had used metaanalysis, and an attempt was made to obtain (at least temporarily) copies of each of the likely candidates. Fortunately, most of these were located in available library sources and many others were located in online-accessible texts. In all 133 references to studies published between 1978 and 2004 using meta-analysis on criminal justice topics were identified. And of these 130 references, copies (or detailed descriptions) of 123 were obtained that allowed at least general coding of the studies principal characteristics. Each of the studies obtained was reviewed and coded into a Filemaker database that recorded: Names of the first three authors, the year of publication, the topic area of the analysis (corrections, courts, policing, crime/delinquency, drugs/alcohol abuse, CJ administration, education, clinical issues), the type of study (quantitative meta-analysis, qualitative review, box score review, methodological discussion), analytical focus (pragmatic, theoretical, methodological, pedagogical), years covered in the review (earliest study included; most recent study included), effect measure used (standardized difference score, correlation coefficient, oddsratio, other), type of analysis (summative, moderator/comparative, prediction/regression/causal modeling, dimensional), disciplinary identification (psychology, sociology, criminal justice, medicine, treatment/clinical), number of authors, number of studies included, number of effect sizes used. When all studies had been entered in the database, the information was exported (as a.dbf-file) to SPSS for basic descriptive analyses. A listing of the variables in the SPSS file is provided in Appendix A. As a supplement to the main quantitative review, additional word searches were done of the Medline database and Dissertation Abstracts Online database. The Medline search--using only the single term meta-analysis as the search criterion-- was done only to get a count of the number of articles in the health-related sciences that dealt with meta-analysis during specific time

6 periods. The Dissertation Abstracts Online was used to identify a list of unpublished Master s and Doctoral theses using meta-analyses of criminal-related topics that were carried out during the last 25 years. All dissertations for which the term meta-analysis appeared in the abstract were selected and these were reviewed manually to identify those on criminal justice-related topics (applying the same criteria as the published literature review). The abstracts for each of the CJ-identified theses were read to confirm that they were applications of meta-analysis, but no attempt was made to obtain copies of them. These extra searches were intended to provide merely some additional descriptive materials for trend comparisons. The Results: The first research question concerns the temporal pattern or use trend for meta-analysis in criminal justice research over the last two-and-a-half decades. How much growth in usage has occurred and when? The yearly distribution of the 123 meta-analysis studies are displayed graphically in Figure 1 (with data for 2004 omitted due to its partial-year coverage). As the line chart in Figure 1 indicates, the growth in meta-analyses in criminal justice research was rather modest and gradual prior to 1995, but usage increased dramatically after that point. In fact, for the 123 studies uncovered in this review slightly over half of them (63) were published after 1999 ( ). Because the results shown in the chart reflect published studies (which may involve a delay due to publication lag, Figure presents a bar chart for the unpublished CJ-related dissertations (and Master s theses) done during this same period. Because the total numbers are much smaller, years are grouped into 5-year (half-decade) intervals. The pattern in Figure 2 shows a somewhat more linear increase in meta-analytic dissertations over the period, with more work in the mid-1990s and a uniform growth trend. Whether the difference between the patterns in Figures 1 and 2 represent any meaningful changes in the field is merely speculation at this point. To address the second research question about forms and varieties of meta-analyses being doing on criminal justice topics, basic frequency and cross-tabulation analyses of the coded study characteristics were computed. The frequency results for all relevant study characteristics are presented in Tables 1 through 7; with additional cross-tabs inserted in Tables 1 and 2. Table 1 shows the distribution of meta-analytic studies across major subject areas in criminal justice and provides the breakdown by Analytical Focus. The largest percentage, although not a majority, are meta-analyses on the correlates and causes of criminal or delinquent behavior, with almost as large a number of corrections/treatment studies. Most of the crime/delinquency meta-analyses involve basic theoretical research while all of the correctional meta-analyses are pragmatic evaluation studies of particular types of interventions. Smaller but still noticeable numbers of meta-analyses have dealt with legal decisions and process, with substance abuse issues, with policing/law enforcement topics, and with clinical diagnostic/prediction issues. The legal studies are more likely to be theoretical studies of basic judgmental and perceptual processes involved in legal decision-making, while the other three (substance abuse, policing, and clinical diagnosis) are much more heavily focused on pragmatic evaluations concerned with identifying effective programs or practices. Table 2 shows the breakdown of meta-analyses by the disciplinary context or setting where they appear. The majority (61%) are readily identified as criminal justice studies, either in the journals where they appear or in the academic departments involved in the research. However, a sizeable number of the meta-analyses appear in psychological journals carried out by

7 persons in psychology departments, in effect reflecting the parent or source discipline of metaanalysis. Much smaller number of CJ-related meta-analyses appear in sociology journals or health-related settings. The right two columns of the table show the cross-tabulation of disciplinary by analytical focus. It shows that meta-analyses in criminal justice setting tend to be mostly applied/pragmatic studies, while the analyses associated with traditional disciplines are predominantly theoretical in focus rather than pragmatic. Not surprisingly, all of the healthsciences meta-analyses are pragmatic studies. Table 3 provides a simple frequency distribution of how many of the studies selfidentified as meta-analyses actually involved meta-analytic procedures. It reveals that most of the analyses uncovered in this review are actually quantitative meta-analyses, while eleven of the studies self-described as meta-analyses turned out to be slightly less rigorous box score reviews. These are quantitative reviews that merely report quantitative results from multiple studies in a summary table, but do not convert all the findings to a standardized metric or attempt a quantitative summary/synthesis of them. Two of the meta-analysis studies located turned out to be criticisms or demonstrations of particular meta-analytic procedures, rather than substantive questions about problems of criminological theory or criminal justice practice. This review turned up no purely pedagogical presentations, aimed simply at instructing how to do a metaanalysis. Such presentations seem to be embedded in substantially meaningful studies rather than as stand alone publications. Tables 4 and 5 report on the computational strategies used in the CJ-related metaanalyses, with Table 4 reporting the type of standard effect size measures used in the analysis and Table 5 reporting on the overall analytical strategies used. Table 4 shows a fairly even split between d-based effect indexes (which measure mean group differences) and r-based effect indexes (which measure correlation between two observed variables). Another sizeable portion of meta-analytic studies used alternative measures, which are largely estimates of prevalence rates measured by percentages. These involve studies using meta-analysis to attempt to derive more reliable pooled population estimates of rare behaviors or unusual outcomes, for which only limited studies with non-representative samples are available. When the outcomes and the independent variables are dichotomous, then odds-ratios recently have been advocated to avoid the attenuating effects on correlation coefficients of uneven marginal splits. Of the studies coded in this review, only 8 used odds-ratios instead of Phi-coefficients, but these do represent a recent trend; all but one were used in studies published in 2000 or later. The results in Table 5 show the distribution of meta-analyses classified by the analytic strategy used. The categories in the table involve the following distinction. Summative studies are those using meta-analysis to combine the results of separate studies into a more reliable overall estimate of effect, either by simple pooling into a global estimate or by using the information about distinctive differences between studies to compute an adjusted estimate of effect. Moderator studies are those using meta-analysis to contrast between studies to identify contingencies or subgroups in which the effects will be very different (and thus cannot meaningfully be combined or pooled). The prediction/regression studies use meta-analysis as a way to combine the bivariate correlations from a variety of separate and different studies (not using all the same variables) to produce a single overall multivariate result. Dimensional studies are those using bivariate correlations from a variety of different studies to produce a single multidimensional or factorial model. As the table shows, the majority of uses of meta-analysis in criminal justice are summative, followed by a slight smaller but substantial number of moderator

8 analyses, and an even smaller percentage (10%) of prediction/regression studies. These are distributed fairly evenly between pragmatic and theoretical applications of meta-analysis (result not shown in the table). However, there has been a slight temporal shift over the past 20+ years toward more summative studies and fewer moderator studies, a trend that is opposite my expectations (where I expected that increasing sophistication of meta-analytic methods would lead to more moderator analyses). Table 6 and 7 report on size dimensions of the meta-analysis reviews: Table 6 reporting on the number of authors involved in each project and Table 7 reporting on the number of studies included in the analysis. These show that multiple author-efforts are the rule, with more than 3 authors are still relatively uncommon (in contrast to the most lab-based health sciences which often feature long lists of authors). There is, however, notable temporal shift in this pattern toward increasing numbers of multi-author efforts (and fewer single-authored studies). Table 7 shows the distribution of meta-analyses by size categories, where size is defined by the number of studies covered by the meta-analysis. The high-water mark remain Lipsey s (1992) metaanalysis of 437 evaluations of juvenile treatment programs, and only 16 meta-analyses have included 100 or more studies. About two-thirds of the criminal justice-related meta-analyses analyzed 50 or fewer studies. There does not appear to be any marked pattern change over time in the sizes of meta-analyses being done. Also, no data were available to gauge how that might compare with meta-analytic studies carried out in other disciplines or research fields. How does criminal justice research compare to other fields in its us of meta-analytic methods? To provide a comparative context for interpreting the trend in meta-analysis in criminal justice relative to other fields of study, Figure 3 presents four bar graphs for the trends (by 5-year intervals) in references to meta-analysis in health-related fields (using the Medline online bibliographic database), in psychology (using the PsychINFO online bibliographic database), in sociology (using the Sociological Abstracts online database), and in the criminal justice database used in this review. As the patterns in these charts suggest, criminal justice seems to have warmed up more slowly to meta-analysis than some other related fields, including psychology and sociology, both of which show more meta-analytic publication in the early 1990s. Not surprisingly, the overall levels for use of meta-analysis in psychology are dramatically higher that in criminal justice and sociology (by a factor of 20 to 30 times high), but the drop in meta-analysis references in psychology after 1999 is quite surprising. It is a reliable pattern in the PsychINFO online database used for this analysis (and confirmed by repeating the search using slightly different query procedures). It would be useful to see if the pattern also occurs with an alternative bibliographic database for psychology. The growth pattern for criminal justice is fairly similar in overall shape to the pattern for medical and health-related sciences--both fields increasing by a factor of 2X over each of the previous 5-year periods, while psychology (surprisingly) shows a decline during those periods. However, the absolute levels (in numbers of studies) are obviously quite different, with 86 meta-analysis references in Criminal Justice Abstract during the past ten years and over 10,000 in the Medline database in the same interval. The temporal pattern for sociology is somewhat similar to criminal justice and divergent from psychology; this seems reasonable given the considerable redundancy between the two databases. Conclusion/Discussion: My analytical aims in this project were rather pedestrian: simply to do an inventory of

9 uses of meta-analysis procedures in criminal justice research and to see how much progress we have made in this direction during the last decade or so. My benchmark was a similar review done in 1991, when slightly over a dozen meta-analyses were published in criminal justice. It is encouraging to notice the great expansion of meta-analytic procedures in other scientific fields in the 1990s along with the general acceptance of meta-analysis as a valid and valuable way to move beyond the atomism and entrepreneurism that hinder the growth of knowledge in most areas of research. As I suggested at the outset, meta-analysis seems a natural fit for many topics and questions in criminal justice; and as Mark Lipsey s landmark review of juvenile intervention programs shows, it can enable a level of clarity and confidence in the yield of our research that otherwise is not available. My review suggests that criminal justice researchers have come slowly to the adoption of meta-analysis, which is not entirely surprising given the pragmatic character of criminal justicerelated research. Meta-analysis does not offer a quick or an easy solution to the problem of fallible and inconclusive research; otherwise, it would have be much more widely used and would have been adopted more quickly. As the best references note, a good meta-analysis entails a time-consuming, labor-intensive, and pains-taking project that seemingly is never quite finished and has a tricky diminishing-returns cost-benefit curve. That was brought home to me again (although I already knew it) as I worked on this modest project. That it requires a substantial commitment to be good means it is certainly possible (and easy) to do a bad meta-analysis, which adds little to the research literature except more clutter. My hope, which is not addressed in the survey I did in this study, is that meta-analytic methods are being made a central component of the graduate research methods training in our Ph.D. programs, so that the growth curve for criminal justice research will not show the peak and decline pattern noted earlier--at least any time soon. REFERENCES Andrews, D.A., Ivan Zinger, Robert D. Hoge, James Bonta, Paul Gendreau, and Francis T. Cullen "Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis." Criminology 28: Cook, Thomas D. and Laura C. Leviton Reviewing the Literature: A Comparison of Traditional Methods with Meta-Analysis. Journal of Personality 48: Cooper, Harris and Larry V. Hedges (eds.) The Handbook of Research Synthesis. New York: Russell Sage Foundation. Gendreau, Paul and D.A. Andrews Tertiary Prevention: What the Meta-Analyses of Offender Treatment Literature Tell Us about What Works. Canadian Journal of Criminology 32(Jan.): Glass, Gene V Primary, Secondary, and Meta-analysis of Research. Educational Researcher 5: 3-8.

10 Glass, Gene V., M. McCaw & M.L. Smith Meta-Analysis in Social Research. Beverly Hills, Ca: Sage. Hedges, Larry V. and I. Olkin Statistical Methods for Meta-Analysis. Orlando: Academic Press. Hunter, John E. and Frank L. Schmidt Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Newbury Park, CA: Sage. Hunter, John E., Frank L. Schmidt, & Gregg B. Jackson Meta-Analysis: Cumulating Research Findings Across Studies. Beverly Hills, Ca: Sage. Lipsey, Mark W Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Juvenile Delinquency Treatment: A Meta-analytic Inquiry into the Variability of Effects. Pp in T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, T.A. Louis, and F. Mosteller (eds.), Meta-Analysis for Explanation: A Casebook. New York: Russell Sage Foundation. Lipsey, Mark W. and David B. Wilson Practical Meta-Analysis. Thousand Oaks, CA: Sage Publications. Mullen, Brian Advanced BASIC Meta-Analysis. Hillsdale, NJ: Lawrence Erlbaum. Mullen, Brian and Robert Rosenthal BASIC Meta-Analysis: Procedures and Programs. Hillsdale, NJ: Lawrence Erlbaum. Rosenthal, Robert Meta-Analytic Procedures for Social Research (Revised Edition). Newbury Park, CA: Sage. Rosenthal, Robert and M.R. Dimatteo Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews. Annual Review of Psychology 52: Wachter, Kenneth W. and Miron L. Straf (eds.) The Future of Meta-Analysis. New York: Russell Sage Foundation. Whitehead, John T. and Steven P. Lab A Meta-Analysis of Juvenile Correctional Treatment. Journal of Research in Crime and Delinquency 26: Wilson, David B Meta-Analytic Methods for Criminology. Annals of the American Academy of Political and Social Science 578(Nov.): Wolf, Fredric M Meta-Analysis: Quantitative Methods for Research Synthesis. Beverly Hills, CA: Sage.

11 FIGURE 1 Meta-Analyses in Criminal Justice Researh Figure 2 Meta-Analyses in CJ Masters /Doctoral Theses

12 Figure 3 Meta-Analysis Reference Trends in Four Academic Fields

13 TABLE 1 Meta-Analyses by Subject Area Type of Analysis Subject Area of Meta-Analysis Pragmatic Theoretical Totals Crime/Delinquent Behavior 8 (19%) Corrections/Intervention 39 (100%) Courts/Law 4 (33%) Drugs/Alcohol 8 (73%) Policing/Law Enforcement 6 (86%) Clinical diagnosis/prediction 7 (78%) Methodology 1 (100%) 35 (81%) 0 (0%) 8 (67%) 3 (27%) 1 (14%) 2 (22%) 0 (0%) 44 (36%) 39 (32%) 12 (10%) 11 (9%) 7 (6%) 9 (7%) 1 (1%) Totals TABLE 2 Meta-Analyses by Discipline Type of Analysis Disciplinary Location Pragmatic Theoretical Totals Criminal Justice 53 (73%) Psychology 16 (40%) Sociology 0 (0%) Clinical/Health-related 3 (75%) 20 (27%) 24 (60%) 4 (100%) 1 (25%) 74 (61%) 40 (33%) 4 (3%) 4 (3%) Totals

14 TABLE 3 Meta-Analyses by Type of Study Study Type Frequency Percentage Quantitative Meta-Analysis % Quantitative Box Score Methodological discussion/criticism Total 123 TABLE 4 Meta-Analyses by Effect Measure Type Effect Measure Used Frequency Percentage Standardized Mean Difference (d) % Correlation Coefficient (r) Odds-Ratio Other coefficient Total 120 TABLE 5 Meta-Analyses by Type of Analysis Analytical type <1989 period period period Totals Summative/combining 7 (44%) 23 (54%) 34 (56%) 64 (53%) Moderator/comparative 8 (50%) 15 (35%) 19 (31%) 42 (35%) Prediction/Regression/Causal Modeling 1 (6%) 4 (10%) 7 (12%) 12 (10%) Dimensional/Scaling 0 (0%) 1 (2%) 1 (2%) 2 (2%) Total

15 TABLE 6 Number of Authors by Time Period Number of Authors Totals 1 5 (31%) 11 (25%) 13 (21%) 29 (24%) 2 6 (38%) 17 (39%) 18 (29%) 41 (34%) 3 or more 5 (31%) 16 (36%) 31 (50%) 52 (43%) [4 or more] (included in above) [22] [18%] Total 16 (100%) 44 (100%) 62 (100%) 122 (100%) TABLE 7 Meta-Analyses by Size (# Studies included) Number of Studies Included in Review Frequency Percentage 4 to 19 studies % 20 to 49 studies to 99 studies to 199 studies and more studies Total 119 Descriptive Statistics: Mean = Median = Mode = Minimum = Maximum =