Scientific Information System

Transcription

1 Scientific Information System Network of Scientific Journals from Latin America, the Caribbean, Spain and Portugal Juan García García, Elena Ortega Campos, Leticia De la Fuente Sánchez The Use of the Effect Size in JCR Spanish Journals of Psychology: From Theory to Fact The Spanish Journal of Psychology, vol. 14, núm. 2, 2011, pp , Universidad Complutense de Madrid España Available in: The Spanish Journal of Psychology, ISSN (Printed Version): Universidad Complutense de Madrid España How to cite Complete issue More information about this article Journal's homepage Non-Profit Academic Project, developed under the Open Acces Initiative

2 The Spanish Journal of Psychology Copyright 2011 by The Spanish Journal of Psychology 2011, Vol. 14, No. 2, ISSN The Use of the Effect Size in JCR Spanish Journals of Psychology: From Theory to Fact Juan García García, Elena Ortega Campos, and Leticia De la Fuente Sánchez Universidad de Almería (Spain) In 1999, Wilkinson and the Task Force on Statistical Inference published Statistical Methods and Psychology: Guidelines and Explanation. The authors made several recommendations about how to improve the quality of Psychology research papers. One of these was to report some effect-size index in the results of the research. In 2001, the fifth edition of the Publication Manual of the American Psychological Association included this recommendation. In Spain, in 2003, scientific journals like Psicothema or the International Journal of Clinical and Health Psychology (IJCHP) published editorials and papers expressing the need to calculate the effect size in the research papers. The aim of this study is to determine whether the papers published from 2003 to 2008 in the four Spanish journals indexed in the Journal Citation Reports have reported some effect-size index of their results. The findings indicate that, in general, the followup of the norm has been scanty, though the evolution over the analyzed period is different depending on the journal. Keywords: effect sizes, statistical significance testing, null hypothesis testing, Spanish journals of psychology. Con la publicación, en 1999, del informe Statistical Methods in Psychology Journals: Guidelines and explanations (Wilkinson & The Task Force on Statistical Inference) se hicieron públicas una serie de recomendaciones para mejorar la calidad en la presentación metodológica de los artículos de investigación en Psicología, siendo una de ellas el cálculo de un estadístico adecuado para valorar el tamaño del efecto en la presentación de los resultados. En 2001, la quinta edición del Manual de Publicación de la Asociación Americana de Psicología ha incluido esta recomendación. En España, en 2003, revistas científicas como Psicothema o el International Journal of Clinical and Health Psychology (IJCHP) publican editoriales y guías en las que expresan la necesidad de calcular el tamaño del efecto en los trabajos de investigación. El objetivo de este estudio es determinar si los artículos publicados desde 2003 hasta 2008 en las cuatro revistas españolas indizadas en el Journal Citation Reports han informado de algún índice del tamaño del efecto en los artículos publicados. Los resultados indican que, en general, el seguimiento de la norma ha sido escaso, aunque la evolución durante el período analizado es diferente dependiendo de la revista. Palabras clave: tamaño del efecto, significación estadística, contraste de hipótesis nula, revistas españolas de psicología. Correspondence concerning this article should be addressed to Juan García García. Facultad de Psicología, Universidad de Almería. Cañada de San Urbano s/n Almería (Spain). jgarciag@ual.es 1050

3 THE EFFECT SIZE IN THE JCR SPANISH JOURNALS OF PSYCHOLOGY 1051 The recommendations to incorporate an index that adequately reports the effect sizes in research articles in Psychology are neither new nor isolated. Both authors and editors of prestigious journals have published normative articles and editorials, urging researchers to present some effect size index in their research articles along with the results obtained in the study. One of the first authors to recommend the use of an effect size index was L.V. Jones, who in 1955 published the article Statistics and Research Design, in the journal Annual Review of Psychology. This was subsequently followed by Kish, in 1959, with the article Some Problems in Research Design, Rozeboom, in 1960, with the article The Fallacy of the Null Hypothesis Significance Test, published in the Psychological Bulletin, or Carver, with the article The Case against Statistical Significance Testing, published in 1978 in the journal Harvard Educational Review. One year later, in September of 1979, Hunter emphasized the importance of using the effect size in the 86 th Annual Convention of the American Psychological Association with a communication entitled Cumulating Results across Studies: A Critique of Factor Analysis, Canonical Correlation, Manova and Statistical Significance Testing. In the decade of the 1990s, Cohen s well-known article Things I have Learned (so far), published in 1990 in the journal American Psychologist, subsequently translated into Spanish and published in Anales de Psicología, is also noteworthy. Also in this decade, journals like The Journal of Experimental Education, in 1993, or Psychological Science, in 1997, published special issues in which the causes and consequences of the existing confusion about the concept of the null hypothesis were studied in depth, and the origins and possible alternatives were discussed. In 1994, the American Psychological Association (APA) published its fourth Publication Manual, in which authors are explicitly reminded of the need to report the effect size with some index that is suitable to the data analyzed. Subsequently, in 1999, the Board of Scientific Affairs of the APA published the important report Statistical Methods in Psychology Journals: Guidelines and Explanations (Wilkinson & Task Force on Statistical Inference -TFSI-, 1999), with the aim of promoting the unification of criteria in research methodology. The ultimate goal of this report was for such recommendations to be included in the APA Publication Manual, which, in effect, in its fifth edition, of 2001, includes some of the recommendations of Wilkinson and the TSFI, and one of them was to calculate an adequate statistic to appraise the effect size of the results. There was no special emphasis that it had to be a specific index, rather it proposed that the index should be freely selected by the researcher, depending on the one that best matched the data and the needs of the investigation. With this recommendation, the goal of the report elaborated by Wilkinson and the TSFI was at least partially met, although the recommendation was much more half-hearted than the authors would have wished; a fact that provoked some critiques and discontent (Fidler, 2002). However, in the new 2009 edition of the APA Publication Manual, a more detailed analysis of the polemic of statistical significance contrasts was included and a more energetic recommendation was made about the obligatory use of the effect size in the journals of its influence, specifically: However, complete reporting of all tested hypotheses and estimates of appropriate effect sizes and confidence intervals are the minimum expectations for all APA journals. The research scientist is always responsible for the accurate and responsible reporting of the results of research studies (APA, 2009, p. 33). In the Spanish context, we note the publication in 1995, in Anales de Psicología, of the article Consideraciones Generales en torno a los Estudios de Potencia [General Considerations about Studies of Power] by Bono and Arnau, an article in which the benefits of reporting the effect size are discussed. Subsequently, in 2000, two articles in Spain were published in the journal Psicothema that endorse the use of the effect size: Tamaño del Efecto del Tratamiento y Significación Estadística [Effect size of the Treatment and its Statistical Significance] by Frías, Pascual, and García, and Significación Estadística, Importancia del Efecto y Replicabilidad de los Datos [Statistical Significance, Importance of the Effect and Replicability of the Data] by Pascual, García, and Frías. In both articles, the use that should be made of the information about the effect size in research articles is explained. However, studies that have shown how classical hypothesis testing is used have continued to point out its incorrect use and interpretation, as well as the need to report the effect size (Borges, San Luis, Sánchez, & Cañadas, 2001; Monterde, Pascual, & Frías, 2006). Subsequently, between 2002 and 2003, both the journals Psicothema and the International Journal of Clinical and Health Psychology (IJCHP) expressed, the former in an editorial, and the latter with the publication of various normative articles (Bobenrieth, 2002; Ramos-Álvarez & Catena, 2003), the need to calculate an effect size index in research articles, thereby echoing the new norms of the APA publication guidelines. Diverse research studies have been conducted in order to quantify the percentage of articles that calculate the effect size in the research results or the type of information provided. Thus, Kirk (1996) reviewed the articles published in 1995 in the journals Journal of Applied Psychology, Journal of Educational Psychology, Journal of Experimental Psychology, Learning y Memory, and Journal of Personality and Social Psychology. His results show that 42% of the 389 articles studied reported the effect size. Thompson and Snyder (1998) found that 60% of the reviewed articles of the Journal of Counseling and Development reported the effect size. However, Vacha-Haase and Nilsson (1998)

4 1052 GARCÍA, ORTEGA, AND DE LA FUENTE reported that the percentage of studies that provided the effect size had dropped to 35%. Vacha-Haase and Ness (1999) published another study on the same topic, and they found that 20% of the studies reported the effect size. In a later study, Bezeau and Graves (2001) indicated that the percentage of studies that reported the effect size was only 9%. Recently, the data found by Fidler et al. (2005) indicate 30%, whereas those provided by Crosby et al. (2008) report a decrease to 17%. Some authors, such as Faulkner, Fidler, and Cumming (2008), report that only 31% out of 193 controlled clinical trials about the effectiveness of psychological therapies interpret the effect size. Lastly, in the Latin-American sphere, García, Ortega, and De la Fuente (2008) analyzed the follow-up in journals indexed in Redalyc in 2007, finding that only 12.26% of the articles analyzed reported the effect size. According to Thompson (1999), concerning the failure of the effectiveness of the APA recommendations in these and other issues, several reasons for the inappropriate use of the effect size could be established: first, because the APA policy is imprecise, like that of other journals that endorse the APA with reference to these recommendations, although we should point out that, in the latest 2009 edition, this topic has been dealt with more extensively. Secondly, the APA policy is self-destructive, that is, more attention is paid to the formal aspects of the article than to the contents, so its message is contradictory. Lastly, the third reason is that many psychologists do not know how to calculate the effect size (Vacha-Haase, Nilsson, Reetz, Lance, & Thompson, 2000), and, hence, how to interpret it. Diverse empirical studies show that some researchers do not adequately understand the logic of statistical significance contrasts (Cumming et al., 2007; Frias, Pascual, & García, 2000; Killeen, 2008; Nelson, Rosenthal, & Rosnow, 1986; Oakes, 1986; Osborne, 2008; Pascual, Frias, & Monterde- Bort, 2000; Velicer et al., 2008; Zuckerman, Hodgins, Zuckerman, & Rosenthal, 1993) and, therefore, they are oblivious of the need to include and interpret the effect size in their articles. Most of the statistical software more frequently used include these aspects, and there is much free and commercial software with which to estimate the effect size, often linked to the use of meta-analysis (see, for example, Botella & Gambara, 2006; Lipsey & Wilson, 2001;) or the so-called evidenced-based practice psychology (i.e., Pascual et. al., 2004; Sánchez-Meca, Boruch, Petrosino, & Rosa, 2002). The goal of this work was to analyze the extent to which the articles published between 2003 and 2008 in the Spanish journals indexed in the databases of the Institute for Scientific Information (ISI) incorporate an adequate effect size index when presenting and discussing their results, thereby following the APA recommendations in this section for the publication of articles. The choice of the year 2003 as the starting point to conduct this study is mainly because of the publication of the recommendations of the journals Psicothema and ICHP, both of them journals with a large diffusion in Spain. However, as the 5th APA Manual was published in 2001, we thought that enough time would have gone by for the authors and investigators to find out about the new publication guidelines and to apply them in their research studies. Sample Method The sample is made up of the articles published in the four Spanish journals indexed in the databases of the ISI during the six-year period of : Psychológica: 70 articles, Psicothema: 550 articles, International Journal of Clinical and Health Psychology (IJCHP): 168 articles, and the Spanish Journal of Psychology: 148 articles. These four journals have maintained an estimation of the impact factor in the Journal Citation Reports (JCR) during this period. Procedure The study was carried out in two stages. In the first phase, all the articles published in the above-mentioned journals during the six-year period under study were reviewed in order to quantify the articles that, due to their nature (design and/or analysis employed), allowed the estimation of an effect size index. After selecting the articles, the second phase consisted of verifying how many of the articles in which the effect size could have been estimated expressly reported some concrete index, and these indexes, to which all the results referred, were the object of our study. To assess possible discrepancies between the interpretations of the articles, we selected 20% of the articles randomly and assessed them concurrently, reaching total agreement, so one of the researchers proceeded to code the remaining 80%. Results The results are presented, firstly, grouped into the total number of articles published by each journal during the six-year period (see Figure 1), indicating the percentage of articles that reported an effect size index in their results in comparison to the articles in which the index could have been calculated and was not. During this six-year period, the journal Psicothema published a total of 116 articles in which the effect size index was calculated, which is 21.1%, 95% CI [17.6, 24.6] of the total amount of articles in which the index could have been estimated or provided and was not (434). In the

5 THE EFFECT SIZE IN THE JCR SPANISH JOURNALS OF PSYCHOLOGY 1053 Figure 1. Percentage and 95% CI of articles published that included the effect size in the four Spanish journals indexed in the JCR during the six-year period , compared to those that should have included it. Figure 2. Evolution of the percentage of articles published that included the effect size in the four journals during the six-year period , compared to those that should have included it. Spanish Journal of Psychology, the number of articles published (47) that reported the effect size was 31.8%, 95% CI [23.9, 39.7] of the total number of articles published that could have reported an index (148). During the sixyear period , the journal Psicológica only published two articles that provided the effect size index (2.9%, 95% CI [0, 10]) versus 68 articles that should have estimated it. Lastly, the journal IJCHP, during the six years studied, published 31 articles in which the effect size was calculated (18.5%, 95% CI [12.4, 24.6]) versus 137 articles that could have reported the index and did not. Figure 2 displays the evolution of the four journals targeted in this study over the six years analyzed, with regard to the percentage of articles published that included an effect size index. An increasing tendency can be seen in all of them until 2006, with some drop in the last two years. Discussion The goal of the present study was to determine the extent to which one of the recommendations made systematically from diverse sources, in an attempt to improve research articles methodologically, has been followed in four Spanish journals indexed in the JCR, which, therefore, usually have higher diffusion, or at least, they incite more follow-up. In view of the results obtained, a low percentage of the articles published in the Spanish journals analyzed report the effect size in their results. Specifically, we have verified that, during the six-year period we studied, the following percentages of articles provided this index: 2.9% in Psicológica; 18.5% in the IJCHP; 21.1% in Psicothema; and 31.8% in the Spanish Journal of Psychology. Therefore, we note that the recommendations published by the APA in its 5th manual are still not being applied in most of the research articles published in the leading journals of Psychology in Spain. Nevertheless, the results also indicate that, except for the journal Psicológica, there is an increasing tendency in the journals analyzed over six years with regard to the articles that could estimate and actually report the effect size in their results, which is a positive indicator in this sense. However, the exception of the journal Psicológica is remarkable, in view of the nature of most of its articles, set in a section called Experimental Psychology. Nevertheless, it seems clear that the editorials and guidelines of the journals, making appeals in this sense, which have been reinforced by the publication of the 6 th edition of the APA Publication Manual (APA, 2009), have substantially improved this practice. But, in view of the current situation, one wonders why most of the researchers continue to neglect this issue, and also why the reviewers of the journals do not demand it. On the one hand, changing the practices acquired over many years regarding the type of results one must obtain may require an additional effort that is frequently not put out, probably more due to not knowing how to do it than to lack of interest, and it may even be due to the belief that the effect size is only complementary. Including an effect size index is not usually complicated, although, if the concept has not been understood, it is somewhat more complex to interpret. In this sense, greater efforts should be made to diffuse and disseminate it. On the other hand, the size or magnitude of the effect can be considered one of the most informative measures of the results of an investigation, because it integrates the different, not directly comparable, results of a study or a series of studies, and it expresses them on a single measurement scale, thus allowing the comparison of the data (Green & Hall, 1984). In this sense, we consider it interesting to point out its advantages when performing quantitative reviews of research reviews increasingly demanded by the scientific community that form the foundation to construct an evidence-based psychology. References American Psychological Association. (1994). Publication Manual of the American Psychological Association (4 th ed). Washington, DC: Author.

6 1054 GARCÍA, ORTEGA, AND DE LA FUENTE American Psychological Association. (2001). Publication Manual of the American Psychological Association (5 th ed). Washington, DC: APA. American Psychological Association. (2009). Publication Manual of the American Psychological Association (6 th ed). Washington, DC: APA. Bezeau, S., & Graves, R. (2001). Statistical power and effect sizes of clinical neuropsychology research. Journal of Clinical and Experimental Neuropsychology, 23, doi: / jcen Bobenrieth, M. A. (2002). Normas para revisión de artículos originales en Ciencias de la Salud [Review guidelines of original articles on Health Sciences]. International Journal of Clinical and Health Psychology, 2, Bono, R., & Arnau, J. (1995). Consideraciones generales en torno a los estudios de potencia [General considerations about statistical power studies]. Anales de Psicología, 11, Borges, A., San Luis, C., Sánchez, J. A., & Cañadas, I. (2001). El juicio contra la hipótesis nula: muchos testigos y una sentencia virtuosa [The judgment against null hypothesis. Many witnesses and a virtuous sentence]. Psicothema, 13, Botella, J., & Gambara, H. (2006). Doing and reporting a metaanalysis. International Journal of Clinical and Health Psychology, 6, Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, doi: // x Cohen, J. (1992). Cosas que he aprendido (hasta ahora) [Things I have learned (so far)]. Anales de Psicología, 8(1-2), Crosby, R. D., Wonderlich, S. A., Mitchell, J. E., De Zwaan, M., Engel, S. G., Connolly, K., Taheri, M. (2006). An empirical analysis of eating disorders and anxiety disorders publications ( ) Part II: Statistical hypothesis testing. International Journal of Eating Disorders, 39, Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Wilson, S. (2007). Statistical reform in psychology: Is anything changing? Psychological Science, 18, doi: /j x Faulkner, C., Fidler, F., & Cumming, G. (2008). The value of RCT evidence depends on the quality of statistical analysis. Behavior Research and Therapy, 46, doi: /j.brat Fidler, F. (2002). The Fifth edition of the APA Publication Manual: Why its statistics recommendations are so controversial. Educational and Psychological Measurement, 62, doi: / Fidler, F., Cumming, G., Thomason, N., Pannuzzo, D., Smith, J., Fyffe, P., Schmit, R. (2005). Toward improved statistical reporting in the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 73, doi: / x Frías, M. D., Pascual, J., & García, F. (2000). Tamaño del efecto del tratamiento y significación estadística. [Effect size and statistical significance]. Psicothema, 12, García, J., Ortega, E., & De la Fuente, L. (2008). Tamaño del Efecto en las revistas de psicología indizadas en Redalyc [Effect size in psychology journals indexed in redalyc]. Informes Psicológicos, 10, Green, B. F., & Hall, J. A. (1984). Quantitative Methods for Literature Reviews. Annual Review of Psychology, 35, doi: /annurev.ps Hunter, J. E. (1979, September). Cumulating results across studies: A critique of factor analysis, canonical correlation, MANOVA and statistical significance testing. Invited address presented at the 86 th Annual Convention of the American Psychological Association. New York, NY. Jones, L. V. (1955). Statistics and research design. Annual Review of Psychology, 6, doi: /annurev.ps Killeen, P. R. (2008). A Rational Foundation for Scientific Decisions: The Case for the Probability of Replication Statistic. In J. W. Osborne (Ed). Best practices in quantitative methods (pp ). Thousand Oaks, CA: Sage. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, doi: / Kish, L. (1959). Some statistical problems in research design. American Sociological Review, 24, doi: / Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. (Applied Social Research Methods Series, Vol. 49). Thousand Oaks, CA: Sage. Monterde, H., Pascual, J., & Frías, M. D. (2006). Errores de interpretación de los métodos estadísticos: importancia y recomendaciones [Interpretation mistakes in statistical methods: Their importance and some recommendations]. Psicothema, 18, Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance levels and effect sizes by psychological researchers. American Psychologist, 41, doi: // X Oakes, M. L. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York, NY: Wiley. Osborne, J. W. (2008). Sweating the small stuff in educational psychology: How effect size and power reporting failed to change from 1969 to 1999, and what that means for the future of changing practices. Educational Psychology, 28, doi: / Pascual J., Frías M. D., & Monterde-Bort, H. (2004). Tratamientos psicológicos con apoyo empírico y práctica clínica basada en la evidencia. [Psychological treatments with empirical evidence and evidence-based practice]. Papeles del Psicólogo, 87,1-8. Pascual, J., García, J., & Frías, M. D. (2000). Significación estadística, importancia del efecto y replicabilidad de los datos. [Statistical significance and replicability of the data]. Psicothema, 12, Ramos-Álvarez, M. M., & Catena, A. (2003). Normas para la elaboración y revisión de artículos originales experimentales en Ciencias del Comportamiento. [Criteria of the peer review process for publication of experimental research in Behavioral

7 THE EFFECT SIZE IN THE JCR SPANISH JOURNALS OF PSYCHOLOGY 1055 Sciences]. Internacional Journal of Clinical and Health Psychology, 4, Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, doi: /h Sánchez-Meca, J., Boruch, R. F., Petrosino, A., & Rosa A. I. (2002). La colaboración Campbell y la práctica basada en la evidencia. [The Campbell collaboration and evidence-based practic]. Papeles del Psicólogo, 83, Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, Thompson, B. (1999). Statistical significance test, effect size reporting, and the vain pursuit of pseudo-objectivity. Theory & Psychology, 9, doi: / Thompson, B. & Snyder, P. A. (1998). Statistical significance testing and reliability analyses in recent JCD research articles. Journal of Counselling and Development, 76, Vacha-Haase, T., & Ness, C. M. (1999). Statistical significance testing as it relates to practice: Use within Professional Psychology: Research and Practice. Professional Psychology: Research and Practice, 30, doi: // Vacha-Haase, T., & Nilsson, J. E. (1998). Further comments on statistical significance tests. Measurement and Evaluation in Counseling and Development, 31, Vacha-Haase, T., Nilsson, J. E., Reetz, D. R., Lance. T. S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, doi: / Velicer, W. F., Cumming, G., Fava, J. L., Rossi, J. S., Prochaska, J. O., & Johnson, J. (2008). Theory testing using quantitative predictions of effect size. Applied Psychology, 57, doi: /j x Wilkinson, L. & Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, doi: / X Zuckerman, M., Hodgins, H. S., Zuckerman, A., & Rosenthal, R. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, doi: /j tb00556.x Received December 7, 2009 Revision received September 17, 2010 Accepted October 27, 2010