Research Quality & Content

Transcription

1 Spring 2001 Y520: 8525 Strategies for Educational Inquiry Research Quality & Content Educational Research: Quality of Methods To evaluate a research article, the reader must attend to theory development, operational definitions, instrumentation/data collection, methods/procedures, data analysis/statistical tests, writing style, and so forth. Even though the theoretical underpinnings may be weak or not well explained in a particular article, our goal is to identify the author s purpose and determine whether the methods used do indeed justify the conclusions. In this course we focus on methods. Theory development is the focus of other courses, such as H510, Foundations of Educational Inquiry. Overview of Quality Beginning students often assume that research articles published in scholarly journals must by virtue of being publish be of high quality. Several experts have investigated the quality of research in educational journals and based on their reports, the reader should not infer that publication insures uniformly high methodological quality. Serious doubt has been expressed as to whether the majority of published educational research articles merit publication on the basis of their quality (Wandt, 1965). Wandt et al. reviewed 125 research articles published in educational journals and report the frequently observed problems: Frequent Problems Results of analysis not clearly presented. Incorrect methods used to analyze data. Inappropriate or defective design. Validity and reliability of the evidence not established. Conclusions not substantiated by the evidence. Organization of Research Reports The sections of a research reports are: introduction (description of theory, statement of the problem, conceptual definitions, synthesis of previous research, testable hypothesis), methods/procedures (design, subjects, sampling, materials, instrumentation/data collection, reliabality of scores, operational definitions, etc.), 1

2 Educational Research: Quality of Methods results (descriptive statistics, interpretation of statistical tests), and conclusion (explication of results to justify problem statement/hypothesis). Notice that the list of frequently occurring problems are found mostly in the methods/procedures and results sections. Nearly a decade ago the National Academy of Science evaluated educational research and characterize the body of work as methodologically weak research, trivial studies, and infatuation with jargon, and a tendency towards fads (Atkinson & Jackson, 1992, p. 20). Several studies have used methodology experts as judges of research quality. The judges in Wall, Hall and Schramm (1975) reported that over 60 percent of published research exhibited serious flaws. Hall, Ward and Comer (1988) reported that over 40 percent of the studies they examined contained serious flaws. In an AERA invited address, Thompson (1998) stated incorrect analyses arise from doctoral methodology instruction that teaches research methods as series of rotely-followed routines, as against thoughtful elements of a reflective enterprise; from doctoral curricula that seemingly have less and less room for quantitative statistics and measurement content, even while our knowledge base in these areas is burgeoning... (p. 4). Erosion of Standards One conference attendee wrote AERA president Alan Schoenfeld to complain that At [the 1998 annual meeting] we had a hard time finding rigorous research that reported actual conclusions. Perhaps we should rename the association the American Educational Discussion Association... This is a serious problem. By encouraging anything that passes for inquiry to be a valid way of discovering answers to complex questions, we support a culture of intuition and artistry rather than the building reliable research bases and robust theories. Incidentally, theory was even harder to find than good research (Anonymous, 1998, p. 41). These issues are discussed at length in Thompson (1999), from which several of the preceding quotations and references were borrowed. Thompson has his own list of suggestions. He states that educational research would be improved by the elimination of five specific practices. Most of these practices are beyond the scope of an introductory course. Thompson s suggestions for improvement Use of stepwise methods in regression. When interpreting results, failure to consider the context specificity of analytical weights (e.g., regression beta weights). When interpreting results, failure to interpret both weights and structure coefficients. Failure to recognize that reliability is a characteristic of scores, not of tests Incorrect interpretation of statistical significance and failure to report and interpret effect sizes. Two additional methodology errors that Thompson lists: Research Quality & Content 2

3 Educational Research: Quality of Methods Use of univariate analyses in the presence of multiple outcomes variables and the use of univariate analyses in post hoc explorations of multivariate effects. The conversion of intervally-scaled predictor variables into nominally-scaled data when performing anova, ancova, manova, mancova. These suggestions focus on the content of the methods/procedures and results sections of research reports. The first three suggestions refer to the use of regression for data analysis. If you have little previous experience with regression these suggestions may not be meaningful at this point. We will explore the implications of these specific suggestions when we scrutinize a research report that uses regression. A very brief explanation of each suggestion follows. Stepwise Methods The stepwise method consists of letting a computer decide the order in which variables are entered into a regression equation. Instead, the investigator should decide the order of variable entry on the basis of theory or simple expection. Other problems with stepwise regression will be mentioned when we study regression. Context Specificity The second suggestion, specify the context specificity of analytical weights such as regression beta weights, means that instead of considering only these weights, it may be important to consider the structure coefficients, which are correlation coefficients between a predictor variable and a composite derived by weighting and aggregating the criterion variable(s) (Thompson and Borrello, 1985, p. 205). This procedure falls in the category variously referred to as beyond the scope of this course or elsewhere as an exercise for the interested reader. Nonetheless, it is important to know that when predictor variables are correlated with each other this condition of collinearity can impact the interpretation of results. The inclusion of structure coefficients may improve interpretation. Reliability of Scores The fourth suggestion, that reliability is a characteristic of scores, not of a test, is a fundamental point that is often reported incorrectly in published articles. An assessment instrument, aka, a test, consists of a series of questions. As such, it is just a piece of paper or a piece of software code, if administered via computer. The instrument is administered to individuals. The individuals respond and we calculate reliability based on the scores of these individuals. We will spend considerable time discussing reliability and validity. Statistical Significance The fifth suggestion, that statistical significance is often misinterpreted and effect size is not reported, is also a fundamental point that we will discuss in virtually every article that we analyze. In simple terms, statistical significance indicates the likelihood that a particular result is not a chance occurrence. We may see some authors state, incorrectly, that statistical significance indicates the importance and/or strength of an effect. Statistical significance indicates nothing about importance. Indeed, social science can be criticized for highly (i.e., improbable due to chance) statistically significant results that are unimportant and even meaningless (cf, Postman, 1975). The effect size, which Research Quality & Content 3

4 Educational Research: Content is often not reported in published research, indicates the strength of an experimental manipulation. Wilkinson et al. (1999), an article we will read later in the semester, discusses the importance of calculating the effect size. Univariate & Multivariate In this course we likely will not have time to consider the implications of Thompson s sixth suggestion, that mulitvarite analysis should be used for post hoc explorations when multivaritate analysis are used. Collapsing Scales Violations of the final suggestion, that one should not throw away information by converting interval scale predictor variables to nominal (e.g., categorical) variables, will be seen in some of the research articles we read and we will discuss the implications at that time. This often happens when an investigator creates categories in order to perform an analysis of variance. This problem can be avoided by using regression instead (cf. Humphreys, 1978). Before we conclude this overview of quality, note that King (1986) describes several serious methodology problems that occur frequently in the political science literature. Rosenthal (1989) and Wilkinson et al. (1999) discuss methodological problems in psychological research and similar critiques of sociology methods can be found. In fact, all branches of social sciences use similar methodologies and experience quality control problems in published articles. Educational Research: Content In addition to research quality, faculty are interested in identifying the particular skills and knowledge students need to acquire in order to evaluate and conduct research. Mundfrom, Shaw, Thomas, Young, and Moore (1998) asked, What should graduate students know about research and statistics after completing an initial course? (p. 2). Instructors of research courses were surveyed and Table 2 presents the mean ratings for each item s importance and the depth to which it should be covered. All of the items in this table should be included in this course, but some may not. Rather than surveying faculty, Elmore and Woehlke (1998) state An approach to determining the essential topics to be included in the doctoral tool sequence for students preparing to be educational researchers is to conduct a content analysis of the methods and techniques used in published articles in educational research journals (p. 2). Following in the footsteps of several previous investigators, they examined the research methods used during the past 20 years in three educational research journals. They report that the methods most frequently used in published articles during that interval were: 1. Descriptive 2. Analysis of variance and analysis of covariance 3. Multiple regression 4. Qualitative 5. Bivariate correlation 6. Multivariate Research Quality & Content 4

5 Educational Research: Content Unfortunately, this list confounds paradigms ( qualitative ), research designs ( descriptive ), and statistical methods (analysis of variance, multiple regression, bivariate correlation, multivariate ). Nonetheless, it provides an additional observation about the knowledge and skills students need in order to critique research reports and to conduct their own investigations. Note that valicity should be validity. Table 2. Means and Standard Deviations for Topics - All Respondents (n=80). Importance Depth of Coverage Topic Mean Std. Dev. Mean Std. Dev. Internal and external valicity Formulation of viable research problems Types of research Formulation of testable hypotheses Scientific method Types of variables Interpreting results (of inferential tests) Literature sources and searching Correlation coefficients Types of sampling Null and alternative hypotheses Measures of center Validity estimates Statistical vs. practical significance Types of experimental designs Reliability estimates Measures of dispersion Ethical and legal issues Type I and Type II errors Significance level Conducting surveys p-values The normal distribution Likert scales Graphical and tabular displays Rating scales Standard error of measurement Standard scores Conducting interviews t-tests Sampling error Contingency tables Sampling distributions Power Analysis of variance Chi-square tests of association Confidence intervals Elements of probability Linear regression Research Quality & Content 5

6 References References Anonymous. (1998). [Untitled letter]. In G. Saxe & A. Schoenfeld, Annual meeting Educational Researcher, 27(5), 41. Atkinson, R.C., & Jackson, G. B. (Eds.). (1992). Research and educational reform: Roles for the Office of Educational Research and Improvement. Washington, DC: National Academy of Sciences (ERIC Document Reproduction Service No. ED ). Elmore, Patricia B., & Woehlke, Paula L. (1998). Twenty years of research methods employed in American Educational Research Journal, Educational Researcher, and Review of Educational Research. Paper presented at annual meeting of the American Educational Research Association, San Diego, CA. (ERIC Document Reproduction Service No ). Hall, B.W., Ward, A.W., & Comer, C.B. (1988). Published educational research: An empirical study of its quality. Journal of Educational Research, 81, Huck, Schuyler W., Cormier, William H., & Bounds, Jr., William G. (1974). Reading Statistics and Research. New York: Harper and Row. Miller, D. W. (1999). The black hole of educational research. The Chronicle of Higher Education, August 6. Mundform, Daniel J., Shaw, Dale G., Thomas, Ann, Young, Suzanne, & Moore, Alan D. (1998). Introductory graduate research courses: An examination of the knowledge base. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. (ERIC Document Reproduction Service No. ED ). Postman, Leo (1975). Verbal learning and memory. Annual Review of Psychology, 26, Rosenthal, Robert, & Rosnow, Ralph L. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44(10), Shott, Susan (1990). Statistics for Health Professionals. Philadelphia: W. B. Sanders. Thompson, B. (1988). Common methodology mistakes in dissertations: Improving dissertation quality. Paper presented at the annual meeting of the Mid-South Educational Research Association, ERIC Document Reproduction Service No. ED ). Thompson, B. (1994). Common methodology mistakes in dissertations, revisited. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED ). Thompson, B. (1999). Common methodology mistakes in educational research, revisited, along with a primer on both effect sizes and the bootstrap. Invited address presented at the annual meeting of the American Educational Research Association, Montreal. Research Quality & Content 6

7 References Thompson, Bruce, & Borrello, Gloria M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, Wandt, Edwin, Adams, Georgia W., Collett, Dorothy M., Michael, William B., Ryans, David G., & Shay, Carleton B. (1965) An evaluation of educational research published in journals. Report of the Committee on Evaluation of Research, American Educational Research Association.Unpublished report. Ward, A.W., Hall, B.W., & Schramm, C.E. (1975). Evaluation of published educational research: A national survey. American Educational Research Journal, 12, Wilkinson, Leland. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), Research Quality & Content 7