Issues in Conducting Rigorous and Relevant Research in Education Richard J. Shavelson Stanford University Keynote Presentation Rigor and Relevance in Educational Research Litteraturhuset, Oslo 4 March 2010
Overview Rigor: Scientific Research in Education (2002) Principles of scientific research Design of scientific research in education Disagreements about quality research in education Relevance: Working in Pasteur s Quadrant Work at the intersection of theory and application Relevance is in the eye (and political agenda) of the beholder
Rigor: Scientific Research In Education Commissioned by the U.S. Department of Education Conducted by the National Research Council of the National Academy of Science Published in 2002 Anticipate legislation under then incoming President George W. Bush proposing a narrow definition of scientific research There is a long-standing debate among scholars, policy makers, and others about the nature and value of scientific research in education and the extent to which it has produced the kind of cumulative knowledge expected of scientific endeavors. Most recently, this skepticism led to proposed legislation that defines what constitutes rigorous scientific methods for conducting educational research. This study sought to examine and clarify the nature of scientific inquiry in education and how the federal government can best foster and support it.
Scientific Research in Education: Unified Conception of Generalizability* SRE Committee Findings Science is fundamentally the same across all disciplines and fields All fields are characterized by a range of legitimate methods and specialization depending on objects of inquiry and context Some differences between social and natural sciences As in other fields, features of education shape inquiry Principles of Scientific Research in Education Pose Significant Questions That Can Be Investigated Empirically Link Research to Relevant Theory Use Methods that Permit Direct Investigation of Question Provide Coherent, Explicit Chain of Reasoning Replicate and Generalize Across Studies Disclose Research to Encourage Professional Scrutiny and Critique * The unified conception needs to pay specific attention to the social importance and other value dimensions of research projects [and this can be done] without compromising their technical standards (Phillips, 2009, p. 511).
Designs For Conduct Of Scientific Research In Education What is happening? Estimates of population characteristics Simple relationships Descriptions of localized educational settings Is there a systematic (causal) effect? Causal relationships when randomization is feasible Causal relationships when randomization is not feasible How or why is it happening? Mechanism when theory is fairly well-established Mechanism when theory is weak
Changes in Quality of Educational Research I: Debate Is There A Systematic (Causal) Effect? General Laws (Tom Cook) Causal main effects are present Context by treatment interactions mask main effects (i.e., error) Interactions (Lee Cronbach) Facts decay and the half-life of a social science fact is very short Interactions between treatment and context abound treatment effect a random variable Even randomized experiments are case studies Policy implications? Policy and causal mechanism?
Changes in Quality of Educational Research II: Debate Persists! Research Context: Questions & Appropriate Research Methods Unified Generalizable Predicts & Explain Interpretative Meaning & Intention in Context Educational Researcher (2009), 38(7), pp. 501-517.
Relevance: Working in Pasteur s Quadrant Personal Experience Relevance doesn t readily appear when looking for it Relevance sneaks up when working on an applied problem Astronaut performance in space and on the moon estimate time and error decrements for first time Military job performance and its predictors direct application of G theory Measurement of students understanding of science re-framing of national assessment Student learning in college and its assessment & value added changing the debate & new measurement issues T h e o r y Donald Stokes Pasteur s Quadrant Bohr Application Pasteur Franklin
Relevance: A Two Edged Sword Research Impact on Policy Political constituencies Values & Beliefs Economics Research findings Policy & Politics Context: Just How Relevant Is... Social Science & Education Research? After Senator Tom Coburn, Republican of Oklahoma, this month proposed prohibiting the National Science Foundation from wasting any federal research funding on political science projects, political scientists rallied in opposition, pointing out that one of this year s Nobel winners had been a frequent recipient of the very program now under attack. NYT 10/20/09 Senator Tom Coburn is a conservative Republican from Oklahoma who is known for his efforts at thwarting expensive legislation. His willingness to block, delay or neuter bills through an array of procedural measures has made him an effective nuisance during his five years in the Senate. NYT 10/30/09 Cohen, P. (Tuesday, October 20, 2009) The New York Times, pp. C1, C7.
Relevance: Or Questionable Use Of Social Science To Settle Policy & Normative Issues Make The Data Fit The Belief Source: Non Sequitur
Relevance: In the Eyes Of The Beholder UM Undergraduate Affirmative Action Case National Association of Scholars files with plaintiffs arguing that the Bakke decision should be interpreted as requiring that structural [racial] diversity directly impact college outcomes. They further argue that the Powell diversity rationale asserts that, holding all other variables constant, students benefit by attending racially diverse colleges. They also argued methodologically this was the only way to legitimately model the relation between diversity and outcomes (eschewing indirect effects) statistically. Amicus brief from SIHER argued that the NAS incorrectly interpreted Powell (and Gurin s analysis) by excluding and ignoring, intentionally, effects of educational activities that depend on the existence of structural diversity. And incorrectly portrayed statistical modeling. That is, structural diversity operates not directly but through the formal and informal educational processes that such diversity affords.
Relevance and Politics Of Instruction: Science or Values At Stake?
The Findings
The Treatments: Discovery & Direct Instruction Discovery Instruction Researcher provided goals and materials; students explored experimental apparatus with no direct guidance or feedback with Direct Instruction Guided Inquiry Researcher provided goals, materials, examples, and explicit explanations of how to control variables; students ran experiments with direct guidance and feedback
Thank You!
Changes in Quality Of Educational Research I: Focus Wide Scale Attention Debate at scholarly meetings both in the U.S. and other areas of the world, most notably Europe that continue today in the U.S. Special issues of journals devoted to critiquing the message in SRE Recognition of the impact of SRE in changing the quality of education research such as this meeting today
Changes in Quality Of Educational Research II: Focus On Standards American Educational Research Association (2006): Standards for Reporting on Empirical Social Science Research in AERA Publications AERA Grants Board White Paper (2007). Estimating causal effects using experimental and observational designs American Psychological Association (2008): A Guide to the Use of Randomized Controlled Trials (RCTs) in Assessing Intervention Effects The American Educational Research Association (AERA) is pleased to provide guidelines for reporting on empirical social science research in AERA publications. These guidelines apply to reports of education research grounded in the empirical traditions of the social sciences. They cover, but are not limited to, what are commonly called qualitative and quantitative methods. Other forms of scholarship equally important to education research include reviews of research; theoretical, conceptual, or methodological essays; critiques of research traditions and practices; and scholarship more grounded in the humanities (e.g., history, philosophy, literary analysis, arts-based inquiry). The latter forms of scholarship are beyond the scope of this document.