Measuring and Assessing Study Quality

Size: px
Start display at page:

Download "Measuring and Assessing Study Quality"

Transcription

1 Measuring and Assessing Study Quality Jeff Valentine, PhD Co-Chair, Campbell Collaboration Training Group & Associate Professor, College of Education and Human Development, University of Louisville Why Is Study Quality Important? Conceptually, most people understand Garbage In, Garbage Out In reality, in a systematic review and meta-analysis things are probably not this simple, but study quality does matter Lots of research in medicine suggests that lower quality studies are associated with exaggerated treatment effects Across the social and medical sciences, safest statement is that study quality covaries with effect size estimates (but in complex ways) 1

2 What Does Study Quality Mean? Study Quality means different things to different audiences: University administrator: Will the study lead to grant funding? Academic: Is the study highly cited? Practitioner: Did the study yield information I can use in my practice? Most of us: Do the study results confirm my pre-existing beliefs? From the standpoint of a systematic review, a quality study is one with a high degree of fit between its goals and the methods deployed in service of those goals Plan for Today Typical (but not recommended) approaches to addressing study quality in systematic reviews Peer Review Quality Quality Scales Truth Suggested approaches to addressing study quality in systematic review Set highly defensible inclusion criteria Code other sources of variability in study quality Univariate tests of individual study quality indicators Multivariate tests of the effects of study quality 2

3 Obligatory Story from Social Psychology A classic study (Lord, Ross, & Lepper, 1979) illustrated how people manage to see what they expect to see Participants were previously identified as firm proponents or opponents of an issue Participants were randomly assigned to receive one of two otherwise identical studies, either supporting or not supporting their position All participants also received commentary critical of the study s methods Participants were asked to rate the quality of the study (DV) Lord et al. s Results Participants whose original position was supported tended to rate the study as being good Participants whose original position was not supported tended to rate the study as flawed They scrutinized the study and paid attention to the critical commentary. This allowed them to discount the conclusions of the study and retain their original positions. Bottom line: Information that confirms pre-existing beliefs is less likely to get scrutiny. Information that goes against pre-existing beliefs sets in motion processes that make it more likely that we will be able to retain our beliefs 3

4 Not Recommended Approaches to Addressing Study Quality I: If it is peer reviewed, it is good Many individuals undertaking a systematic review will include only peer-reviewed studies Explicit or implicit goal of using publication status as a proxy for study quality Unfortunately, peer review is at best an imperfect marker for study quality Some of these reasons are outlined on the next slides Publication Status as a Proxy for Study Quality Peer review was not designed solely as a methodological quality screen This means that even if peer review was a perfect process, papers would go unpublished for a variety of reasons Study methods and/or analyses are of low quality Study fails to make a significant contribution to theory Peer review process is biased against replication research This is particularly problematic for systematic reviewers, who are hoping for many highly similar studies! Study authors are unable to find a journal that is a good fit for the research question, the population studied, or the outcomes measured 4

5 But the Peer Review Process Doesn t Work All That Well I just argued that even if the peer review process was perfect, studies would go unpublished for reasons unrelated to their quality But the peer review process is far from perfect. In addition to being biased against replication research, studies suggest that the peer review process is biased against the null hypothesis biased in favor of the reviewers pre-existing beliefs perhaps biased in favor of famous scholars characterized by low inter-rater reliability And, almost everyone agrees that low quality studies do in fact get published in peer reviewed outlets Publication is not Always a Goal of Research In addition, publication is not always a goal of research Examples include applied dissertations, work done for governments or non-profits, etc. It is one thing to believe that peer review guarantees truth it is quite another to believe that all truth appears in peer reviewed journals (Glass, 2000) Need to recognize that published and unpublished studies overlap in terms of their quality As such systematic reviews should not be limited to published studies 5

6 Not Recommended Approaches to Addressing Study Quality II: I know, let s develop a quality scale! Many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many, many quality scales have been developed. These scales vary in terms of their specificity coverage Quality Scale Specificity What do I mean by specificity? The extent to which the items on the scale are operationally defined As specificity increases, reliance on the cognitive algebra of the persons making the judgments decreases Cognitive algebra = idiosyncratic perspectives that different individuals bring to making a judgment People can apply different standards to the same judgment task People can weight the same attributes differently Specificity serves to increase inter-rater reliability 6

7 Quality Scale Coverage What do I mean by coverage? Some quality scales address only one aspect of quality, such as internal validity Others address more than one, and some attempt to address many, including things like ethics and utility Theoretically, the validity of one aspect of quality is unrelated to the validity of another aspect of study quality Assessing multiple aspects of study quality is probably most desirable But a single number describing multiple aspects of study quality probably does not make sense Quality Scales Do Not Work as Intended Jüni et al. (1999) found an existing meta-analysis on the effects of a new drug relative to the standard version Also located 25 quality scales Most of these (24 or 25) had been published in peer reviewed medical journals They then conducted 25 separate meta-analyses Each meta-analysis used a different quality scale Were interested in what they high quality studies said about the drug relative to the low quality studies What did they find? 7

8 Jüni et al. s Results In about 50% of the meta-analyses, the high and low quality studies agreed about the effectiveness of the drug In about 25% of the meta-analyses high quality studies suggested that the new drug was more beneficial than the old one low quality studies suggested it was no better than the old drug In about 25% of the meta-analyses high quality studies suggested the new drug was no better than the old drug low quality studies suggested the new drug was more beneficial than the old one Implications of Jüni et al. s Study The conclusion about the effectiveness of the new drug depended (in part) on the specific quality scale chosen This could mean that study quality doesn t matter in this area Or it could mean that the study quality scales were so bad that they masked the effects of study quality 8

9 The Jüni et al. (1999) findings go beyond the immediate debate about the effectiveness of the treatment in question and strike at the heart of the scientific enterprise. Even though the quality scales Jüni et al. examined were all developed by well-qualified medical researchers, these researchers apparently disagreed enough about the definition of quality in medical research that applying their standards individually led to a hopelessly confused picture of the state of evidence. As such, Jüni et al. s findings cast serious doubt on the growing practice of using quality scales to evaluate evidence (Valentine & Cooper, 2008, p. 130). Study Quality Scales Revisited Specificity Scales vary in the extent to which they operationally define their criteria As such they vary in terms of how much is left to the reviewers judgment (and/or imagination) Coverage Scales vary in terms of how many different aspects of study quality that they attempt to assess 9

10 Weaknesses of Study Quality Scales Very different numbers of items In Jüni et al. s study, quality scales had between 3 and 34 items This suggests differences in coverage (some are more comprehensive than others) Different weightings of the same criteria Even among studies with the same number of criteria, the same aspect of study quality can have different weights across scales Two scales with 14 items Beckerman et al. said masking worth 7% of total points, Brown said masking worth 23% of the total points There is no empirical justification for any weighting scheme (equal weight or otherwise) Weaknesses of Study Quality Scales, cont d. Study quality problems are often thought to work in the same direction It is entirely possible that some problems positively affect effect size estimates, and others negatively bias effect size estimates No quality scale has ever attempted to address this Very little empirical work to help figure this out The difference between study quality and reporting quality is often ignored Just because a report is silent about a quality dimension, it does not necessarily mean that study quality is low (description could have been omitted due to space concerns, etc.) 10

11 Weaknesses of Study Quality Scales, cont d. Most quality scales rely on a single score to represent study quality Usually, items on quality scales are summed or perhaps weights are applied and then items are summed but the result is a single number As mentioned, there is no theoretical reason to expect one aspect of study quality to be related to other aspects of study quality When unrelated dimensions are summed, the results do not have meaning Study A: High internal validity but low external validity = 80 Study B: Low internal validity but high external validity = 80 Important Challenges to Study Quality Assessments The standards for quality judgments need to be open to the inspection of others (i.e., less reliance on the reviewer s cognitive algebra) Assessment instruments must recognize that study quality is a multidimensional construct Study context probably matters a lot Valentine & McHugh (2007) found that attrition in school-based studies is probably ignorable. In other studies (e.g., drug treatment interventions), this is unlikely to be true. Much more empirical work is needed to help understand the relative importance of quality dimensions and this work will likely be contextually bound 11

12 So What Should I Do About Study Quality in My Review? Best practice is to use a combination of two strategies a priori rules that govern what gets included in the review empirical investigation of the extent to which variations in study quality are associated with variability in effect size estimates Both strategies require deep substantive, methodological, and statistical knowledge Substantive, Methodological, and Statistical Knowledge Near certainty that all quality dimensions are contextually bound A threat to the validity of one research question might not be a threat to validity of another research question Need to think very carefully about how potential threats to validity might play out for your research question 12

13 a priori Rules In doing a systematic review you will almost certainly adopt some rules before data collection that relate to study quality. For example you might exclude non-randomized experiments non-randomized studies that don t control for a pretest of the outcome studies lacking a comparison group studies that fail to use a gold standard way of measuring the outcome (when one is available) Regardless of the decisions you make, they should be explicit, and you should justify them (ideally, these a priori decisions will be highly defensible) Empirical Investigation of the Extent to which Other Study Quality Indicators Covary with Effect Size Except for a few highly defensible exclusion criteria, I suggest that you code other study quality indicators, and test whether the presence of these covaries with effect size For example, one might include both randomized and nonrandomized experiments, then code studies for whether or not a random process was used to allocate participants to groups, conduct a moderator test comparing the effects observed in randomized experiments relative to non-randomized experiments This process will help build the evidence base on how study quality indicators are related to study outcomes in your field of study 13

14 Multivariate Approach to Assessing Study Quality The drawback of the approach I just outlined is that it is univariate Any intercorrelation between study characteristics makes it harder to interpret the results of the moderator test Meta-regression allows us to examine the independent effects of a study quality indicator, controlling for other study quality indicators (and indeed other study characteristics) One drawback is that generally we need a lot of studies to reasonably estimate these models (rough guess is > 30) Example of Process (Screening and Coding) To wrap up, the next few slides show how I might approach study quality issues in a systematic review focused on mentoring interventions for youth (based on DuBois et al., 2011) To keep things relatively simple, I will focus on study quality issues as they relate to internal validity 14

15 Addressing Study Quality at the Screening Stage Focus of review is on effectiveness, so we will include Randomized experiments Quasi-randomized experiments Non-randomized experiments, but only if the study authors control for a pretest of the outcome During title and abstract screening, one question we will ask is What was the research design? We will exclude studies if they did not use one of the above designs Study Quality at the Coding Stage Once we have the set of studies we want to meta-analyze, we will code studies for a variety of characteristics Design (random assignment vs. quasi-random vs. non-random with statistical control) Baseline equivalence on important variables Attrition: loss of participants, both overall and for each group Effect size 15

16 Empirical Assessment of Variations in Study Quality Once coded, I can compare the effects observed in studies with better characteristics with the effects observed in studies with less desirable characteristics. For example: Do randomized and quasi-randomized experiments yield effects similar to those seen in non-randomized experiments? Do study effects vary as a function of the level of overall attrition? Do study effects vary as a function of the magnitude of differential attrition? What is the effect of design (randomized and quasi-randomized vs. non-randomized) while holding the effects of differential attrition constant? Summary The relation between study quality and outcomes is complex Publication in a peer reviewed journal is not a good proxy for study quality Study quality scales don t work as intended and should be avoided Best practice is to Think carefully about how different study quality indicators might operate for your research question Set a few highly defensible study inclusion criteria related to study quality Code other study quality characteristics and investigate their relations to effect size empirically 16

17 P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway Recommended Readings Books Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). Handbook of Research Synthesis and Meta-Analysis, 2 nd ed. New York: Russell Sage. See especially my chapter, on which this presentation was based Articles Glass G. V. (2000). Meta-analysis at 25. Available online at On the web Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version [updated March 2011]. The Cochrane Collaboration, Available from See especially chapter 8 and chapter 13 17

Meta-Analysis De-Mystified: A Step-by-Step Workshop

Meta-Analysis De-Mystified: A Step-by-Step Workshop Meta-Analysis De-Mystified: A Step-by-Step Workshop Eval Café Evaluation Center at WMU January, 2012 Robert McCowen, IDPE robert.h.mccowen@wmich.edu Overview Background and Context Applications of Meta-Analysis

More information

Essential Skills for Evidence-based Practice Understanding and Using Systematic Reviews

Essential Skills for Evidence-based Practice Understanding and Using Systematic Reviews J Nurs Sci Vol.28 No.4 Oct - Dec 2010 Essential Skills for Evidence-based Practice Understanding and Using Systematic Reviews Jeanne Grace Corresponding author: J Grace E-mail: Jeanne_Grace@urmc.rochester.edu

More information

Meta-analyses: analyses:

Meta-analyses: analyses: Meta-analyses: analyses: how do they help, and when can they not? Lee Hooper Senior Lecturer in research synthesis & nutrition l.hooper@uea.ac.uk 01603 591268 Aims Systematic Reviews Discuss the scientific

More information

Controlled Trials. Spyros Kitsiou, PhD

Controlled Trials. Spyros Kitsiou, PhD Assessing Risk of Bias in Randomized Controlled Trials Spyros Kitsiou, PhD Assistant Professor Department of Biomedical and Health Information Sciences College of Applied Health Sciences University of

More information

Introductory: Coding

Introductory: Coding Introductory: Coding Sandra Jo Wilson Editor, Education Coordinating Group Associate Director, Peabody Research Institute Research Assistant Professor, Dept. of Special Education Vanderbilt University,

More information

Cochrane Pregnancy and Childbirth Group Methodological Guidelines

Cochrane Pregnancy and Childbirth Group Methodological Guidelines Cochrane Pregnancy and Childbirth Group Methodological Guidelines [Prepared by Simon Gates: July 2009, updated July 2012] These guidelines are intended to aid quality and consistency across the reviews

More information

In this chapter we discuss validity issues for quantitative research and for qualitative research.

In this chapter we discuss validity issues for quantitative research and for qualitative research. Chapter 8 Validity of Research Results (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.) In this chapter we discuss validity issues for

More information

Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews

Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews Author's response to reviews Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews Authors: Khalid M. AlFaleh (kmfaleh@hotmail.com) Mohammed AlOmran (m_alomran@hotmail.com)

More information

Recent developments for combining evidence within evidence streams: bias-adjusted meta-analysis

Recent developments for combining evidence within evidence streams: bias-adjusted meta-analysis EFSA/EBTC Colloquium, 25 October 2017 Recent developments for combining evidence within evidence streams: bias-adjusted meta-analysis Julian Higgins University of Bristol 1 Introduction to concepts Standard

More information

Fixed Effect Combining

Fixed Effect Combining Meta-Analysis Workshop (part 2) Michael LaValley December 12 th 2014 Villanova University Fixed Effect Combining Each study i provides an effect size estimate d i of the population value For the inverse

More information

1 The conceptual underpinnings of statistical power

1 The conceptual underpinnings of statistical power 1 The conceptual underpinnings of statistical power The importance of statistical power As currently practiced in the social and health sciences, inferential statistics rest solidly upon two pillars: statistical

More information

Free Will and Agency: A Scoping Review and Map

Free Will and Agency: A Scoping Review and Map Kent State University From the SelectedWorks of Paul Fehrmann Summer August 6, 2015 Free Will and Agency: A Scoping Review and Map Paul Fehrmann, Kent State University - Kent Campus Available at: https://works.bepress.com/paul_fehrmann/9/

More information

TRANSLATING RESEARCH INTO ACTION. Why randomize? Dan Levy. Harvard Kennedy School

TRANSLATING RESEARCH INTO ACTION. Why randomize? Dan Levy. Harvard Kennedy School TRANSLATING RESEARCH INTO ACTION Why randomize? Dan Levy Harvard Kennedy School Your background Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How

More information

Foundations of Research Methods

Foundations of Research Methods 1 Foundations of Research Methods Relevant dates and information 09/03/2017 Intermediate test (final test) >> to be defined 1 Things to do: Enrol yourself in the page of the course (obligatory) Sign every

More information

PHO MetaQAT Guide. Critical appraisal in public health. PHO Meta-tool for quality appraisal

PHO MetaQAT Guide. Critical appraisal in public health. PHO Meta-tool for quality appraisal PHO MetaQAT Guide Critical appraisal in public health Critical appraisal is a necessary part of evidence-based practice and decision-making, allowing us to understand the strengths and weaknesses of evidence,

More information

Regression Discontinuity Analysis

Regression Discontinuity Analysis Regression Discontinuity Analysis A researcher wants to determine whether tutoring underachieving middle school students improves their math grades. Another wonders whether providing financial aid to low-income

More information

Previous Example. New. Tradition

Previous Example. New. Tradition Experimental Design Previous Example New Tradition Goal? New Tradition =? Challenges Internal validity How to guarantee what you have observed is true? External validity How to guarantee what you have

More information

Differential Item Functioning

Differential Item Functioning Differential Item Functioning Lecture #11 ICPSR Item Response Theory Workshop Lecture #11: 1of 62 Lecture Overview Detection of Differential Item Functioning (DIF) Distinguish Bias from DIF Test vs. Item

More information

Threats to validity in intervention studies. Potential problems Issues to consider in planning

Threats to validity in intervention studies. Potential problems Issues to consider in planning Threats to validity in intervention studies Potential problems Issues to consider in planning An important distinction Credited to Campbell & Stanley (1963) Threats to Internal validity Threats to External

More information

A Short Primer on Power Calculations for Meta-analysis

A Short Primer on Power Calculations for Meta-analysis A Short Primer on Power Calculations for Meta-analysis Terri Pigott, Associate Provost for Research, Loyola University Chicago Editor, Methods Coordinating Group, Campbell Collaboration A webcast series

More information

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission. The Concept of External Validity Author(s): Bobby J. Calder, Lynn W. Phillips, Alice M. Tybout Source: The Journal of Consumer Research, Vol. 9, No. 3 (Dec., 1982), pp. 240-244 Published by: The University

More information

Impact Evaluation Methods: Why Randomize? Meghan Mahoney Policy Manager, J-PAL Global

Impact Evaluation Methods: Why Randomize? Meghan Mahoney Policy Manager, J-PAL Global Impact Evaluation Methods: Why Randomize? Meghan Mahoney Policy Manager, J-PAL Global Course Overview 1. What is Evaluation? 2. Outcomes, Impact, and Indicators 3. Why Randomize? 4. How to Randomize? 5.

More information

The Regression-Discontinuity Design

The Regression-Discontinuity Design Page 1 of 10 Home» Design» Quasi-Experimental Design» The Regression-Discontinuity Design The regression-discontinuity design. What a terrible name! In everyday language both parts of the term have connotations

More information

Multiple Regression Models

Multiple Regression Models Multiple Regression Models Advantages of multiple regression Parts of a multiple regression model & interpretation Raw score vs. Standardized models Differences between r, b biv, b mult & β mult Steps

More information

GOLDSMITHS Research Online Article (refereed)

GOLDSMITHS Research Online Article (refereed) GOLDSMITHS Research Online Article (refereed) Davies, Martin F. Irrational beliefs and unconditional self-acceptance. II. Experimental evidence linking two key features of REBT Originally published in

More information

Research Questions, Variables, and Hypotheses: Part 2. Review. Hypotheses RCS /7/04. What are research questions? What are variables?

Research Questions, Variables, and Hypotheses: Part 2. Review. Hypotheses RCS /7/04. What are research questions? What are variables? Research Questions, Variables, and Hypotheses: Part 2 RCS 6740 6/7/04 1 Review What are research questions? What are variables? Definition Function Measurement Scale 2 Hypotheses OK, now that we know how

More information

Assignment 4: True or Quasi-Experiment

Assignment 4: True or Quasi-Experiment Assignment 4: True or Quasi-Experiment Objectives: After completing this assignment, you will be able to Evaluate when you must use an experiment to answer a research question Develop statistical hypotheses

More information

Comparing Direct and Indirect Measures of Just Rewards: What Have We Learned?

Comparing Direct and Indirect Measures of Just Rewards: What Have We Learned? Comparing Direct and Indirect Measures of Just Rewards: What Have We Learned? BARRY MARKOVSKY University of South Carolina KIMMO ERIKSSON Mälardalen University We appreciate the opportunity to comment

More information

Lorne A. Becker MD Emeritus Professor SUNY Upstate Medical University. Co-Chair, Cochrane Collaboration Steering Group

Lorne A. Becker MD Emeritus Professor SUNY Upstate Medical University. Co-Chair, Cochrane Collaboration Steering Group Lorne A. Becker MD Emeritus Professor SUNY Upstate Medical University Co-Chair, Cochrane Collaboration Steering Group Many different types Vary in Complexity Trustworthiness of conclusions http://www.ahrq.gov/clinic/uspstf/uspscopd.htm

More information

CHAPTER LEARNING OUTCOMES

CHAPTER LEARNING OUTCOMES EXPERIIMENTAL METHODOLOGY CHAPTER LEARNING OUTCOMES When you have completed reading this article you will be able to: Define what is an experiment Explain the role of theory in educational research Justify

More information

Supplementary Study A: Do the exemplars that represent a category influence IAT effects?

Supplementary Study A: Do the exemplars that represent a category influence IAT effects? Supplement A to Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the Implicit Association Test: II. Method Variables and Construct Validity. Personality and Social Psychology

More information

Choose an approach for your research problem

Choose an approach for your research problem Choose an approach for your research problem This course is about doing empirical research with experiments, so your general approach to research has already been chosen by your professor. It s important

More information

04/12/2014. Research Methods in Psychology. Chapter 6: Independent Groups Designs. What is your ideas? Testing

04/12/2014. Research Methods in Psychology. Chapter 6: Independent Groups Designs. What is your ideas? Testing Research Methods in Psychology Chapter 6: Independent Groups Designs 1 Why Psychologists Conduct Experiments? What is your ideas? 2 Why Psychologists Conduct Experiments? Testing Hypotheses derived from

More information

Math 140 Introductory Statistics

Math 140 Introductory Statistics Math 140 Introductory Statistics Professor Silvia Fernández Sample surveys and experiments Most of what we ve done so far is data exploration ways to uncover, display, and describe patterns in data. Unfortunately,

More information

Meta-Analysis David Wilson, Ph.D. Upcoming Seminar: October 20-21, 2017, Philadelphia, Pennsylvania

Meta-Analysis David Wilson, Ph.D. Upcoming Seminar: October 20-21, 2017, Philadelphia, Pennsylvania Meta-Analysis David Wilson, Ph.D. Upcoming Seminar: October 20-21, 2017, Philadelphia, Pennsylvania Meta-Analysis Workshop David B. Wilson, PhD September 16, 2016 George Mason University Department of

More information

Editorial. From the Editors: Perspectives on Turnaround Time

Editorial. From the Editors: Perspectives on Turnaround Time Editorial From the Editors: Perspectives on Turnaround Time Editors are frequently asked about the turnaround time at their journals, but there is no standard way this simple statistic is defined, making

More information

Psych 1Chapter 2 Overview

Psych 1Chapter 2 Overview Psych 1Chapter 2 Overview After studying this chapter, you should be able to answer the following questions: 1) What are five characteristics of an ideal scientist? 2) What are the defining elements of

More information

A brief history of the Fail Safe Number in Applied Research. Moritz Heene. University of Graz, Austria

A brief history of the Fail Safe Number in Applied Research. Moritz Heene. University of Graz, Austria History of the Fail Safe Number 1 A brief history of the Fail Safe Number in Applied Research Moritz Heene University of Graz, Austria History of the Fail Safe Number 2 Introduction Rosenthal s (1979)

More information

Title:Continuity of GP care is associated with lower use of complementary and alternative medical providers A population-based cross-sectional survey

Title:Continuity of GP care is associated with lower use of complementary and alternative medical providers A population-based cross-sectional survey Author's response to reviews Title:Continuity of GP care is associated with lower use of complementary and alternative medical providers A population-based cross-sectional survey Authors: Anne Helen Hansen

More information

Internal Validity and Experimental Design

Internal Validity and Experimental Design Internal Validity and Experimental Design February 18 1 / 24 Outline 1. Share examples from TESS 2. Talk about how to write experimental designs 3. Internal validity Why randomization works Threats to

More information

RESEARCH METHODS. Winfred, research methods, ; rv ; rv

RESEARCH METHODS. Winfred, research methods, ; rv ; rv RESEARCH METHODS 1 Research Methods means of discovering truth 2 Research Methods means of discovering truth what is truth? 3 Research Methods means of discovering truth what is truth? Riveda Sandhyavandanam

More information

We Can Test the Experience Machine. Response to Basil SMITH Can We Test the Experience Machine? Ethical Perspectives 18 (2011):

We Can Test the Experience Machine. Response to Basil SMITH Can We Test the Experience Machine? Ethical Perspectives 18 (2011): We Can Test the Experience Machine Response to Basil SMITH Can We Test the Experience Machine? Ethical Perspectives 18 (2011): 29-51. In his provocative Can We Test the Experience Machine?, Basil Smith

More information

Guidelines for reviewers

Guidelines for reviewers Guidelines for reviewers Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. This format of

More information

CALIFORNIA STATE UNIVERSITY STANISLAUS DEPARTMENT OF SOCIOLOGY ASSESSMENT MODEL

CALIFORNIA STATE UNIVERSITY STANISLAUS DEPARTMENT OF SOCIOLOGY ASSESSMENT MODEL CALIFORNIA STATE UNIVERSITY STANISLAUS DEPARTMENT OF SOCIOLOGY ASSESSMENT MODEL Introduction The purpose of assessment in education is to create a model that can quantify the degree of program success

More information

CSC2130: Empirical Research Methods for Software Engineering

CSC2130: Empirical Research Methods for Software Engineering CSC2130: Empirical Research Methods for Software Engineering Steve Easterbrook sme@cs.toronto.edu www.cs.toronto.edu/~sme/csc2130/ 2004-5 Steve Easterbrook. This presentation is available free for non-commercial

More information

Chapter 9 Experimental Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.

Chapter 9 Experimental Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters. Chapter 9 Experimental Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.) In this chapter we talk about what experiments are, we

More information

RESEARCH METHODS. Winfred, research methods,

RESEARCH METHODS. Winfred, research methods, RESEARCH METHODS Winfred, research methods, 04-23-10 1 Research Methods means of discovering truth Winfred, research methods, 04-23-10 2 Research Methods means of discovering truth what is truth? Winfred,

More information

The Cochrane Collaboration

The Cochrane Collaboration The Cochrane Collaboration Version and date: V1, 29 October 2012 Guideline notes for consumer referees You have been invited to provide consumer comments on a Cochrane Review or Cochrane Protocol. This

More information

Political Science 15, Winter 2014 Final Review

Political Science 15, Winter 2014 Final Review Political Science 15, Winter 2014 Final Review The major topics covered in class are listed below. You should also take a look at the readings listed on the class website. Studying Politics Scientifically

More information

Experimental Design Part II

Experimental Design Part II Experimental Design Part II Keith Smolkowski April 30, 2008 Where Are We Now? esearch eview esearch Design: The Plan Internal Validity Statements of Causality External Validity Statements of Generalizability

More information

Conclusion. The international conflicts related to identity issues are a contemporary concern of societies

Conclusion. The international conflicts related to identity issues are a contemporary concern of societies 105 Conclusion 1. Summary of the argument The international conflicts related to identity issues are a contemporary concern of societies around the world. It is only necessary to watch the news for few

More information

Meta-Analysis. Zifei Liu. Biological and Agricultural Engineering

Meta-Analysis. Zifei Liu. Biological and Agricultural Engineering Meta-Analysis Zifei Liu What is a meta-analysis; why perform a metaanalysis? How a meta-analysis work some basic concepts and principles Steps of Meta-analysis Cautions on meta-analysis 2 What is Meta-analysis

More information

Structural Approach to Bias in Meta-analyses

Structural Approach to Bias in Meta-analyses Original Article Received 26 July 2011, Revised 22 November 2011, Accepted 12 December 2011 Published online 2 February 2012 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/jrsm.52 Structural

More information

VIRGINIA LAW REVIEW IN BRIEF

VIRGINIA LAW REVIEW IN BRIEF VIRGINIA LAW REVIEW IN BRIEF VOLUME 96 JUNE 15, 2010 PAGES 35 39 REPLY GOOD INTENTIONS MATTER Katharine T. Bartlett * W HILE writing the article to which Professors Mitchell and Bielby have published responses,

More information

Research Prospectus. Your major writing assignment for the quarter is to prepare a twelve-page research prospectus.

Research Prospectus. Your major writing assignment for the quarter is to prepare a twelve-page research prospectus. Department of Political Science UNIVERSITY OF CALIFORNIA, SAN DIEGO Philip G. Roeder Research Prospectus Your major writing assignment for the quarter is to prepare a twelve-page research prospectus. A

More information

Assignment Research Article (in Progress)

Assignment Research Article (in Progress) Political Science 225 UNIVERSITY OF CALIFORNIA, SAN DIEGO Assignment Research Article (in Progress) Philip G. Roeder Your major writing assignment for the quarter is to prepare a twelve-page research article

More information

WWC STUDY REVIEW STANDARDS

WWC STUDY REVIEW STANDARDS WWC STUDY REVIEW STANDARDS INTRODUCTION The What Works Clearinghouse (WWC) reviews studies in three stages. First, the WWC screens studies to determine whether they meet criteria for inclusion within the

More information

Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education.

Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education. The Reliability of PLATO Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO M. Ken Cor Stanford University School of Education April,

More information

Checklist for Randomized Controlled Trials. The Joanna Briggs Institute Critical Appraisal tools for use in JBI Systematic Reviews

Checklist for Randomized Controlled Trials. The Joanna Briggs Institute Critical Appraisal tools for use in JBI Systematic Reviews The Joanna Briggs Institute Critical Appraisal tools for use in JBI Systematic Reviews Checklist for Randomized Controlled Trials http://joannabriggs.org/research/critical-appraisal-tools.html www.joannabriggs.org

More information

Validity and Reliability. PDF Created with deskpdf PDF Writer - Trial ::

Validity and Reliability. PDF Created with deskpdf PDF Writer - Trial :: Validity and Reliability PDF Created with deskpdf PDF Writer - Trial :: http://www.docudesk.com Validity Is the translation from concept to operationalization accurately representing the underlying concept.

More information

Pooling Subjective Confidence Intervals

Pooling Subjective Confidence Intervals Spring, 1999 1 Administrative Things Pooling Subjective Confidence Intervals Assignment 7 due Friday You should consider only two indices, the S&P and the Nikkei. Sorry for causing the confusion. Reading

More information

Better Must Come: Exiting Homelessness in Two Global Cities

Better Must Come: Exiting Homelessness in Two Global Cities 311 Matthew D. Marr (2015) Better Must Come: Exiting Homelessness in Two Global Cities Ithaca, New York: Cornell University Press, pp. 223, $24.95 Marr s book is described as a longitudinal, ethnographic

More information

Reduce Tension by Making the Desired Choice Easier

Reduce Tension by Making the Desired Choice Easier Daniel Kahneman Talk at Social and Behavioral Sciences Meeting at OEOB Reduce Tension by Making the Desired Choice Easier Here is one of the best theoretical ideas that psychology has to offer developed

More information

Opening Comments of Session Chairperson Edward J Mullen, Professor, Columbia University

Opening Comments of Session Chairperson Edward J Mullen, Professor, Columbia University NC2 Methods Network Workshop Systematic Reviews in Social Science Some Essential Challenges Finding a Workable Review Title: Intervention, Target Group & Outcome Session 2: How to choose outcome measures

More information

ON GENERALISING: SEEING THE GENERAL IN THE PARTICULAR AND THE PARTICULAR IN THE GENERAL IN MARKETING RESEARCH

ON GENERALISING: SEEING THE GENERAL IN THE PARTICULAR AND THE PARTICULAR IN THE GENERAL IN MARKETING RESEARCH ON GENERALISING: SEEING THE GENERAL IN THE PARTICULAR AND THE PARTICULAR IN THE GENERAL IN MARKETING RESEARCH Ian F. Wilkinson University of New South Wales Track: Marketing Education Keywords: Marketing

More information

Chapter 11 Nonexperimental Quantitative Research Steps in Nonexperimental Research

Chapter 11 Nonexperimental Quantitative Research Steps in Nonexperimental Research Chapter 11 Nonexperimental Quantitative Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.) Nonexperimental research is needed because

More information

Confidence Intervals On Subsets May Be Misleading

Confidence Intervals On Subsets May Be Misleading Journal of Modern Applied Statistical Methods Volume 3 Issue 2 Article 2 11-1-2004 Confidence Intervals On Subsets May Be Misleading Juliet Popper Shaffer University of California, Berkeley, shaffer@stat.berkeley.edu

More information

PLS 506 Mark T. Imperial, Ph.D. Lecture Notes: Reliability & Validity

PLS 506 Mark T. Imperial, Ph.D. Lecture Notes: Reliability & Validity PLS 506 Mark T. Imperial, Ph.D. Lecture Notes: Reliability & Validity Measurement & Variables - Initial step is to conceptualize and clarify the concepts embedded in a hypothesis or research question with

More information

Methodological Issues in Measuring the Development of Character

Methodological Issues in Measuring the Development of Character Methodological Issues in Measuring the Development of Character Noel A. Card Department of Human Development and Family Studies College of Liberal Arts and Sciences Supported by a grant from the John Templeton

More information

The Science of Research Synthesis: Limiting Bias and Error in Reviews

The Science of Research Synthesis: Limiting Bias and Error in Reviews Bryn Mawr College Scholarship, Research, and Creative Work at Bryn Mawr College Graduate School of Social Work and Social Research Faculty Research and Scholarship Graduate School of Social Work and Social

More information

Asking and answering research questions. What s it about?

Asking and answering research questions. What s it about? 2 Asking and answering research questions What s it about? (Social Psychology pp. 24 54) Social psychologists strive to reach general conclusions by developing scientific theories about why people behave

More information

School of Dentistry. What is a systematic review?

School of Dentistry. What is a systematic review? School of Dentistry What is a systematic review? Screen Shot 2012-12-12 at 09.38.42 Where do I find the best evidence? The Literature Information overload 2 million articles published a year 20,000 biomedical

More information

Standards for the reporting of new Cochrane Intervention Reviews

Standards for the reporting of new Cochrane Intervention Reviews Methodological Expectations of Cochrane Intervention Reviews (MECIR) Standards for the reporting of new Cochrane Intervention Reviews 24 September 2012 Preface The standards below summarize proposed attributes

More information

Instrument for the assessment of systematic reviews and meta-analysis

Instrument for the assessment of systematic reviews and meta-analysis Appendix II Annex II Instruments for the assessment of evidence As detailed in the main body of the methodological appendix (Appendix II, "Description of the methodology utilised for the collection, assessment

More information

Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews

Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews Reviewer's report Title: Reporting and Methodologic Quality of Cochrane Neonatal Review Group Systematic Reviews Version: 1 Date: 26 August 2008 Reviewer: Andreas Lundh Reviewer's report: General The authors

More information

Evidence Informed Practice Online Learning Module Glossary

Evidence Informed Practice Online Learning Module Glossary Term Abstract Associations Attrition Bias Background and Significance Baseline Basic Science Bias Blinding Definition An abstract is a summary of a research article. It usually includes the purpose, methods,

More information

3/12/2011. You Know The Story. What is missing? Balance, Fairness and Bias

3/12/2011. You Know The Story. What is missing? Balance, Fairness and Bias You Know The Story What is missing? Balance, Fairness and Bias 1 Balance, Fairness and Bias Balance, Fairness and Bias Balance, Fairness and Bias What s Missing? 2 What s your opinion? Most news outlets

More information

TACKLING WITH REVIEWER S COMMENTS:

TACKLING WITH REVIEWER S COMMENTS: TACKLING WITH REVIEWER S COMMENTS: Comment (a): The abstract of the research paper does not provide a bird s eye view (snapshot view) of what is being discussed throughout the paper. The reader is likely

More information

Conducting Strong Quasi-experiments

Conducting Strong Quasi-experiments Analytic Technical Assistance and Development Conducting Strong Quasi-experiments Version 1 May 2015 This report was prepared for the Institute of Education Sciences (IES) by Decision Information Resources,

More information

Clever Hans the horse could do simple math and spell out the answers to simple questions. He wasn t always correct, but he was most of the time.

Clever Hans the horse could do simple math and spell out the answers to simple questions. He wasn t always correct, but he was most of the time. Clever Hans the horse could do simple math and spell out the answers to simple questions. He wasn t always correct, but he was most of the time. While a team of scientists, veterinarians, zoologists and

More information

RESPONSE FORMATS HOUSEKEEPING 4/4/2016

RESPONSE FORMATS HOUSEKEEPING 4/4/2016 RESPONSE FORMATS Allyson L. Holbrook Associate Professor of Public Administration and Psychology & Associate Research Professor at the Survey Research Laboratory of the University of Illinois at Chicago

More information

Guidelines for Writing and Reviewing an Informed Consent Manuscript From the Editors of Clinical Research in Practice: The Journal of Team Hippocrates

Guidelines for Writing and Reviewing an Informed Consent Manuscript From the Editors of Clinical Research in Practice: The Journal of Team Hippocrates Guidelines for Writing and Reviewing an Informed Consent Manuscript From the Editors of Clinical Research in Practice: The Journal of Team Hippocrates 1. Title a. Emphasize the clinical utility of the

More information

Having your cake and eating it too: multiple dimensions and a composite

Having your cake and eating it too: multiple dimensions and a composite Having your cake and eating it too: multiple dimensions and a composite Perman Gochyyev and Mark Wilson UC Berkeley BEAR Seminar October, 2018 outline Motivating example Different modeling approaches Composite

More information

INTRODUCTION. Evidence standards for justifiable evidence claims, June 2016

INTRODUCTION. Evidence standards for justifiable evidence claims, June 2016 EVIDENCE STANDARDS: A DIMENSIONS OF DIFFERENCE FRAMEWORK FOR APPRAISING JUSTIFIABLE EVIDENCE CLAIMS 1 David Gough, EPPI-Centre, SSRU, UCL Institute of Education, University College London INTRODUCTION

More information

Chapter 2 Methodological Approach to Assessing the Evidence

Chapter 2 Methodological Approach to Assessing the Evidence Chapter 2 Methodological Approach to Assessing the Evidence Joan Kennelly The approach taken in this book to guide authors in assessing the evidence for their respective topic areas was generated by the

More information

Principles of publishing

Principles of publishing Principles of publishing Issues of authorship, duplicate publication and plagiarism in scientific journal papers can cause considerable conflict among members of research teams and embarrassment for both

More information

Social Biases and Pressures. Critical Thinking

Social Biases and Pressures. Critical Thinking Social Biases and Pressures Critical Thinking Humans are Social Animals We are very sensitive to what others do and think. Most of our deep-rooted beliefs and values come from family and society we grow

More information

VERDIN MANUSCRIPT REVIEW HISTORY REVISION NOTES FROM AUTHORS (ROUND 2)

VERDIN MANUSCRIPT REVIEW HISTORY REVISION NOTES FROM AUTHORS (ROUND 2) 1 VERDIN MANUSCRIPT REVIEW HISTORY REVISION NOTES FROM AUTHORS (ROUND 2) Thank you for providing us with the opportunity to revise our paper. We have revised the manuscript according to the editors and

More information

Introduction to meta-analysis

Introduction to meta-analysis Introduction to meta-analysis Steps of a Cochrane review 1. define the question 2. plan eligibility criteria 3. plan methods 4. search for studies 5. apply eligibility criteria 6. collect data 7. assess

More information

Experimental Design. Dewayne E Perry ENS C Empirical Studies in Software Engineering Lecture 8

Experimental Design. Dewayne E Perry ENS C Empirical Studies in Software Engineering Lecture 8 Experimental Design Dewayne E Perry ENS 623 Perry@ece.utexas.edu 1 Problems in Experimental Design 2 True Experimental Design Goal: uncover causal mechanisms Primary characteristic: random assignment to

More information

Research and science: Qualitative methods

Research and science: Qualitative methods Research and science: Qualitative methods Urban Bilstrup (E327) Urban.Bilstrup@hh.se 140922 2 INTRODUCTION TO RESEARCH AND SCIENTIFIC METHODS Outline Definitions Problem formulation? Aim and goal operational

More information

A Case Study: Two-sample categorical data

A Case Study: Two-sample categorical data A Case Study: Two-sample categorical data Patrick Breheny January 31 Patrick Breheny BST 701: Bayesian Modeling in Biostatistics 1/43 Introduction Model specification Continuous vs. mixture priors Choice

More information

Meta-Analysis and Subgroups

Meta-Analysis and Subgroups Prev Sci (2013) 14:134 143 DOI 10.1007/s11121-013-0377-7 Meta-Analysis and Subgroups Michael Borenstein & Julian P. T. Higgins Published online: 13 March 2013 # Society for Prevention Research 2013 Abstract

More information

Use of the Quantitative-Methods Approach in Scientific Inquiry. Du Feng, Ph.D. Professor School of Nursing University of Nevada, Las Vegas

Use of the Quantitative-Methods Approach in Scientific Inquiry. Du Feng, Ph.D. Professor School of Nursing University of Nevada, Las Vegas Use of the Quantitative-Methods Approach in Scientific Inquiry Du Feng, Ph.D. Professor School of Nursing University of Nevada, Las Vegas The Scientific Approach to Knowledge Two Criteria of the Scientific

More information

Our goal in this section is to explain a few more concepts about experiments. Don t be concerned with the details.

Our goal in this section is to explain a few more concepts about experiments. Don t be concerned with the details. Our goal in this section is to explain a few more concepts about experiments. Don t be concerned with the details. 1 We already mentioned an example with two explanatory variables or factors the case of

More information

Statistical considerations in indirect comparisons and network meta-analysis

Statistical considerations in indirect comparisons and network meta-analysis Statistical considerations in indirect comparisons and network meta-analysis Said Business School, Oxford, UK March 18-19, 2013 Cochrane Comparing Multiple Interventions Methods Group Oxford Training event,

More information

ORIGINS AND DISCUSSION OF EMERGENETICS RESEARCH

ORIGINS AND DISCUSSION OF EMERGENETICS RESEARCH ORIGINS AND DISCUSSION OF EMERGENETICS RESEARCH The following document provides background information on the research and development of the Emergenetics Profile instrument. Emergenetics Defined 1. Emergenetics

More information

THE DEPARTMENT OF VETERANS AFFAIRS HEALTH SERVICES RESEARCH & DEVELOPMENT SERVICE 810 Vermont Avenue Washington, DC 20420

THE DEPARTMENT OF VETERANS AFFAIRS HEALTH SERVICES RESEARCH & DEVELOPMENT SERVICE 810 Vermont Avenue Washington, DC 20420 HSR&D Service Document #39-32-010 THE DEPARTMENT OF VETERANS AFFAIRS HEALTH SERVICES RESEARCH & DEVELOPMENT SERVICE 810 Vermont Avenue Washington, DC 20420 June 1989 Common Methodological Problems in Investigator

More information

Chapter 1 Data Types and Data Collection. Brian Habing Department of Statistics University of South Carolina. Outline

Chapter 1 Data Types and Data Collection. Brian Habing Department of Statistics University of South Carolina. Outline STAT 515 Statistical Methods I Chapter 1 Data Types and Data Collection Brian Habing Department of Statistics University of South Carolina Redistribution of these slides without permission is a violation

More information

Sampling Controlled experiments Summary. Study design. Patrick Breheny. January 22. Patrick Breheny Introduction to Biostatistics (BIOS 4120) 1/34

Sampling Controlled experiments Summary. Study design. Patrick Breheny. January 22. Patrick Breheny Introduction to Biostatistics (BIOS 4120) 1/34 Sampling Study design Patrick Breheny January 22 Patrick Breheny to Biostatistics (BIOS 4120) 1/34 Sampling Sampling in the ideal world The 1936 Presidential Election Pharmaceutical trials and children

More information