SOCQ121/BIOQ121. Session 8. Methods and Methodology. Department of Social Science. endeavour.edu.au

Transcription

1 SOCQ121/BIOQ121 Session 8 Methods and Methodology Department of Social Science endeavour.edu.au Endeavour College of Natural Health endeavour.edu.au 1

2 Review Progress report to the person next to you on your literature review assignment. Online students: please consider how you would explain your topic to one of your peers. You may like to post something online. By now you should understand exactly what is required to write your next assignment. If you are unsure ask. Endeavour College of Natural Health endeavour.edu.au 2

3 Review Types of research used in health Endeavour College of Natural Health endeavour.edu.au 3

4 Today s Topics 1. Methods and methodology 2. Sampling 3. Validity & Reliability 4. Observation 5. Surveys, Interviews & Questionnaires 6. Methods for CM research Endeavour College of Natural Health endeavour.edu.au 4

5 Introduction to today lecture Today s session covers methods and methodology and the questions we need to ask ourselves to help make decisions about the quality of the research Questions like does the method of research match the question, are the participants well chosen, are the conclusions well founded? Endeavour College of Natural Health endeavour.edu.au 5

6 Section 1: Methods and Methodologies In practice method and methodology are often used interchangeably, however: A methodology is the theory of the method o the underpinning knowledge or ideas that inform the method o the underlying theoretical model, epistemological framework or research paradigm o it describes the overall research strategy, including the methods A method is the story of the journey through the research process o it describes a series of processes or models Endeavour College of Natural Health endeavour.edu.au 6

7 A methods section should: o contain information on what was done to obtain the data o describe the study in enough detail that other researchers could replicate it (statistical data) o should provide enough detail for the method to be adapted for other studies (small scale or qualitative) Endeavour College of Natural Health endeavour.edu.au 7

8 Methods sections: may include but are not limited to o Study Design o Population group o Sampling strategy o Inclusion and Exclusion criteria o Data collection tools o Treatment and placebo described in detail o Including dosage and form if applicable o Length of the trial o Statement of ethical procedure o Sample size calculation o Statistical analysis methods o Theoretical underpinnings o Any disclosures that may affect the study Endeavour College of Natural Health endeavour.edu.au 8

9 Section 2: Scientific Sampling o Identify Target population Strategies inclusion/exclusion e.g. all people who consult naturopaths o Identify accessible population o may be limited by budget, geography e.g. people who consult naturopaths in person at a clinic o Construct sampling frame o e.g. all patients at a particular clinic o Choose sampling method o e.g. random o Determine sample size. Endeavour College of Natural Health endeavour.edu.au 9

10 Choosing a Population o Population or Target Population All units of a group to which the findings of a study are to be applied o Representative Sample Sample that matches a population for all core characteristics o Accessible population A portion of the target population that has a chance to be selected Endeavour College of Natural Health endeavour.edu.au 10

11 o Sample sub-set of a population Population serves as a reference group for drawing conclusions about a population Endeavour College of Natural Health endeavour.edu.au 11

12 Sampling Error o A census of an entire population gives us accurate statistics The Australian Bureau of Statistics o If we don t survey a whole population then our statistics are estimates of what might be happening in a particular population. It is likely that our estimate is not 100% accurate o The difference between the estimate and the true measure can be calculated the margin of error o The larger the margin of error the less confidence we have in the accuracy of our results o The larger the sample the smaller the margin of error Endeavour College of Natural Health endeavour.edu.au 12

13 Section 3: Reliability & Validity o Reliability = are our measures consistent o Validity = are we researching what we intend to research Endeavour College of Natural Health endeavour.edu.au 13

14 Reliability If something is reliable it means we can trust it e.g. the car will start In research that means that we are getting the same results when we retest under the same or similar conditions, so we can be reasonably sure of our results If we are getting many different results our measurement instrument or technique is probably not very reliable we can t rely on our research instrument to get us a reliable result Endeavour College of Natural Health endeavour.edu.au 14

15 Reliability Reliability is the extent to which a measurement s outcomes are consistent when undertaken in similar conditions. In other words, it is the measurement s reproducibility st measure Low reliability 2nd measure 3rd measure 4th measure st measure High reliability 2nd measure 3rd measure 4th measure Coaley, 2014 Endeavour College of Natural Health endeavour.edu.au 15

16 Validity If we say something is valid we generally mean it is well grounded, true or logically correct Endeavour College of Natural Health endeavour.edu.au 16

17 Validity Validity is the degree to which the measurement (method) is accurate and measures what it was said to be measuring. measured phenomenon Low validity measurement outcome measured phenomenon measurement outcome High validity Coaley, 2014 Endeavour College of Natural Health endeavour.edu.au 17

18 Validity If we research something and come to a conclusion we generally wish our conclusions to be correct, at least for a particular group of people, or in a particular situation, and even sometimes for entire populations of people. If we can consider our conclusions to be correct (disprove our null hypothesis) then we consider the research to be valid. However, if, for example, our results are being affected by something outside the research, or something that has not been taken into account in the research design, or investigator bias is causing us to interpret something incorrectly, then our results are not valid (well grounded, true, correct ) Endeavour College of Natural Health endeavour.edu.au 18

19 Factors that affect validity There are many different types of validity in qualitative, quantitative and mixed methods research and a variety of factors and practices that affect validity For example: effects and biases poor sample selection controls such as blinding not enforced the variation in the dependent variable (the results) are attributable to something other than the treatment or intervention researchers choice of material for publication researchers propensity to interpret data in certain ways Endeavour College of Natural Health endeavour.edu.au 19

20 Factors that affect validity Man prefers to believe what he prefers to be true Francis Bacon Endeavour College of Natural Health endeavour.edu.au 20

21 Factors that affect validity Effects - Error and Bias Hawthorne Effect o People change their behaviour, simply because they are being studied o One of the hardest inbuilt biases to eliminate or factor into the design Rosenthal Effect o The research participants start to act according to the researchers expectations Endeavour College of Natural Health endeavour.edu.au 21

22 Section 4: Observation Observation = using our senses to recognize and record facts Facts are events that can be directly, empirically & repeatedly observed Observation is a cornerstone of empirical research Endeavour College of Natural Health endeavour.edu.au 22

23 In history empirical observation techniques have been used to both support and refute complementary medicine e.g. homeopathy Hahnemann - Materia Medica Pura In a reply to critics of his Organon of the Art of Healing entitled Nota Bene For My Reviewers Hahnemann notes it does what no medical doctrine, no system of physic, no so-called therapeutics ever did or could do, it insists on being judged by the result Endeavour College of Natural Health endeavour.edu.au 23

24 Section 5: Surveys, Interviews & Questionnaires The hierarchy of evidence gives us a means to compare different studies against one another However not all research papers are equal It is important to be critical and consistent in your analysis of research Having a set of questions that you ask yourself can help you develop your skills in critical analysis Endeavour College of Natural Health endeavour.edu.au 24

25 Case control and cohort studies RCT s and confounding variables Endeavour College of Natural Health endeavour.edu.au 25

26 Questions to ask about methods used in surveys, interviews & questionnaires If a quantitative method, is the sample representative of the target population? How has the external validity been affected by the exclusion criteria? Has an appropriate sampling method been selected? If a qualitative method, are the participants appropriate to the research question? Endeavour College of Natural Health endeavour.edu.au 26

27 Questions to ask about methods used in surveys, interviews & questionnaires. Has sample size appropriate to the method been selected? Do the research findings relate to and support the authors conclusions? Do you understand how the findings from the paper relate to the wider knowledge in the field? Have limitations been acknowledged? Endeavour College of Natural Health endeavour.edu.au 27

28 Section 6: Methods for CM Research Endeavour College of Natural Health endeavour.edu.au 28

29 Methods for CM research Lack of an established pharmacological or physiological basis for therapies in CM = vulnerability to criticism of some CM modalities Pseudoscience Quakery Charlatanism. We will discuss critiques of CM in session 12 Research into practice Endeavour College of Natural Health endeavour.edu.au 29

30 CM Industry response to critiques Emphasis on scientific evidence EBM & EBP, meta-analyses, randomised controlled double blind studies and the hierarchy of evidence Where such information exists it is important to be able to distinguish the quality of the research Endeavour College of Natural Health endeavour.edu.au 30

31 CM methods & ongoing critical evaluation Current responses to best practice in health and wellbeing are founded on scientific research findings However, it is important to be critical of how the scientific method is used to legitimate poor practices The Cochrane Collaboration in partnership with WHO systematically organises medical information according to the principles of EBM Endeavour College of Natural Health endeavour.edu.au 31

32 Exercise - thinking critically about research and evidence Peter Gotzsche, founder of the Cochrane Collaboration, visits Australia to talk about dangers of prescription drugs Please follow the link below, read the article & answer the questions on the next slide Endeavour College of Natural Health endeavour.edu.au 32

33 Exercise questions Answer the following questions (in groups). In class discussion. 1. What is the name of the worlds foremost body in assessing clinical trials? 2. Does the system of drug regulation offer protections for patients? 3. Who funds clinical trials and why? 4. Think of some reasons why drug companies might withhold information about drugs 5. How do researcher access unpublished and withheld information? 6. Why should we not be prescribing drugs on the smallest indication that they might be successful? 7. What are some of the ethical issues evident in the video and article? 8. Why might drugs for urinary incontinence be prescribed even though evidence suggests they don t work? Endeavour College of Natural Health endeavour.edu.au 33

34 Consumer response to critiques of CM Despite critiques, in Australia, CM is a 4 Billion dollar industry with around 4% predicted annual growth Increased health consciousness and health literacy are a key driver of growth in the industry Endeavour College of Natural Health endeavour.edu.au 34

35 Methodological response to critiques Alongside methods emphasised in the hierarchy of evidence exist a range of qualitative & mixed methods, health service reviews evaluation research and other methods designed to measure utility, efficacy studies, outcomes and effectiveness research, public impact studies of patient relevance in the real world And a re-valuation of other types of knowledge intuitive, experiential Endeavour College of Natural Health endeavour.edu.au 35

36 Discussion Forum question session 9 Please post your answers in discussions forum for session 9 on LMS. Due date for this post is pm on Sunday following session 9. Pick one (any) of the research studies that you are reviewing for assessment 3 (literature review). On the Discussion Forum post a correct reference of the research article and answer following questions: How would you evaluate reliability of this study? Why? How would you evaluate validity of this study? Why? Endeavour College of Natural Health endeavour.edu.au 36

37 What you should now know o You should now understand the difference between a method and a methodology and know that the terms are often used interchangeably o You should understand the significance of reliability and validity in research o You should understand that best practice in research is constantly contested Endeavour College of Natural Health endeavour.edu.au 37

38 Large scale studies Next Session o Understanding basic statistics that you can find in research papers o Understanding figures, charts and tables o You should read chapter five of your textbook: Statistics for non-statisticians (How to Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh) Endeavour College of Natural Health endeavour.edu.au 38

39 Next session: Reading You should be able to answer following questions: 1. Why is it necessary to check baseline differences in groups in comparative clinical trials? 2. What is the difference between ordinal and nominal data? 3. What is the difference between parametric and non-parametric tests? 4. Greenhalgh states that if data cannot be transformed into a smooth pattern, such as a bell curve, then linear regression is inappropriate. Why? 5. Is transforming data to achieve a normal distribution cheating? (p. 63). 6. Why should you be suspicious of strange sounding tests? 7. Why should you be suspicious of statistics based on sub-samples of the original research sample? 8. What does the term paired data refer to? (p. 68) 9. What do p values and confidence intervals tell us? Endeavour College of Natural Health endeavour.edu.au 39