Experiments and Observational Studies. Characteristics of a Study

Transcription

1 Experiments and Observational Studies Characteristics of a Study

2 Thought Questions 1. In studies to determine the relationship between two conditions (activities, traits, etc.), one of them is often defined as the explanatory (independent) variable and the other as the outcome or response (dependent) variable. In an experiment to determine whether the drug Memox improves cognition of patients with moderate to severe Alzheimer s disease, whether or not the patient received Memox is one variable, and cognitive score is the other. Which is the explanatory variable (EV) and which is the response variable (RV)? A. Memox is RV, Cognitive Score is EV B. Memox is EV, Cognitive Score is RV

3 Thought Questions 2. When experimenters want to compare two treatments, such as an old and a new drug, they use randomization to assign the participants to the two conditions. If you had 50 people participate in such a study, how would you go about randomizing them? Why shouldn t the experimenter decide which people should get which treatment?

4 Randomized, Comparative Experiments An experiment is a study design that allows us to show a cause-andeffect relationship. In an experiment, the experimenter must identify at least one explanatory variable, called a factor (Example: Drug), to manipulate and at least one response variable to measure. An experiment: Manipulates factor levels to create treatments (Example: Drug A, Drug B, Dummy Drug (Placebo)) Randomly assigns subjects to these treatment levels. Compares the responses of the subject groups across treatment levels. A treatment maybe the combination of more than one factor (Example: Drug/Chemo for cancer patients)

5 Randomized, Comparative Experiments In general, the individuals on whom or which we experiment are called experimental units. When humans are involved, they are commonly called subjects or participants.

6 Example of a Randomized Experiment Does Aspirin Prevent Heart Attacks? Physicians Health Study (1988) 5-year randomized experiment 22,071 male physicians (40 to 84 years old). Group 1: took ordinary aspirin tablet every other day. Group 2: took placebo (looked like aspirin but no active ingredients).

7 The Four Principles of Experimental Design 1. Control (of certain variables and the environment) We control sources of variation other than the factors we are testing by making conditions as similar as possible for all treatment groups. Making Generalizations from the experiment to other levels of the controlled factor can be risky. Example: Suppose we test two laundry detergents and carefully control the water temperature at 180 degrees Fahrenheit. This would reduce the variation in our results due to water temperature, but what would that say about the detergents performance in cold water?

8 The Four Principles of Experimental Design 2. Randomize: Randomization allows us to equalize the effects of unknown or uncontrollable sources of variation. It does not eliminate the effects of these sources, but it spreads them out across the treatment levels so that we can see past them. Without randomization, you do not have a valid experiment and will not be able to use the powerful methods of Statistics to draw conclusions from your study.

9 The Four Principles of Experimental Design Randomization Example: Quitting Smoking with Nicotine Patches Nicotine and control "look-alike" patches were randomly assigned to participants. Baseline values were compared for the two groups on numerous characteristics. For example, the mean age of those receiving the nicotine patches was 42.8, while it was 43.6 for those with the control patches. Because they examined comparability on 28 characteristics, a few showed substantial differences in the two groups, as would be expected just by chance. For instance, 85% of those receiving the nicotine patches were employed, while only 73.3% of those receiving the control patches were employed.

10 The Four Principles of Experimental Design 3. Replicate: Repeat the experiment, applying the treatments to a number of subjects. The outcome of an experiment on a single subject is an anecdote, not data.

11 The Four Principles of Experimental Design 4. Block: Sometimes, attributes of the experimental units that we are not studying and that we can t control may nevertheless affect the outcomes of an experiment. If we group similar individuals together and then randomize within each of these blocks, we can remove much of the variability due to the difference among the blocks. Example: Allocating two six player teams from a pool of 12 children. If there were two 12 year olds and ten 6-years. Randomization might be a problem.

12 The Four Principles of Experimental Design Example of Blocking Blood pressure study comparing either active treatment to a placebo. You are randomly assigning the drug to mild/moderate hypertensive. The study is 8 weeks in duration. In this class of people the baseline diastolics will be between You will see efficacy (change from baseline) for an active agent (drug) of the order of 0-40, while for a placebo you may see values from 0-20.

13 The Four Principles of Experimental Design Example of Blocking One critical baseline factor- newly diagnosed patients vs previously diagnosed patients. Newly diagnosed patients, will have a strong response regardless of whether they are in the active or placebo group. Why?

14 Diagrams of Experiments It s often helpful to diagram the procedure of an experiment. The following diagram emphasizes the random allocation of subjects to treatment groups, the separate treatments applied to these groups, and the ultimate comparison of results: Slide 13-14

15 SAS Programing A SAS program is a sequence of steps that the user submits for execution. Raw Data SAS Data Set DATA steps are typically used to create SAS data sets. DATA Step SAS Data Set PROC Step Report PROC steps are typically used to process SAS data sets (that is, generate reports and graphs, manage data, and sort data). 15

16 SAS Programs data work.newsalesemps; length First_Name $ 12 Last_Name $ 18 Job_Title $ 25; infile 'newemps.csv' dlm=','; input First_Name $ Last_Name $ Job_Title $ Salary; run; DATA Step proc print data=work.newsalesemps; run; proc means data=work.newsalesemps; class Job_Title; var Salary; run; PROC Step PROC Step 16 p102d01

17 SAS Windowing Environment In the SAS windowing environment, windows are used to edit and execute programming statements, display the log, output, and Help facility, and more. 17

18 Three Primary Windows In the SAS windowing environment, you submit and view the results of a SAS program using three primary windows. contains the SAS program to submit. contains information about the processing of the SAS program, including any warning and error messages. contains reports generated by the SAS program. 18

19 Editor Windows Enhanced Editor Program Editor Only available in the Windows operating environment Default editor for Windows operating environment Multiple instances of the editor can be open at one time Code does not disappear after it is submitted Incorporates color-coding as you type Available in all operating environments Default editor for all operating environments except Windows Only one instance of the editor can be open at one time Code disappears after it is submitted Incorporates color-coding after you press ENTER 19

20 Editor Windows Enhanced Editor Program Editor 20

21 Log Window Partial SAS Log 33 data work.newsalesemps; 34 length First_Name $ 12 Last_Name $ Job_Title $ 25; 36 infile 'newemps.csv' dlm=','; 37 input First_Name $ Last_Name $ 38 Job_Title $ Salary; 39 run; NOTE: The infile 'newemps.csv' is: File Name=S:\Workshop\newemps.csv, RECFM=V,LRECL=256 NOTE: 71 records were read from the infile 'newemps.csv'. The minimum record length was 28. The maximum record length was 47. NOTE: The data set WORK.NEWSALESEMPS has 71 observations and 4 variables proc print data=work.newsalesemps; 42 run; NOTE: There were 71 observations read from the data set WORK.NEWSALESEMPS. 21

22 Output Window Partial PROC PRINT Output Obs First_Name Last_Name Job_Title Salary 1 Satyakam Denny Sales Rep. II Monica Kletschkus Sales Rep. IV Kevin Lyon Sales Rep. I Petrea Soltau Sales Rep. II Marina Iyengar Sales Rep. III Shani Duckett Sales Rep. I Fang Wilson Sales Rep. II Michael Minas Sales Rep. I Amanda Liebman Sales Rep. II Vincent Eastley Sales Rep. III Viney Barbis Sales Rep. III Skev Rusli Sales Rep. II Narelle James Sales Rep. III Gerry Snellings Sales Rep. I Leonid Karavdic Sales Rep. II 27860

23 Output Window PROC MEANS Output The MEANS Procedure Analysis Variable : Salary N Job_Title Obs N Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ Sales Rep. I Sales Rep. II Sales Rep. III Sales Rep. IV Temp. Sales Rep ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ 23

24 SAS Code Random Allocation of Subjects to Treatments DATA RANDOM; DO SUBJ = 1 TO 20; GROUP=RANUNI(0); OUTPUT; END; RUN; proc print;run;

25 SAS Output Random Allocation of Subjects to Treatments

26 SAS Random Allocation of Subjects to Treatments PROC RANK DATA=RANDOM GROUPS=2 OUT=SPLIT; RUN; VAR GROUP; PROC FORMAT; RUN; VALUE GRPFMT 0='CONTROL' 1='TREATMENT'; PROC PRINT DATA=SPLIT NOOBS; TITLE "Subject Group Assignments"; VAR SUBJ GROUP; FORMAT GROUP GRPFMT.; RUN;

27 SAS Output Random Allocation of Subjects to Treatments

28 Does the Difference Make a Difference? How large do the differences need to be to say that there is a difference in the treatments? Differences that are larger than we d get just from chance alone are called statistically significant. We ll talk more about statistical significance later on. For now, the important point is that a difference is statistically significant if we don t believe that it s likely to have occurred only by chance.

29 Control Treatments - Placebo Often, we want to compare a situation involving a specific treatment to the status quo situation. Often simply applying any treatment can induce an improvement. To separate out the effects of the treatment of interest, we can use a control treatment that mimics the treatment itself. A fake treatment that looks just like the treatment being tested is called a placebo. Placebos are the best way to blind subjects from knowing whether they are receiving the treatment or not.

30 Blinding There are two main classes of individuals who can affect the outcome of the experiment: those who could influence the results (subjects, treatment administrators, technicians) those who evaluate the results (judges, treating physicians, etc.) When all individuals in either one of these classes are blinded, an experiment is said to be single-blind. When everyone in both classes is blinded, the experiment is called double-blind.

31 The Best Experiments are usually: randomized. comparative. double-blind. placebo-controlled.

32 Eletriptan for the Acute Treatment of Migraine in Adolescents: Results of a Double-Blind, Placebo- Controlled Trial Background. Eletriptan is a potent agonist with proven efficacy in the acute treatment of migraine in adults. Objective. To evaluate the efficacy and tolerability of eletriptan 40 mg versus placebo in adolescent patients (aged 12-17).

33 Eletriptan for the Acute Treatment of Migraine in Adolescents: Results of a Double-Blind, Placebo- Controlled Trial Methods. A multicenter, double-blind, placebocontrolled trial was conducted comparing 40 mg of oral eletriptan with placebo for the treatment of migraine in adolescent patients. The primary efficacy endpoint was 2-hour headache response, and a number of secondary endpoints were also evaluated.

34 Eletriptan for the Acute Treatment of Migraine in Adolescents: Results of a Double-Blind, Placebo- Controlled Trial Results. Of 274 patients who treated a migraine attack, 267 were evaluated for efficacy (n = 138 eletriptan; n=129 placebo) at 2 hours post-dose. There was no significant difference in 2-hour headache response for eletriptan 40 mg versus placebo (57% vs 57%), and no significant improvements were observed for any of the outcomes at 1 or 2 hours post-dose.

35 Designing a Good Observational Study In an observational study, researchers don t assign choices; they simply observe them. Manipulation occurs naturally, not imposed. Can t assume the explanatory variable is the only one responsible for any observed differences in the response variable. Case-Control Studies attempt to include an appropriate control group.

36 Designing a Good Observational Study Retrospective or Prospective Studies Prospective: participants followed into future, and events recorded. Retrospective: participants are asked to recall past events. Which is better and why?

37 Using the Past as a Source of Data The problem: In some observational studies participants are required to remember the past. Partial solution: Use official records if they exist, rather than relying on memory. If possible, use prospective study. Example: Pearson, Ross and Dawes (1992, pgs ) reported on a study in which chronic headache sufferers were asked to keep a diary of the intensity of their pain for a week. At the end of the week, they were asked to rate their current level of pain, then to recall the maximum and minimum pain they had felt during the week. Those who were currently suffering from a high level of pain overestimated their prior levels of pain, while those with current low levels underestimated their prior pain.

38 Example: Baldness and Heart Attacks Observational study: compared 665 men admitted to hospital with 1 st heart attack to 772 men (same age group) admitted to same hospitals for other reasons. Percent with pattern baldness higher for heart attack group (42%) compared to no heart attack (34%). Included adjustments for age and other heart attack risk factors. Can you think of any differences in the two groups that might causes baldness?

39 Difficulties in Observational Studies Extending the Results Inappropriately The problem: Because many observational studies use convenience or volunteer samples, it may not be valid to extend the results to other groups. Example : Baldness and Heart Attack Observational study only used men who were hospitalized. Should consider whether results should be extended to all men.

40 Difficulties in Observational Studies Confounding Variables A confounding variable is related to the explanatory variable, and affects the response variable. The effect of a confounding variable on the response variable cannot be separated from the effect of the explanatory variable on the response variable.

41 Difficulties and Disasters in Observational Studies Study of the relationship between smoking during pregnancy and child s subsequent IQ a few years after birth. Explanatory variable: whether or not the mother smoked during pregnancy Response variable: subsequent IQ of the child Women who smoke also have poor nutrition, lower levels of education, or lower income. Possible Confounding Variables: Mother s nutrition, education, and income. The effect of these variables (confounding effect) on the child s IQ cannot be separated from the effects of smoking. They are likely to differ for smokers and non smokers and are likely to also affect the child s IQ.

42 Temporal Patterns of Cocaine Use in Pregnancy Seventy-five cocaine-using women enrolled in a comprehensive perinatal care program were divided into two groups: those who used cocaine in only the first trimester of pregnancy (group 1 [N = 23]) and those who used cocaine throughout pregnancy (group 2 [N = 52]). Perinatal outcomes of these pregnancies were compared with perinatal outcomes of a matched group of obstetric patients with no history or evidence of substance abuse (group 3 [N = 40]).

43 Temporal Patterns of Cocaine Use in Pregnancy Can we conclude that cocaine use causes reduction in mean birth weight?

44 Go easy on the coffee, you could start seeing things Daily Mail, January 14 th, 2009 Drinking cup after cup of coffee dramatically increases the risk of hallucinating, research shows. Healthy young men and women who had more than seven cups of instant coffee a day were three times more likely to hear or see things that were not there than those who limited their intake to less than a cup. Confirming the link could lead to new treatments for those who suffer severe hallucinations, including schizophrenics, some victims of child abuse and the recently bereaved.

45 Go easy on the coffee, you could start seeing things Daily Mail, January 14 th, 2009 The Durham University researchers asked 219 students to document their caffeine intake, working on the principle that a cup of instant coffee contains 45mg of caffeine. The volunteers were also asked how often they suffered hallucinations. The high caffeine users were three times as likely to have had problems as those who rarely drank coffee. Large amounts of caffeine also made people more likely to think they could sense the presence of ghosts, the journal Personality and Individual Differences reports.

46 Study: The effect of caffeine and stress on auditory hallucinations in a non-clinical sample Abstract Both the diathesis-stress model and the continuum theory of schizophrenia attempt to explain the mechanism by which stress may facilitate the expression of the symptoms of schizophrenia in non-clinical samples. Caffeine has also recently been reported to increase proneness to hallucinate. In this study, 92 non-clinical participants were assigned to either a high or a low stress condition and a high or allow caffeine condition on the basis of self-report.

47 Study: The effect of caffeine and stress on auditory hallucinations in a non-clinical sample After they had been primed, the participants were asked to listen to white noise and to report each time they hear the song White Christmas during the white noise. The song was never played. The results indicated that the interaction of stress and caffeine had a significant effect on the reported frequency of hearing White Christmas. The results demonstrated that high caffeine levels in association with high levels of stressful life events interacted to produce higher levels of hallucination in non-clinical participants,

48 Study: The effect of caffeine and stress on auditory hallucinations in a non-clinical sample Ninety-two, non-clinical volunteer individuals (35 males and 57 females) participated in the study. The two predictor variables, caffeine and stress were measured using self-report questionnaires The self-reported level of stress during the past year was measured using the perceived stress questionnaire (PSQ) which has been previously demonstrated to have high test retest reliability. Hallucination proneness was also measured using a selfreport scale.

49 Study: The effect of caffeine and stress on auditory hallucinations in a non-clinical sample

50 Study: The effect of caffeine and stress on auditory hallucinations in a non-clinical sample

51 Study: Erroneous analyses of interactions in neuroscience: a problem of significance In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05).

52 Study: Erroneous analyses of interactions in neuroscience: a problem of significance We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience.