Introduction to Design of Experiments

Transcription

1 Introduction to Design of Experiments Martin L Lesser, PhD Director, Biostatistics Unit Feinstein Institute for Medical Research Professor, Department of Molecular Medicine, Department of Population Health, Hofstra North Shore-LIJ School of Medicine Acknowledgments: Nina Kohn, MA and Lisa Rosen, ScM

2 CME Disclosure Statement The North Shore LIJ Health System adheres to the ACCME s new Standards for Commercial Support. Any individuals in a position to control the content of a CME activity, including faculty, planners, and managers, are required to disclose all financial relationships with commercial interests. All identified potential conflicts of interest are thoroughly vetted by the North Shore-LIJ for fair balance and scientific objectivity and to ensure appropriateness of patient care recommendations. Course Director and Course Planners, Kevin Tracey, MD, Cynthia Hahn, Emmelyn Kim, MPH, Tina Chuck, MPH have nothing to disclose. Martin L Lesser, PhD has nothing to disclose.

3 Introduction The goal of proper experimental design is to: Reduce or eliminate confounding Promote efficient use of resources to attain maximum information, precision, and accuracy Design and method of analysis are interrelated Objective of this presentation: Student should appreciate the variety of experimental designs available

4 The Feinstein Institute Investigator at Work

5 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

6 Overview We will consider experimental designs that address: H 0 : μ 1 = μ 2 = = μ k H A : Not all μ s are equal where μ i is the mean for treatment or condition i

7 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

8 Example Comparison of 5-FU vs Paclitaxel in colon cancer cells (adenocarcinoma) Outcome= cell kill (% of cells killed) Based on resources, can do 14 experiments: 14 wells of cancer cells: 7 randomly assigned to 5- FU treatment; 7 to paclitaxel

9 Single Factor (2 levels) Completely Randomized Design Treatment Treatment A Treatment B (5-FU) Y 11 Y 12 Y 13 Y 14 Y 15 Y 16 Y 17 (Paclitaxel) Y 21 Y 22 Y 23 Y 24 Y 25 Y 26 Y 27 Y ij = % cells killed in sample * Typically analyzed using two-sample t-test (or Mann-Whitney test)

10 A Bit of a Twist For practical reasons, only 2 experiments can be processed on a given day Here s how the experiments were actually conducted

11 Schedule of Experiments Date Treatments Tested 7/13 A, A 7/15 A, B 7/16 A, B 7/17 A, A A = 5-FU B = Paclitaxel 7/20 B, B 7/22 A, B 7/23 B, B What if there is an effect of date on the results? e.g., due to conditions, temperature, operator, or an unknown factor? If so, then date is known as a confounding variable or confounder. Is there an improved design that eliminates the effect of date?

12 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

13 The Randomized Block Design Date Treatments Tested 7/13 A, B 7/15 A, B 7/16 A, B 7/17 A, B 7/20 A, B A = 5-FU B = Paclitaxel 7/22 A, B 7/23 A, B The experiments are blocked on date. Date is the blocking factor. This allows for a comparison between A and B within a date, thus cancelling out the effect of date in the full analysis. Date is no longer a confounder. [Typically analyzed as a randomized block ANOVA (or Friedman test)]

14 Single Factor (2 levels) Completely Randomized Design Treatment Date Treatment A Treatment B (5-FU) (Paclitaxel) 7/13 Y 11 Y 21 7/15 Y 12 Y 22 7/16 Y 13 Y 23 7/17 Y 14 Y 24 7/20 Y 15 Y 25 7/22 Y 16 Y 26 7/23 Y 17 Y 27

15 Common Blocking Factors Patient Date Gender Cell line Type of disease Any variable that might be a strong confounder

16 What is Confounding? Aim: to determine whether or not there is an association between an explanatory factor (E) and an outcome (O) A third variable (C) is a confounding variable if C is related to E and, independent of E, C is related to O. A confounding variable can create the appearance of an association that doesn t exist or can mask an association that does exist.

17 Confounding E O C E O C E O C E is associated with O and C is NOT a confounder E is associated with O and C is a confounder E is not associated with O and C is a confounder

18 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

19 Single Factor (k>2 levels) Completely Randomized Design Treatment Treatment 1 Treatment 2 Treatment 3 Treatment 4 Treatment 5 Y 11 Y 12 Y 13 Y 14 Y 15 Y 16 Y 17 Y 21 Y 22 Y 23 Y 24 Y 25 Y 26 Y 27 Y 31 Y 32 Y 33 Y 34 Y 35 Y 36 Y 37 Y 41 Y 42 Y 43 Y 44 Y 45 Y 46 Y 47 Y 51 Y 52 Y 53 Y 54 Y 55 Y 56 Y 57 * Typically analyzed using one-way ANOVA (or Kruskal-Wallis test)

20 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

21 The Randomized Block Design Date Treatments Tested 7/13 A, B, C, D, E 7/15 A, B, C, D, E 7/16 A, B, C, D, E 7/17 A, B, C, D, E 7/20 A, B, C, D, E 7/22 A, B, C, D, E 7/23 A, B, C, D, E The experiments are blocked on date. Date is the blocking factor. This allows for a comparison between all 5 conditions (levels) within a date, thus cancelling out the effect of date in the full analysis. Date is no longer a confounder. [Typically analyzed as a randomized block ANOVA (Friedman test)]

22 Single Factor (k>2 levels) Randomized Complete Block Design Treatment Date Treatment 1 Treatment 2 Treatment 3 Treatment 4 7/13 7/15 7/16 7/17 7/20 7/22 7/23 Y 11 Y 12 Y 13 Y 14 Y 15 Y 16 Y 17 Y 21 Y 22 Y 23 Y 24 Y 25 Y 26 Y 27 Y 31 Y 32 Y 33 Y 34 Y 35 Y 36 Y 37 Y 41 Y 42 Y 43 Y 44 Y 45 Y 46 Y 47 * Typically analyzed using two-way ANOVA (or Friedman test)

23 What Happens if there are Restrictions on Experimentation? Suppose there are k=4 treatment conditions Suppose that blocking on a factor (e.g., date) is deemed necessary You can only process 3 experiments on a given day What s the design? The randomized incomplete block design

24 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

25 This design is wrong!! Treatment Date Treatment 1 Treatment 2 Treatment 3 Treatment 4 7/13 7/15 7/16 7/17 7/20 7/22 7/23 Y 11 Y Y 14 Y 15 Y Y 22 Y 23 Y 24 Y Y 27 Y Y 33 Y Y 36 Y 37 Y 41 Y 42 Y Y 45 Y 46 Y 47

26 because the design is unbalanced.

27 Welcome to The Balanced Incomplete Block Design

28 Consider Balanced Incomplete Block Design (BIBD) with 4 Dates Treatment Date Treatment 1 Treatment 2 Treatment 3 Treatment 4 7/13 7/15 7/16 7/17 Y 11 Y Y Y 22 Y 23 Y 24 Y Y 33 Y 34 Y 41 Y 42 Y b= # blocks = 4 k= # treatments=4 t= # treatments per block=3 r= # reps of a trt throughout the experiment=3 n= total number of observations =bt =kr = 12 v= # time each pair of trts appears together= r(t-1)/(k-1) = 2 If n needs to be increased, just add more BIBDs of similar structure

29 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

30 Latin Square Design

31 Latin Square Design Want to compare n treatment conditions, blocking on two factors, each with n levels. Example: 3x3x3 Latin Square: 3 treatments A, B, C Blocking Factor A B C Blocking Factor 2 2 C A B 3 B C A n Number of LS Designs ,851,200

32 Latin Square Design: Colon Cancer Example 3 Treatments: (A) 5-FU, (B) paclitaxel, (C) Vitamin C Blocking Factor 1: Date Blocking Factor 2: Histologic grade (differentiation): Well Differentiated, Moderately Differentiated, None/Poorly Differentiated Well Mod None/Poor 8/4 A B C 8/5 C A B 8/14 B C A

33 Other Block Designs Greco-Latin Squares: 3 blocking factors n x n x n x n Youden Squares -- incomplete version of Latin Square Many other designs for different settings

34 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

35 2x2 Factorial Designs Experiments involving two or more categorical factors that are of interest to the investigators (blocking factors are not of interest they re just nuisances) In a complete factorial experiment, the response is observed at every combination of factor levels; i.e., every level of each factor appears with every level of every other factor 2x2 factorial refers to two factors, each with two levels Factorial designs are used to study synergism and antagonism ( interaction )

36 2x2 Factorial Design (balanced) Factor Factor 2 1 Y 111 Y 112 Y 113 Y 114 Y 115 etc. 2 Y 211 Y 212 Y 213 Y 214 Y 215 etc. Y 121 Y 122 Y 123 Y 124 Y 125 etc. Y 221 Y 222 Y 223 Y 224 Y 225 etc. * Typically analyzed using two-way ANOVA

37 2x2 Factorial Example: Breast Cancer Effect of docetaxel ( doc ), panobinostat ( pan ), and their combination on breast cancer cells Outcome = caspase-3 (pg) Cells are exposed to the 4 possible combinations of treatment and caspase is measured 2 days later n = 8 observations per treatment combination (total=32)

38 2x2 Factorial Design: Breast Cancer Example Data from Day 2 Day 2 Panobinostat + - Docetaxel , 198, 195, , 169, 167, , 169, 180, , 150, 151,...

39 Plot of Means for Each of the Four Treatment Combinations On Day 2 Day= No Panobinostat Panobinostat Mean pg of Caspase pg 198 pg 166 pg pg 140 No Docetaxel Docetaxel Docetaxel Note: Effect of doc without pan= +18pg; with pan = +22pg -- parallelism

40 Expansion of Example Experiment is performed at 4 time points: Day 0 Day 1 Day 2 Day 3 Let s look at the Day 1 result

41 2x2 Factorial Design: Breast Cancer Example Data from Day 1 Day 1 Panobinostat + - Docetaxel , 193, 188, , 158, 160, , 170, 166, , 154, 151,...

42 Plot of Means for Each of the Four Treatment Combinations On Day 1 Day= No Panobinostat Panobinostat 200 Mean pg of Caspase pg 189 pg No Docetaxel 156 pg 158 pg Docetaxel Docetaxel Note: Effect of doc without pan= +2pg; with pan = +22pg -- non-parallelism or, doc x pan interaction

43 Data from the Entire Set of Experiments: Days 0,1,2,3 Day 0 Day 1 Panobinostat + - Docetaxel , 150, 153, , 156, 153, , 149, 157, , 154, 147,... Panobinostat + - Docetaxel , 193, 188, , 158, 160, , 170, 166, , 154, 151,... Day 2 Day 3 Panobinostat + - Docetaxel , 198, 195, , 169, 167, , 169, 180, , 150, 151,... Panobinostat + - Docetaxel , 221, 227, , 188, 186, , 190, 185, , 149, 150,...

44 Plot of Means for Each of the Four Treatment Combinations On Each of the Four Days Day =0 Day =2 220 No Panobinostat Panobinostat 220 No Panobinostat Panobinostat Day 0 Mean pg of Caspase Mean pg of Caspase Day No Docetaxel Docetaxel No Docetaxel Docetaxel Docetaxel Docetaxel Day =1 Day =3 220 No Panobinostat Panobinostat 220 No Panobinostat Panobinostat Mean pg of Caspase Day 1 Day Mean pg of Caspase No Docetaxel Docetaxel No Docetaxel Docetaxel Docetaxel Docetaxel These plots show a 3-way interaction: time x doc x pan

45 No Yes 12 No Interaction 10 % Cells Killed RT 2 0 No Yes No Chemotherapy Yes % Cells Killed RT Quantitative Interaction No Yes No Chemotherapy Yes Qualitative Interaction % Cells Killed RT No Chemotherapy Yes

46 Experimental Designs to be Discussed Single factor with 2 levels no blocking factor blocking factor Single factor with k>2 levels no blocking factor blocking factor incomplete blocks multiple blocking factors Two factors 2x2 factorial rxc factorial interactions More than Two Factors 2x2x2 factorial L1xL2xL3x...xLF Interactions Nested designs crossed vs. nested factors split plot split split plot

47 Extension of 2x2 Factorial to Factorial Designs of Higher Dimensions r x c factorial: refers to two factors, one with r levels, the other with c levels (i.e., r rows, c columns) 2x2x2 factorial: 3 factors, each with 2 levels 2 n factorial: n factors, each with 2 levels 3x3, 3x3x3, 4x2, 4x2x2, etc.

48 Nested Designs

49 Nested Designs Crossed Factors when every level of each factor is tested with every level of every other factor Nested Factors factors that are not crossed Nested designs require more sophisticated methods of analysis Examples: split plot designs, repeated measures designs

50 Nested Design 5-FU Paclitaxel Vitamin C Cell line J225 Cell line J345 Cell line JQ23 Cell line K544 Cell line A3200 Cell line BB509 Y 11 Y 12 Y 13 Y 14 Y 15 Y 16 Y 17 Y 21 Y 22 Y 23 Y 24 Y 25 Y 26 Y 27 Y 31 Y 32 Y 33 Y 34 Y 35 Y 36 Y 37 Y 41 Y 42 Y 43 Y 44 Y 45 Y 46 Y 47 Y 51 Y 52 Y 53 Y 54 Y 55 Y 56 Y 57 Y 61 Y 62 Y 63 Y 64 Y 65 Y 66 Y 67 * Typically analyzed using mixed model ANOVA

51 Moral of the Story There are many types of study designs that may be applicable to both simple and complex experiments or studies. These designs are useful in both the laboratory or the clinical setting. It s a good idea to consult with a statistician during the planning phase of a study.

52