Tutorial 3: Pekka Malo 30E00500 Quantitative Empirical Research Spring 2016
Step 1: Research design Adequacy of sample size Choice of dependent variables Choice of independent variables (treatment effects) Optional: Use of covariates (MANCOVA) 2
Choice of dependent variables Don t try to include too many, use only as many variables as necessary Single badly chosen variable can distort the entire result Use only variables with a strong theoretical support 3
Choice of independent variables (factors or treatments ) Often researcher is interested in examining effects of several independent variables rather than using only a single variable à factorial design Sometimes researchers are for compelled to add more independent variables to control for characteristic that affect the dependent variables but is not part of the study design (e.g. geographic location, gender) à blocking factors Remember sample size limitations 4
Use of covariates (MANCOVA) Sometimes the research also has to control for effects of metric variables à covariates Objective of covariates is to 1. Eliminate any effects that influence only a portion of the respondents or 2. Effects that vary among the respondents Key benefits: 1. Helps to eliminate systematic error outside the control of the researcher (effects which could bias results) 2. Accounts for differences in responses due to unique characteristics of respondents Note: Number of covariates should not exceed the following thumbrule: 0.1 x sample size (number of groups -1) 5
Sample size Need to use larger samples than in ANOVA Minimum sample in each group must be greater than the number of dependent variables Recommended minimum number per group is 20 observations The higher the number of dependent variables, the greater the sample size needed to maintain statistical power 6
Sample size with power at 0.80 7
Step 2: Check assumptions Independence of observations Homogeneity of variance/covariance matrices Levene s univariate tests (similar process as in ANOVA) Box s M-test for covariance matrices Normality Check histograms and Kolmogorov-Smirnov tests Multicollinearity of dependent variables Sensitivity to outliers 8
Normality and independence Multivariate normality: All dependent variables should be normally distributed Any linear combination of the dependent variables should be also normally distributed Luckily is robust test and survives departures from multivariate normality When multivariate normality not satisfied, lesser power to detect main or interaction effects Independence of observations must not be violated! 9
Explore by variable Analyze > Descriptive statistics > Explore (plots) 10
Equality of covariance matrices For k multivariate populations, the hypothesis of equality of covariance matrices is Commonly tested using Box s M-test: Sensitive to the size of covariance matrices and the number of groups in the analysis Highly sensitive to departures from normality: always check univariate normality of measures before performing the test However, violation of the assumption doesn t generally have severe impact when groups are of approximately equal size 11
Equality of covariance matrices (2) What if group sizes differ more and Box s M-test is not within acceptable levels (alpha <.001)? Apply one of the many variance-stabilizing transformations on variables Use adjusted tests which do not assume equal variances: E.g. Games-Howell 12
Box s M-test in SPSS The test is obtained at the estimation step as part of GLM routine (see Step 3) 13
Step 3: Estimation and significance testing Choice of criteria for significance testing Assessment of statistical power Effects of dependent variable multicollinearity 14
Estimation with the General Linear Model (GLM) Classical techniques established more than 70 years ago, but nowadays models are commonly fitted using GLM can be viewed as a multivariate multiple regression model Reasons for using GLM: Single estimation procedure can be used to handle wide range of specifications Available in most statistical packages (e.g. SPSS, SAS) 15
Multivariate GLM in SPSS 16
Multivariate Tests () Wilks lambda (or U-statistic): Roy s greatest characteristic root (GCR): Pillai s criterion and Hotelling s T 2 Often referred to as the multivariate F-test Preferred when basic requirements (sample size, no violations, approximately equal sized groups) are met Advantages in power and specificity Most appropriate when dependent variables are strongly interrelated on one dimension Strongly affected by violations of assumptions Similar to Wilks lambda More robust and preferred when sample size decreases, unequal groups, or homogeneity of covariances is violated 17
18
Step 4: Interpretation of Results Interpret the effects of covariates, if employed Assess which dependent variable(s) exhibited differences across the groups of each treatment Identify whether the groups differ on a single dependent variable or the entire dependent variate 19
Interpreting Covariates If covariates are involved, they need to be interpreted first, since they act as controls on the dependent variate Assessment of overall impact: Evaluate impacts with and without covariates in the model Effective covariates should improve statistical power and reduce within-group variance Interpretation of covariates: Similar to regression analysis If their overall impact is significant, it is of interest to examine the strength of their predictive relationship with dependent variables 20
Main Effects of Treatments Main effect is typically described by the difference between groups on the dependent variables in the analysis Example: if gender has a significant main effect, we could look to the difference in means as a way of describing the impact (e.g. 7.5 for women vs. 6.0 for men) When treatment or factor has more than two levels, significant main effect does not guarantee that all groups are significantly different (at least one pair of groups is) 21
Identifying differences between individual groups Once you are confident that there are differences between groups, it is possible to perform a variety of tests to understand which groups differ Contrasts (~a priori tests): Planned comparisons based on scientific goals Post-hoc tests (~ decide after experiment): Performed for each dependent variable separately Used for situations where you can decide which comparisons you want to make after looking at the data 22
Impact of Interactions Any time more than one treatment or factor is used, there are interaction effects Interaction terms represent the joint effect of 2 or more factors No interaction: The effect of one factor is the same for each level of the other factor; allows direct interpretation of main effects independently Ordinal interaction Effects differ across the levels of another factor, but the group differences are always in the same direction Disordinal interaction Factors don t represent any consistent effect 23
Profile plots Profile plots (interaction plots) are useful for comparing marginal means in your model. A profile plot is a line plot in which each point indicates the estimated marginal mean of a dependent variable (adjusted for any covariates) at one level of a factor. If you have more than 1 independent variable: The levels of a second factor can be used to make separate lines. Each level in a third factor can be used to create a separate plot. 24
25
Example: Cereal evaluation Shape (ball, cube, star) Vs. Color (red, blue, green) Source: bestofbritish.ca 26
No interaction effect 27
Ordinal interaction 28
Disordinal interaction 29
Identifying differences between individual groups Once you are confident that there are differences between groups, it is possible to perform a variety of tests to understand which groups differ Contrasts (~a priori tests): Planned comparisons based on scientific goals Post-hoc tests (~ decide after experiment): Performed for each dependent variable separately Used for situations where you can decide which comparisons you want to make after looking at the data 30
Post-hoc Multiple Comparisons Commonly used tests for multiple comparisons: Bonferroni s test: based on Student s t-test, adjusts the significance level for multiple comparisons Tukey s honestly significant difference test: uses Studentized range statistic to make all pairwise comparisons between groups Choose a test based on the number of pairwise comparisons: Use Tukey for large number of pairs: More powerful than Bonferroni when several pairs of means considered Use Bonferroni for small number of pairs 31