CHAPTER ELEVEN. NON-EXPERIMENTAL RESEARCH of GROUP DIFFERENCES

Transcription

1 CHAPTER ELEVEN NON-EXPERIMENTAL RESEARCH of GROUP DIFFERENCES Chapter Objectives: Understand how non-experimental studies of group differences compare to experimental studies of groups differences. Understand the nature of the independent variable in nonexperimental studies of group differences Understand methods of data collection in non-experimental studies of group differences Understand the procedures for conducting a non-experimental studies of group differences Understand inferential tests of significance used for continuous and categorical dependent variables Understand how non-experimental studies of group differences are evaluated for validity 1

2 Purpose and Comparison to Experiments Non-Experimental Studies of Group Differences provide a new angle from which to view group differences; they investigate existing differences between and among groups that differ on a categorical variable like gender or ethnicity, grade level, etc. Experiments and non-experiments of group differences are similar in the following ways: (1) they investigate theory-based differences between groups in order to determine whether the differences exist and whether they are statistically significant, (2) they select the sample for the study and seek to make groups as equivalent as possible, (3) they operationalize the dependent variable, (4) they select a valid and reliable measure for the dependent variable and use the measure to collect data for both groups, (5) they sort the data into two groups and use inferential tests to determine statistical significance, and (6) they make inferences about results. 2

3 Non-experiments of group differences are different from experiments in these ways: (1) Experiments demonstrate a cause and effect relationship between groups variables by introducing an intervention or treatment and controlling conditions of the study by assigning subjects to groups and manipulating the treatment so that one group receives it and another does not. Non-experiments of group differences do not demonstrate a cause and effect relationship, nor do they introduce an intervention or treatment, or assign subjects to groups. As a result, while they can explore the connections between groups and may suggest an explanation for differences, they cannot attribute a cause. This is due to the nature of the subjects, who are divided into groups before the study begins and that are based on a shared, immutable characteristic like their gender, ethnicity, grade level, etc. (2) Experiments almost always use continuous variables for the dependent variable. Nonexperiments of group differences may use either continuous or categorical variables as the dependent variable. In the case of a categorical dependent variable, different tests of significance are used. By way of Review: A continuous variable can be measured and quantified. Continuous variables may include scaled scores on assessments of achievement or aptitude or responses on a Likert scale survey or questionnaire -- all of which can be quantified and described statistically as means and standard deviations. 3

4 A categorical variable is an attribute that cannot be measured or quantified. Categorical variables may be on the order of innate characteristics like gender or ethnicity or on the order of extrinsic characteristics such as, highest level of education attained, grade level taught, or political party affiliation. Numerous labels describe this form of research, including ex-post facto, causal-comparative, and pre-experimental studies. We prefer the term non-experiments of group differences to indicate how they are similar to and different from the way that experiments and quasi-experiments investigate differences in groups. Data Collection and Analysis 4

5 Surveys and large-scale tests are commonly used as measures for collecting data in non-experimental group comparisons. A survey usually has a section that asks questions about the attributes of subject demographics like their gender, race, grade level, last degree earned, etc. Having this information allows the researcher to divide responses into different categorical groups for analysis. As an example, researcher may want to investigate whether there are gender differences in job satisfaction within a sample of teachers in a school district. IV- gender DV= job satisfaction The researcher would collect data through a survey that asks for demographic information like gender and that uses a Likert scale to assess job satisfaction as a continuous variable. After administering the survey to all teachers in the sample, the researcher (1) divides the sample into two groups, male and female, (2) calculates means and standard deviations for each group, and (3) selects an inferential test of significance to determine whether differences do exist and whether they are significant. Because the dependent variable is a continuous variable, the researcher would use a t- test, ANOVA/MANOVA, or ANCOVA/MANCOVA as the test of significance. Now consider another example. Here the researcher also wants to use the survey to investigate whether there are significant gender differences in the highest level of education attained by teachers in the same sample. 5

6 IV= gender DV= highest level of education In this instance, the dependent variable is a categorical variable. After administering the survey to all teachers in the sample, the researcher would divide the sample into two groups: male and female. Since the dependent variable is categorical, it cannot be quantified in terms of means and standard deviations and therefore cannot depend on t-tests, ANOVA/MANOVA, or ANCOVA/MANCOVA as test of significance. Instead it uses the Chi-Square Statistic that relies on frequencies and proportions/ percentages, to determine statistical significance. The process for determining statistical significance is similar to that used in other tests of significance. The researcher calculates a Chi-Square value and then consults a table of critical values to establish a p-value. Summary of Tests of Significance Used in Non-Experiments Type of Dependent Variable Continuous t-test Significance Test ANOVA/MANOVA ANCOVA/MANCOVA Categorical Chi-Square 6

7 Examples of Non Experimental Studies of Group Differences Continuous DV Study Al-Yagon and Cinamom (2008) investigated the work-family relations of two groups of mothers: those with and without children with disabilities. The researchers used a survey that measured six continuous dependent variables: attachment, affect, sense of coherence, work-tofamily facilitation, family-to-work conflict, and family cohesion. IV = Mothers of disabled and typical children DV = attachment, affect, sense of coherence, work to family facilitation, family to work conflict, family cohesion The first level of analysis was to calculate the means and standard deviations for each group of mothers on each of the six continuous dependent variables. The second step was to conduct the analysis of variance (ANOVA) for each dependent variable. Altogether there were fourteen comparisons, as represented in the table below, which looks like a table you might find in an experimental or quasi-experimental study. 7

8 Two of the variables showed significant differences: work-to-family facilitation (F=4.20, p<.05) and family-to-work conflict (F=4.70, p<.05). Both are starred (*) to indicate p= <.05, which appears at the bottom of the table. According to the authors As a group, mothers of children with LD reported a higher level of Family-to-Work conflict and a higher level of Work-to-Family facilitation compared to mothers of non-ld children (2009, p. 99). The inference might be that having a child with learning disabilities influences mothers engagement in facilitation and their experience of conflict when it comes to work-family issues. 8

9 Categorical DV Study Li (2006) investigated whether there were gender differences in occurrences of bullying, which she divided into four categories: bully, bully victim, cyberbully, and cyberbully victim IV= gender DV= bullying (bully, bully victim, cyberbully, cyberbully victim) Li administered a survey to a sample of students that asked for demographic data like gender and that also asked students to report on their status in the bullying culture of the school, using a yes/no response format. The answers to questions like those below allowed the researcher to create the four categorical variables for comparison. Examples of questions are: 1. I have been bullied during school: Yes No 2. I have bullied others: Yes No 3. I have been cyber-bullied (e.g. via , chat room cell phone): Yes No 4. I have cyber-bullied others: Yes No The researcher began the analysis of data by reporting the percentages of students that fell into each category by gender, as indicated below (p. 163). 9

10 By looking at these percentages, it appears there is a difference in boys and girls. However, as we know from previous chapters, descriptive data like percentages are not sufficient to make inferences about the degree of difference or whether the difference is statistically significant. Because the variables against which the two groups are being compared are categorical, Li used Chi-Square as the statistical tool to determine whether gender makes a significant difference in bullying. The Chi-Square (X2) values and p-values are reported in the table below: IV DV s X2 p-value Male vs. female Male vs. female Male vs. female Male vs. female Traditional Bully victim Cyberbully victim Being a traditional bully Being a cyberbully * * 10

11 * p.05 The results showed significant differences between girls and boys in terms of being a bully and being a cyber-bully, but showed no significant differences between girls and boys in terms of being a bully or victim. The inference would be that gender has a significant influence on whether one becomes a bully and a cyber bully. Evaluating the Validity of Non-Experimental Group Differences Non-experiments cannot be evaluated for internal validity because they do not show causal relationships; they are evaluated on the basis of statistical conclusion validity and external population validity, as represented in the table below. 11

12 Criterion Theory Statistical Conclusion Validity Evidence of fishing? External (Population) Validity A sound theoretical construct generated through an extensive research review, Clear identification of independent and dependent variables Sample Adequate sample size Random selection of the sample or sample matching Detailed description of who is in the sample and the characteristics of subjects Measure Rating Reliability of measure = >.7 Appropriate inferential statistics Appropriate alpha level for significance testing H M L Demonstrated construct validity of the measures in relation to the theoretical construct Evidence of the reliability of the measure. H M L 12

13 Though based on weaker causal inferences, non-experimental group difference studies can explore the complexity of differences between existing intrinsic and membership groups on different outcomes. 13

14 Chapter Summary Non-experimental studies of group differences are similar to group experiments because they investigate the statistical significance of group differences on a dependent variable. Non-experimental studies of group differences are different from group experiments because they are not causal, do not involve an intervention or treatment, do not assign subjects to groups, and use both continuous or categorical variables for the dependent variable 14

15 Data are commonly collected through surveys, questionnaires, and achievement and aptitude measures that ask for demographic information as well as for responses. Data analysis for differences and statistical significance are different for continuous and categorical variables; continuous variables are tested on the same statistics as experiments; categorical variables are tested on the Chi Square Statistic. Studies are evaluated for statistical conclusion validity and external validity. Terms and Concepts categorical variable independent variable Chi Square statistic continuous variable dependent variable statistical conclusion validity external validity Review, Consolidation, and Extension of Knowledge 1. In your own words, a. describe how non- experiments of group differences are similar to and different from experiments b. identify inferential tests of significance for studies that have a continuous variable as the DV. 15

16 c. Identify the inferential test of significance for studies that have a categorical variable as the DV. 2. Using an electronic database, search for non-experimental group difference study (look for ex-post facto and casual comparative in your search. on a topic of interest. Read the article and answer the questions in the Guide below. 3. Using the guide as a template write a critique of about 750 words of the non-experimental group differences study you selected. See the Appendix for an Exemplar.. Guide to Reading and Critiquing a Non- Experimental Group Study Research Review and Theory: What is the purpose of the research review? Does it establish an underlying theory (big ideas) for the research? Purpose and Design: What is the purpose of the study? 16

17 Is there a hypothesis or a research question? If so, what is it? If not, can you infer the question from the text of the article? What is the basic research design and type? What are the groups being studied for differences? Is the dependent variable continuous or categorical? Sampling: How is the sample selected: random or non-random? Are the groups matched on characteristics other than their defining characteristic?s If so, how? Who is in the sample? What are the characteristics of the sample? What is the sample size? Data Collection: How are data collected: survey, questionnaire, test reports? What information does the instrument collect? For continuous dependent variable studies, What is the response format of the questions? Are there indications of validity and reliability of the measure? What are they (r-values)? Data Analysis and Results: What statistical tests are used to analyze the data? Were the results (p-values) significant or non-significant? 17

18 What does the researcher conclude about the findings? Evaluation of Validity: What is the quality of statistical conclusion validity? What is the rationale for this judgment in terms of theory, sample, and measurement? What is the quality of external validity? What is the rationale for this judgment in terms of theory, sample, and measurement? 18

19 19