Developing a Coding Scheme to Evaluate Gender Bias in Student Comments on Teaching Evaluations

Transcription

1 Developing a Coding Scheme to Evaluate Gender Bias in Student Comments on Teaching Evaluations Stephanie C. Payne, Ph.D., and Estefany Saracay Department of Psychology Payne, S. C., & Saracay, E. (2015, March). Developing a coding scheme to evaluate gender bias in student comments on teaching evaluations. Presentation at the annual Texas A&M University Student Research Week, College Station, TX. This material is based upon work supported by the National Science Foundation under NSF Cooperative Agreement No. HRD

2 Gender Roles & Descriptors Gender roles can be described as agentic or communal (Eagly et al., 2000) Communal attributes women affectionate, helpful, caring, and sympathetic Agentic attributes men confident, aggressive, ambitious, and intellectual

3 Student Evaluations of Teaching (SET) Millions of evaluations completed every semester Quantitative ratings Not gender biased (Feldman, 1993) Qualitative comments More challenging data to analyze and time consuming Most analyses are subjective and descriptive

4 Research on SET Comments Braskamp et al., 1982 Examined 3,240 student comments to 4 questions (2 were related to the course) 1 out of every 2 comments (1607) was about the instructor Ability to make clear presentations Knowledge of the field Instructor s personality and relationship with students Overall convergence between ratings and valence of comments Sex differences not examined

5 Descriptors Associated with Effective College Teaching (Lowman, 1995) Dimension 1: Intellectual Excitement Adjective Frequency Adjective Frequency Enthusiastic 68 Engaging 18 Knowledgeable 45 Prepared 16 Inspiring 43 Energetic 15 Humorous 34 Fun 13 Interesting 31 Stimulating 13 Clear 25 Creative 12 Organized 22 Lectures well 11 Exciting 22 Communicator 10

6 Descriptors Associated with Effective College Teaching (Lowman, 1995) Dimension 2a: Interpersonal Concern Adjective Frequency Adjective Frequency Concerned 45 Approachable 12 Caring 33 Interested 12 Available 27 Respectful 11 Friendly 18 Understanding 11 Accessible 17 Personable 10

7 Descriptors Associated with Effective College Teaching (Lowman, 1995) Dimension 2b: Effective Motivation Adjective Frequency Adjective Frequency Helpful 41 Demanding 14 Encouraging 29 Patient 13 Challenging 28 Motivating 11 Fair 19

8 Descriptors Associated with Effective College Teaching (Lowman, 1995) Commitment to Teaching and General Positive Descriptors Adjective Frequency Adjective Frequency Effective 17 Dedicated 35 Excellent 17 Committed 19 Outstanding 14 Great 10

9 Agency & Communion in Letters of Recommendation (Madera et al., 2009) Letters of recommendation for academia Women described as more communal and less agentic than men Communal characteristics negatively related to hiring decisions Linguistic Inquiry and Word Count (LIWC) used to code the descriptive terms

10 Hypotheses H1: Female professors will be described with more interpersonal concern adjectives than male professors. H2: Female professors will be described with significantly more (a) communal descriptors and less (b) agentic descriptors than male professors. H3: STEM professors will be described as (a) more agentic and (b) less communal than non-stem professors. H4: Female STEM professors will be described as significantly (a) more agentic and (b) less communal than female non-stem professors.

11 Method Spring 2013 online evaluations 3038 evaluations for male faculty 2746 evaluations for female faculty LIWC software used Lowman s adjectives Madera s communal and agentic descriptors

12 Results Dimension 1: Intellectual Excitement Adjective % Adjective % Men Women Men Women Enthusiastic Engaging Knowledgeable* Prepared Inspiring Energetic Humorous Fun Interesting* Stimulating Clear Creative Organized Communicator Exciting.01.02

13 Results Dimension 2a: Interpersonal Concern Adjective % Adjective % Men Women Men Women Concerned Approachable Caring Interested Available Respectful Friendly Understanding Accessible Personable.01.01

14 Results Dimension 2b: Effective Motivation Adjective % Adjective % Men Women Men Women Helpful Demanding Encouraging Patient Challenging* Motivating Fair.11.07

15 Results Commitment to Teaching and General Positive Descriptors Adjective % Adjective % Men Women Men Women Effective* Dedicated Excellent Committed Outstanding Great

16 Agency/Communion Results Male Faculty (n = 3038) M (SD) Female Faculty (n = 2746) M (SD) t(df) Agentic 0.43 (1.63) 0.41 (1.88).43 (5473).67 Communal 0.34 (1.31) 0.37 (1.40) -.91 (5620).37 STEM (n = 3375) nonstem (n = 2409) t(df) Agentic 0.49 (1.91) 0.33 (1.51) -3.5 (5717).00* Communal 0.32 (1.33) 0.40 (1.39) 2.22 (5044).03* STEM *Sex Interaction Term F p Agentic Communal p p

17 Agency/Communion Results Agentic M (SD) Communal M (SD) Male STEM 0.47 (0.04) 0.30 (0.30) Male nonstem 0.36 (0.06) 0.43 (0.04) Female STEM 0.52 (0.05) a 0.36 (0.04) Female nonstem 0.32 (0.05) a 0.39 (0.04) a significantly different from each other

18 Discussion Very few differences between male and female professors on Lowman s adjectives Although students describe male and female professors differently and consistent with gender roles, these differences are not meaningful different STEM professors are described as more agentic and less communal than non-stem professors Female STEM professors are described as significantly more agentic than female nonstem professors

19 Discussion Limitations One semester of online data from one university LIWC not perfect Future Directions Correlate descriptors with quantitative ratings Examine opposite descriptors (e.g., demotivating)