2014 Assessment Report: Department of Psychology Division of Behavioral and Social Sciences, Quincy University

Transcription

1 2014 Assessment Report: Department of Psychology Division of Behavioral and Social Sciences, Quincy University Methods of Assessment The Department of Psychology has been using two primary methods of assessment to evaluate the program s effectiveness. These methods include the Major Field Test for psychology, developed by the Educational Testing Service and an internally developed examination: the QU Test of Psychological Knowledge, developed by the Psychology faculty. The Major Field test is used to assess how our graduates compare to national norms. The QU Test of Psychological Knowledge is used to evaluate how well our department is serving students taking Introduction to Psychology as a General Education Requirement and how much Psychology majors learn through the entire Psychology program. This document describes each method of assessment, how it is used, any identified weaknesses, and any plans to alter assessment methods and/or courses to better reflect the goals of the department. Educational Testing Service s Major Field Test for Psychology Description: According to the Educational Testing Service (2011), The ETS Major Field Tests are comprehensive undergraduate and MBA outcomes assessments designed to measure the critical knowledge and understanding obtained by students in a major field of study. The Major Field Tests go beyond the measurement of factual knowledge by helping you evaluate students ability to analyze and solve problems, understand relationships and interpret material from their major field of study. ETS offers comprehensive national comparative data for the Major Field Tests, enabling you to evaluate your students performance and compare your program's effectiveness to programs at similar institutions nationwide. Objectives: Our objectives concerning this assessment are as follows: 1. Collectively, graduating seniors total scores on the Major Field Test will be as good as or better than the national average. 2. Collectively, graduating seniors normalized subscores (a, b, c, d) on the Major Field Test will be as good as or better than the national average. ETS Data: ETS offers normalized subscores in 4 areas (for each individual student): a. Learning, Cognition, Memory (Learn, Cog, & Mem) b. Perception, Sensation, and Physiology (Perc, Sen, & Phy) c. Clinical, Abnormal, and Personality (Clin, Abn, & Pers) d. Developmental and Social (Dev & Soc) ETS offers Institutional Assessment Indicators as mean raw percent correct subscores in 6 areas (institutional level only): e. Memory and Cognition (Mem & Cog) f. Perception, Sensation, and Physiology (Perc, Sen, & Phy) g. Developmental (Dev) h. Clinical and Abnormal (Clin & Abn) i. Social (Soc) j. Measurement and Methodology (M & M)

2 Results and Analysis: Table 1: Scores, ETS - MFT Psychology (2014) QU Mean* QU Range QU SD Nat. Mean** Nat. SD Normalized Subscores Test Total Learn, Cog, & Mem Perc, Sen, & Phy Clin, Abn, & Pers Dev & Soc Assessment Indicators 1 Mem & Cog 42 Reported as None Perc, Sen, & Phy 44 Institutional Given Dev 47 Means Clin & Abn 61 Only # Soc M & M *Note: QU scores reflect the mean of our students (n = 20 students). **Note: These scores are according to ETS (2012) and reflect the national institutional mean score over the period of September 2010 to June 2014 (n = 357 schools). # Note: An inquiry was made in July 2014 via to ETS to get additional details for this data. ETS indicated that Unfortunately we do not offer a report that displays assessment indicators on a student-by-student basis. Table 2: Statistical Comparisons, ETS - MFT Psychology (2014) QU Mean %tile** Nat. Mean** t-value p-value* Normalized Subscores Test Total th p > Learn, Cog, & Mem th p > Perc, Sen, & Phy th p > Clin, Abn, & Pers th p > Dev & Soc th p > 0.1 Assessment Indicators 1 Mem & Cog st 43.6 No Perc, Sen, & Phy th 48.7 Statistical Dev th 50.9 Comparison Clin & Abn st 62.2 Possible # Soc th M & M th *A p-value above at least 0.05 would indicate that QU student scores did not differ significantly from the national average. **Note: These scores are according to ETS (2012) and reflect the national institutional mean score over the period of September 2010 to June 2014 (n = 357 schools). # Note: Since only the mean scores are given, it is not possible to do a valid statistical analysis of this data.

3 Normalized Scores ( ) Percentile QU MFT Performance Score Percentile Year Figure 1: QU Psychology Major Field Test Performance over the past 4 years using the overall score and percentile rank each year. Summary: Objectives 1 and 2, as indicated above, have been satisfied. Objective 1: The mean overall score on the Psychology Major Field Test for QU graduating seniors was (out of 200). This places out students in the 47 th percentile were not significantly different than the national average, t (19) = , p>0.1. Objective 2a: The mean subscore on the Learning and Cognition section for QU graduating seniors was 53.7 (out of 100). This places out students in the 48 th percentile were not significantly different than the national average, t (19) = 0.000, p>0.1. Objective 2b: The mean subscore on the Perception, Sensation, and Physiology section for QU graduating seniors was 51.1 (out of 100). This places out students in the 29 th percentile were not significantly different than the national average, t (19) = , p>0.1. Objective 2c: The mean subscore on the Clinical, Abnormal, and Personality section for QU graduating seniors was 55.9 (out of 100). This places out students in the 48 th percentile were not significantly different than the national average, t (19) = 0.150, p>0.1. Objective 2d: The mean subscore on the Developmental and Social section for QU graduating seniors was 52.9 (out of 100). This places out students in the 37 th percentile were not significantly different than the national average, t (19) = , p>0.1. Assessment Indicators: As for the Assessment Indicators provided by ETS, no specific goals had been set previously. Additionally, further analysis is complicated by the fact that ETS does not provide enough information for a detailed analysis. However, given this limitation, an impression can be made from the data available. The percentile ranks put our QU graduating seniors below the 50 th percentile in every category. However, when you look at the percentage score our students are no more than a few percentage points different than the mean in every category. This suggests that we are not far away from being in the upper 50% of psychology programs nationally.

4 Weaknesses and plans for improvement for the MFT assessment: Our programs strives to set major requirements that provide our students with the broad foundation in Psychology expected by graduate programs in Psychology, and still remain within the maximum number of credit hours allowed in the major under Quincy University policy. Students typically fulfill the major requirements by selecting one course from a list of options in each of several major subfields of Psychology. Thus, for example, if students take Learning, they are unlikely, unless they have a special interest in the area, to also take Cognition. Because of the inevitable peaks and valleys of the students learning, it is likely that students subscores will vary somewhat from year to year depending on the emphases of the test that year and on the set of options chosen by that year s students. The fact that our students do not differ significantly from the national average of each of the subfields of Psychology year after year speaks well of the depth and breadth of our program. That being said, there is always room for improvement. One, it has been a challenge to get all graduating seniors to take the test in a timely manner. Several strategies are being considered to address this concern including having the test administered as part of a psychology capstone course. Two, it is not clear that all students are making their best effort to do well on this test. It is our intention to develop better strategies to incentivize good performance. Three, not all students take exactly the same classes and are not equally prepared for a test such as the MFT in Psychology. Some thought could be given to the program course structure and/or advising could be done to encourage students to get a more well-rounded psychology background. QU Test of Psychological Knowledge (QU-TPK) Description: This internally developed assessment method was originally a 100-item multiple choice test, used to assess a student s knowledge about the field of Psychology. It was recently modified (summer 2013) to include 65 items. This was done mostly because it was a recurring observation that students had difficulty finishing 100 questions in a timely manner. Please note that the data reported here is using the older 100 question test. Items within this assessment examine important subfields in Psychology (e.g. Social, Developmental, Physiological, Learning, & Research Methods). Students who major in Psychology and spend their entire undergraduate careers here take this assessment three times: once at the beginning and once at the end of our Introduction to Psychology course (PSY 100) and once shortly before graduating. The first administration of the examination assesses how much students know about Psychology coming into Introduction to Psychology. The second administration assesses how much students know about Psychology at the end of Introduction to Psychology. Comparing the scores on the first and second examinations allows us to assess how well we are meeting our content objectives for Introduction to Psychology, a General Education Requirement. The third examination assesses how much our graduating majors know about Psychology shortly before they graduate. Comparing scores from the first and second assessments with the third allows us to determine how effectively the Department is meeting its goals for Psychology majors. Since its development in 2003, 1,492 students (as of May 2014) have taken the assessment at the beginning and end of the introductory level class. Objectives: Our objectives assessed by this measure are as follows: 1. Students in Introduction to Psychology will improve their scores from the beginning of the semester to the end. 2. Graduating seniors who major in Psychology will perform better on their final assessments than students assessed at the beginning of Intro. to Psychology.

5 3. Graduating seniors who major in Psychology will perform better on their final assessments than students assessed at the end of Intro. to Psychology. 4. Graduating seniors, who major in Psychology, on average, will perform better on their final assessments than their own performance assessed at the beginning of Intro. to Psychology. 5. Graduating seniors, who major in Psychology, on average, will perform better on their final assessments than their own performance assessed at the end of Intro. to Psychology. 6. Performance on the QU Test of Psychological Knowledge correlates with performance on the Major Field Test. Results and Analysis: Table 3: QU Test of Psychological Knowledge ( ) Summary Data Assessment Time PSY 100 Students (n = 164) Graduating Seniors (n = 20) Beginning of PSY End of PSY End of Program Table 4: QU Test of Psychological Knowledge ( ) Statistics Comparison t-value p-value* All PSY 100 (beginning) vs. All PSY 100 (end) p < All PSY 100 (beginning) vs. Seniors (End) p < All PSY 100 (end) vs. Seniors (End) 5.44 p < Seniors previous PSY 100 (beginning) vs. Seniors (End) 5.73 p < Seniors previous PSY 100 (end) vs. Seniors (End) 3.87 p < *A p-value less than 0.05 indicates a significant improvement in each comparison. Summary: Objectives 1 through 6, as indicated above, have been satisfied. Objective 1: The mean score for PSY 100 students showed an increase from the beginning (M = 38.7) to the end (M = 46.7) of the course. A two-tailed t-test was performed on this data which indicated a significant improvement, t (163) = 12.52, p< Objective 2: The mean score for Seniors at the end of the program (M = 63.2) showed an increase compared to PSY 100 students the beginning of PSY 100 (M = 38.7). A two-tailed t-test was performed on this data which indicated a significant improvement, t (182) = 11.00, p< Objective 3: The mean score for Seniors at the end of the program (M = 63.2) showed an increase compared to PSY 100 students the end of PSY 100 (M = 46.9). A two-tailed t-test was performed on this data which indicated a significant improvement, t (182) = 5.44, p< Objective 4: The mean score for Seniors at the end of the program (M = 63.2) showed an increase compared to their previous scores at the beginning of PSY 100 (M = 39.3). A two-tailed t-test was performed on this data which indicated a significant improvement, t (182) = 5.73, p< Objective 5: The mean score for Seniors at the end of the program (M = 63.2) showed an increase compared to their previous scores at the end of PSY 100 (M = 46.3). A two-tailed t-test was performed on this data which indicated a significant improvement, t (182) = 3.87, p< Objective 6: A Pearson s r correlation coefficient was calculated comparing MFT and QU-TPK. The results show a strong correlation in the scores of these two tests, r = 0.742, t (18) = 4.70, p<0.001, indicating a high degree of construct validity.

6 Weaknesses and plans for improvement for the QU-TPK assessment: [To be completed later.] Additional Assessment: QU - Psychology Program Learning Outcomes Description: This internal assessment method is currently under development. [To be completed later.] [To be completed later.] Overall Summary and Future Plans