[ASSESSMENT DOCUMENT]

Transcription

1 [ASSESSMENT DOCUMENT]

2 TABLE OF CONTENTS Student Learning Assessment... 3 Institutional Objectives... 3 Program Objectives Institutional Effectiveness Assessment UIE System Institutional Effectiveness Data Evaluative Conclusion Student Learning Assessment Institutional Effectiveness Assessment God s Bible School and College 2017 Assessment Document

3 STUDENT LEARNING ASSESSMENT Institutional Objectives Objective 1: Upon graduation, students from God s Bible School and College will demonstrate a foundational knowledge of the Bible. Measures: ABHE Bible Content Test; GBSC Theology Test Both tests are given to a selection of incoming students annually; those who earn a BA complete the tests again during Senior Worldview Seminar, to provide longitudinal data. ABHE Bible Content Test results Faculty-developed targets for senior performance are shown in Table 1, along with results; a passing score is defined as 60% or higher. Longitudinal analysis is not yet possible, due to data-warehousing problems in , so the improvement target cannot be assessed. Table 1. ABHE Bible Content Test targets and data Area Target Data Percent SRs passing 80% 54% Percent SRs improved 100% N/A GBSC Theology Test results In 2005 faculty who teach theology courses in the Bible/Theology Core collaborated to develop a multiple-choice test of material covered in these courses. (Bible courses were not included in this test, as they are assessed using the ABHE Bible Content Test.) The GBSC Theology Test includes 110 items distributed across required theology courses. Test content is heavily Biblical in nature. Table 2 reports results and targets for this test. Table 2. GBSC Theology Test targets and data Area Target Data Percent SRs passing 80% 29% Percent SRs improved 90% 97% Discussion There is no updated data yet this year for this objective. The plan presented last year is provided again this year, with modifications. Passing-score targets are not achieved on either measure, requiring follow up. 1. Both tests content should be examined. 3 God s Bible School and College 2017 Assessment Document

4 a. Does test content align with desired student learning? b. If not, the in-house test should be revised and the Bible Content Test targets should be revised accordingly. 2. If test content aligns with desired student learning, course content and assessment will be reviewed. 3. This work should be completed before the Fall 2018 assessment report. 4 God s Bible School and College 2017 Assessment Document

5 Objective 2: Upon graduation, students from God s Bible School and College will demonstrate a commitment to Biblical truth and to lifelong study and application of its meaning. Measures: Graduate Exit Survey (selected items); Cross-Curricular Concern embedded assessment The Graduate Exit Survey is administered each semester to graduating students. The selected items are used with permission from the Barna Group. Cross-Curricular Concern assessment is embedded in courses, with faculty using a holistic rubric to assess students in a course. Assessment is on a 5-point (0-4) rubric, with 3 the target outcome. Graduate Exit Survey items Table 3 shows results from selected items. The target for each item is 95% endorsing the preferred response. Table 3. Selected Graduate Exit Survey data Item % endorsing preferred response Moral truth is absolute 86.36% Moral/ethical choices are based on principles 94.12% Bible/God as basis for principles 98.53% The Bible is totally accurate in its teachings 95.52% Read the Bible in the last week 93.85% Totally committed to spiritual development 95.38% Cross-Curricular Concern Assessment Current Biblical Worldview Development data is shown in Table 4; see rubric here. Currently there are scores for only 11 seniors, limiting this measure s validity until more data is accumulated. This data has improved since the last report (where the average was 2.60, SD was 1.14 and percentage scoring 3 or higher was 60%). Table 4. Biblical Worldview Development data SR average (SD) % scoring 3 or higher 3.00 (0.89) 82% Discussion While not all targets are currently achieved, most Barna items are nearly achieved, and embedded assessment of seniors, while limited, currently meets the target on average with more than 80% of seniors achieving it. Results will be monitored as they accumulate. 5 God s Bible School and College 2017 Assessment Document

6 Objective 3: Upon graduation, students from God s Bible School and College will demonstrate Christlike attitudes and actions. Measures: No direct assessment Graduate Exit Survey items shows results from selected items that indirectly address this outcome. Target levels are 3.5 mean agreement on a 5-point scale. Table 5. Selected Graduate Exit Survey data Item Mean agreement Chapel services contributed to my spiritual development 4.35 D-groups contributed to my spiritual development 3.91 I improved in development of a Christian worldview 4.49 I improved in worship/devotional life 4.30 I improved in personal ethics 4.33 I improved in moral values 4.33 Additional related items are part of the Barna Group items; the target is 95% endorsement of preferred responses. Table 6. Selected Graduate Exit Survey data Item % endorsing preferred response Moral truth is absolute 87.14% I have a responsibility to tell other people my 87.88% religious beliefs. I know some ways the Bible applies to the 94.96% vocation I plan to enter Faith is not relevant to my career or interests 90.77% The church s teachings on sexuality and birth 75.00% control are out of date Totally committed to spiritual development 95.38% Completely dependent upon God 92.31% Surrendered control of my life to God 93.94% Submitted to God s will for my life 93.94% Devoted myself to loving and serving God 93.94% and other people 6 God s Bible School and College 2017 Assessment Document

7 Discussion An assessment gap remains, with no direct assessment. As the student development plan (part of Strengthen for Growth) continues to be implemented, some direct data may be created. Graduate Exit Survey data largely supports a conclusion that the outcome is being met. Data for some of the Barna items is not meeting expectations: the Assessment Committee should investigate this and propose action steps before next year s report. 7 God s Bible School and College 2017 Assessment Document

8 Objective 4: Upon graduation, students from God s Bible School and College will demonstrate a desire and commitment to fulfill the Great Commission. Measures: Alumni Survey supportive; Commissional Christians Survey (under development); Graduate Exit Survey items Information from the Alumni Survey provides some evidence for this objective. Exhaustive searching has found no standardized instrument to assess this objective, so the Assessment Committee has worked to develop an instrument and has done some testing, although work is needed before it can be used effectively. Finally, selected Barna Group items from the Graduate Exit Survey are relevant here. Alumni Survey Alumni Surveys show a high percentage of graduates involved in ministry. In the 2008 survey, 92.31%. A number over 90% supports a conclusion that this objective is achieved. Commissional Christians Survey Drawing on Krathwohl s affective domain taxonomy, the Assessment Committee developed a draft instrument to address this objective. Initial results are positive: as anticipated, there is a general trend toward less agreement as respondents progress through the items. More testing, with a broader respondent pool including non-students, is needed before the instrument can be effectively used here. Graduate Exit Survey Selected items from the Graduate Exit Survey address this objective. Results are shown in Table 7; for each the target is 95%. Table 7. Selected Graduate Exit Survey results Item % endorsing preferred response I have a responsibility to tell others my religious beliefs 87.88% I have devoted myself to loving and serving God and 93.94% other people Discussion Existing data is limited, but what exists supports a conclusion that this objective is being realized in part, but data is insufficient for strong conclusions. Further development of the Commissional Christians Survey would provide a useful additional measure, including one that 8 God s Bible School and College 2017 Assessment Document

9 could be used for longitudinal analysis. As noted above, the Assessment Committee will address the Barna-item data that does not meet expectations. 9 God s Bible School and College 2017 Assessment Document

10 Objective 5: Upon graduation, students from God s Bible School and College will demonstrate an awareness of their spiritual gifts and evidence of the fruit of the Spirit as they minister to others. Measures: Graduate Exit Survey data One item from the Graduate Exit Survey (a Barna Group item) relates to this objective. Graduate Exit Survey One item relates to this; data is show in Table 8. Table 8. Selected Graduate Exit Survey data Item I know some ways the Bible applies to the vocation I plan to enter % endorsing preferred response 95.38% Discussion Clearly more data is needed to assess this objective effectively. While individual courses include spiritual-gifts inventories or similar items, this objective can best be assessed through the Christian Service program. The Assessment Committee will develop a plan for improvement before next year s report. 10 God s Bible School and College 2017 Assessment Document

11 Objective 6: Upon graduation, students from God s Bible School and College will demonstrate awareness of diverse cultural elements and the ability to evaluate them from a Christian worldview. Measures: Cross-Curricular Concern embedded assessment; Graduate Exit Survey Diversity appreciation is a Cross-Curricular Concern assessed by faculty in selected courses. Additionally, selected items on the Graduate Exit Survey address this objective. Cross-Curricular Concern Assessment Current Diversity Appreciation data is shown in Table 9; see rubric here. Currently there are scores for only 12 juniors and 6 seniors, limiting this measure s usefulness until more data is accumulated. The trend, though, is in a positive direction except for seniors, where only 2 of 6 achieved the target score of 3 or higher. Table 9. Diversity Appreciation Cross-Curricular Concern data Class standing Average (SD) Freshman 2.24 (0.96) Sophomore 2.41 (0.93) Junior 2.50 (0.67) Senior 2.33 (0.52) Graduate Exit Survey One Graduate Exit Survey item asks students to evaluate their grown in appreciation of cultural diversity. Other (Barna Group) items address diversity differently, probing whether respondents have a perspective in line with GBSC s faculty statement on diversity, which includes seeing some diversity as outside God s approval. As with other Graduate Exit Survey items, the target is 95% endorsing the preferred response; see results in Table 10. Table 10. Selected Graduate Exit Survey data Item % endorsing preferred response I have improved in appreciation of cultural 92.65% diversity The Bible specifically condemns homosexuality 95.45% It doesn t matter what religious faith you follow 96.97% Christians and Muslims worship the same God 95.31% 11 God s Bible School and College 2017 Assessment Document

12 Discussion While additional data would strengthen conclusions, existing data suggests that this objective is being partially achieved. Of particular value will be additional course-embedded assessment results. 12 God s Bible School and College 2017 Assessment Document

13 Objective 7: Upon graduation, students from God s Bible School and College will demonstrate proficiency in critical thinking, writing, public speaking and technology use. Measures: ETS Proficiency Profile/CAAP modules; Cross-Curricular Concern assessment The ETS Proficiency Profile and ACT s CAAP have been used for assessment in General Education areas. The Proficiency Profile has been phased out in favor of the CAAP. Data is analyzed longitudinally. Cross-Curricular Concern course-embedded assessment includes all four areas listed in this objective as well. ETS Proficiency Profile/CAAP For all areas, two targets exist: 80% of students will improve their scores and 60% of seniors will score above national means for specialized institutions. Table 11 shows current data. Table 11. Proficiency Profile/CAAP results Area % improved % above national mean Proficiency Profile overall 76% 94% Humanities 82% 93% Social science 68% 79% Critical thinking 77% 86% Reading 77% 93% Writing 59% 93% Math 77% 71% CAAP Writing 69% 75% Reading 58% 33% Critical thinking 58% 82% Math 27% 13% Cross-Curricular Concern assessment Faculty complete course-embedded assessment of critical thinking, writing, public speaking and technology use; the target senior score is 3 (0-4 scale). See results in Table God s Bible School and College 2017 Assessment Document

14 Table 12. Cross-Curricular Concern data Concern SR average (SD) % scoring 3 or higher Critical thinking 2.66 (0.97) 59% Public speaking 2.74 (0.78) 65% Technology use 2.72 (0.96) 58% Writing 2.54 (1.06) 53% Discussion The data presents a mixed picture: CAAP and Proficiency Profile data do not agree, and seniors outperform national norms at a better rate than they show improvement. In Cross-Curricular Concern data, seniors who have been evaluated are not achieving targets. Last year s planned efforts to understand these scores were not completed. An inter-rater reliability check is planned for the January 2018 Assessment Week, which addresses the Cross- Curricular Concern data, to determine if rubric adjustments are needed. It is also likely, though, that this outcome will be eliminated in the future. In response to the most recent HLC site visit, GBSC is developing new outcome statements for General Education, which should be presented for Board approval in May Those statements likely will replace this outcome. 14 God s Bible School and College 2017 Assessment Document

15 Objective 8: Upon graduation, students from God s Bible School and College will demonstrate knowledge of the arts and sciences. Measures: ETS Proficiency Profile/CAAP modules Data for Objective 8 includes that for Objective 7. The same targets also apply. Additional information from the Proficiency Profile applies here. ETS Proficiency Profile Math proficiency is measured as a skill subscore; Figure 1 shows results for students for whom longitudinal data on the standard Proficiency Profile is available (n=19). Note that the number of proficient or marginally proficient students increases for seniors at each math level. It is not clear, though, how these levels correspond with intended GBSC outcomes. Figure 1: Proficiency Profile math proficiency classifications Proficient Marginal Not proficient Figure 2Figure 2 shows results for Proficiency Profile subscores. Again, the data included are for the students for whom longitudinal data on the standard Proficiency Profile exists; norm data is for specialized institutions. In each case, GBSC s average performance exceeds the norm group. 15 God s Bible School and College 2017 Assessment Document

16 In every case, the senior-level difference between GBSC and the norm group is greater than the freshman-level difference, suggesting that GBSC s students show greater improvement than the norm group. Figure 2. Proficiency Profile comparative data Humanities Science Social science Critical thinking Reading Writing Math Freshman-Specialized mean Senior-Specialized mean Freshman-GBSC mean Senior-GBSC mean Discussion As noted above, senior performance presents a mixed picture. As the Assessment Committee carries out the tasks noted for Objective 7, they should also consider Objective 8 and this consideration should include whether additional or different assessment methods should be used. 16 God s Bible School and College 2017 Assessment Document

17 Objective 9: Upon graduation, students from God s Bible School and College will demonstrate knowledge and skills consistent with their professional areas. Measures: N/A Discussion Objective 9 is intended as a pass through objective, pointing toward students individual divisional and programmatic outcomes. Thus, it is not assessed at the institutional level, but serves as a placeholder to remind students and other stakeholders that outcomes do exist at other levels. See programmatic information in the following pages. 17 God s Bible School and College 2017 Assessment Document

18 Objective 10: Upon graduation, students from God s Bible School and College will demonstrate a written Christian servant-leadership philosophy statement that identifies leadership skills appropriate to their majors. Measures: Graduate Survey item (direct measure) Graduate Survey item Graduating students are asked this question: What is your philosophy of servant leadership? What skills does servant leadership include? Responses, along with the student s major, are evaluated by the Assessment Committee as meeting expectations or not meeting expectations. The target is for at least 70% of graduates to meet expectations. Table 13 presents Spring 2017 results. Table 13. Evaluation of servant-leadership philosophy statements Category 2017 Total responses 27 AA/AAS 6 (22%) BA 21 (78%) Rated 100% meets expectations 15 (56%) AA/AAS 3 (50%) BA 12 (57%) Rated >50% meets expectations 4 (15%) AA/AAS 1 (17%) BA 3 (14%) Rated 0% meets expectations 3 (11%) AA/AAS 1 (17%) BA 2 (10%) Discussion Results from this initial assessment round show students meeting the target, with 71% of respondents (overall and BA; 67% of AA/AAS) rated by 3 or more of the 4 evaluators as meets expectations. Existing results provide a starting point for divisions, and individual faculty, to consider how to strengthen students leadership growth. 18 God s Bible School and College 2017 Assessment Document

19 Program Objectives In recent Assessment Weeks, academic divisions have developed outcomes assessment plans for all degree programs, calling for cycles of assessment. The discussion below links to those plans and to existing data, when available. Divisions will incorporate assessment results into their Board reports. These reports will also include planned changes resulting from assessment of student learning. (Board reports from past semesters already show divisions engaging in this process with effectiveness assessment.) Division of Education and Professional Studies Programmatic assessment plan (note that the ISWM assessment plan needs to be incorporated in the future) Spring 2017 assessment update is included in the division s Assessment Week report Division of Ministerial Education Programmatic assessment plan Spring 2017 assessment update is included in the division s Assessment Week report, with more details given here Graduate program assessment information is included in a separate Assessment Week report, as well as the summer 2017 Ohio Department of Higher Education progress report Division of Music Programmatic assessment plan Spring 2017 assessment update is summarized in the division s Assessment Week report, with more details given here 19 God s Bible School and College 2017 Assessment Document

20 INSTITUTIONAL EFFECTIVENESS ASSESSMENT UIE System Assessment Document For a number of years GBSC has worked to implement an institutional-effectiveness assessment system controlled at the office/unit level. This was the Unit of Institutional Effectiveness (UIE) system. Reports for several years are available via the Institutional Research site. While progress has been made in seeing more reports, it is unclear if this system promotes as much improvement as the institution desires. Institutional Effectiveness Data The data presented below comes from the Noel-Levitz Student Satisfaction Inventory (SSI) and the internally developed Student Life Survey (SLS). Both use 7-point satisfaction scales, and averages of 5 or higher are seen as acceptable. Both instruments combine individual items into categories or scales. Scores here are category satisfaction averages. No scores are below target levels, suggesting good institutional effectiveness. Asterisks indicate the statistical significance of the difference between GBSC averages and national norms (*=0.05 level, **=0.01 level, ***=0.001 level). Table 14. SSI Scale Averages Scale (national norm) Student Centeredness (5.48)** Campus Life (4.92)*** Instructional Effectiveness (5.60)* Recruitment/Financial Aid Effectiveness (5.30)** Campus Services (5.59) Academic Advising Effectiveness (5.53) Registration Effectiveness (5.23)* Safety and Security (5.17)* Campus Climate (5.48)** Table 15. SLS Category Averages Scale Policies Recreation Services Spirituality Staff God s Bible School and College 2017 Assessment Document

21 EVALUATIVE CONCLUSION Student Learning Assessment Existing institutional data supports a conclusion that student learning objectives are largely achieved, but below target levels in some cases. This warrants investigation by the Assessment Committee. Following educational-measurement standard practices, the following steps should be completed: 1. Instrumentation review to ensure that (1) instruments are valid/reliable as they are used and (2) scoring or other mechanical errors are not occurring. 2. Review of performance targets to determine their reasonableness. 3. Review of relevant courses/curriculum to investigate (1) alignment with objectives, (2) alignment with instruments and (3) course-level student learning. Further, the Assessment Committee should identify or develop additional measures for several outcomes, as noted above. Programmatic assessment is on track. Existing plans should be implemented, including ongoing collection of data, review of results against established targets, planning for changes based on data, implementation of planned changes and review of new data to determine the effectiveness of changes. Institutional Effectiveness Assessment Institutional effectiveness data is strongly positive, with no scale/category scores below target levels. The Assessment Committee should continue work to replace the UIE system with administrative-level assessment reports. A target for initial implementation of a new approach is Spring 2018, for inclusion in next year s assessment report. 21 God s Bible School and College 2017 Assessment Document