China January 2009 International Business Trip Analysis

Transcription

1 1 China January 2009 International Business Trip Analysis Researchers and Presenters: Dr. Stephen Snyder, Natalie Nunes, and Heather Bartlett Presented to: Jeff Sherlock

2 2 Purposes of Investigation and Report August 2009 Executive Summary Pretest/Posttest Comparisons The first purpose of this study was to assess intercultural developmental changes of students involved in the International Business China (Hong Kong) program during January 2009 by comparing their scores on a pretest compared to their posttest. The analysis of this purpose includes a comparison of all 23 scales of the Taylor University Intercultural Inventory (TUII) which measures holistic intercultural development. The next analysis assessed the degree of change that took place in students in regard to Taylor University s Common Learning Objectives. Control Group Comparisons The second purpose was to compare those that went on the International Business China (Hong Kong) trip in January of 2009 with a control group. The analysis compared the development of the IDS of Taylor University of those who went on the China trip (N=18) with a combined control group (N=115) of freshman and seniors who have had no intercultural experience before or during their time at Taylor University. Results of Investigation Descriptive Means Summary Of the 23 scales on the Taylor University Intercultural Inventory (TUII), students who went on the China trip in January 2009 increased on 20 of the scale means from pretest to posttest. All five of the summary scales of the TUII focusing on cognitive understanding levels means increased, which include Knowledge, Awareness, Behavior, Intrapersonal Growth, and Attitudes. Students mean scores on two of the three personal development scales, Intellectual and Spiritual, increased from the pretest to posttest. The means remained the same for the development scale of Interpersonal. Pretest/Posttest Comparisons IDS Pre and Post Test Analysis Students who participated in the China trip had statistically significant gains due to their experience on 4 of the 23 scales of the TUII. A statistically significant increase was found for Knowledge Intellectual, Knowledge Spiritual, Total Behavior, and Awareness Spiritual. There was a significant decrease in Intrapersonal Spiritual. See Figure 1 and Table 1. Common Learning Objectives Pre and Post Test Analysis All seven of the common learning objectives had mean scores on the posttest that were found to be higher than the mean scores on the pretest. Significant differences were found between the preand posttests of students going on the China trip for five of the seven objectives (71%). The only objectives that did not have a significant increase from pretest to posttest were Lifelong Learning and Responsible Stewardship. See Appendix B for definitions of the Common Learning Objectives.

3 Cognitive Understanding Levels 3 IDS China and Control Group Comparison In 16 of the 19 IDS that were compared (the Intrapersonal scales could not be used because previous intercultural experience is required for them to be applicable), all but three (Attitude Intellectual, Attitude Interpersonal, and Total Attitude) were statistically significant. There was practical significance in 10 of the 23 IDS. Method Instrument The survey was comprised of demographic information and 130 inventory questions developed by the Taylor University research team, a Statistics and Design class and findings from a literature review conducted in the fall and spring of The survey was then reviewed by experts working with students who are involved in intercultural experiences and revised accordingly. Individual questions were based on a Likert scale and designed to evaluate holistic changes in students after cross cultural experiences. A total of 23 Intercultural Developmental Scales (IDS) shown below was generated from specific groups of questions to compare growth between and within subject groups. The survey is structured around personal development connections (3 categories) and cognitive understanding levels (5 categories). The number in parentheses shows the number of questions included in each scale. Figure 1: Intercultural Developmental Scales Personal Development Connections Intellectual Interpersonal Spiritual Knowledge Intellectual Knowledge (11) Interpersonal Knowledge (3) Spiritual Knowledge (10) Awareness Intellectual Awareness (6) Interpersonal Awareness (6) Spiritual Awareness (4) Attitudes Intellectual Attitudes (4) Interpersonal Attitudes (7) Spiritual Attitudes (9) Behavior Intellectual Behavior (18) Interpersonal Behavior (10) Spiritual Behavior (11) Intrapersonal Growth Intellectual Intrapersonal Growth (20) Interpersonal Intrapersonal Growth (6) Spiritual Intrapersonal Growth (5) Reliability Cronbach Alpha and the Test-Retest reliability methods were used to determine the degree of reliability that the inventory possesses. The Cronbach Alpha for overall internal consistency showed r(67) =.933 and r(419)=.965. To perform test-retest reliability, a group of 38 students were administrated the Taylor University Intercultural Inventory and then they retook the test after 3 weeks. The test-retest coefficient was r(38) =.915 for all questions. These two methods indicate that the Taylor University Intercultural Inventory has extremely high reliability.

4 4 Validity Based on strong reliability, content and construct validity was assessed. Content validity was established by basing the survey on current literature, reviewing prior study abroad assessment tools and allowing experts in intercultural study abroad trips to evaluate the instrument's accuracy and comprehensiveness. Construct validity was assessed using the Known-Group Difference method by analyzing the 419 subjects in two dimensions: amounts of experience and gender (See Table 2.0 and 2.1). Categories were created to distinguish between the different amounts of intercultural experience that students had. These categories were created by assigning students 1 point for each short-term experience and 3 points for each full-term experience they had, and then adding up their total number of points. The four categories were "none" (0 points), "low" (1 point), "medium" (2-4 points) and "high" (5-7). The amount of student intercultural experience formed a normal distribution. A One-Way ANOVA comparing the levels of intercultural experience showed significant differences between the students in 23 out of 24 IDS. Significant changes were seen in intercultural knowledge, F(3, 403) = 14.92, p <.001, awareness of cultures, F(3, 403) = 7.02, p <.001, attitudes towards different cultures, F(3, 403) = 13.01, p <.001, behavioral development, F(3, 403) = 9.13, p <.001, intrapersonal growth, F(3, 403) = 12.81, p <.001, intellectual development, F(3, 403) = 13.43, p <.001, interpersonal development, F(3, 403) = , p <.001, and spiritual development, F(3, 403) = 13.56, p <.001. Significant changes were also seen in nearly all subscales. Construct validity was also seen by the significant differences found between genders. Females scored significantly higher than males in intellectual awareness, general intrapersonal growth, and intellectual intrapersonal (p <.01). On all other categories except knowledge intellectual, females scored significantly higher than males (p <.001). Participants Overview A total of two groups were used in these analyses. The first was the China 2009 group (N=18). There were 18 participants in the 2009 January China program and 18 filled out a pre and post inventory for a return rate of 100%. The control group was made up of two groups: freshman and seniors who have had no intercultural experience prior or during their time at Taylor University (N=121). Demographics (See Table 1) The 2009 China group that we analyzed consisted of a total of 18 students. There were 11 females and 7 males in the program. Of that group, there were 3 seniors, 12 juniors, and 3 sophomores. Two of the students in this group lived in a country other than the United States prior to attending Taylor University. Also, 17 (94%) of the 18 students had been on an intercultural trip before attending Taylor University, while 1 had not. There were 2 students who were fluent in another language (15.5%). Of those students, 1 could speak Spanish and 1 Russian. While attending Taylor University, 11 (61%) had gone on an intercultural trip and 7 had not gone on any trips other than the China trip. It is important to note that 100% of the students in the China group have had some sort of intercultural experience, whether before or during their time at Taylor. For this reason, their pre/post analysis will not show as much change as it would if they had never had an intercultural experience.

5 5 Table 1: Participant Information Demographics China 2009 Class Seniors: 3 Juniors: 12 Sophomores: 3 Gender Female: 11 Male: 7 Prior to TU Experience 17 (94%) Lived in a country other than the U.S. before TU 2 students Intercultural experience at TU (other than Lighthouse) 11 (61%) Fluent in a language other than English 2 1 student: Spanish 1 student: Russian Analysis Descriptive statistics, Friedman s One Way ANOVA, and Kruskall Wallis tests were conducted on all 23 scales of the TUII and seven of Taylor University s Common Learning Objective scales. Practical significance (effect size) was found for all the significant scales in the above mentioned areas. Statistical significance indicates that there is a real difference in the impact of the China group between an individual s pre and post scores or between groups of individuals. Practical significance means that the change in student s intercultural development is at such a high level that the China procedures should be applied to other intercultural programs. It should be noted that non-parametric tests were used because the number of students measured was small (N=18). Our goal is not to generalize our findings to other universities with other China programs but to assess only the Taylor University China program. Non-parametric tests carry less power than parametric testing procedures. Results Descriptive Mean Summary We conducted a descriptive analysis on the data and received the means for the 23 intercultural development scales (IDS) for the pretest and posttest that the students took. Of the 23 scales, 20 increased from pre- to posttest. The means for the pretest ranged from compared to the posttest scores of It should be noted that the Attitude scales were extremely high (M = ) on the pretest which created a ceiling effect. The ceiling effect will prevent improvement on the posttest of students. The only two scales that decreased were Attitude Intellectual and Intrapersonal Spiritual. Means for Total Intrapersonal remained the same. See Table 2. Mean Increase Mean Decrease No Pre/Post Change

6 6 Table 2: Pre/Post Mean Comparison Pre-Test Mean Post-Test Mean Knowledge Intellectual Knowledge Interpersonal Knowledge Spiritual Total Knowledge Awareness Intellectual Awareness Interpersonal Awareness Spiritual Total Awareness Attitude Intellectual Attitude Interpersonal Attitude Spiritual Total Attitude Behavior Intellectual Behavior Interpersonal Behavior Spiritual Total Behavior Intrapersonal Intellectual Intrapersonal Interpersonal Intrapersonal Spiritual Total Intrapersonal Total Intellectual Total Interpersonal Total Spiritual *Means are based on a 6 point likert scale with 6 being high and 1 being low. Pre and Post Test Analysis of Intercultural Development Development of the China students was assessed according to the 23 IDS using the Friedman One-Way ANOVA. The results are organized into three broad categories: Personal Development Connections, Cognitive Understanding Levels, and Nested Subscales. See Appendix A for variable definitions. Of the 23 IDS, 20 means increased from pretest to posttest. Of those increased scales, 4 of the 23 increased significantly. The two means that did not increase were Attitude Intellectual and Intrapersonal Spiritual, and the Total Intrapersonal means remained the same. Intrapersonal Spiritual indicated a significant decrease from the pre to post testing. Out of the 5 significant areas, all 5 showed effect sizes in the small to medium range. Knowledge Intellectual had an effect size of.45 which is almost in the large range for effect sizes. Knowledge Spiritual, Intrapersonal Spiritual (practically significant decrease), Awareness Spiritual, and Total Behavior indicated small practical significance (.18,.26,.10, and.24). See Figure 2 and Table 3.

7 Cognitive Understanding Levels 7 Figure 2: Intercultural Developmental Scales Personal Development Connections Intellectual Interpersonal Spiritual Knowledge Knowledge Intellectual (11) Knowledge Interpersonal (3) Knowledge Spiritual (10) Awareness Awareness Intellectual (6) Awareness Interpersonal (6) Spiritual Awareness (4) Attitudes Attitudes Intellectual (4) Attitudes Interpersonal (7) Attitudes Spiritual (9) Behavior Behavior Intellectual (18) Behavior Interpersonal (10) Behavior Spiritual (11) Intrapersonal Growth Intrapersonal Growth Intellectual (20) Intrapersonal Growth Interpersonal (6) Intrapersonal Growth Spiritual (5) *** Yellow Cells indicate areas students had significant improvement in from the beginning to the end of their China International Business experience. Table 3: Significance: p<.05 p<.01 p<.001 p>.05 Effect Size: Small:.21 Medium: Large:.31+ None Intercultural Development Scales: Mean N Chi df Sig Difference Sq. Knowledge Intellectual: Knowledge Interpersonal: Knowledge Spiritual: Total Knowledge: Awareness Intellectual: Awareness Interpersonal: Awareness Spiritual: Total Awareness: Attitude Intellectual:

8 8 Attitude Interpersonal: Attitude Spiritual: Total Attitude: Behavior Intellectual: Behavior Interpersonal: Behavior Spiritual: Total Behavior: Intrapersonal Intellectual: Intrapersonal Interpersonal: Intrapersonal Spiritual: Total Intrapersonal: Total Intellectual: Total Interpersonal: Total Spiritual: Taylor University Common Learning Objectives Analysis Taylor University has eight common learning objectives which students are intended to achieve as academic outcomes. These learning objectives are: Spiritual Activity, Critical Thinking, Competent Communication, Scientific Literacy, Aesthetic Literacy, Civic Mindedness, Responsible Stewardship, and Lifelong Learning. Matching these objectives with questions from the Taylor University Intercultural Inventory (TUII), we assessed how well the goals were being met according to the pre- and posttest scores of the January 2009 China students using a Friedman One-Way ANOVA test. Only seven of the eight objectives were actually tested because there were no questions that matched the Scientific Literacy objective in the TUII. All seven of the common learning objectives had mean scores on the posttest that were found to be higher than the mean scores on the pretest. Significant differences were found between the preand posttests for five of the seven objectives (71%). The only objectives that did not have a significant increase from pretest to posttest were Lifelong Learning and Responsible Stewardship. Practical significance was found in five of the seven areas: Spiritual Activity, Competent Communication, Critical Thinking, Civic Mindedness, and Aesthetic Literacy. See Figure 3 and Table 4.

9 9 Figure 3: Taylor University Common Learning Objectives Pre/Post Mean Comparison Pretest Posttest 0 Table 4: Common Learning Objectives Pre/Post Analysis Common Learning Objectives Means N Chi Sq. df Sig C 2 Spiritual Activity: Pre: 4.82 Post: Responsible Stewardship: Pre: 4.76 Post: Lifelong Learning: Pre: 5.00 Post: Competent Communication: Pre: 3.67 Post: Critical Thinking: Pre: 4.64 Post: Civic Mindedness: Pre: 4.42 Post: Aesthetic Literacy: Pre: 4.28 Post:

10 10 China Group vs. Control Group Comparison Comparing the scores given by the control group with the posttest scores of the China group using a Kruskall-Wallis test and mean comparisons, the effects of participating in an intercultural experience are clearly seen. Of the 19 intercultural developmental scales that were compared, all but two (Intellectual Attitude and Total Behavior) were statistically and practically significant (the Intrapersonal scales could not be used because previous intercultural experience is required for them to be applicable). This indicates the importance for business majors to go on this trip compared to not going on any kind of intercultural trip. It should be kept in mind that all of the students participating in the China trip had some type of previous intercultural experience, so it can be expected that they would have higher means than the control group before going on the trip. See Table 5 for more details. Table 5: Control Group vs. China Group IDS Comparison IDS Mean Chi Sq. df Sig. Knowledge Intellectual Knowledge Interpersonal Knowledge Spiritual Total Knowledge Awareness Intellectual Awareness Interpersonal Awareness Spiritual Total Awareness Attitude Intellectual Mean: China Group Post Control Scores Significance: p<.05 p<.01 p<.001 p>.05 Effect Size: Small: Medium: Large:.50+ None Attitude Interpersonal C 2

11 Attitude Spiritual Total Attitude Behavior Intellectual Behavior Interpersonal Behavior Spiritual Total Behavior Total Intellectual Total Interpersonal Total Spiritual Recommendations Intercultural Development Due to the changes in intercultural development in students participating in the January 2009 International Business trip to China program on 4 of the 23 scales of the TUII, it is recommended that this program be continued with goals for improvement. With the exception of three areas, all posttest means were higher than pretest. This indicates that growth took place among students who enrolled in the China program, but the growth is not statistically significant in all areas. We recommend increased preparation beforehand, reflection time during the trip, and debriefing time after the experience to improve posttest scores. More reflection by the students could significantly increase the benefits of the program. The Lighthouse program has seen significant and practical changes in all Personal Development Connections and Cognitive Understanding Levels due to the program s strong emphasis on preparation, refection during the experience, and debriefing. Common Learning Objectives of Taylor University The China program seems to be very instrumental in accomplishing the common learning objectives, considering that five of the seven objectives were statistically significant due to the experience abroad. It is recommended that the findings from this analysis be integrated into a brochure for next year s trip and also be promoted in the Business Department as a valuable experience for business majors.

12 12 Appendix A Definitions of Taylor University Intercultural Inventory (TUII) All questions were put into categories which were named the Intercultural Developmental Scales (IDS; Table 1). Intercultural Development is defined as cultural growth in an individual s cognitive understanding in relation to intellectual, interpersonal, and spiritual connections. The first three categories formed the Personal Development Dimension. The next five categories are termed Cognitive Understanding Levels to reflect the depth of processing and understanding an individual may possess in a content domain. Instructions were given, and participants were asked to rate each question on a 6- point Likert scale from strongly disagree to strongly agree. The nested subscales of this instrument contained 143 items. Personal Development Connections 1. Intellectual: It describes the formation of cognitive thoughts and processes concerning a unique culture. This category stresses how a person will internally relate to one self. 2. Interpersonal: It describes the interaction occurring between individuals of different cultures. This category stresses how an individual relates to other people on the vertical level. 3. Spiritual: It describes the progressive and dynamic interactions with a personal God and spiritual community of a unique culture. This category stresses how an individual relates to a world that cannot be seen but accepts by faith focusing on a horizontal dimension. Cognitive Understanding Levels 4. Knowledge: Understanding of specific facts, truths and principles of different cultures 5. Awareness: Perceiving differences between aspects of unique cultures 6. Attitudes: Personally held dispositions concerning different cultures 7. Behavior/Skills: The ability to act effectively and appropriately on facts, truths and principles of a unique culture 8. Intrapersonal Growth: Internal development due to knowledge, awareness, attitudes and behavior/skills from cultural experiences. It stresses the personal internal incorporation of what was learned. Nested Subscales The personal development dimension and the cognitive understanding levels dimension were structured in a matrix that formed the subscales listed below: 9. Intellectual Knowledge: The formation of cognitive thoughts and processes based on the understanding of specific facts, truths and principles of unique cultures 10. Interpersonal Knowledge: Understanding of specific cultural facts, truths and principles in order to effectively interact with people of different cultures 11. Spiritual Knowledge: Understanding of the culture s perspective on specific cultural facts, truths, and principles about God and the effect it has on the individual s interaction with God 12. Intellectual Awareness: The formation of cognitive thoughts and processes by recognizing the existence of the facts, truths, and principles of unique cultures 13. Interpersonal Awareness: The interaction between individuals based on an accurate perception of unique cultures 14. Spiritual Awareness: Perceiving the spiritual significance between aspects of unique cultures 15. Intellectual Attitudes: Personally held dispositions concerning different cultures based on the formation of cognitive thoughts and processes due to a cultural experience

13 Interpersonal Attitudes: Personally held dispositions concerning different cultures that change the interaction occurring between people of different cultures 17. Spiritual Attitudes: Personally held dispositions concerning different cultures that change the individual s interaction with God and a spiritual community 18. Intellectual Behavior/Skills: The use of abilities to act on the facts, truths, and principles found through the formation of cognitive thoughts and processes developed from experiencing a unique culture 19. Interpersonal Behavior/Skills: The ability to act effectively and appropriately during the interaction between individuals of different cultures 20. Spiritual Behavior/Skills: The interaction of God and spiritual community demonstrated through the use of an individual s cultural skills 21. Intellectual Intrapersonal Growth: The development of the cognitive thoughts and processes of an individual concerning different cultures 22. Interpersonal Intrapersonal Growth: The development within the individual due to the interaction with individuals of different cultures 23. Spiritual Intrapersonal Growth: The development that occurs within the individual due to their interaction with God Appendix B Definitions of Common Learning Objectives Taylor University s educational Common Learning Objectives include: Critical Thinking, Spiritual Activity, Competent Communication, Scientific Literacy, Aesthetic Literacy, Civic Mindedness, Responsible Stewardship, and Lifelong Learning. Scientific Literacy was not included in these findings because there were no questions in the TUII that pertained to this objective. Critical Thinking: Students who are critical thinkers have learned the intellectually disciplined process of conceptualizing, quantifying, analyzing, synthesizing, and evaluating information gathered from or generated by observation, experience, reflection, reason or communication as a guide for action or belief. Spiritual Activity: Students who are spiritually active have developed an intellectual and experiential understanding of the Christian heritage enacted in a consistent lifestyle of study, worship, service, stewardship and world outreach. Competent Communications: Students who are competent communicators have developed understanding and skill in the nature and practice of private and public discourse as instruments for creating, interpreting, and evaluating beliefs, attitudes, policies and values in the context of various communities. Aesthetic Literacy: Students who are aesthetically literate have developed knowledge of the unique epistemology represented by the arts including a discerning awareness of the language and literature of diverse art forms. Civic Mindedness: Students who are civic minded have attained systematic knowledge of the structure and processes of domestic and international cultural, economic, political, and social systems, and value participation in those structures and processes. Responsible Stewardship: Students who are responsible stewards have developed an understanding of God s command to be good caretakers of His creation, and practice individual accountability in managing spiritual, intellectual, personal, physical, and economic resources. Lifelong Learning: Students who are lifelong learners are committed to learning as a means of continuous growth and development, which expresses itself in ministry and reverence toward God in an ever changing world.