Meaning-Making Among Medical Students: Development of a Quantitative Measure of Self- Authorship

Transcription

1 Seton Hall University Seton Hall Seton Hall University Dissertations and Theses (ETDs) Seton Hall University Dissertations and Theses Spring Meaning-Making Among Medical Students: Development of a Quantitative Measure of Self- Authorship Robert Fallar robert.fallar@student.shu.edu Follow this and additional works at: Part of the Educational Assessment, Evaluation, and Research Commons Recommended Citation Fallar, Robert, "Meaning-Making Among Medical Students: Development of a Quantitative Measure of Self-Authorship" (2014). Seton Hall University Dissertations and Theses (ETDs)

2 MEANING-MAKING AMONG MEDICAL STUDENTS: DEVELOPMENT OF A QUANTITATIVE MEASURE OF SELF-AUTHORSHIP ROBERT FALLAR Dissertation Committee Eunyoung Kim, Ph.D., Mentor Elaine Walker, Ph.D. Erica Friedman, M.D. Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Seton Hall University 2014

3 2014 Robert Fallar

4

5 ABSTRACT Preparation for and application to medical school, as well as the subsequent medical training of matriculating students, can have an important impact on psychosocial development. The premedical baccalaureate is the traditional preparation for medical school, although many medical schools also offer a separate entry path through early assurance programs that provide conditional acceptance in the sophomore year of college. These programs may provide freedom in the remainder of the baccalaureate program to explore the liberal arts, which could be a source of differential development of noncognitive skills among medical students. Self-authorship is defined as the ability to define one s beliefs, identity, and social relations and provides the basis to operate in a complex, ambiguous environment. Such a capacity is relevant to the medical education, where students face changing roles and expectations as they progress through four years of medical school and then graduate training. This study investigated the application of a previously validated quantitative measure of self-authorship across the new population of medical students. Principal component analysis identified a 16-item instrument that measured three dimensions and one phase of self-authorship. Content analysis of responses validated the components. The findings identified the challenges in applying a quantitative measure of selfauthorship to medical students. These students may be at a different level of development and require different contexts than those in previous studies. Further research is needed to investigate differences in development across students based upon the entry path to medical school. ii

6 ACKNOWLEDGMENTS While you can read numerous books about the process of writing a dissertation, nothing really prepares you for the actuality when it s time to write your proposal and then follow through with the proposed research. I have to thank some people who have helped make this process less painful and more productive than I thought it could be at times. First, I extend my biggest thanks to my co-researchers at the Icahn School of Medicine who graciously allowed me to co-opt the data from our survey as the basis for my research. Thanks go to Dr. Basil Hanss, Dr. Craig Katz, and Roberta Sefcik, MS2. Second, my research committee provided me with invaluable help and perspective as I sought to understand the potential of a new measure of self-authorship. My sincerest thanks go to my mentor, Dr. Eunyoung Kim, as well as my committee members, Dr. Elaine Walker and Dr. Erica Friedman. I also have to extend thanks to Dr. MaryDee McEvoy, who was a colleague and confidante for many years at The Mount Sinai Hospital. While she has moved on to enjoy retirement, she has continued to provide me with moral support and encouragement in these last few months as this project has come to an end. Finally, I wanted to thank my partner, my family, and my friends by name for their support during this process and realized I would forget someone unintentionally. Better that I just say a big thanks to each and every one of you. You know who you are. iii

7 TABLE OF CONTENTS CHAPTER 1. INTRODUCTION... 1 Background... 1 Theoretical Framework Conceptual Model Justification for the Study Research Context and Motivation Purpose of Study and Research Questions Significance Overview of Chapters CHAPTER 2. REVIEW OF THE LITERATURE Theories of Student Development Self-Authorship and Medical Education Quantitative Methods to Assess Self-Authorship CHAPTER 3. METHODS Pilot Study Research Project CHAPTER 4. RESULTS Descriptive Statistics Reliability Principal Component Analysis Sub-Group Analyses Qualitative Analysis of Open-ended Responses CHAPTER 5. CONCLUSION Summary of Research Findings Implications for Theory Implications for Practice Limitations Suggestions for Future Research REFERENCES APPENDIX A. Comparison of Survey Items from Creamer, Baxter Magolda, and Yue (2010) with Study Pilot Phase Items APPENDIX B. Revised Survey Items for the Research Project with Indicators for Proposed Dimension and Phase of Self-Authorship APPENDIX C. Correlation and Covariance Matrix for the Full Data Set iv

8 LIST OF TABLES Table Page 1. Number of Survey Items in Pilot Phase by Developmental Stage and Dimension Representative Characteristics as a Percentage of Respondents and Population Component Loadings Using Principal Components with Promax Rotation Correlation Matrix for Pilot Study Principal Component Analysis with Promax Rotation Number of Survey Items in Study Phase by Developmental Stage and Dimension Demographic Characteristics of Study Respondents Representative Characteristics of Study Respondents as a Percentage of Respondents and Population Mean and Standard Deviation by Survey Item and Study Cohort Reliability Statistics in Total and by Subgroup Component Loadings Using Principal Components Analysis with Promax Rotation Component Correlation Matrix after Using Principal Components Analysis with Promax Rotation Factor Loadings and Components Using Principal Components Analysis with Promax Rotation FINAL MODEL Component Correlation Matrix after Using Principal Components Analysis with Promax Rotation Factor Loadings Using Principal-axis Factor Analysis with Promax Rotation FINAL MODEL Component Loadings Using Principal Components Analysis with Promax Rotation PREMED STUDENTS v

9 16. Communalities for Principal Component Analysis Early Assurance Students Frequency Distribution of Responses by 159 Students to Open-end Question Summary Table Illustrating Examples of Qualitative Analysis of Responses...84 vi

10 LIST OF FIGURES Figure Page 1. Phases and Dimensions of Self Authorship , Relationship between Experience and Meaning-Making among Early Assurance and Traditional Premed Students Scree Plot for Exploratory Factor Analysis Using Principal Components Scree Plot for Principal Component Analysis with Promax Rotation Using 19 Variables...71 vii

11 CHAPTER 1 INTRODUCTION Background The process of preparation for and application to medical school, as well as the subsequent matriculation into and completion of training, can have an important impact on the psychosocial development of students. The current process of preparation for medical school and the medical school curriculum itself have antecedents dating back to the early 1900s. At that time there were a large number of poorly trained doctors coming from schools that did not provide a proper medical education. In 1910 the Flexner Report, sponsored by the Carnegie Foundation for the Advancement of Teaching, summarized the standards for a good medical education and provided an analysis of each of the medical schools, citing how each performed against these standards (Flexner, 1960). This report, along with subsequent work by the American Medical Association, led to the primacy of a premedical (premed) bachelor s degree as the necessary scientific preparation for medical school (Gross, Mommaerts, Earl, & De Vries, 2008). However, since the emergence of premed programs at the baccalaureate level, there have been concerns over both the curriculum and the high-pressure culture it breeds among students. As part of taking the premed curriculum, students focus on achieving high grades in classes and on the MCAT standardized admissions test in order to improve their chances of admission to medical school, which is decided in their senior year of college (Gunderman & Kanter, 2008). Students may study for and take the MCAT, often up to three times, in order to obtain the 1

12 highest score they can. This has become the traditional path upon which students endeavor to apply to medical school. In addition to the traditional entry path, there are several alternatives offered at some medical schools that provide entry for subgroups of students alongside their traditional premed peers. There are combined BS/MD degree programs, which can run from five to eight years, with the BS taking anywhere from one to four years before the student moves on to medical classes (Eaglen et al., 2012). There are also early assurance programs, which are a loosely defined set of programs that allow a student to be accepted to medical school in his or her sophomore year; however, matriculation into medical school does not occur until after completion of the bachelor s degree. To date, empirical literature evaluating the purposes or goals of early assurance programs is almost non-existent. An informal interview of the 11 medical schools with the largest classes of early assurance students identified that even after acceptance, many still require the MCAT as part of the requirements for matriculation (S. Shadravan, personal communication, November 27, 2012). A review of online resources at three of the schools reveals that these programs tend to recruit honors students and often have a goal of educating a particular type of doctor, such as rural physicians (TTUHSC School of Medicine, 2013; University of Maryland, College Park, 2013; Upstate Medical University, State University of New York, 2013). Several programs do mention on their web sites that the goals of the programs include the potential to reduce the stresses of the traditional premed program and to allow the students more freedom to explore their undergraduate education. However, no empirical research is available to identify the non-cognitive 2

13 effects or benefits of the assumed freedom early assurance students attain during their last two years of college. In a recent study, Muller and Kase (2010) demonstrated how students from an early assurance program with some unique features (elimination of the MCAT entrance exam and discouragement of a premed undergraduate degree) perform just as well as, if not better than, their traditional premed-trained classmates on many measures of academic performance during medical school. When the findings were highlighted in a front-page article of the New York Times, the subsequent online discussion was swift, rather onesided against the idea of such a program, and somewhat vitriolic in tone (Hartcollis, 2010). A common theme across the comments was that the students were less prepared for medical school and graduates of this particular early assurance program would be less qualified doctors than their peers, incapable of understanding concepts such as prescription drug interactions or advanced testing procedures. Many comments seemed to assume that the basic science in a premed education is the only legitimate source of this knowledge and failed to acknowledge that the same science knowledge is addressed in medical school itself. While admittedly not representative, these comments highlight a view held by the general public about the requirements to become a doctor, as well as the type of physician this particular program graduates. Though Muller and Kase (2010) provided the first evidence to identify possible differences on cognitive measures between the early assurance students and traditional premed students by the end of medical school, they did not look at possible differences at the start of medical school nor investigate any non-cognitive differences in students either at the beginning or end of medical school. If there are differences in non-cognitive skills 3

14 among different groups of medical students, one potential influence could be the different paths these students took prior to matriculation, including removal of the academic pressures of the premed undergraduate program and possible differences in preparing for medical school. Preparation for Medical School Historically, students prepare for medical school with a premed undergraduate degree and by taking the MCAT, often more than once during their college years in an effort to achieve the best score they can (Zhao, Oppler, Dunleavy, & Kroopnick, 2010). The premed curriculum includes general chemistry, organic chemistry, biology, physics, and calculus. Critics are concerned that the content of the curriculum places too much or too little focus on science and thereby a premed program might instill some negative behaviors in students, such as competitiveness and a focus on grades (Gross et al., 2008). While no one suggests the premed curriculum should be abolished, the content of a premedical program has received the most consistent criticism over time. The primary arguments against the curriculum relate to the desire to maintain exposure to the liberal arts as part of a student s education (Brieger, 1999; Gross et al., 2008). The development of clinical skills is of primary importance in becoming a doctor, but so is the ability to communicate well, a sense of objectivity, and an understanding of the world from a natural, social, and cultural perspective (Weingartner, 1980). Further, Weingartner suggested that while maintaining some basic science courses, a premed program should require a humanities or social science major or concentration. Some critics also note that the purpose of a premed program is not to make medical school easier, but instead to prepare a student for the rigors of a medical education. Therefore, the actual science of 4

15 medical school should not be taught beforehand in a premed program; rather, discipline and a well-rounded education are paramount (Brieger, 1999). What purpose should an undergraduate education serve for a prospective medical student? An undergraduate degree must provide a foundation in liberal arts for all students. The premed program should not be viewed as just a prerequisite for medical school but as an undergraduate degree that provides a well-rounded, holistic education (Gunderman & Kanter, 2008). The science should be limited to that which is most relevant to medicine, opening up the opportunity to explore the liberal arts (Dienstag, 2008). In particular, students need to learn how to acquire knowledge and understand selfreflection, both of which are regarded as important qualities for a doctor s career. From a broader perspective, there have been long-standing issues with the definition, value, and purpose of a liberal education itself. A liberal arts education has been a fundamental pillar of American higher education since the colonial period, building on the concepts of the European system (Thelin, 2004). In 1947, the Truman Commission proposed a set of principles for providing access to higher education and guidelines for what every student should learn. The report distinguished liberal arts from a general education, which included the development of basic skills, responsible citizenship, and multicultural relationships in the world (Hutcheson, 2007). The Commission report was instrumental in improving access to higher education with programs like the GI Bill, but it did not effectively lead to initiating change in teaching general education skills. By the late 20 th century, an explosion of new technology vastly affected society s ability to automate processes in business and manufacturing, as well as the ways in which we communicate and interact with others. In this more complex work environment, the 5

16 curriculum of higher education shifted from a utilitarian focus on specific skills to an interest in the more non-cognitive skills that are desirable for college graduates and the new work force (Baxter Magolda & King, 2012). The higher learning skills would help to better prepare students to work in a complex environment where they would have to deal with uncertainty, think critically, and be self-sufficient to find solutions to problems (Hodge, 2009). In 2005, the Association of American Colleges and Universities began a decadelong project called Liberal Education and America s Promise (LEAP) in an effort to change the goals of higher education to meet the needs of a more complex world (The National Leadership Council for Liberal Education and America's Promise, 2007). The proposed new outcomes for college graduates included the following: Knowledge of cultures and the physical and natural world Intellectual and practical skills (critical thinking, problem solving, and teamwork) Personal and social responsibility (fostering responsibilities beyond oneself) Integrative learning across subject areas The educational leaders who participated in the creation of LEAP understood the complexity of improving these skills and acknowledged that one program would not be effective across all types of programs/institutions. Rather, programs should be designed to include interventions that are specific to their pedagogy and environment. The proposed skills are, however, somewhat vague and left to the individual institutions to specify and define. LEAP does not identify specific tools to measure these skills but recommends that any assessments be linked to the specific learning situations. They offer broad examples such as portfolios and capstone projects, providing evidence to identify if students have 6

17 met the goals or not; but these examples are not specific enough to allow for any measure of progress along a continuum up to and beyond meeting the goal. In July 2012, the National Academies published Education for Life and Work, based upon the work of another expert panel charged with defining 21 st century skills that should be the goal of higher education (Board on Testing and Assessment, 2012). The identified skills fall into three domains: cognitive (reasoning), interpersonal (expressing oneself and interpreting messages from others), and intrapersonal (the ability to regulate oneself when working with others towards goals). This report reflects recent theoretical work on student development, specifically the concept of self-authorship, calling for more research to understand the link between these skills and outcomes after college. The reports from the National Academies and LEAP indicate there is a growing public interest to improve the outcomes of higher education and provide accountability towards higher learning skills over and above the accumulation of specific knowledge within an academic discipline. A common theme to the criticism and discussion thus far points to the need for students to accumulate more non-cognitive skills that can be helpful in both the educational experience and in the students eventual careers. The National Academies goals apply to all types of higher education, and some critics have particularly specified the benefits for medical students to develop these same types of skills after having exposure to the liberal arts as undergraduates (Brieger, 1999; Dienstag, 2008; Weingartner, 1980). Regardless of whatever efforts may arise from the LEAP and National Academies calls for improvement in liberal arts education, the opportunity for 7

18 early assurance students to have more exposure to liberal arts than their premed peers may lead to the potential for differences in non-cognitive skill development between the two. A second issue with premedical education involves the culture it perpetuates among the students in the programs: that the students become too fixated on grades and the likelihood of being admitted to medical school while missing out on a richer, more holistic undergraduate experience (Gross et al., 2008; Weingartner, 1980). Although early assurance programs can be the opportunity to enhance the liberal education of aspiring medical students while removing some of the pressure to achieve the highest grades, there is no empirical evidence to support any potential benefits from these programs. Assessing Medical School Applicants Since concerns about the content of the premed curriculum have existed since the early 20 th century, it can be assumed that the general nature of the premed curriculum will not change quickly. The focus of criticism then shifts to how medical schools select applicants, with the primary suggestion being to move away from relying on one s GPA and MCAT scores as the most significant indicators of success. Acceptance to medical school is usually determined based on cognitive measures (GPA, MCAT) as well as other aspects of personal development of the student, often assessed based upon interviews and written personal statements. These measures are used to identify the students who are most capable of succeeding in medical school and becoming the kind of doctors the schools seeks to train. There is a large volume of literature on the positive correlation between MCAT scores and GPA with performance in the first two years of medical school (Peskun, Detsky, & Shandling, 2007). For example, the MCAT and GPA have been shown to be valid predictors of written knowledge assessment throughout medical school, 8

19 as well as in clinical skills assessments (Kreiter & Kreiter, 2007). However, there is inconsistent evidence in the literature on the relationship between these scores and performance in the last two years of medical school. For instance, research has shown that the MCAT has little predictive validity for clinical performance among minority students (White, Dey, & Fantone, 2009). The undergraduate GPA has also been found to not be predictive of performance in the clinical phase (third and fourth years) of medical school (Silver & Hodgson, 1997). Other researchers have indicated that science indicators (MCAT and GPA) are actually inversely related to non-cognitive performance, suggesting that those with higher MCAT or GPA scores may not be as advanced as others in the domain of non-cognitive skills (Barr, 2010). Furthermore, the issue of bias within these measures has been raised. Standardized tests are designed to distinguish between students and highlight those who are more or less knowledgeable. These types of tests may be biased against minorities, providing inadequate measurement of other desirable skills among these students (Sedlacek, 2004). Therefore, other non-cognitive measures may provide better information to support disadvantaged students in aspects such as gaining access to medical schools. Admissions decisions can have different purposes, including being a means to identify who is most prepared for the rigors of medical school and/or to select prospective students who will become the type of doctor the school wishes to train. Reliance on standardized cognitive indicators such as the MCAT and the focus on premed training can impact admissions decisions regarding who will be able to be trained to become a physician. Any potential bias introduced by these indicators may also influence who ultimately applies and gets accepted to medical school. 9

20 Given the questionable utility of these cognitive indicators in predicting success through various stages of medical training, non-cognitive measures including essays, interviews, and multiple mini-assessments have become more common in the application process as alternative methods of assessing candidates. These non-cognitive measures have been found to be more predictive of academic success in the first two years of medical school than the traditional cognitive measures (Peskun et al., 2007). For example, multiple mini-interviews provide the opportunity for prospective students to react to various situations that measure competencies such as ethical behavior that cannot be measured by the traditional cognitive methods. Mini-interviews have been found to be a fair and engaging method of assessing students without the pressure of traditional assessments (Razack et al., 2009). Despite little consensus on which non-cognitive skills are the best criteria for evaluating applicants, many medical schools have incorporated some type of non-cognitive measure of applicants into their review process (Bardes, Best, Kremer, & Dienstag, 2009). Many of these non-cognitive measures require significant human effort in terms of reviews of written documents or ratings of interactive activities. These are qualitative in nature, requiring significant resources to implement on any largescale evaluation of students. Alternatives to the Traditional Entry Path Along with changes to the applicant review process for the traditional medical school entry path, there are now additional programs to gain entry to medical school. One popular non-traditional entry path to medical school is a combined baccalaureate-md program. Many of these programs are at medical schools within a university setting, where the early acceptance is for undergraduates at the same institution. As of 2011, 44% (57) of 10

21 the medical schools in the United States offered some type of combined program (Eaglen et al., 2012). In addition, there are 30 medical schools in the United States that have an early assurance program (Hutcheson, 2007). Early assurance programs guarantee admission to medical school during the sophomore year. There is no standardized format for such programs; but most, if not all, require some type of premedical education while in college and may or may not require the MCAT. These programs guarantee acceptance to medical school prior to the third year of college, thus eliminating some of the traditional premed application steps. More than half of the early assurance programs report that the primary goal of the program is to attract honors students into medicine, and 22% report that the mission is to integrate the liberal arts into medicine (Eaglen et al., 2012). The types of students applying to medical school are still predominantly biology majors. While the composition of MCAT examinees is not fully reflective of those who attend medical school, the data show that students from non-premed backgrounds are still a minority taking the exam as part of the traditional path into medical school (Association of American Medical Colleges, 2011). No data on the demographic characteristics (race, academic performance, gender) of students nationally who apply to the early assurance programs are available. How do early assurance programs counter the traditional criticism of medical school admissions? First, they may remove the MCAT scores from the admission decision process and, in so doing, remove the pressure on the prospective student to study for, and take, the exam during their undergraduate education. Second, they provide the student with the opportunity to explore classes of their choice in the last two years of college, free of the need to have their transcript include courses they might think imperative to get 11

22 accepted to medical school. Third, the opportunity to broaden the liberal arts education for early assurance students might provide a more humanistic experience than traditional premedical programs. Taken together, these differences may create a subgroup of medical students different than their premed-trained peers. Still, there has been little empirical evidence of the effects of an early assurance program. As Eaglen et al. (2012) state, For those programs that strive to reduce competitive pressures or emphasize humanistic qualities, there have been no studies to date indicating whether students in those programs are different in any meaningful way by the time they begin medical school (p. 5). The historical arguments regarding premedical undergraduate programs and the criteria used for acceptance to medical school, as well as the number of alternative entry paths to medical school, all suggest the need to investigate non-cognitive development among medical students along the trajectory from baccalaureate through medical school. However, the literature provides no evidence of any broad measure of non-cognitive development among this population. This study sought to validate a quantitative measure that could provide a first effort to more broadly determine potential non-cognitive development among medical students, with analyses to determine validity across subgroups. Theoretical Framework When studying the cognitive and non-cognitive development of students as they enter and progress through medical school, self-authorship provides a useful theoretical lens. Self-authorship, defined as the internal capacity to define one s beliefs, identity, and social relations (Baxter Magolda, 2008, p. 269) seeks to explain how a person is able to consider external influences and persevere in a complex, ambiguous environment through 12

23 a threefold set of dimensions: epistemological (what one believes), intrapersonal (what one values), and interpersonal (how one relates with others in mutually beneficial ways) (Baxter Magolda, 2008). The epistemological dimension refers to how a student deals with ambiguity and conflicting information and grows from accepting knowledge as coming from authority figures to the point where they understand that knowledge can be contextually driven and interpreted differently by different people. According to Pascarella and Terenzini (2005), epistemological skills provide the foundation for making choices among truth claims, values, and behaviors (p. 49). The intrapersonal element refers to how one grows to understand one s own values and beliefs and becomes comfortable using these to interpret information and deal with others. Similarly, the interpersonal element identifies how one relates with others while maintaining one s sense of self. These three elements are intertwined and equally important to the development of self-authorship. Baxter Magolda (2008) identifies three phases in the development towards selfauthorship. In the External formulas phase, students base decisions and actions on influences and knowledge provided by authority figures. As they move to the Crossroads phase, students question knowledge and formulas that conflict with their own ideas, yet they do not know how to resolve these conflicts. By the Early self-authorship phase, students can understand their own beliefs and how they coincide or conflict with External formulas. Subsequent decisions can be made by taking into account their own values and beliefs and working with others to achieve mutually beneficial goals. Figure 1 displays the dimensions and phases of self-authorship. 13

24 Figure 1. Phases and Dimensions of Self-Authorship. Adapted from Joseph Boehman (2011, April 28). Who are you? Self-authorship defined [Web log post]. Retrieved from Self-authorship is relevant to medical education for several reasons. Firstly, premed education relies heavily on external authorities as one learns how to prepare and apply to medical school (Gross et al., 2008). Students who bypass the traditional premed programs may have different perspectives on External formulas and different senses of internal beliefs given the preparation and motivations necessary to apply to medical school as a sophomore. Next, developmental movement from one phase to another (External to Crossroads) reflects the period in which a person comes across information that conflicts with, or contradicts, the messages from external influences. In the Crossroads phase, a person may not know exactly how to resolve the conflict, but he or she becomes aware of 14

25 his or her own beliefs and values and realizes the need to resolve these with the conflicting external messages. Baxter Magolda (2004) found that the break with External formulas was often caused by a perceived conflict regarding the student s current career plans. This conflict is relevant to the pressures of the premed curriculum as well as the experience students have once they matriculate. It also provides a lens to determine both the differences in expectations prior to medical school and the potential conflict between expectations and reality the students face once they matriculate into medical school. These conflicts may be different for early assurance students; they are more likely than traditional premed students to face increased pressures at least for the first year of medical school as they begin studying medical science. Finally, medical education emphasizes experiential learning, which is key to medical student development and combines basic sciences commonly taught in a classroom setting with clinical learning that takes place in a medical practice setting (Yardley, Teunissen, & Dornan, 2012). Medical education is a progressive path where students identities and knowledge are formed and revisited over time as students move from undergraduate to resident to practice (Slotnick, 2001). Students face different expectations from teachers and others (e.g., patients) along their education path and need to resolve these changing expectations along with their personal needs and perspectives. Self-authorship can be an important skill necessary to succeed in the arc of experiences during medical school. Self-authorship supports the cognitive as well as sociocultural dimensions within Slotnick s (2001) view of a life-span of learning. In light of cognitive development, Slotnick also adds that epistemological development is important in building knowledge and skills in problem solving and in satisfying self-esteem needs. The 15

26 problem-solving skills of medical students parallel Baxter Magolda s epistemologic dimension. The sociocultural dimension relates to the many roles a medical student plays in life, both in school and in practice as a physician after graduation. The ability to maintain relationships with others (family, peers, patients) is key to successfully resolving expectations and meeting one s own needs. This dimension coincides with the interpersonal and intrapersonal dimensions of Baxter Magolda s model. Conceptual Model Suggestions to incorporate self-authorship into the curriculum to support different ways of knowing include providing opportunities for self-discipline, as well as providing freedom to exercise choice and develop one s voice. (Baxter Magolda, 1992). The comparative freedom of an early assurance program student s final years of college as compared to a premed student provides an excellent context for the application of selfauthorship and testing whether differences in development do exist. This conceptual model is not intended to explain the possible differences in development but instead to illustrate the potential differences that could occur within the cyclical nature of development towards self-authorship (see Figure 2). The development of self-authorship is a recursive process based upon reflection and revision of beliefs, values, and knowledge when a person is faced with new challenges to his or her current state. These challenges can come from sources such as educators, family, and work situations (Baxter Magolda, 2004). It is hypothesized that the college experiences of early assurance students as compared to those of premeds can create different influences towards self-authorship. These influences can take several potential forms: 16

27 1. The freedom to take liberal arts courses for the early assurance students may allow them to grow and develop skills towards self-authorship. 2. The freedom for the early assurance students may remove pressures and challenges to beliefs, values, and knowledge in the traditional premed curriculum that would otherwise spur development toward self-authorship. 3. Once the early assurance students arrive in medical school, they may be faced with additional challenges of feeling less prepared than their peers because of their lack of basic science training during their undergraduate degree program. 17

28 Experiences (education, family, community) Personal Characteristics (personal history, meaning making capacity) Person's Interpretations of Experiences (grounded in meaning making) Developmental Growth Figure 2. Relationship between Experience and Meaning Making among Early Assurance and Traditional Premed Students. Adapted from Development of the Ten Positions in the Journey Toward Self-Authorship by M. B. Baxter Magolda and J. P. Barber, 2012, Assessing Meaning Making and Self-Authorship: Theory, Research, and Application: ASHE Higher Education Report 38:3, p. 41. Copyright by John Wiley & Sons. Self-authorship is a movement away from External formulas when faced with complexity, ambiguity, and conflict. The early assurance program does not force students to face External formulas but instead removes some of them. The challenge for academics has been how to measure self-authorship on a large-scale method and then apply this knowledge to the educational experience as it relates to student outcomes. This research sought to identify a tool that could be used to measure self-authorship; it sought to allow 18

29 for measurement among sub-groups of medical students along the trajectory of their educational paths. Justification for this Study The nature of self-authorship lends itself to qualitative methods, as researchers have sought to understand what it is and what factors influence non-cognitive development. Significant training is necessary for a researcher to properly conduct interviews and guide conversations in order to elicit responses that help explain the thought processes that can exhibit self-authorship or a lack thereof (Baxter Magolda & King, 2007). Qualitative studies of self-authorship can be resource-intensive but provide rich data to help understand the phases and dimensions of self-authorship. If, however, a researcher intends to conduct large scale studies of how a population stands in terms of their positioning within the phases and dimensions of self-authorship, qualitative methods are too cumbersome and subject to the potential for variation between reviewers (Baxter Magolda & King, 2007). A quantitative scale that measures self-authorship could be used in large-scale evaluations, providing the opportunity to identify group differences and correlates with development (Pizzolato, 2005). Such a tool could also provide economies of scale over qualitative efforts in allocating resources to study self-authorship. Very few quantitative tools have been developed to measure self-authorship. Pizzolato (2005) created a scale based upon the skills needed to manifest self-authorship, such as autonomy and relationships with authorities. The scale ultimately was modeled by four skill factors, which Pizzolato called Capacity for Autonomous Action, Problem Solving Orientation, Perceptions of Volitional Competence, and Self-Regulation in Challenging Situations. Creamer et al. (2010) created a scale that attempted to mirror the 19

30 phases and dimensions of self-authorship as theorized by Baxter Magolda. This scale identified factors representing the three dimensions and the three phases of selfauthorship. In both studies, the sample populations were largely White female undergraduates at a Midwestern university. Neither instrument was directly transferable to the medical student population. Extant literature fails to provide any evidence of the study of self-authorship specific to the medical student population. Within the last two years, however, there has been discussion regarding the desire for self-authorship to be studied as it relates to medical education curriculum (Jarvis-Selinger, Pratt, & Regehr, 2012; Prober & Khan, 2013). Researchers argue the importance of self-authorship for students and physicians during their education and training. (Jarvis-Selinger et al., 2012). In particular, these discussions relate to measuring phases of self-authorship and how they pertain to medical education. Medical education curricula, as it relates to the experiences of traditional premeds and early assurance students, provide one specific example of the need for a quantitative measure that could be validated for medical students in general and across sub-groups. However, given the potential to apply the concept of self-authorship to medical students, there are no validated quantitative instruments to allow for comparative studies among medical students. Research Context and Motivation This study took place at a large (more than 500 students) medical school in the northeast United States that has an early assurance program. The Icahn School of Medicine at Mount Sinai has a limited early assurance admission program and is not affiliated with any university; therefore, the program is available to students from a variety 20

31 of undergraduate institutions. Students are able to apply to the medical school in their sophomore year of college. The application process includes a review of the student s college GPA up to that point, as well as personal essays and interviews. Upon acceptance, these students are not required to take the traditional premed undergraduate program, nor are they required to take the MCAT admissions test. Rather, they agree not to take a premed undergraduate program and instead choose a major in a liberal arts program. They must take two semesters each of biology and chemistry. The students are also encouraged to take a year off after completing their bachelor s degree for further exploration of interests they might have, such as travel, research opportunities, or volunteering efforts nationally or internationally. Those who are accepted spend part of the summer between their junior and senior years at the medical school doing an immersion class in organic chemistry. In the summer before they start medical school, they are also provided the opportunity to attend an optional six-week preparation program that covers fundamentals in biochemistry and anatomy, as well as sessions on study methods and acclimating to medical school. The premise of the early assurance program is that it graduates a different type of physician someone who would be different in a non-cognitive skill such as empathy or morality. A previous study has identified the differences and similarities in cognitive performance between early assurance and premed students during the four years of medical school (Muller & Kase, 2010). Although the early assurance program has been in place for 25 years, there has been no formalized effort to date at this medical school to identify the non-cognitive development of the early assurance (or any other) students, either upon entry to, or at the completion of, medical school. In 2011, a team of three 21

32 researchers (including myself) came together for a four-year project to design a method to identify the non-cognitive development of the same student groups. The team reviewed literature on several non-cognitive measures, including empathy, morality, actualization, and self-authorship. We originally viewed the empathy and morality of medical students as very specific characteristics, whereas there may not be much differentiation among students at the start of medical school. Therefore, we decided these were not appropriate for our first analysis. Instead, we selected broader concepts of student development, specifically self-authorship and self-actualization. Self-actualization refers to the selffulfillment of one s perceived potential (LeClerc, Lefrancois, Dube, Hebert, & Gaulin, 1999). This research primarily focused on the development of the measure of the selfauthorship construct because the self-actualization items had been empirically validated previously. Self-authorship has predominantly been examined qualitatively through in-depth interviews, which can be time-intensive and require skilled interviewers (Baxter Magolda & King, 2012). Previous efforts to create quantitative measures of self-authorship, using both contextual and non-contextual items within the construct, have had mixed results when data from the scale are correlated with qualitative data that is collected concurrently (Creamer, Baxter Magolda, & Yue, 2010; Pizzolato, 2005). In particular, respondents may answer questions in a manner that would indicate self-authorship yet when they explain their reasoning through qualitative measures, they do not actually reflect self-authored behaviors. Still, there is no consensus on whether efforts to build a quantitative measure are worthwhile or not. 22

33 The research team decided to use the tool created by Creamer et al. (2010) as the basis for a new quantitative measure of self-authorship among medical students. The challenge was to create construct items that measure the thought process the student goes through, not the actual decision, and apply these concepts to medical students. A pilot study was conducted in 2011 using a test version of a self-authorship scale created by the research team. The analysis of this pilot formed the basis for this research. Purpose of the Study and Research Questions The purpose of this study was to perform the analyses necessary to create a valid and reliable scale of self-authorship that can be applied to the medical student population. This study incorporated qualitative data with the scale for contextual verification. Given the scant literature regarding self-authorship in medical school as well as the prior studies of early assurance programs at this site, this study aimed to answer two central research questions: 1. (a) Does data collected for students at the beginning of the academic year support the validity and reliability of the proposed instrument, and (b) does the reliability and validity extend to the premed and early assurance subpopulations? 2. Does a content analysis of the qualitative responses provide evidence to validate the constructs of self-authorship among medical students? Significance of the Study This study focused on the development of an instrument to measure the construct of self-authorship among medical students. This research is significant in several respects. First, it expanded upon the literature of self-authorship by extending efforts to validate the 23

34 concept among the medical student population. This analysis explored the dimensions and phases of self-authorship among a new population and highlights differences in the theoretical interrelationships developed among the undergraduate population. Second, most of the previous research has been exploratory and qualitative in defining selfauthorship among populations. This research built upon the earlier research into defining quantitative measures of movement within self-authorship. Finally, the potential to identify development among subpopulations of medical students could help in understanding the tool s applicability in evaluating admissions criteria. Early assurance programs in general provide an opportunity to diversify the types of students who enter medical school (Eaglen et al., 2012). As early assurance students are a key subgroup of the student population, this evaluation is the first step towards validating the tool in relation to this population. Overview of Chapters This first chapter has provided the background information regarding selfauthorship and its relation to medical students. This chapter also included the theoretical framework, justification of the study, the research context, purpose statement, research questions, and significance of the study. In Chapter 2, there is a two-part literature review examining the development of self-authorship theory and its applicability to medical students, followed by a review of the literature regarding efforts at quantitative measures of self-authorship. Chapter 3 describes the methods used in the first wave of data collection, the pilot for this research project, followed by a description of the methods used in the second wave. Next, Chapter 4 presents the results from the second wave. In 24

35 conclusion, Chapter 5 discusses the implications of research findings for theory and practice and provides suggestions for future research. 25

36 CHAPTER 2 REVIEW OF LITERATURE Given that little research has been conducted examining the development of noncognitive skills among early assurance medical students, this chapter provides a review of the literature on the development of self-authorship as a theoretical guide and analyzes the empirical evidence of methods to investigate self-authorship. Theories of Student Development Theories of student development explain how students mature and grow toward more complex and sophisticated thinking during their educational years. While some theorists describe development from early childhood on, higher education has been the primary focus of much student development work. There are four general classes of development theory that have been most commonly used with college students since the 1960s (Kuh, Gonyea, & Rodriguez, 2002). In brief, they are the following: 1. Psychosocial Theories these focus on development of the person and his or her identity, how one builds relationships and becomes self-sufficient. Conceptually, development can occur in response to pressures from society as one seeks to understand one s identity as it fits in with others. The locus of development is through sociocultural, psychological, and/or biological influences (Pascarella & Terenzini, 2005). Development/regression occurs when one needs to make a choice among alternatives in regard to self-identity. In terms of college students, this relates to how students develop in the face of a new college environment, peer pressure, and academic responsibilities. 26

37 2. Cognitive Structural Theories a class of theories which explain personal development not necessarily from societal pressure but rather from internal questioning when faced with information contrary to one s beliefs or assumptions. Influential theorists include Kegan (1982) and Perry (1997). Baxter Magolda (2008) refers to how one makes meaning of experiences by incorporating beliefs, values, and knowledge. Unlike the psychosocial theorists, cognitive structural theorists include an epistemic factor into development. The movement from accepting knowledge from authority to being able to understand knowledge as contextual and having multiple meanings is an integral part of these theories of development. 3. Person-environment Interaction Theories these theories do not specifically describe development; instead, they seek to explain how a person goes about having one s needs match with demands of the environment. These models help to explain the match between student and college and why some succeed and others do not, as well as how one identifies a good match. 4. Typology Models here, theorists categorize students by how alike or different they are in responding to situations within the college setting. Again, these models do not necessarily describe development but instead reflect inventories that provide the ability to predict how a person might react in various situations. In the case of student development, this can relate to situations such as classroom environment or dormitory living assignments. Of the four broad theoretical perspectives, cognitive-structural theories would be the most relevant to the study of deeper learning skills because of the inclusion of 27

38 epistemological development, which can be related more to learning and introspection. All of the theories include conceptualizations of higher order skills that support the LEAP goals for a liberal arts education, such as building relationships, autonomy, and acceptance of diversity. However, without considering epistemology, many of these theories are more applicable to issues of student life than they are to explaining the growth of non-cognitive skills. This is not to say that the psycho-social theories ignore epistemology, but more that they focus on socialization, acceptance into the higher education environment, and the ability to succeed and continue to develop as adults. A review of key cognitive structural theorists further clarifies the applicability to deeper learning skills. Perry s Theory of Intellectual Development When one thinks of development over time, it is helpful to think about indicators of progress or milestones that mark change. Many cognitive-structural theories identify varying numbers of developmental stages, but all see a recursive process of growth within the student s development of cognitive skills. One of the earliest studies into cognitive structural theories of student development consisted of qualitative interviews with White male students during their four years in college (Perry, 1997). Perry proposed that development occurred in nine progressive positions, from what he called dualism to relativism. In the lowest position (dualism), knowledge is either right or wrong and is handed down from authority. At this stage, there is no regard for others or diversity of thought. By the time students develop to the ninth position (relativism), they are aware that there are a variety of opinions and values and they can analyze and compare these in order to make a decision on what is the best interpretation of an experience or problem. The path through the positions is not linear, nor is it exactly the same for everyone. Perry 28

39 explains how people may retreat, or digress, in their development when faced with too difficult situations, but progress may continue again in the future. On the other hand, others may escape, as Perry puts it, and not progress any further or may even revert toward a dualistic state. Kegan s Theory of the Evolution of Consciousness Robert Kegan provided a complementary theory that introduced the construct of self-authorship (Love & Guthrie, 2002). Kegan s seminal work relates to growth over a lifetime and does not focus on just the period of academic experiences (Kegan, 1982). His theory is based upon the concept of how we view experiences as either subject or object. [Emphasis added.] When we are responsible for an experience or can work to fix problems with it, it is considered in our control, or an object. Conversely, if there are experiences we are not responsible for, or are not aware of, then these are considered to be subject, as in we are subject to them. Moving from subject to object shows development in knowing and independence. Kegan proposes five stages of development and believes that most people will plateau at the fourth stage. Similar to Perry, he does not see the path through the stages as linear, but more like a helix; movement upward requires reflection on past learning and assumptions and subsequent reconciliation of conflicted material. The ability to reflect is important to these cognitive theories. Of the five stages, three are most relevant to the college years. Stage 2 starts at around age six and goes through adolescence. At this stage, a person s impulses are object, but the person is subject to his needs and desires. Through awareness and interaction with others, growth into the third stage occurs during post-adolescence and includes movement of one s needs and desires to become object, where one is able to control them in relation to one s interactions with 29

40 others. On the other hand, at this point people become subject to interpersonal relationships and can be influenced by the needs and desires of others (peer pressure), often at the cost of their own needs. As an individual moves into the fourth stage, he or she acquires what Kegan termed self-authorship. In addition to having one s own needs and desires become object, at this time interpersonal relationships move into a phase in which the person understands that his or her relationships are mutually created with the other participants. The self-authored individual can mesh his or her own needs with those of the other person to achieve mutually satisfying solutions. The fifth stage, in which one has the ability to have multiple systems of thinking, occurs, if at all, in the adult years after college. One key to understanding Kegan s theory is that it is not necessary for every person to achieve the fifth, or even fourth, stage of development. Instead, he explains how achieving each stage provides a new set of skills that can benefit one as one moves into more complex life situations. Baxter Magolda s Theory of Self-Authorship The theory of self-authorship has been further refined by Baxter Magolda based upon her 25-year longitudinal study of a cohort of college students (Baxter Magolda, 2008). Her qualitative research included annual interviews with a cohort of men and women as they went through college, entered careers, and began families. She clarified self-authorship as a necessary foundation for adults to meet typical expectations they face at work, home, and school, such as the ability to be self-initiating, guided by their own visions, responsible for their experience, and able to develop interdependent relationships with diverse others (p. 269). One key skill of self-authorship is the ability to learn on 30

41 one s own and not be uncritically assimilating from others; another is the ability to reflect on experiences and understand their relation to one s internal needs and desires. Baxter Magolda s (2008) model can be conceptualized in a 3x3 matrix. The dimensions of self-authorship are epistemological (understanding where knowledge comes from), intrapersonal (understanding one s own values and beliefs), and interpersonal (how one builds relationships with others). The three dimensions are considered equally important to self-authorship, with no priority given for one over the others. Across each dimension there are then three phases of growth: External formulas, Crossroads, and Early self-authorship. The External formulas phase represents the place where thoughts and behaviors are based upon input from external sources. Knowledge is handed down as rote from authority and relationships are often predicated on trying to please or obey the other instead of understanding one s own internal needs and desires. Moving into the Crossroads phase, one becomes aware of and comfortable with one s own needs, how one is responsible for them, and how relationships need to incorporate the perspectives of both individuals. Knowledge becomes questioned as alternative perspectives are understood in a more complex environment and one realizes there are limits to certainty when faced with problems. In the final phase, self-authorship, one understands the complexity of knowledge and how it is individually constructed. Similarly, the interpersonal and intrapersonal dimensions develop to the point that one becomes comfortable creating one s own sense of values and that, when acting with others, one does not lose sense of oneself but can balance the needs of both parties. One can act on one s own senses and not necessarily just assimilate the knowledge and beliefs of others. Similar to Perry and 31

42 Kegan, Baxter Magolda also sees the distinction between the three stages as somewhat fuzzy and the path towards self-authorship as not necessarily the same for every person. The third phase of development (Early self-authorship) has been shown to exist during the college years (Pizzolato, 2004). One common theme across these cognitive-structural theories is the understanding that conflict of some type usually helps to promote development to a further phase. This conflict could relate to new knowledge that counters what the individual has always believed or that leads to questions about the assumed knowledge of a trusted authority, or a diverse opinion from someone that they had never heard before. Having to react to an affront to one s current comfort levels can provide the opportunity to better understand the conflicting information and how to incorporate it into one s thought processes going forward. The more self-authored the person, the better the person is at taking a challenge to his or her belief systems, comparing and evaluating the information, and using it to learn and grow. A second common thread is the need for self-reflection if one is to move to different phases of development. These theories focus on the inner voice, being able to understand one s needs and beliefs, and interpreting conflicts to one s prior understanding. Development is not organic or linear. It requires an effort, whether intentional or not, to learn from new information and to reconcile it with past beliefs and values. Selfauthorization is thus a thought process and not something that is measured by actions. Through sources of conflict and reflection on one s past experiences, a person learns how to move forward. What has been shown in some of the research is that the ability to reflect is not automatic and needs to be nurtured. Not only must one be able to understand one s 32

43 internal voice, but one must also learn to trust it and then reconcile it with the perceived conflicts one experiences with others (Baxter Magolda, 2008). Third, contextual knowing is key to understanding the development of selfauthorship. Other people s ideas continue to be important, but no longer simply as a source of ideas that coexist with the learner s; rather, they are potential elements that, when judged to be valid, can be incorporated into the learner s own thinking and views. (Pascarella & Terenzini, 2005) All of the qualitative studies of self-authorship center themselves on the context of the student s experiences, either in school, work, or with family, and how these particular situations can spur movement towards self-authorship. When faced with a situation or issue, self-authored individuals seek out expert advice but integrate it with their own views in deciding what to think (Baxter Magolda, 2004). In summary, cognitive-structural theories help to illuminate the growth of the ways individuals construct meaning as they develop skills to question knowledge, understand their own perspectives, and relate to others in a manner that accounts for diverse worldviews. As one develops towards self-authorship, one s ability to build higher learning skills such as personal responsibility, integrative learning, and critical analysis will be easier. Self-authorship can provide the foundation to foster development of the LEAP goals for the 21 st century such as critical thinking and social responsibility. Among undergraduate students planning to enter medical school, the foundation to develop selfauthorship may be different for those in an early assurance entry path when compared to the traditional premed student because of their different exposures to liberal arts courses and the requirements of medical school preparation. 33

44 Self-Authorship and Medical Education Literature regarding the study of self-authorship specifically within the medical education setting is negligible, although there is evidence of its utility as a lens for study. Slotnick (2001) argued that the journey through medical education from student in the classroom to student in the clinical setting to graduate in residency requires the individual to model and acquire new roles at each step. Each new step incorporates new expectations upon the individual and a new set of actors (faculty, preceptors) with whom the individual must interact. However, each progressive step can also retain some of the prior expectations/roles as well. His arguments for cognitive and socio-cultural skills to help adapt to these changes were prescient of Baxter Magolda s conception of self-authorship. Once medical students start learning in the clinical arena, the multiple roles that they must progressively master also include that of doctor and student (Shuval, 1975). This creates a conflicting set of messages to the student regarding expected behaviors from teachers and patients, each of whom may have very different expectations of the student as a learner and expert, respectively. Self-authorship has been shown to provide an understanding of how meaning-making can support the perceptions of multiple identities (Abes, Jones, & McEwen, 2007). Abes, Jones, and McEwen did not study medical students, but their conception of the conflict of multiple expectations is relevant to medical education as well, considering the multiple roles students must exhibit. In addition to medical education requiring the achievement of new roles over time, it is also an experiential process and becomes progressively self-directed as one goes through medical school and residency (Yardley et al., 2012). Yardley et al (2012) argue for an experience-based learning theory that illustrates the relative importance of 34

45 understanding the change from classroom-based lectures to the clinical setting and how students experience multiple sources of information and expectations, including the ability to rehearse the act of a doctor (p. e111). This interaction creates a process where the student is both learner and student, facilitated by the faculty. Applying the concept of self-authorship highlights the importance with regards to the interpersonal and intrapersonal dimensions of self-authorship as the student incorporates the multiple messages with his understanding of what it means to be a student and doctor. Understanding the experiential environment can provide a framework to understand how students enter and experience medical school. Medical education, especially in the clinical years, can also expose students to complicated problems where conflicting information may be presented between all of the different levels of clinicians (or even patients) who may be involved in the teaching experience. The ability to reconcile all of this information within the clinical context may be better achieved by those who are more self-authored. Finally, within the experiential process of medical education, the literature points to the significant influence of the physician-teachers with whom the students interact throughout the education process. Teachers provide knowledge and exhibit behaviors in the clinical setting that students see and emulate. The behaviors they see can often conflict with their own beliefs and understanding of expected behavior (Shuval, 1975). The influence of physician-teachers is significant to both knowledge and identity formation as the student develops into a physician (Jarvis-Selinger et al., 2012; Yardley et al., 2012). Self-authorship is founded on the idea that a person moves from External formulas to internalized beliefs of one s own. The significant teacher-student relationship within 35

46 medical education highlights the importance of being self-authored as a medical student internalizes his or her own beliefs and values and reconciles that with all of the individuals he or she interacts with during the educational process. Evidence of the study of self-authorship within medical education is rare but there are compelling arguments for the utility of its study. Medical education is progressive, with each new step (science years, clinical rotations, residency) a potential starting point where the student may feel unprepared and Externally motivated to prove oneself. White, Kumagai, Ross, and Fantone (2009) reported that students who are more developed with regard to self-authorship have been found to complete their clinical rotations while remaining more patient-centered and less subject to compromising their values. The authors provided evidence to support efforts within the curriculum to develop selfauthorship but fell short of providing methods to systematically measure this development within the curriculum. Similarly, arguments have been made for widening medical education beyond clinical competencies to incorporate identity formation, calling for a systematic measure of self-authorship to support these specific changes to the curriculum (Jarvis-Selinger et al., 2012). Time-consuming and resource-intensive qualitative methods have been the most common format to study self-authorship. Reliance on qualitative measures will not be sufficient if the medical education system seeks to meet these calls for change. Support to develop self-authorship along the medical education path can help improve the learning experience. The availability of some method to measure selfauthorship development in a standardized, economical way can support the appropriate evaluation of curricular efforts. 36

47 Quantitative Methods to Assess Self-Authorship The literature on methods to measure self-authorship is somewhat limited and fairly recent. Given the limited literature, it is helpful to review the empirical evidence regarding the development of assessment methods in a chronological fashion and see how it has accumulated. Prior to Baxter Magolda s introduction in 2001 of her conception of self-authorship, there were some attempts to quantitatively measure concepts that are akin to self-authorship among medical students. Researchers at the Harvard Medical School created a cognitive behavioral survey to measure epistemological beliefs among medical students when comparing three curriculums (Mitchell, 1994). This survey of second-year students included three newly created scales of learning. With 96 responses, a 71% response rate, the findings were appropriate for generalization to the student population at Harvard as part of a program evaluation, but there were not enough responses to allow for validation of the actual scales. Still, the authors were confident in their quantitative methodology to collect useful data on these concepts. Empathy can be viewed as another characteristic of the self-authored person. One study looked at empathy and how it correlates to the quality of counseling provided by graduate counseling students (Lovell, 1999). Lovell used two previously validated psychological batteries in a national survey in which over half of the respondents had a liberal arts undergraduate degree and found that those in higher cognitive stages of growth showed more empathy. The use of a standardized quantitative measurement tool showed that this methodology could be appropriate for measuring non-cognitive skills (i.e., empathy) as well as cognitive factors. However, his research did not focus on the methodology itself and possible implications for further research. 37

48 The National Study of Student Learning (NSSL) was a three-year program from to survey students orientation to learning and its relationship to academic experiences. The standardized tools included various constructs that did not explore noncognitive skills but did include measures on constructs such as acceptance of diversity and critical thinking; both skills could fall into the epistemological and interpersonal dimensions of self-authorship (Pascarella, 2001). However, the purpose of NSSL was to measure changes in general cognitive skills and how they were related to varied activities such as Greek life, diversity training, and volunteer work. By creating a construct on openness to diversity and trending each student over the three years, the researchers were able to identify changes in the student s development within the construct and how that might be related to other academic characteristics. Conceptually, this provides a model for using some quantitative measure of self-authorship and monitoring changes over time within a particular context. The NSSL itself is more applicable to person-environment interaction theories, but the use of a survey to measure skills related to self-authorship supports further research into the applicability of this methodology. Baxter Magolda s (2008) research was a labor-intensive effort that supported the development of a theory as well as a specific qualitative method in which to collect the data. Starting in 1986, she followed 101 students, conducting annual semi-structured interviews. By 2000, there were still 39 former students participating each year. In her methodology, the interviewer starts the conversation by asking about important learning experiences in the past year, thus giving the interviewee the opportunity to set the context. Baxter Magolda uses a constructivist lens, allowing each person to describe their unique experiences before trying to find commonality in the data to help explain self-authorship. 38

49 The method requires significant training and practice for the interviewer to ensure the quality of the data and to ensure the interviewer understands how to properly prompt and lead the interviewee to reflect and express his or her experiences. Her qualitative methodology set the groundwork for other subsequent studies of self-authorship. Pizzolato (2004) followed Baxter Magolda s method with a qualitative study of marginalized Hispanic students in order to understand the theory and its application to student retention and success. This study of 27 students was exploratory in nature and identified movement towards self-authorship upon entering college but found regression by these students as they initially faced challenges in college similar to the Crossroads stage as identified by Baxter Magolda. The basis of studying self-authorship using qualitative methods continued in a study to explore ethnic identity and its relationship to self-authorship, which was not part of the original hypothesis but arose out of the coding of the interviews (Torres & Hernandez, 2007). The study supports the idea of a long-term process to measure development of students over the four years of college. In terms of concepts within selfauthorship, the researchers identified the need to incorporate students expectations and how these related to experiences and development of self-authorship. While Baxter Magolda s original work identified the influence of conflict and pressure to spur the development of self-authorship, Torres and Hernandez identified how positive experiences also promoted development. Their research among Latino students followed Baxter Magolda s method, but they did not include any explanation of how they structured their interviews, nor did they explain the challenges of working with this particular population. 39

50 Empirical evidence of quantitative measures of self-authorship appeared around this same time. For example, Wawrzynski and Pizzolato (2006) published on the development of a quantitative tool to measure self-authorship because, as they pointed out, the theory of self-authorship was built upon qualitative methods that are not practical for large-scale studies of development. In a study of 368 students at one university, a survey that included a 29-item questionnaire of self-authorship was conducted at the beginning and end of one semester. The authors acknowledged that they did not hypothesize that they would find significant growth in self-authorship in such a short time frame, but instead were looking to see how personal characteristics and school environmental factors might be related to self-authorship. The data identified a link between the students precollege experiences (high school grades and test scores) and development on subscales of self-authorship. However, this article did not provide any indication of the actual content of the survey, nor was there an explanation of how their concepts may differ theoretically from Baxter Magolda s conception. Pizzolato (2007) further described the challenges of developing an assessment method for self-authorship that combined a quantitative Self-Authorship Survey (SAS) and a qualitative text exercise called the Experience Survey (ES). Her goal was to develop a measure that would allow for assessment of development towards self-authorship and could support program evaluations of curricular efforts to encourage development. She found moderate (r=.51) correlations between scores on the constructs and ratings of the text responses. However, she noted that the text item only asked the student to write about an important experience, not specifically one in which the student was an actor and would need to reflect on his or her thought processes. Thus the language in the responses made it 40

51 difficult to identify self-authored reasoning. Pizzolato s constructs are also related to skills necessary for self-authorship, but the items themselves were not situated in any context but instead related to experiences in general. In a study that included surveys among 467 men and women and follow-up interviews with 40 female college students, there were somewhat conflicting data regarding the epistemological and interpersonal dimensions of self-authorship (Creamer & Laughlin, 2005). The survey asked for sources of information in deciding on a career, but the responses were hard to interpret given the subsequent analysis of the interviews. The study highlighted the difficulty in writing questions that can identify the thought process involved in self-authorship as opposed to behavioral actions that are not representative of self-authorship. The most recent evidence of a quantitative measure of self-authorship is an 18-item scale that was designed to reflect constructs for both the three dimensions of authorship and the three phases of growth (Creamer et al., 2010). It was included as part of a larger survey on career development for women in information technology, and the scale was shown to have acceptable reliability and consistency. The authors included the important point that Baxter Magolda s original qualitative methodology depended on the interviewee identifying a notable experience from the past year and thus directing the conversation toward the topic to ultimately explore self-authorship. A survey of selfauthorship would also need a context in which to measure the construct; in this case it was choice of a career, which may not have resonated with all of the respondents. Also, in a 3x3 matrix, the 18 items were limited in measuring each of the nine dimensional stages and self-authorship phases. However, their work did show utility to the scale when the analysis looked more globally at either the three dimensions or the three phases separately. 41

52 Although the analysis did not explore the relative importance of any of the dimensions in regard to the others, it gave equal weight to each in the end. Given the acknowledged fuzziness between the three stages of development, it is encouraging that the suggested items did produce evidence of the separate stages. More recent work using mixed-methods to combine both qualitative and quantitative methods attempted to triangulate the measurement of self-authorship and provided an interesting perspective on assessing outcomes of liberal education. Researchers at the Wabash National Study conducted a longitudinal mixed methods study using a national sample to measure liberal arts outcomes among students (Seifert, Goodman, King, & Baxter Magolda, 2010). In the first wave of the project, the researchers collaborated on designing the methods for the study; but then the quantitative members worked separately from the qualitative members in terms of data collection and initial data interpretation. They then jointly compared findings, identified common themes, and incorporated the qualitative data to enrich findings from the survey. The quantitative researchers did not create any new survey items or scales to measure higher learning outcomes, including self-authorship. Rather, they used validated national surveys such as the National Survey of Student Engagement and the Wabash National Study of Student Experiences Survey. The utilized instruments included scales to measure constructs such as reasoning, moral character, and intention for lifelong learning, none of which specifically target self-authorship but many of which related to concepts within self-authorship. On the qualitative side, interviews were conducted using Baxter Magolda s suggested methodology to explore self-authorship. The findings highlighted many correlations between the themes identified in the qualitative and quantitative data, 42

53 illustrating how the two methodologies may provide richer information to measure selfauthorship than using either one alone. Some of the validated scales may also be helpful in measuring concepts of self-authorship, either alone or incorporated with new scales specific to self-authorship. Some progress has been made in the move toward creating a quantitative method to measure self-authorship. For the most part, the construct items were validated and/or substantiated through a concurrent analysis of either interviews or reviews of student writing (Baxter Magolda, 2008; Torres & Hernandez, 2007). This study sought to validate a new measure of self-authorship and to build upon these methods through the analysis of survey items and open-ended responses that were captured simultaneously, as described in the next chapter. 43

54 CHAPTER 3 METHODS This study used a mixed-methods approach quantitatively driven with simultaneous qualitative analysis (Morse, 2010). The development and validation of the instrument occurred in two stages. Construct definition, item generation, and a pilot test occurred in the first stage, which was part of an ongoing research project at the study site. At the end of the first stage, the instrument was revised based upon findings in the pilot. Data collection using the revised instrument represents the second stage, which was the basis for this dissertation. This chapter first describes the pilot study in Stage 1, including the steps that were taken to create, analyze and refine the proposed measure of selfauthorship, followed by a description of Stage 2, which represents the data collected from the revised instrument and analyzed for this study. Pilot Study Research Setting The setting for this research was a medical school in the northeastern United States. The majority of students self-identify as White (53.2%), Asian (23.2%), Hispanic or Latino (12.1%), and Black or African American (6.6%); this is slightly more diverse than national averages (Association of American Medical Colleges, 2012). Just over half of the students in 2012 were male (53.3%). Similar to national averages, more than three quarters (78.2%) of applicants are from out-of-state; and 69.8% of matriculants are from out-of-state, which is higher than the 38.8% national average. All medical degree candidates from Years 1 through 4 were included in the pilot phase (N = 569). Master s degree students were excluded because they do not have the same undergraduate 44

55 requirements, nor do they follow the same curriculum in medical school. Consent to participate in the pilot included acknowledgment that the research database would include identifying information such as address in order to allow longitudinal tracking of changes in responses over the course of the four years of medical school. Item Development For the pilot study in Stage 1, the team consulted telephonically with Baxter Magolda and Creamer in 2009 to discuss revising and utilizing their published tool. (see Appendix A.) Much of the previously reviewed literature accounted for identifying selfauthorship qualitatively within a specific context such as career or education experiences (Baxter Magolda, 2004; Creamer & Laughlin, 2005; Seifert et al., 2010). Quantitative research on self-authorship by Creamer et al. (2010) used the context of selecting a career for the survey items, while the tool developed by Pizzolato (2007) is not context bound. We identified selection of a specialty to practice medicine as an appropriate context to develop the construct items. Using a 3x3 matrix of the three dimensions of self-authorship (epistemological, intrapersonal, and interpersonal) and the three phases of development (External, Crossroads, and Early), the research team aligned the original 18 items from Creamer s scale and then created an additional six items so that there were at least two items per cell (see Table 1). According to Baxter Magolda and King (2007), assessing meaningmaking structures in the journey toward self-authorship requires unearthing both object and subject aspects of meaning-making (p. 495). Therefore, the research team used item stems already in the scale or based upon the experience of physician-teachers on the research team that would add options related to object or subject where lacking in the 45

56 original scale. The team tried to have an equal number of items per dimension of selfauthorship under the assumption that each could be equally important. However, it was challenging to create an equal number of items relevant to the interpersonal dimension as compared to the other dimensions. Similarly, without prior knowledge or assumptions of how medical students develop self-authorship, the team tried to allow for an equal number of items representing each phase of development. Again, it was challenging to create items to represent the Crossroads phase. As a result, there were fewer items generated representing this phase. All of the original items from Creamer (2010) were revised to reflect selection of a medical specialty where appropriate (see Appendix A). Table 1 Number of Survey Items in Pilot Phase by Developmental Stage and Dimension Epistemological Interpersonal Intrapersonal Total External Formulas Crossroads Early Selfauthorship Total Item Structure and Scaling For the pilot, the self-authorship scale consisted of 24 items that were all rated on a 4-point Likert scale of strongly agree, agree, disagree, strongly disagree. The items were contextualized about choice of medical specialty in a manner similar to the original survey by Creamer et al. (2010), with stems that included To make a good choice about a medical specialty..., The most important role of a mentor or advisor..., and My 46

57 primary role in choosing a medical specialty... There were two additional stems that were not related to choice of specialty. They were Experts are divided on some scientific issues such as the causes of global warming. I think..., and When some people have different interpretations of a book, I think... Order bias occurs when the order in which survey items are presented can affect responses to subsequent questions by creating thought processes based upon initial items (Dillman, 2000). Using an online method, the questions were randomly ordered for each participant in order to remove the potential for order bias. The pilot version of the instrument was pretested in 2011 with a group of new interns at the medical center since they were recent medical school graduates and could reflect on the process of selecting a specialty. This relates directly to establishing the reliability of the instrument, as the discussion could help identify ambiguous or confusing items that may introduce unintended information to the construct (DeVellis, 2012). Question wording was clarified on several items before data collection began. Appendix A compares the original version by Creamer, Baxter Magolda, and Yue (2010) with the final revised pilot version of the survey. Procedures Data collection in the pilot wave included all four current classes. The first wave of data collection began in August invitations with links to the online survey were sent to 569 students, with reminder s sent to non-responders every 2-3 weeks through September. To increase the response rate, an incentive of a $5 gift card to a coffee shop on campus was incorporated into the process after the initial three weeks of data collection. The gift cards were also provided to those who had already responded. 47

58 Analyses Analyses after data collection for the pilot phase were twofold. First, reliability testing and exploratory principal component analysis were conducted. Second, focus groups were conducted with students to further refine the instrument. All data were analyzed using SPSS, Version With development of a survey instrument, it is key to determine the reliability of the instrument to ensure consistent data over time and the validity of the instrument to accurately measure its intended concept(s) (Abell, Springer, & Kamata, 2009). Reliability also refers to the extent to which an instrument differentiates among individuals accurately. It is intimately linked to the population being measured (Streiner & Norman, 2008). This was particularly relevant to this study for two reasons: (1) the instrument which was the basis for this study had never before been applied to the medical student population, and (2) the conceptual framework was based upon the hypothesis that early assurance students and traditional premed students may start medical school in different phases of self-authorship. From a quantitative perspective, the reliability was determined using Cohen s alpha internal consistency estimate to measure internal consistency of the construct. In addition, agreement coefficients were calculated using random halves criteria since a test-retest design was not possible. The two random halves were a reasonable proxy for a test-retest (Downing, 2004). If random errors proved to be low, this would be good evidence of score reproducibility. The construct validity of a measure requires multiple sources of evidence which relate to test content, the response process, internal structure of the construct, its relation to 48

59 other variables, and consequences of testing (Downing, 2003). Internal consistency and factor analysis are sources of internal structure evidence. Regarding factor analysis, there is significant discussion in the literature about the choice between principal component analysis and exploratory factor analysis techniques (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Russell, 2002). Principal components analysis seeks to model all of the variance and focuses on data reduction. Exploratory factor analysis techniques instead model only the shared variance between variables. Exploratory factor analysis is appropriate for instrument development and identifying underlying constructs (Wetzel, 2011). However, in many cases the difference in methods can be negligible, in which case principal components can provide a simpler structure for analysis. Child (2006) in particular recommends starting with a principal components analysis and then using an exploratory factor analysis technique for verification of the initial findings. The pilot study used principal components analysis only, and the research in this study also used principal components. The set of potential variables should have a good range of response choices which are at equal appearing intervals and are normally distributed (Child, 2006).The data should also be sufficiently correlated to warrant factor analysis as a method to identify an underlying structure. The data from the pilot test were analyzed with the Kaiser-Meyer- Olkin measure of sampling adequacy, which identifies whether the number of data records is factorable in relation to the number of variables, and Bartlett s test of sphericity, where the null hypothesis is that the variables are uncorrelated. Significant results for each test would indicate that the items are sufficiently correlated and factor analytic techniques are justified. 49

60 When interpreting principal component or exploratory factor analysis results, a salient variable is defined as one that has a sufficiently high loading and ensures a relationship between that variable and the component (Gorsuch, 1983). Guidelines for interpretation can vary depending on the level of significance one desires and the size of the dataset. Gorsuch included guidelines for interpretation such as using a minimum load of.4 if a sample is 100 and a minimum load of.3 when a sample is at least 175. The decision on an appropriate cutoff can be arbitrary depending on the circumstances of the research (Abell et al., 2009). This study used a more conservative.400 cutoff for the interpretation of principal components. Initial Assessment of Self-Authorship Component Structure Principal component analysis was performed using Promax rotation of the component loadings. Promax is an oblique rotation that allows for correlated factors. American psychologists believe that most behavioral characteristics of human beings are so interrelated that we should allow for this in any kind of analysis used. In other words the resulting factors may be correlated (Child, 2006). Components were identified using two criteria: (1) eigenvalues greater than 1.0 and (2) a visual review of a Cattel s scree plot. The eigenvalue rule alone might be too loose in allowing factors, especially when discriminating between factors just above or below the 1.0 threshold. Applying the scree test involves some reliance on subjective criteria when reviewing factor interpretability and the visual drop off of the plot (DeVellis, 2012). The pilot wave of data collection elicited 165 responses (25% response rate). Low response rates alone are not biasing so long as the respondent characteristics are representative of non-respondents (Dillman, 1991). However, Table 2 shows that 50

61 respondents characteristics significantly (p<.05) under-represented females and White students. Still, within the responses, there were 42 early assurance students, allowing for some preliminary analysis of early assurance and traditional premed students. Table 2 Representative Characteristics as a Percentage of Respondents and Population Characteristic Survey Respondents (n = 149) School Population (N = 569) Early assurance students Female White Univariate analysis of the responses to each question identified a range of responses to nearly all items from 98% strongly agree or agree to 1% strongly agree or agree over the 24 items, visually confirming that there was variation among the responses. All variables had kurtosis < 7 and skewness < 2, so none were deleted because of significant non-normality (Fabrigar et al., 1999). Using the guideline of a minimum 5 records per variable and 100 total records, the data were determined to be sufficient for principal component analysis methods (Child, 2006; Gorsuch, 1983). Internal consistency of the construct was measured using Cronbach s alpha internal consistency estimate to check the reliability of the scale and the value was.755, which is acceptable for a formative assessment (Downing, 2004). Removing any of the items would improve the alpha by less than +/-.005; thus, no items were deleted. Reliability was tested using split-halves. The correlation between halves was moderate at.569 with Cronbach s alpha of.618 for one half and.609 for the other half. 51

62 The Kaiser-Meyer r-olkin index of sampling was.736 for the pilot, indicating that the data were a homogenous collection suitable for factor analysis. Bartlett s test of sphericity was significant ( , p<.001), indicating that the inter-item correlations were significantly different from 0 and suitable for factor analysis. The principal component analysis with Promax rotation of 24 items producedd eigenvalues that showed six factors which met the rule for an eigenvalue greater than 1, although the sixth factor just met thatt criterion. However, review of the scree plot showed that four factors were more reasonable to interpret. (see Figure 3.) Figure 3. Scree Plot for Exploratory Factor Analysis Using Principal Components Four components were selected as being most salient; ; the fifth and sixth components were not selected based upon review of the screee plot and the fact that the variables thatt loaded on these two components could not be easily interpreted as 52