An Integrated Framework for Conducting and Assessing Interpretive Information Systems (IS) Research

An Integrated Framework for Conducting and Assessing Interpretive Information Systems (IS) Research Shirish C. Srivastava NUS, Business School National University of Singapore shirish@nus.edu.sg Thompson S. H. Teo NUS, Business School National University of Singapore bizteosh@nus.edu.sg Abstract Though the number of interpretive research in information systems (IS) has increased substantially in recent years, the development and proliferation of methodology guiding such research is relatively minimal. The IS research methodology in interpretive research is still not institutionalized. This lack of common standards poses problems both for IS researchers using the interpretive paradigm and readers of such research articles. Drawing from the two major methodological developments in the area: seven principles for conducting interpretive IS field studies (Klein and Myers, 1999) and the three criteria for assessing ethnographic research (Golden-Biddle and Locke, 1993), we examine the linkage between the two methodological lenses and propose an integrated framework for conducting and assessing interpretive IS research. Keywords: Interpretivist, Positivist, Paradigms, Methodology, Qualitative 1. Introduction In their seminal paper about research approaches and assumptions used in studying information technology in organizations, Orlikowski and Baroudi (1991) discovered that even though information systems (IS) research is not rooted in a single overarching perspective, it does exhibit a single set of philosophical assumptions regarding the underlying nature of the phenomenon being investigated. In their paper they exhorted an agenda for greater paradigmatic diversity in IS research. Since they believed that a single research perspective for studying the information systems phenomenon is unnecessarily restrictive, they argued that studies which examine relationships between information technology, people, and organizations can be enriched if a diversity of research perspectives is employed. Since the pronouncements of Orlikowski and Baroudi (1991), significant paradigmatic and methodological upheavals have taken place in the field of information systems. As of now, interpretive research is a recognized paradigm used for conducting contextual IS research. Interpretive ethnographic research is especially popular among the qualitative researchers and seeks to inform the subject on second order interpretations, which would otherwise be left unaddressed by positivist research. A recent research about the diversity in IS discipline shows that the use of interpretive paradigm has increased substantially (Vessey, Ramesh and Glass, 2002). In spite of the increasing numbers of qualitative researchers using the interpretive paradigm in IS research, the methodological development in this stream has been minimal. Researchers trained in the positivist traditions are sometimes not able to appreciate the richness of the qualitative studies. It is a common experience for many positivist researchers to review or understand articles which follow the interpretivist paradigm. Absence of an integrated framework for assessing the quality of interpretive research makes assessment of interpretive articles by positivist researchers an exercise of cogency assessment rather than a thoughtful 1646

application of some common underlying rules. Apart from the positivist researchers, many interpretive researchers find themselves incapable of understanding and justifying the contributions made by their research. The lack of a cumulative tradition (Keen, 1980) in terms of methodology for interpretive research is the motivation for the genesis of this paper. By bringing together various methodological deliberations about interpretive IS research scattered throughout the literature, we seek to build an integrated framework for conducting and assessing interpretive IS research. 2. Integrated Framework For Interpretive IS Research Though there are established and accepted norms and procedures for conducting and assessing positivist IS research which have been institutionalized over time, there are very few institutionalized procedures for conducting and assessing interpretive IS research. In this section we draw upon the recent methodological developments for IS research to present an integrated framework for conducting and assessing interpretive IS research. For providing an integrated framework, we use the seven principles for interpretive IS research expounded by Klein and Myers (1999) as the point of departure and integrate it with the framework given by Golden-Biddle and Locke (1993) for assessing the convincing ability of ethnographic texts. In the process, we also see the linkage between the two and propose that for a comprehensive interpretive research, application of an integration of both these frameworks is desirable. 2.1. Principles of interpretive research Klein and Myers (1999) enumerated seven principles for conducting interpretive research. These principles have emerged as de-facto standards for interpretive IS research. These principles are discussed below:- The fundamental principle of hermeneutic circle: - This is the fundamental meta-principle on which the other six principles are based. The concept of the hermeneutic circle implies that human beings understand a complex whole from the meanings of its parts and their interrelationships. This means that our sense making is a constant iteration of the parts and the whole they form. For example, Markus (1994) traces the love and hate for email in various parts of the organization to understand the whole. The principle of contextualization: - This principle implies that in interpretive research, the context of the research setting plays a very important role in attributing meaning to a particular action. Hence it is important to be knowledgeable about the social and historical background to understand how the current situation has emerged. Walsham and Sahay (1999) detail the socio-political context for GIS application in Indian districts. The principle of interaction between researchers and subjects: - This principle reiterates the importance of interaction between researchers and subjects. It emphasizes to researchers that they must understand that research data produced in interpretive studies are socially constructed. During research, subjects may also act as analysts and researchers may become native to the study. Both these situations might lead to confounding of the data; hence interpretive researchers must be wary of these potential pitfalls. For example, Barley (1986) describes how the subjects started treating him as one of them and how he interacted with them to help them even in their professional lives The principle of abstraction and generalization: - This requires interpretative researchers to relate the data interpretation through the application of theories. This means that their data are not really only a story or an anecdote but are explained by the application of general concepts or theories that describe the nature of human understanding and social action. The difference from the positivist way of theory application is that the researchers are not interested in falsifying a theory; rather they use theories as a sensitizing device to view the 1647

world in a particular way. For example, Barley (1986) uses structuration theory (Giddens, 1976) to develop his study of CT scanners. The principle of dialogical reasoning: - It means that the researcher has to be sensitive to the possible contradictions between the theoretical preconceptions guiding the research and the design and the actual findings. These findings should be derived from a cycle of revisions and iterations to understand how the data contradicts the assumptions of the research design, and it should help explain the phenomenon in a different way. Walsham and Sahay (1999) mention how they started their research with structuration theory (Giddens, 1976) as the theoretical preconception guiding their study, but gradually in the process of conducting their research on GIS implementation in Indian districts, they switched to actor network theory (Callon, 1986; Latour, 1987) as it was explaining their research more closely. The principle of multiple interpretations: - This principle lays down the importance of sensitivity to possible differences in interpretations among the participants. The researcher should try and analyze the account and interpretation of the same phenomenon from different participants so as to understand the underlying truth from multiple accounts. In Markus (1994), the same email is interpreted in multifarious ways by different actors. These multiple interpretations are used to explain the way in which the happy medium is understood by the organization. The principle of suspicion: - Interpretative researchers have to understand the biases and the distortions which might be present in the narratives collected from the participants. This implies that the researcher should take into account the social world of the actor, his or her vested interests, power structure, his or her motivations and then take all the narratives with a pinch of salt. This implies that the researcher should not take the participant as a person insulated from the social world and understand the context for the actor completely to interpret the data or narratives in a more realistic way. This principle is visible quite clearly in Robey and Newman (1996) paper on sequential patterns in information system development, in which, in each phase, they consider the opinions of different relevant actors to understand what went wrong at each phase. They are also successful in communicating this element and suspicion in the mind of the reader who may attribute another set of reasons for failure at each stage. The seven principles serve as guidelines for interpretive IS researchers but it must be remembered that these principles cannot be applied mechanistically. This means that there has to be sensitivity towards the case for applying these principles. It is possible that for a particular case, one or two of these principles may not be relevant or may be implied. The principle of hermeneutic circle is perhaps the most important meta-principle and must emerge explicitly in the qualitative research to make it persuasive and relevant. 2.2. Quality of ethnographic work Golden-Biddle and Locke (1993) gave a framework for assessing the convincing ability of ethnographic texts. This framework spells out the criteria of authenticity, plausibility and criticality. Past IS studies like Walsham and Sahay (1999) used this framework for convincing the readers about the quality of their interpretive research. In contrast to the positivist studies which stress on reliability, validity and statistical generalizability, interpretive studies focus on plausibility and cogency. This framework when applied in conjunction with the seven principles gives a complete guidance and assessment framework for interpretive IS studies. The three criteria of authenticity, plausibility and criticality reflect some of the attributes described by the seven principles. We discuss the three criteria and also see the overlap and linkages with the seven principles with suitable examples. Criterion 1: Authenticity implies convincing the readers through their text that the researcher has actually been to the field and has understood the meaning of subjects 1648

world. This implies that for authenticity to be present, the researcher needs not only to be physically there but also needs to exhibit that he has understood the phenomenon from the perspective of the subjects. This can be done by the researcher by incorporating the following strategies:- Particularizing everyday life: The researcher gives excerpts and instances about the understanding of the everyday life of the subjects. This can be done by giving relevant quotes which exhibit the fact that researcher has immersed himself in the subjects world. For example, Walsham and Sahay (1999) do it by describing the vivid details of heat and dust of an Indian summer day. This has some elements of the principle of contextualization and also makes an attempt to understand the parts of the whole following principle of hermeneutic circle (Klein and Myers, 1999) Delineating the relationship between the researcher and organization members: The authors should make a conscious attempt to delineate their relationship with the subjects and be like a professional stranger rather than becoming native to the situation. This criterion has an underlying reflection of the principle of interaction between the researcher and the subjects (Klein and Myers, 1999). For example Barley (1986) describes how he delineates from the subjects in his research on CT scanners to avoid becoming native to the situation. Depicting the disciplined pursuit and analysis of data: It is important for the readers to be convinced that the data collection and analysis were indeed a tough job and have been done with utmost rigor and in a systematic manner. The results and the analysis must be cogently and logically tied to the data collected. This criteria reflects the principle of dialogical reasoning, which states that the researcher while analyzing data must be sensitive to the possible contradictions of the stories which the data tell with the theoretical preconceptions guiding the research. This is exhibited in Hsiao (2003) who started off with differences in technological frames of adopters (Orlikowski and Gash, 1994) for explaining the technology fears but later after data collection, modified the theoretical base to reliability-related and value-oriented distrust (Sitkin and Roth, 1993) which was further examined using the concept of value-oriented distrust with regard to the influence of cultural beliefs on economic actions (Greif, 1994). This indicates a disciplined pursuit and analysis of data which initiated the revision of their theoretical preconception. Qualifying personal biases: The researcher must at the outset make clear to the readers about his biases and his own background specifying from where he comes from. In Walsham and Sahay (1999), we observe that the authors make clear about their backgrounds and biases for the GIS technology. We also observe the application of this criterion by Klein and Myers (1999) in which they provide information on their background. This helps in better assessment of the article in terms of the context of the article and incorporates the essence of the principle of contextualization. Criterion 2: Plausibility implies that the ethnographic text makes a distinctive contribution to issues of common concern. It can be exhibited by researchers in their texts in the following ways:- Normalize unorthodox methodologies: It implies making the reader understand the methodologies used in research by making it simpler for them to comprehend systematically, for example, summarizing in tables, etc. Recruit or draft the reader: It means involving the reader to get involved in the thinking process within the text so that she becomes a part of the context. This is incorporated into the texts by a careful use of language, for example, using the word we implicitly implying the readers as well. Legitimate atypical situations: This implies giving authority or legitimating situations which may otherwise be not so convincing for the users. It can be incorporated into the 1649

texts in various ways but one of the ways is similar to the principle of abstraction and generalization (Klein and Myers, 1999), which means applying a relevant theoretical base for legitimating situations. For example, Barley (1986) shows how structuration theory explains the phenomenon in question. Smoothe contestable assertions: In ethnographic texts, there may be some assertions which may not appear to be true to the reader at the first instance. The author through the convincing use of language and involvement of the reader has to try to smoothe his assertions. This technique is also reflected in the principle of multiple interpretations and also the principle of hermeneutic circle (Klein and Myers, 1999) to highlight varying perspectives on a particular issue to finally synthesize an assertion. For example, Markus (1994) uses multiple respondents in the organization to understand the happy medium (e-mail) in a holistic manner. This also has an undercurrent of the principle of suspicion which tries to move beyond the obvious in search for the true understanding, for example in Walsham and Sahay (1999), through an extensive discussion about India having map based culture or not, the authors are able to make their point. Build dramatic anticipation: This aspect in ethnographic texts highlights the importance of making the text as exciting and gripping as a story (Dyer and Wilkins, 1991). It reasserts the careful use and importance of effective language in convincing the readers. For example in Walsham and Sahay (1999), the explanation of role compartmentalism through the Naipaul s description of cleaning of Bombay hotel steps highlights this aspect. Differentiate the findings: It is very important for the readers to understand that the study makes some distinctive findings which enhances the body of knowledge. This spells out the value element of the ethnographic study. Criterion 3: Criticality implies that the ethnographic text probes the reader to question his taken-for-granted assumptions. Golden-Biddle and Locke (1993) mention that not all ethnographic texts fulfill this criterion satisfactorily. This criterion creates a space for the reader to think, analyze and ponder over the research. This criterion incorporates the principle of dialogical reasoning (Klein and Myers, 1999) at a much broader level enabling readers to reflect upon the possible differences from their expectations. It can be incorporated into the research through the following ways: Carving out room to reflect: This implies that the researcher has given food for thought to the reader by leaving and opening up certain possibilities for her to reflect upon. Provoking the recognition and examination of differences: This means that the researcher is able to evince sufficient interest in the reader to recognize and examine various differences highlighted in the text. From the vantage point of good interpretive research, this aspect of criticality is supposed to stimulate dialectical thinking in quest for a synthesized solution. Enabling the readers to imagine new possibilities: This extends the theme of criticality further in the potential of the ethnographic text to stimulate the reader to think in creative ways. It can be incorporated in studies by the use of appropriate metaphors or suggesting unorthodox possibilities. The three criteria for assessing the quality of ethnographic work have to be present in the study to make it convincing. The three criteria have some underlying linkages with the seven principles as shown in Table 1. Table 1: Linkage between the seven principles and the three criteria for interpretive IS research Criteria Sub criteria (Golden-Biddle and Locke, 1993) Authenticity Particularizing everyday life Principle (Klein and Myers, 1999) Principle of contextualization Principle of hermeneutic circle 1650

Plausibility Criticality Delineating authors relationship in the field Depicting the disciplined pursuit and analysis of data Qualifying personal biases Normalizing unorthodox methodologies Recruiting or drafting the reader Legitimating the atypical Smoothing the contestable Building dramatic anticipation Differentiating their findings Carving out room to reflect Provoking the recognition and examination of differences Imagining new possibilities Principle of interaction between researchers and the subjects Principle of contextualization Principle of abstraction and generalization Principle of multiple interpretations Principle of hermeneutic circle Principle of suspicion Though there are linkages between the seven principles and the three criteria, the two address the research in different ways. Some of the sub criteria of the plausibility criterion are not linked to the seven principles: normalizing unorthodox methodologies, recruiting or drafting the reader, building dramatic anticipation, and differentiating their findings. Though there are overlaps between the two methodological lenses, the application of both provides a better understanding of the research. Hence the application of the seven principles of interpretive IS research (Klein and Myers, 1999) along with the three criteria for assessing the convincing ability of ethnographic texts provides a holistic framework for conducting and interpreting interpretive IS research. 3. Contributions and Conclusions Recent increase in interpretive work in information systems is not marked by a proliferation of the current methodologies for conducting and assessing it. The underlying principle of exhibiting cogency and plausibility in interpretive research has been elucidated through a number of principles and criteria. Though these rules are available in literature, they are disbursed in the vastness of the body of knowledge. There are two main contributions of this paper. First, we explain the importance of conducting interpretive research in IS, by mentioning how a lack of perspectives in a social field like IS will result in lop sided development of the field. Second, by bringing together the two methodological lenses used for conducting and assessing qualitative IS research, namely the seven principles for conducting and evaluating interpretive field studies (Klein and Myers, 1999) and the three criteria for assessing ethnographic qualitative studies (Golden- Biddle and Locke, 1993), we expound an integrated framework for conducting and assessing interpretive IS research. We also demonstrate that the three criteria for assessing quality of ethnographic research have an intertwined relationship with the seven principles. An explicit assessment, using the integrated framework, by the researcher in the conduct and writing of an interpretive IS research will not only be helpful in convincing the readers but also putting the ideas cogently in a logical fashion. The framework presented in this study will also be useful for the reviewers and readers of qualitative work to appreciate and assess interpretive IS studies. 1651

4. References Barley, S. R. Technology as an Occasion for Structuring: Evidence from Observations of CT Scanners and the Social Order of Radiology Departments, Administrative Science Quarterly: (31), 1986, pp. 78-108. Callon, M. Some Elements of a Sociology of Transformation: Domestication of the Scallops of the Fishermen, In J.Law (Ed.), Power, action and belief: a new sociology of knowledge: 196-233. London: Roultedge & Kegan Paul, 1986. Dyer Jr., W. B. and Wilkins, A. L. Better Stories, Not Better Constructs, To Generate Better Theory: A Rejoinder to Eisenhardt, Academy of Management Review, (16:3), 1991, pp. 613-619. Giddens, A. New rules of sociological method. London: Hutchinson, 1976. Golden-Biddle, K. and K. Locke. Appealing Work: An Investigation of How Ethnographic Texts Convince, Organization Science (4), 1993, pp. 595-616. Greif, A. Cultural Beliefs and the Organization of Society: A Historical and Theoretical Reflection on Collectivist and Individualist Societies, Journal of Political Economy (102: 5), 1994, pp. 912 950. Hsiao, R. Technology Fears: Distrust and Cultural Persistence in Electronic Marketplace Adoption, The Journal of Strategic Information Systems, (12:3), 2003, pp. 169-199. Keen, P. G. W. MIS Research: Reference Disciplines and a Cumulative Tradition, in Proceedings of the First International Conference on Information Systems, E. McLean (ed.), Philadelphia, P.A., 1980, pp. 9-18, Klein, H. K. and M. D. Myers. A Set of Principles for Conducting and Evaluating Interpretative Field Studies in Information Systems, MIS Quarterly (23:1), 1999, pp. 67-94. Latour, B. Science in Action: How to Follow Scientists and Engineers through Society. Cambridge, MA: Harvard University Press, 1987. Markus, M. L. Finding a Happy Medium: Explaining the Negative Effects of Electronic Communication on Social Life at Work, ACM Transactions on Information Systems (26:6), 1994, pp.119-149. Orlikowski, W. J. and Baraoudi, J. J. Studying Information Technology in Organizations: Research Approaches and Assumptions, Information Systems Research, (2:1), 1991, pp. 1-28. Orlikowski, W. J. and D. C. Gash. Technology Frames Making Sense of Information Technology in Organisations, ACM Transactions on Information Systems. (12:2), 1994, pp.174-207. Robey, D. and M. Newman. Sequential Patterns in Information Systems Development: An Application of a Social Process Model, ACM Transactions on Information Systems (14:1), 1996, pp. 30-63. Sitkin, S. and Roth, N. L. Exploring the Limited Effectiveness of Legalistic Remedies for Trust/Distrust, Organization Science (4:3), August 1993, pp. 367-392. Vessey, I., Ramesh, V., and Glass, R. L. Research in Information Systems: An Empirical Study of Diversity in the Discipline and Its Journals, Journal of Management Information Systems, (19:2), 2002, pp.129-174. Walsham, G. and S. Sahay. GIS for District Level Administration in India: Problems and Opportunities, MIS Quarterly (23:1), 1999, pp. 39-66. 1652