Information Systems Research Methodology: From Paradigm to Practice (Roelien Goede) The use of Interpretive Narratives in the analysis of Project Failures. (Lynette Drevin) Measuring Information Security Awareness: A Comparison Of Different Research Paradigms (Hennie Kruger en Roelien Goede)
Information Systems Research Methodology: From Paradigm To Practice Roelien Goede
Paradigms Positivism Interpretivism Critical social theory Research Practice (methods) Questionnaires, Hypothesis, Stat. analysis Case Studies, Ethnography, Narratives Action Research Research Methodology Empirical methods Hermeneutics Intervention, Construction, Deconstruction, Emancipation Philosophy of Research Bacon (1561-1626), Descartes (1596-1650) Comte (1798-1857) Dilthey (1831-1911) Habermas (1929-)
The use of Interpretive Narratives in the analysis of Project Failures. Lynette Drevin
Interpretive Narratives and Project Failures Outline What are failures? What type of methods are used to investigate failures is it enough? Narrative and ante-narrative narrative methods Conclusion 17 September 2004 5
Interpretive Narratives and Project Failures What are failures Projects that are never completed Projects that are late, over budget, don t meet requirements, questionable quality Terms: challenged projects, runaways, disasters, expectation failures 1995 US: 31 % projects cancelled Cost of failed projects - $81 Billion 2000: 65000 projects failed 2004 UK: 16 % projects are successful 17 September 2004 6
What are failures (2) Examples London Ambulance system Ambulances not arriving, long waiting periods, loss of lives Faulty system ineffective SE practices Denver Airport Baggage handling system Late opening of the Denver airport baggage handling system Frequent changes to requirements, external political involvement, massive overruns in time and costs Losses of revenue to airlines and Denver city Co-op op Group vs ICL 11 m Failed to deliver goods in time Legal battle ongoing South Africa? Interpretive Narratives and Project Failures 17 September 2004 7
Interpretive Narratives and Project Failures What type of methods are used to investigate failures is it enough? Software crisis go back in time : problems are not new 1968, 1969 Nato conferences Primary activity: evidence collection interviews, observation, documentation, surveys, eyewitness accounts, Problem: after the incident Researcher can t plan for data gathering biases,many views, interaction between sub-systems systems Aim of researcher is to conduct a forensic investigation using different sources, different perspectives CHALLENGE 17 September 2004 8
Interpretive Narratives and Project Failures What type of methods are used to investigate failures is it enough? (2) Choice of research methods is coupled with type of information that is available to researcher Positivistic norm not suitable for meaning-rich contextual work Qualitative methods originated in social sciences look at social & cultural phenomena Qualitative perspectives relies on words Meaning of words must be interpreted process of sense-making CASE STUDY 17 September 2004 9
Interpretive Narratives and Project Failures Narrative and ante-narrative narrative methods Main method of presenting findings: Case study Research methods more qualitative, take into account many perspectives and interaction Case history - investigation after the incident Main tool for researcher: case history Good case study: effective storyline, style, plot personal stories valid source of data, interaction A narrative description (story) is valid if the resulting narrative adds knowledge Researchers: how can information from stories be structured to produce valid research findings? Understanding failures: complex and not only a chronology of events 17 September 2004 10
Interpretive Narratives and Project Failures Narrative and ante-narrative narrative methods (2) Failure stories lack collective consensus Accounts are subjective, contradictory & fragmented Boje proposes ante-narrative narrative methods to take into account : improper storytelling fragmented stories polyphonic stories Incoherent and unplotted stories Stories in a fragmented, incoherent and polyphonic state is in an ante state of affairs before it is constructed in a narrative 17 September 2004 11
Interpretive Narratives and Project Failures Narrative and ante-narrative narrative methods (3) Boje described 8 alternative ways that focus on multi-stranded stories of experience that lack collective consensus 1. Deconstruction 2. Grand narrative 3. Microstoria 4. Story network 5. Intertextuality 6. Causality 7. Plot 8. Theme 17 September 2004 12
Interpretive Narratives and Project Failures Narrative and ante-narrative narrative methods - Example Grand narrative When stories are analysed it is important that grand narratives must be looked into and see how many smaller stories exist within. Each story is an intertextual network. Other voices can be embedded in the grand narrative 17 September 2004 13
Interpretive Narratives and Project Failures Conclusion Questions for research How can interpretive narrative or ante- narrative methods be used to describe software failure situations? How can conflicting and contradictory accounts (stories) of stakeholders and witnesses be combined in meaningful ways to make sense of complex situations 17 September 2004 14
Interpretive Narratives and Project Failures Possible Outcomes Conclusion (2) A new approach to investigating software failures. Assess the potential value of narrative, discourse analysis and their usefulness in the software failure investigation process A framework for selecting the most suitable method(s) for describing failures Insights into the applicability of research methods from other disciplines and their usefulness in the software failure domain derived from experiments An improved method for making sense and generating failure stories in complex environments 17 September 2004 15
A Comparison Of Different Research Paradigms HA Kruger, R Goede and WD Kearney North-West University (Potchefstroom Campus) AngloGold (Australia)
Background 2003 Information Security Awareness program implemented Global South-Africa, North-Africa, North- America, South-America, Australia ± R1 million Pamphlets,CD s,videos,posters,presentations Difficulties: distributed operations,languages 17 September 2004 17
Background - Objective Build prototype model (using positivistic approach) that provides meaningful information in measuring information security awareness at AngloGold (globally and regionally) Specific to AngloGold Comply with principles of sustainability Ease of use Scientifically sound 17 September 2004 18
Background Main focus 6 golden rules 1. Adhere to AngloGold policies 2. Keep passwords secret 3. Use e-mail and Internet with care 4. Be careful when using mobile equipment 5. Report security incidents 6. All actions carry consequences 17 September 2004 19
Methodology Methodology based on 3 dimensions What does a person know (knowledge) How do they feel about the topic (attitude) What do they do (behavior) Each dimension categorized into the six focus areas 17 September 2004 20
Methodology What to measure Overall global awareness level measure at regional (country) level, combine in meaningful way One set of aspects measured at all regions not of equal importance in all regions 17 September 2004 21
What to measure 3 dimensions (KAB) Methodology Each dimension subdivided in 6 focus areas Each focus area further subdivided into specific factors (if appropriate) Measure importance of contributing Regions/Dimensions/Focus area/factors 17 September 2004 22
Regions w North Africa Dimensions 6 Focus areas Overall Awareness Level w Australia w w w w Attitude Knowledge Behavior w w Policies Passwords w w Factors Purpose Confd Factors w Write Give w Brazil w = importance weight 17 September 2004 23
How to measure Methodology Scorecard: V(a) = v i (a)w i n i = 1 V(a) = overall value of alternative (branch) a v i (a) = value score reflecting alternative a s performance on criterion i w i = weight assigned to reflect the importance of criterion i 17 September 2004 24
How to measure Methodology To determine performance v i (a): Questionnaire to test Knowledge, Attitude, Behavior of focus areas To determine importance weights w i : Analytic Hierarchy Process (AHP) 17 September 2004 25
How to measure Methodology To determine performance v i (a): Questionnaire 35 questions True, False, Don t Know Example Knowledge Internet access on AngloGold systems is a corporate resource and should be used for business purposes only 17 September 2004 26
How to measure Methodology To determine importance w i : AHP Pairwise comparisons preference scale numerical values Square matrix eigenvector associated with largest eigenvalue normalize to sum to one unique numerical measure of decision maker s perception Consistency index can be computed 17 September 2004 27
Methodology What did we get out of the method? Awareness levels can be measured/reported at different levels region/dimension/focus area/factor Apply at regular intervals measure change index Drill down facility 17 September 2004 28
Application Start with ± 100 questions Tests iterative process - open-ended questions, monkey puzzle, one-on-one contact, e- mail Final 35 questions Perth 22 respondents Weights AHP IS Security Manager 17 September 2004 29
Application Results Awareness scale: Good (80% - 100%) Average (79% - 60%) Poor (59% and less) Non-awareness 35% Overall Awareness Overall Awareness: 65.2% Awareness 65% Non-awareness: 34.8% 17 September 2004 30
Application Results Australia 6 Focus areas Focus Area 1: Policies Focus Area 2: Passwords Non-awareness 17% Awareness 44% Non-awareness 56% Awareness 83% 17 September 2004 31
Application Recommendations for effective ongoing use Comprehensive bank of questions Importance weights relevant managers Use of practical system data nr of virus infections, unauthorized websites Automation web-based 17 September 2004 32
Application Positivism right approach? Possible involvement of wrong level of management to do importance ratings No guarantee - right questions were asked Dishonest answers from respondents Respondents confer with each other Respondents learn answers 17 September 2004 33
Other Paradigms? 17 September 2004 34
Interpretive Problem Orientation Wider angle = Increase the boundary FROM DATA TO THEORY Areas not specified Researcher Subjectively part of organisation: Researcher needs time and personal involvement to understand the real attitude of the employees. Holistic view of security awareness. 17 September 2004 35
Data collection Interpretive Methods Case studies, ethnography Analysis Narratives, grounded theory Research Conclusions as Rules: Staff members only read security messages if they are not engaged in work with deadlines and if they know exactly where to find such messages 17 September 2004 36
Critical Social Problem Orientation Researcher is agent for change Aim of research is to improve ISA Emancipate organisation from burden resulting from poor ISA. Focus shifts from measuring to change 17 September 2004 37
Critical Social Methods Action research 2 phases Phase 1 : Diagnosis Positivistic or Interpretive Phase 2: Action learning cycle ISA improvement programmes Cyclic approach of measure and change 17 September 2004 38
Advantages (+) and Disadvantages (-) Positivistic + Repeatable gives quantitative results - Possible dishonesty and weight allocation Interpretive + Holistic understanding - Time consuming qualitative result Critical social theory + Change part of research - Difficult to generalize results looks like consultation 17 September 2004 39
Conclusions Know your problem boundary judgment Know your methods ( + and -) Know the outcomes of each method Select an appropriate method! 17 September 2004 40
INFORMATION SYSTEMS RESEARCH METHODOLOGY Thank you 17 September 2004 41