Assessing and evaluating the quality of management research Hertta.vuorenmaa@aalto.fi
Questions?
Today we will Learn to assess in a correct manner Do a role play and later a group exercise Go over the general peculiarities of scientific evaluation
After today s lecture you should be able to: Evaluate the quality of research: rigour in the world of academia universal quality criteria contingent quality criteria (depend on the researcher s philosophical position) criteria originating from research practice Appreciate the quality of research from the perspective of practicing managers: relevance in the world of management practice
Quality criteria from the viewpoint of management practice: Relevance of research
Class exercise Your group has the possibility of being granted access to Orton hospital to study the perception of quality in health care services provided to Russian patients. The initial contact with Orton is through the training manager, who is a former employee of the university. She has arranged a meeting with the managing director Juha Aarvala to discuss the possible research. The meeting will take place onsite and the managing director has a busy schedule.
Class exercise (cont d) Form a group of 4 persons: Student 1: Your aim is to negotiate access for your MSc thesis project that meets the academic criteria of quality. Managing director Juha Aarvala: He is prepared to provide access assuming the project uses minimal resources/time, offers benefit to Orton and carries absolutely no risks for the hospital or the patients. Marketing manager: Her job security at Orton is questionable. Student 4: Your task is to act as an observer during the roleplay and debrief the rest of the class about your observations.
What was the key learning of this role play for you?
Quality criteria from the viewpoint of various philosophical positions: Rigour of research
Universal positivist quality criteria 1. Construct validity: identifying correct operational measures for concepts being studied e.g. multiple sources of evidence, member validation 2. Internal validity: establishing a causal relationship e.g. pattern-matching (comparing empirically observed patterns with predicted ones), addressing rival explanations 3. External validity: establishing the domain to which a study s findings can be generalised e.g. replication logic for multiple case studies 4. Reliability: demonstrating that the operations of a study can be repeated with the same results e.g. case study protocol/database, auditing Source: Yin (2014, pp. 45-49); see also Yin (2009, pp. 40-45)
Dominance of positivist criteria Analysis of case studies published in 10 management journals published 1995-2000 We failed to identify in our sample a single case study that used, and explicitly reported, rigor criteria other than the [positivist] validity and reliability notions discussed here (p. 1473). Source: M. Gibbert, W. Ruigrok and B. Wicki (2008), What Passes as a Rigorous Case Study?, Strategic Management Journal, 29, 1465-1474.
Compare the following views on validity the degree to which the finding is interpreted in a correct way (Kirk and Miller 1986) we define validity as how accurately the account represents participants reality of the social phenomena and is credible to them (Creswell and Miller 2000, pp. 124-125)
The limits of positivist quality criteria is it really imperative to meet all four criteria [of Yin s]? The first two criteria are essential whenever we make a plausible argument in any sphere of social discourse. We must pay careful attention to whether our argument is really supported by the appropriate evidence (construct validity) and is looked upon as more plausible than other possible arguments (internal validity). However, the other two criteria, reliability/replicability and external validity, are relevant only if the social researcher is searching for an invariant and universal law. Source: Numagami (1998, p. 30)
Alternative criteria for quality evaluation Positivist criteria Internal validity Generalizability Reliability Objectivity Naturalistic criteria Credibility Transferability Dependability Confirmability Source: Symon and Cassell (2013)
Guba and Lincoln s naturalistic (neo-empiricist) criteria Credibility: achieving fit between the constructed realities of research participants and researcher s reconstructions Transferability: providing sufficient detail about the research situation so readers can judge to which other contexts the findings are relevant Dependability: providing an audit process that tracks and accounts for changes in the methodological process followed Confirmability: grounding the conclusions in data by showing where the data came from (data collection) and how they were transformed into findings (data analysis) Source: Symon and Cassell (2013)
Validity procedures Triangulation: search for convergence among multiple/different sources of data, methods, investigators, theories Disconfirming evidence/negative instances: after establishing preliminary themes/categories, search for evidence that (dis)confirms it Researcher reflexivity: researchers selfdisclose values and biases, and reflect on their role in the study Source: Creswell and Miller (2000)
Validity procedures cont d Member checking: taking data and accounts back to participants for comment and review Prolonged engagement in the field: repeated observations and encounters, building trust and understanding Collaboration: close collaboration with participants throughout the process of research Source: Creswell and Miller (2000)
Validity procedures cont d Audit trail: clear documentation allowing an external person to examine the process as well as outcome of the study Rich description: high contextualised and detailed accounts, establishing verisimilitude Peer debriefing: review of data and research process by someone familiar with the research phenomenon Source: Creswell and Miller (2000)
Transparency as a criterion Transparency concerns whether the article reported sufficient information in both data collection and analysis for the study to be replicated to a reasonable extent (p. 1874) High transparency helps remove the stigma of qualitative research as second-class research by allowing the traditionalist reader to determine the accuracy of the conclusions drawn (pp. 1880-1881) High transparency allows readers to learn not just from the findings of the research, but also from the methodologies, allowing comparison of methods across articles towards a standard of the best practices in qualitative research (p. 1881) Source: Bluhm, Harman, Lee, and Mitchell (2011)
Criteria stemming from research practice Quality of research output making a contribution that s interesting! Quality of research process detailing the steps technically accomplished non-linearity Quality of research performance a logical and internally consistent and compelling story rhetorical skills Source: Symon & Cassell (2013, p. 214)
A brief class discussion: Why are the positivist quality criteria problematic for the social constructivist researcher?
However, sometimes qualitative research does go into boxes and is excellent.
Grouping text before moving it to the tables Davide Ravasi slide from autumn 2016
Micro material practices (1 st order codes) Aggregate material practices (2 nd order codes) Macro-phases of sensemaking Cognitive processes (2 nd order codes) Cognitive sub-processes (1 st order codes) Browsing and collecting Material memory (cues) Noticing & Bracketing Extracting cues from flow of experiences Recreating experiences Material assemblage Bucketing Material classification Verbal articulation Articulation Linking material cues and abstract categories Visual referencing Sorting things out Brain dumping Thumbnailing Frameworking Group Sketching Storybuilding Material memory (provisional interpretations ) Interactive talk Visual Integration Elaboration Influence Storing, sharing and retrieving mental content Integrating and refining emerging mental structures Parking ideas Connecting brains Getting in the right frame of mind Keeping the bread crumbs Capturing ideas Organizing thoughts Building on each other s ideas Walking the client through Davide Ravasi slide from autumn 2016
Micro- practices (material) Browsing & collecting Material assemblage Bucketing Brain dumping Thumbnailing Cognitive subprocesses Recreating experiences Visual referencing Sorting things out Parking ideas Keeping the bread crumbs Connecting brains Getting in the right frame of mind Capturing ideas Aggregate material practices Material memory (cues) Material classification Material memory (provisional interpretations) Aggregate conversational practices Verbal articulation Interactive talk Macro-phases of sensemaking Noticing & Bracketing Articulation Elaboration Frameworking Group sketching Storybuilding Organizing thoughts Building on each other s ideas Walking the client through Visual Integration Influence Davide Ravasi slide from autumn 2016
Conclusion Evaluation criteria are constituted by particular philosophical conventions (p. 133); they are not value-free Yet it is often assumed that one size (i.e. positivist criteria) fits all This may lead to using inappropriate criteria to evaluate qualitative research (positivist criteria as the commonsense benchmarks, p. 136) Solution? Contingent criteriology that enables different sets of evaluation criteria to be contingently deployed so that they fit the researcher s mode of engagement (p. 134) Source: Johnson, Buehring, Cassell and Symon (2006)
Wrapping the course up.
My part sum up 1. Reflect on you, your surroundings, why do you do/think the way you do? Then do it again and be honest. Unpack why do people act the way they do/learn the way they do, write the way they do? Be aware of your academic community and where you come from and how that influences your thinking.
My part sum up 2. Know your epistemology & ontology and where you are in the midst of it all. Evaluate research in a manner that shows that you understand both the context and the nature of the argument and can place both, thus can assess both. Know the classics of your chosen line of thinking, understand where the thought structures come from. Give your feedback in constructive manner
Anne s part sum up. Write a lot and edit fiercely. Learn all the rules by heart. Manage your time. Do not wait for the right moment and don t be afraid. the single most useful piece of equipment for a writer is a bucket of glue. First you spread some on your chair, and then you sit down. - Joan Bolker: Writing Your Dissertation in Fifteen Minutes a Day (1998, pp.32)
Please give feedback! You have all been sent a link to a survey, please fill it in and offer some feedback to us, it will help us improve the quality of the course Please note that even if you have not attended any of the lectures you might still comment on other matters
Thank You and Good Luck!