Module 3 Dr. Maxwell Reema Alsweel EDRS 797 November 13, 2009 Matrix Potential validity threat Looking for answers Leading Overlooking important data that does not seem to fit Asking the right questions Possible strategies - Letting research questions emerge from data (EMIC) - Having open ended questions, letting participants take it where they want to - Analyzing/Using all data - Negative case analysis - Pilot study - Peer review Being native: - Interpretation - Questions - Analysis - Bias - Reactivity - Triangulation - Member checks - Peer review - Researcher identity statement 1
Memo To begin, I want to explain the outline of my matrix followed by a more detailed clarification of each section. I have placed possible validity threats that I feel go together in one box. I began with my main concern of looking for answers: that is the problem of trying to force my data to fit my theory. I have placed leading and overlooking other data in that box as well because I believe they are associated with that problem. This is also related to the issue of asking the right questions; that is what is the purpose/ aim of my question, what am I trying to get to. The concern of being a native is discussed next; what affects effects does that have on my questions, analysis and interpretations. And finally the issue of generalization pops out at me; it is an issue because I believe that my study will not be generalize-able, and how that fact my affect it. Generalizable to what? Certainly in terms of analytic generalization (see my Design book, chapter 6), your results will have some relevance to other Saudi women of this generation. Our trouble is that when we make a single hypothesis we become attached to it (Platt, 1966, p.210), I find this to be true. I first realized that this was a concern when doing a mock pilot study using my friends; I found that I was trying to get to my answer. It was frustrating, and after discussing it with my professor at the time, I discovered that by doing that I was leading the participants and overlooking rich data that did not seem to fit but was in fact very useful to my overall idea. There springs up also unwittingly a pressing of the theory to make it fit the facts and a pressing of the facts to make them fit the theory (Platt, 1966, p.211). It was a double problem, trying to fit pieces together that don t necessarily go together was both frustrating and blinding. One of the strategies to resolve this validity threat was using a pilot study and peer review. By conducting first a mock pilot study using peers and, hopefully next semester, a small 2
pilot study on my intended population, I hope to better understand my role as a researcher, my theories and most importantly asking good questions ( I get to test my questions). By conducting a pilot study and a peer review, my interview and survey questions are looked at from different angles and I believe better assessed. It is very scary to go into something not having a fully defined plan, but I believe that no matter how much planning is done, it is never finalized. By letting research questions emerge from data (emic model), I find that it will lessen the hold a theory will have on my research. Also by asking open ended questions and allowing the participants to speak freely and openly, letting them take me where they want to, will lead to what Maxwell (2005) calls Rich Data : data that are detailed and varied enough that they provide a full and revealing picture of what is going on (p.110). This leads to the immediate analysis of all data, which can help in understanding my general theory and creating new research questions and alternative theories and a better understanding of the phenomenon being researched. By looking at all data, this also leads to what Maxwell (2004) calls searching for discrepant evidence and negative cases ; identifying and analyzing discrepant data and negative cases as a key part of assessing a proposed conclusion (p.21). One pressing possible validity threat is my being a native ; I am studying a group of people like myself. By being a part of the participant criteria, am I blinded by my insider knowledge? This can lead to issues in participants reactivity, interpretation, question formulation, analysis and bias. Yes, but it can also be an advantage; you will understand some thing a non-native might miss. As Maxwell (2005) states it is important to understand how you are influencing what the informant says, and how this affects the validity of the inferences you can draw from the interview (p. 109). One particular consequence of this is how you will be 3
seen by your participants, and what they might be willing to say or not say because of this. For example, are there things that they might not explain fully because they assume that you already know these? Also, you have to be careful not to assume that you do know what they ve experienced and mean, without asking specifically; you might be wrong. By being aware of my role as a researcher and what influences I may have on the conduct and conclusions of the study is a task that is part and partial to my proposal (Maxwell, 2005). Getting others perspectives on my research is another possible strategy. Member checking and peer review can lend a different lens, and give validation to my interpretations. I am hoping that triangulation will help in eliminating bias, but I could still see where I could be bias in my different methods and have the same validity threat. I believe what is needed is a triangulation of different perspectives in addition to different methods but I agree with Greene that triangulation is often overemphasized as a strength of mixed method research; Validity threats are ruled out by evidence, not methods (Maxwell, 2005, p. 112). The researcher tries to identify the potential validity threats, or alternative explanations, that would threaten the proposed explanation, and then search for clues as to whether these processes were operating and if they had the causal influence hypothesized (Maxwell, 2004, p.20); I believe that this is the goal of this memo. By having possible validity threats in mind and being open to change and movement, I believe is very enriching to our data and research, we do not realize the added power that the regular and explicit use of alternative hypotheses and sharp exclusions can give us at every step of our research (Platt, 1966, p.205). Now I think that s easier said than done. I am hoping that these memos will keep me on track, and after listing to Ann talk about her process and how memo-ing helped her, some of the anxiousness is wearing off. 4
Reema: This is an excellent memo and matrix (nice use of Platt!), and your memo does a good job of explaining your matrix. I really don t have anything to add to my comments above. Can I use this assignment as an example in future semesters? Grade: A. 5
References Maxwell, J. A. (2004). Using qualitative methods for causal explanation. Field Methods, 16 (3), 243-264. Maxwell, J. (2005). Qualitative research design: An interactive approach (2 nd ed.). Thousand Oaks, CA: Sage. Platt, J. R. (1966). Strong inferences. In H.S. Broudy, R.H. Ennis, & L.I. Krimerman (Eds.), Philosophy of Educational Research (203-217). New York: Wiley. 6