Assignment 4: True or Quasi-Experiment

Transcription

1 Assignment 4: True or Quasi-Experiment Objectives: After completing this assignment, you will be able to Evaluate when you must use an experiment to answer a research question Develop statistical hypotheses that can be tested through experiments Identify the specific design needed to best answer a research question Develop a sampling protocol that will meet the requirements of a specific experimental design; Select the appropriate methods of data analysis to answer the research question Analyze the strengths and weaknesses of a proposed research design in terms of internal and external validity and explanatory power. Rules of the Game for Assignment 4: True or Quasi-Experiment Work in Class. I am setting aside most of the class period on Week 11 for you to work as a group on this assignment. It is due the following week so you need to make good use of this time. The point of this assignment is for you to first create and then critically examine an experimental design that tests a theory-based intervention. By far the most important part of the process is the critical examination. I am not grading the flow chart per se, but rather your justification for decisions and your overall final reflective examination of the design you created in the flow chart. You should not spend hours and hours working on the flow chart. Your group must POST a completed flow chart that all group members contributed to and approved by 4:00 PM on October 30. ONE of you should post it to the Assignment on Canvas called Flow Chart for Assignment 4. List all group members on the submission. This requirement is to ensure that you can spend your time in class on explaining and justifying your decisions (answers to questions 1-8 in the assignment), on identifying resources (references) that you can use to justify your decisions, and on answering the final questions (self-critique). Theory. You must use a theoretically based intervention as the basis for your experiment. Since your task in this assignment is to demonstrate that you know how to use theory in research, I have selected a group of theories that have well-defined constructs and an ample literature base. All of these theories are used to explain health and risk behaviors, but they can and are applied to many other kinds of human behaviors, including adoption of new technologies, educational choices, decisions about finances, use of natural resources, and others. I will not prohibit the use of other theories for the assignment, but I strongly advise you to use one of the selected theories to save yourselves time and effort. I will not say that you absolutely have to use one of these two theories, but I do not want you spending hours on agreeing a theory, trying to learn enough about it to identify the constructs, trying to develop thorough and complete definitions of the constructs, and looking at previous literature to see how people have used the theory. The theories are (1) Elaboration Likelihood Model, (2) Health Belief Model, (3) Protection Motivation Theory, (4) Social Cognitive Theory (although this one will be harder to use), (5) Social Support Theory, (6) the Theory of Planned Behavior, and (7) The Transactional Model of Stress and coping. There is enough information about each of these theories to get you started on the assignment at: Do not spend many hours looking for more information about the theory itself. Spend your time looking at literature and thinking about research design. I think for most of these Assignment 4: True or Quasi-Experiment, FYC 6800, Page 1

2 theories, the information at this web site will be sufficient to design an experiment. You do not have to be highly sophisticated in the use of the theory. You do have to understand enough about the theory to figure out a reasonable treatment based on it, what the independent or predictor and dependent or outcome variables would be and what other variables that could affect the outcome (non-experimental, non-theoretical variables) that you would need to measure in an experiment. I do not expect you to become expert in the theory. Look up a few studies done using the theory or key constructs in the theory to see how people have used the theory in research. You may get lucky and find an example of an experiment using the theory, but I doubt it. Here are some good places to start looking. Your sole objective is to understand the theory well enough to set up a fairly simple experiment! Context for the Experiment. I do not care what context you choose for your experiment The content of the intervention refers basically to the theoretical population of interest, the goals of the intervention, and the PIN of interest in the intervention. You will greatly reduce your workload and probably do a better job on this assignment if you select a context with which one of you is thoroughly familiar. Do not spend hours reviewing the subject matter literature (topical). Spend your time on research design. Submit two Word documents for this assignment A. A completed flow chart. Using each team member s last name listed in alphabetical order, title this document LastName1_LastName2_LastName3_6800_A4_FLOW B. An explanation of the context, theoretical framework, factors you considered in the decisions you made at each step in designing the experiment, and your final critical assessment of your own work. Using each team member s last name listed in alphabetical order, title this document LastName1_LastName2_LastName3_6800_A4_EXPLAIN DOCUMENT 1: FLOW CHART Complete the template. The more detail you provide in the template, the more easily I can understand what you plan to do. You don t need pages and pages, but if all you put under comparison groups is treatment and control, you haven t told me much. You need to develop the flow chart in conjunction with each other. This will be a back and forth process where you adjust what you enter in the flow chart as you think through your decisions. It will be helpful to start by agreeing on the general outlines of what you want to do with your experiment. Then each of you should develop a flow chart the individual charts will save time because team meetings are notoriously ineffective when there are no written documents to serve as the basis for decision making. You can then compare the flow charts to arrive at your final group product. I know this sounds time consuming, but you will save time and effort and do a better job on the assignment if you work together on the flow chart and get one final product that everything thinks is good BEFORE the Oct. 30 class meeting. The worst case is that you try to patch together three different ideas. That will not work. You must have one, coherent flow chart. You all have to make the decisions, understand exactly what you are proposing, and reach agreement. Otherwise, you cannot complete the rest of this assignment. DOCUMENT 2: EXPLANATION Research Goals & Theory Assignment 4: True or Quasi-Experiment, FYC 6800, Page 2

3 1. Define the nature of the problem, issue or need that you want to address (500 words maximum, 150 is fine) and define the theoretical population of interest. This is a description of the what and who of interest to you. 2. What is the theoretical basis for your research? a. Include a diagram or model of the theory like the ones I provided during our first class session and the University of Twente provides a model for the theories I selected for this assignment. Provide a table with the definitions of the constructs You have to be specific about this. MORE detail is better. It makes the assignment easier and it s more likely that I will understand your experiment well. No simple big boxes with lots of arrows going both ways, overlapping circles that include things like proximal, intermediate and distal environment. Be explicit and precise in the definitions. Here is an example of a diagram of a learning theory (Bandura s social learning theory) Explanation of Decision-Making in Construction of the Flow Chart 3. Explain your decision-making process as you completed the flow chart. Include each Box in the flow chart. If you will not use one of the two approaches to data analysis, just write will not use in the box on the flow chart. Indicate each flow chart in the discussion. I cannot grade your assignment if I do not know which decision you are discussing. Do not tell me what you decided here. You have done that in the flow chart. Explain what you considered and why you made the choices you made from the perspective of research design. I do not grade on length of answers. Be brief. You can use bullet points to indicate your thinking process. Go through the flow chart box by box as a group and make notes about what you considered as you reached consensus about what to put in the boxes. Use some of your time to work on this assignment in class to make sure you have all the important points covered. Write down the points you think your group should make in Assignment 4: True or Quasi-Experiment, FYC 6800, Page 3

4 this component of the assignment and bring them to class on Oct. 30 so that you can use your time in class to discuss your ideas and arrive at consensus. As you write down the points you think are important write down one or two references that influenced your thinking. Bring those to class. Focus your discussion on (1) disagreements about procedures and (2) making sure you think through the logic of each decision. I provide a complete (OK, long) discussion of the kinds of information that I expect for this component in the assignment in the document Assignment 4: Examples and Guidance for Discussion of Decisions. I try to describe in some detail what I want to see in each section. Please use this document as you complete this part of the assignment. Otherwise, you are unlikely to provide sufficient information for me to assess your mastery of research design principles. Self-Assessment of the Design This is where you should spend at least 50% of the time you spend on this assignment. Spend most of your class working time on this. To be successful, this must be a group product. This is where you show me that you have a thorough grasp of the concepts we have studied and that you can use the research design literature to improve your own work. Think through what you set out to do. Review the research question(s) and the objectives. Then review your answers and think about the strengths in your design and the weaknesses. Do not try to redo the assignment. DON T CHANGE THINGS in the flow chart or the discussion. That is not the objective here. The objective is for you to think through what you have proposed and identify the strengths and the weaknesses that you cannot reduce and, probably, some weaknesses that you could address through better design decisions. It s ok to point out what you could do to fix weaknesses. However, the main point of this portion of the assignment is for you to critically analyze your own work. I won t be impressed by a discussion that just repeats why you decided to do X. You ve told me that. I want you to treat your work as though YOU were the reader, not the researcher. Try to answer these two questions: (1) As a reader, what aspects of your design would convince you of the value of the work? (2) What weaknesses in the design would you spot? 1. Consider your design and protocol as a whole, including sampling and planned statistical analyses, with regard to internal validity. Remember that internal validity, for a realist, refers to the confidence that the conclusions that you reach are valid and justified. Internal validity involves both providing evidence that the explanation you propose (my idea that by increasing perceived power I can increase change in behavior) is valid and by your ability to show that other things probably did not cause any change you observe (eliminating alternatives). a. What are the greatest strengths of your design and protocol with regard to internal validity? Put another way, what aspects of your design and protocol would reassure a reader that your conclusions are justified? Refer to the specific threats to internal validity that we have discussed in class. Consider each of them carefully as you answer this question. Be specific in your responses. Do not just repeat broad generalities about the kinds of strengths your design has in general. Respond in terms of your study and what you can do to make your study as strong as possible. For example, in my proposed study I did four things that would give me some confidence in my conclusions. First, I set up a pre- and post-test of behavior for each. Any change that occurred should be the result of the training, and the Assignment 4: True or Quasi-Experiment, FYC 6800, Page 4

5 treatment group should show a greater change in behavior. Second, I tested perceived power pre- and post-test for both treatment and control groups. I am proposing that I can change this condition by training. Any difference between preand post-training scores should be due to the training. Further, I should see a greater difference in the treatment than in the control group. So including this variable increased the internal validity of my study. Third, I also collected data about the intervening variable (behavioral intent). This helped, too. Finally, I plan to run regression analysis where I can confirm (or not) the proposed linkages between all these variables and, more important, I can see if perceived power is a more important (higher beta value) predictor of behavior in the treatment than in the control group. If so, this is one more piece of evidence that my idea is valid and justified. To eliminate other variables, I included a pre- and post-test of knowledge (this is something I really do usually do). Knowledge is not a construct in the theory of planned behavior, but we do know that knowledge is a pre-requisite for behavior. Therefore, I need to determine whether prior knowledge affected the outcome. Maybe all my participants were very knowledgeable before the training resulting in no affect. Worst yet, perhaps there was a pre-training difference in knowledge for the two groups. By including knowledge as a construct and testing pre- and post-training, I should be able to eliminate or at least evaluate the importance of knowledge in the outcome. b. What are the greatest weaknesses of your design with regard to internal validity? Consider each of them carefully as you answer this question. Be specific in your responses. Do not just repeat broad generalities about the kinds of weaknesses your design has in general. Respond in terms of your specific study. For example, in my study I have a potential problem because my sample is essentially a volunteer one. I cannot force people into my training sessions. They sign up because they are interested. For example, people who want to lose weight sign up in weight loss classes. If my training were Diet Management for Weight Loss, I could very possibly attract people who really want to change their eating habits. This could mean that almost any training approach will work, that I will not see a difference due to training method. On the other hand, since I conduct my training in a one-time, short-duration setting, I probably will not have to worry about mortality (dropouts) so that s a strength. 2. Consider your design and protocol as a whole, including sampling and planned statistical analyses, with regard to external validity. Remember that external validity, for a realist, refers to the degree to which we can extend our findings, and especially our conclusions, to people, places, conditions, or circumstances other than those under which the study was conducted. Sometimes we want to generalize statistically and we almost always want to generalize theoretically. Address both types of generalization in your responses. a. Identify the key strengths of your study with regard to external validity. Consider both statistical and theoretical generalization. Be specific. Do not just include generalities about the design in general. Refer to what you plan to do. For example, unlike some experiments, my study is not at all contrived, so artificiality should not be a problem. By including a pre- and post-training test of knowledge and including this in the regression model, I strengthen theoretical generalization because I can compare the significance of knowledge (a key construct in other theories) and perceived power as predictors of change in behavior. If it turns out (which it does when I do this) that knowledge does change, but that it does not predict change in behavior, I go a long way Assignment 4: True or Quasi-Experiment, FYC 6800, Page 5

6 toward saying that the theory of planned behavior is a good one when you have to deal with these short-term exposures to trainees. b. Identify the key weaknesses of your study with regard to external validity. Consider both statistical and theoretical generalization. Be specific. Do not just include generalities about the design in general. Refer to what you plan to do. People volunteer for my study because they want the training they presumably have high motivation. This may pose some problems for statistical generalization. I probably can generalize the general conclusion that discovery-based learning will, in general, with lots of kinds of people, topics, places, etc., be superior to the traditional lecture-based learning in creating behavioral change. However, given the potential for high motivation on the part of my participants, I probably cannot generalize the degree of change in behavior from my study to other settings. In situations where motivation is not nearly as high, the amount of change might be considerably less. I also need to be careful about extending my conclusions to other kinds of training settings. I get a one-time, shortduration exposure to trainees. We cannot, based on my findings alone, conclude that the focus on discovery learning would result in similar behavioral changes under very different conditions. This is an opportunity for more research for example, in classroom settings where there is longer-term, repeated exposure. 3. Discuss the contributions to explanatory power of your study your contributions to the body of knowledge. Remember that explanatory power rests on the degree to which the researcher can explain the phenomenon under study in a complete and robust way. Explanatory power rests, to a large degree, on accumulating multiple kinds of evidence to support (or not) a theoretical explanation. Contributions to explanatory power often come from providing a different kind of evidence. For example, if most of the research about a specific topic has relied heavily on a single type of design, the scientific realist would argue that evidence from the other design groups is needed to enhance explanatory power. Contributions may also come from addressing parts of the theory that have been largely ignored in the research base. This may include, for example, verification of some aspects of the theory, extension of the theory to areas where it has not been tested before, or building new components into the theory, to name a few. What are the contributions of your study to explanatory power? What kinds of evidence will we have that we have not had before? What will we know or understand after you complete your study that we did not know or understand before? Assessment Criteria Possible Points Followed instructions, including full APA citations and references 5 Question was sophisticated, a how or why question rather than a simple 25 what question and is clearly based on the theory The question moves beyond a single pathway of cause and effect to incorporate complicating factors like gender, age, prior experience in the research design (e.g., additional experimental factors beyond the treatment or intervention) Identified ways to make contributions to the body of knowledge that draw on the strengths of the experimental or control group of designs Specifically included design considerations in explaining and justifying the objectives of the research Developed robust research hypotheses that are not simple and obvious but Your Points Assignment 4: True or Quasi-Experiment, FYC 6800, Page 6

7 rather incorporate more complex approaches to understanding causality and using controls to establish evidence of cause and effect Was able to distinguish between constructs and variables and between predictor or independent and outcome or dependent variables and selected appropriate variables to represent the constructs of interest. Demonstrated an understanding of the importance of the concept of control in experiments in the decisions made regarding design choice, sample selection and use of statistical control of non-experimental factors Selected and justified the selection of a specific design that is appropriate for the research question and the treatment(s) that will be imposed Selected and explained how the design reduces non-experimental sources of variance in the outcome Provided a logical argument that explains why the treatment is theorybased and explains how the treatment is appropriate for testing the stated research hypotheses Demonstrates an understanding of the difference between the theoretical population, accessible population, and sampling frame The decisions regarding the definition of the theoretical and accessible populations are well justified Can distinguish between probability and non-probability samples and between random and non-random samples and identified an appropriate sampling strategy The sampling approach, including selection of screening criteria, minimizes the effect of non-experimental variables on internal validity Incorporated sampling procedures that increase internal and external validity and maximize the researcher s ability to detect direct cause and effect relationships. Developed appropriate procedures for analyzing the data that result from the experiment based on the statistical hypotheses stated and other considerations like the level of measurement of the variables. The analytic procedures will maximize the researcher s ability to demonstrate direct cause and effect Incorporates data analysis techniques to test the effect of the predictor variables on the outcome variable(s) Where needed, incorporated data analysis techniques to employ statistical elimination of potential threats to internal and external validity. Was able to explain the techniques chosen and justify their use. Correctly identified the major strengths of the study with regard to internal and external validity, especially ability to draw conclusions about direct cause and effect, maximize explanatory power, and permit statistical and/or theoretical generalization Was able to identify specific threats to internal and external validity and explanatory power that would be of concern in the design did not just discuss general threats common to all experiments Identified any steps taken to reduce the potential impact of these threats on the researcher s ability to draw conclusions about direct cause and effect relationships. Was able to explain why the specific threats identified are of concern in this design, despite steps taken to eliminate them, and explained why the threats could not be eliminated either through alternative designs, Assignment 4: True or Quasi-Experiment, FYC 6800, Page 7

8 sample selection, or statistical techniques Consulted, used, cited and referenced extensive required and additional 20 materials about research design, and particularly true and quasiexperiments Consulted, cited and referenced the literature about sampling to justify the approach selected Consulted, cited and referenced the research design literature about the nature of the research question, development of hypotheses, and the use of theory in research Used and referenced the literature about internal and external validity Total 150 Assignment 4: True or Quasi-Experiment, FYC 6800, Page 8