Assignment 2: Sampling & Data Analysis COMBINED (Group or Individual Assignment Your Choice)

Transcription

1 Assignment 2: Sampling & Data Analysis COMBINED (Group or Individual Assignment Your Choice) Objectives: After completing this assignment, you will be able to Explain the sampling and analytic procedures used in research reports Determine whether the researcher wants to generalize his/her specific findings and/or conclusions to a group of people larger than those included in the sample and evaluate the degree to which a given sample and analytic procedures are adequate for the desired generalization Assess the degree to which the results or findings support (or warrant) the claims that the researcher makes, the conclusions that s/he draws, and the implications or recommendations for practice that the researcher makes Assess the contribution of the study to the body of knowledge based on research design considerations This can be a group or an individual assignment. Groups can consist of two or three individuals. I think there is an advantage to working in a group because it allows you to exchange ideas, correct each other s misperceptions, and divide some tasks like looking at the relevant research design (sampling, not topical) literature. If you work in a group, your grade will be a group grade and ONE member of your team should submit the assignment. I understand from personal experience the problems with working in a group for students and particularly for people in distance degree programs who do not see each other in person and who often have many responsibilities outside their degree program. Therefore, the choice is yours. If you want to work in a group, please submit the names of the individuals in your group no later than September 19. Each member of the group should develop a flow chart for both the List A and the List B article, using the template for Flow Chart for ARTICLES YOU READ before you try to complete the assignment as a group. Whether in a group or as an individual, you will upload four Word documents, two for each article. Document 1 is a completed Flow Chart for Articles You Read for the article you selected from List A, using the template linked at the course home page under Documents by Swisher and also linked under Assignment 2 on Canvas. Document 2 consists of your responses to the Discussion Questions listed below. Document 3 is a completed Flow Chart for Articles You Read for the article you selected from List B, using the template linked at the course home page under Documents by Swisher and linked under Assignment 2 on Canvas. Document 4 consists of your responses to the Discussion Questions listed below. If you work in a group, develop your responses to the Discussion Questions as a group process. Do NOT try to divide up the work. That always fails because the answers are not consistent. E.g., someone discusses the sample as though it is a probability sample in Q1-3 and then later in discussing generalization makes comments that are applicable only to nonrandom samples. Each of you should decide your answer independently. Write down your ideas a phrase or a few words are all you need. You can then have a fruitful team meeting to reach agreement. Use the following identifiers for your submissions, listing list the names of your team members by last name only in alphabetical order or, if you are submitting an individual assignment, your last name only: 1. LastName1_LastName2_LastName3_A3_Article_A_Flow 2. LastName1_LastName2_LastName3_A3_Article_A_Discuss 3. LastName1_LastName2_LastName3_A3_Article_B_Flow 4. LastName1_LastName2_LastName3_A3_Article_B_Discuss Assignment 3, FYC 6800 Page 1

2 I strongly encourage you to look at the example of a completed flow chart for articles you read. It is a TEMPLATE for what your flow chart should look like all the components, the level of detail. Please use this resource. State responses to all questions in your own words, including what you put in the flow chart. Do not copy and paste from the article. I base my assessment of your comprehension and ability to apply key concerns in large part on your ability to state things in your own terms. When you can explain things in your own words, I know whether you understand the concepts or not. Copying and pasting and lots of paraphrases from the article do not show me that you understand the concepts of research design. Repeating what you ve read or what I have said earns no points. Apply what you have learned. COMPLETING THE FLOW CHART First, read the introduction carefully to find the research question (sometimes not explicitly stated) and the research objectives (almost always stated). The introduction explains the author s motivation (what they wanted to fix or solve or why they find existing theoretical explanations inadequate or why they thing there are aspects of the phenomenon under study that are not well explained. The discussion lays the basis for the objectives. Usually the objectives are in the last few paragraphs or even just the last paragraph of the introduction. Second, read the conclusions really do go to the conclusions immediately after you read the introduction. Results are specific to a study. Conclusions are what we want to generalize and they are not specific to a given study. The conclusions in a study flow directly from the research objectives. I find that students confuse results and conclusions and then criticize authors because the results cannot be generalized. This shows poor understanding of the nature of scientific research and how studies fit together to generate knowledge. Focus on the sampling and analytic processes in your responses to this assignment keeping in mind that each step in the process is like a step in building a car. Mess up a step and the final product may be very flawed. This is not easy you need to consider both the pitfalls in specific procedures used and how the choice of procedures and the degree to which they were successfully completed affects the conclusions. The conclusions, as we know, comprise the author s answer to his/her research question and his/her contributions to the body of knowledge. We judge the value of the conclusions based on the whole process from developing a good, thick question to the sampling and analytic procedures used, to the claims made. Demonstrate that you understand this process. Please take care to make sure that you do not mis-state the author s objectives, research question, and theoretical hypotheses. If you get these wrong, everything else in your analysis is likely to be wrong. For example, your assessment of the adequacy of the sample will depend on whether the sample was good enough to answer the research question. If you misunderstand or misstate the research question, you will not be able to assess the adequacy of the sample. You may not like the author s objectives or question. You may think s/he should have asked a different or broader question. However, the researcher determines the question and objectives not the reader. One very common error is to confuse the problem the author wants to address or the potential uses of the new knowledge s/he creates with the research question and objectives. In one article that I have used for the example of a flow chart (not this year) I have seen students say that the author s objectives are to improve people s stress management skills or to improve women s stress management skills. It is true that the author of this article does want to improve workplace stress management for employees and he is specifically concerned about stress management for women because of the dual family/workplace stress many women experience. However, research deals with creating knowledge that we can then use to solve problems. Therefore, his objective for this research is not to implement some training or fix the problem through some program. He has two main objectives: (1) determine if Assignment 3, FYC 6800 Page 2

3 training actually does improve stress management and (2) if gender affects response to training. A training program is his intervention or treatment in a quasi-experimental study. It is NOT the objective of his research. Provide enough detail in the flow chart for me to understand how well you grasp the material in the article and your ability to apply what we are learning in this class to your work on this assignment. If you provide super short answers of a couple of words on the flow chart, I will not be able to assess whether you understood the article and were able to identify the specific components in the article that you have to address in this assignment. Do not write paragraphs or long discussion, but do not be vague: specific but brief answers. For example, for sampling do not say something like random sample specify the specific type of random sample, e.g., systematic random sample. You do need a lot of detail for Boxes 5-7 about sampling. Look at the sample I provide of a completed flow chart for an article I read. I listed every single statistical hypothesis. You need this level of detail. If you do not provide detail, I will be unable to assess your understanding of the research strategy and the specific steps in the process. For example, do not say something like statistical test -- specify the specific test, e.g., student T test for two independent samples. DISCUSSION QUESTIONS Discussion Questions are the core of the assignment. You describe what the author did in the flow chart. You assess the quality of the procedures used and the overall value of the work in your responses to the discussion questions. Do not repeat what you said in the flow chart. Concentrate on evaluating what the author(s) did. Answer the discussion questions in narrative form. I suggest that you make each key point in sentence in bold typeface that states the point clearly and follow it with the support for your argument. Long rambling paragraphs filled with unsubstantiated statements will convince me that you do not understand the key concepts we discuss in this class and that you did not use the research design literature to assess the article. Here is an example of the kind of response that would be appropriate as part of an answer to Q3 about qualitative data analysis (Assignment 3), but this approach will work for every assignment. The authors did not specify the analytic procedures used. The authors do not include any details about the specific steps they used to reach conclusions, saying only that they used a grounded theory approach in their work. Saini & Shlonsky (2012, p. 116) argue that regardless of the epistemological or ontological assumptions guiding a particularly qualitative study, the story should be told in a consistent, transparent way and should adhere to the highest standard of methods associated with the philosophical traditions the investigators purportedly draw from. Saying grounded theory fails to meet this standard. The authors do not explain which of the many approaches to grounded theory they used in this study, leaving me unsure about the reliability of the interesting conclusions they reached. Hardy and Bryman (2008, p. 626) make comments that are relevant to the approach taken in this study: However, like Rennie (2000), we view most core methodological writings on grounded theory as rather insular, placing too little emphasis on making connections with other traditions of qualitative inquiry and ways of conceptualizing, justifying and practicing social science research. Assignment 3, FYC 6800 Page 3

4 In their final discussion, the authors employed none of the four kinds of questions that researchers should address when they use qualitative analysis identified by Patton (2002, p.467). Two of these were particularly important omissions. The authors provided no discussion of the degree to which their findings are supported by previous research (qualitative or quantitative in nature). They do not discuss the degree to which their findings are new, or innovative. Practice critical thinking, not criticism. Be neither over critical nor too willing to accept pretty much anything. I am not upset if you give answers that are kind of yes and kind of no as in we thought maybe the sample was adequate because. But then we also thought there were some problems with the sample because I want to know how well you understand the principles of research design. I am most interested in your explanations how sophisticated they are and whether they show a good grasp of the materials we have covered. Consider multiple perspectives about research design in your response. For example, you will find that Yin and Gorard disagree about the value of case study designs. If you are discussing a case study design, I want to see that you understand their differences. Focus on showing that you can think about design in a sophisticated way. There are no right and wrong answers to the discussion questions. You may have actual wrong things on the flow chart but here I want to see your thought processes. I m not looking for a single right answer. There isn t one. You have to demonstrate what you ve learned. In fact, as you answer the discussion questions you may spot errors in the flow chart. Do not spend time changing the flow chart. Rather use this as an opportunity to demonstrate that you thought about your responses carefully. Draw attention to the error, explain what is wrong with your comment on the flow chart and explain what you now think is a better interpretation of the material in the article. Use, cite and reference the research design literature, including required readings, additional materials provided at the course website, and materials that you find that are relevant to the specific design or type of analysis used. Please review pages 4-6 of the syllabus to understand what use, cite and reference means. Do not do the assignment first and then throw in references. That does not work. DISCUSSION QUESTIONS 1. Explanatory Power Adding to the body of knowledge 1A. Sampling & Explanatory Power Assess the explanatory power of this study based on the SAMPLING PROCEDURES used. Focus on the degree to which the procedures are appropriate and sufficiently rigorous to justify the author s primary intended contributions to (1) the overall body of empirical evidence we have about the topic, (2) our ability to understand and explain the social phenomenon of interest, and (3) the adequacy of the theoretical base on which our knowledge rests. Focus on the sampling considerations. I know that you may not know much about the theory or the topic of the study or the theoretical basis of the research and I am not grading this based on your expertise in that regard. Make sure you specifically address how the author s sampling approach and sampling decisions and the quality of the final sample affected all three components listed above, remembering that theory-building may not be an objective in which case that consideration is not relevant. However, Assignment 3, FYC 6800 Page 4

5 most scientific research (excluding purely descriptive studies) do use theories and failure to use theory is a weakness for many reasons, including generating several sampling problems. Remember, research does not have to produce earthshaking results to be good, solid work that contributes to the body of knowledge. However, it is also true that not all research really adds to what we know. To answer this question, you first need to assess the overall quality of the research question. It is hard to achieve good explanatory power if you start with a poor research question. As we have discussed, good research questions are how and why questions not just what questions. They rest on theory but push beyond simply providing one more demonstration of a well-established connection between theoretical constructs or one more example of applying Theory A to explain phenomenon Y. Answering good questions tells us something new. Further, good research questions lead to surprising hypotheses or propositions for the future. They lead us to move beyond predicting the obvious to making novel or unexpected predictions about things. Think through the entire study as you answer this. Make sure you examine the conclusions. The conclusions should respond directly to the author s research question and his/her specific objectives. Conclusions that are weak or do not respond to the research question are often a sign of poor explanatory power, sometimes because the question itself was weak and often because the design decisions (type of study, sampling, data analysis) were not robust did not lend rigor to the conclusions the author could reach. With regard to sampling, focus on the degree to which the sampling decisions permitted the author to develop new knowledge about the topic, the phenomenon, or the theory. 1B. Analytic Strategy and Explanatory Power Assess the explanatory power of this study based on the analytic strategy and the execution of each step in the process. Focus on the degree to which the strategy is appropriate and sufficiently rigorous to justify the author s primary intended contributions to (1) the overall body of empirical evidence we have about the topic, (2) our ability to understand and explain the social phenomenon of interest, and (3) the adequacy of the theoretical base on which our knowledge rests. A good way to go about answering this question is to think first about the overall process. I often draw a sort of flow diagram of the steps because it helps me understand the author s total strategy the totality of what s/he does in the analysis. Identify the steps first. Then look at the results. Make sure you understand them fully. This means getting to the detail. Look at my example of a completed flow chart. The results are very specific test by test, variable by variable. You must get the details right. Otherwise, you cannot assess the conclusions. You cannot answer the question if you focus purely on the results. Examine the conclusions. The conclusions should respond directly to the author s research question and his/her specific objectives. Conclusions that are weak or do not respond to the research question are often a sign of poor explanatory power, sometimes because the question itself was weak and often because the design decisions (type of study, sampling, data analysis) were not robust did not lend rigor to the conclusions the author could reach. With regard to analysis, focus on the degree to which the analysis strategy as a whole permitted the author to develop new knowledge about the topic, the phenomenon, or the theory. Again, remember that negative findings not getting the anticipated result is NOT a bad thing. 2. Internal Validity Producing reliable conclusions 2A. Sampling and Internal Validity What is your assessment of the confidence (internal validity) that one can have in the conclusions reached by the author based on the sampling decisions? Show that you have a good understanding the relationships between the research question, the author s hypotheses or propositions and the sampling procedures employed. Think about how all of these together, as a Assignment 3, FYC 6800 Page 5

6 system, affect the confidence that we can have in the conclusions (or claims) that the author makes. A good approach to answering this question is to identify both the strengths and weaknesses in the study. For example, use of multiple comparison groups greatly improves ability to reach conclusions about causal relationships. Think about the other features of the design. Remember Gorard s comments about the studies where people conclude that some group like African-American youth perform more poorly than others. Yet, the study involved no direct comparison of African-American and other youth, did nothing to control for things like poverty or educational level of parents or exposure to positive role models in school. This conclusion is not warranted and cannot be generalized because the study never provided any evidence at all about comparative performance it was a single group crosssectional design. Claims are warranted when the design of the study is adequate to answer the question, the procedures used in sampling and data analysis are rigorous, and the conclusions are not overdrawn. This is what you are judging here. Avoid the pitfall of restating the author s question or objectives and therefore holding him/her accountable for conclusions that s/he never intended to make. Also remember that virtually every research study runs into unforeseen problems (like non-response) and that researchers face many obstacles in getting the perfect sample. It is not easy. One thing that researchers do is to list the limitations to their work. I, personally, prefer a study where modest claims are made and the researcher clearly delimits what s/he accomplished modest conclusions. Please think carefully and make sure you do not insert comparisons that the author never claimed or wanted to make. Also consider carefully the ramifications of criticisms of samples based purely on sociodemographic deviance from some general population. As I have said many times, I have rarely ever wanted a general population sample. Make extensive use of the literature as you answer this question. Consult, cite and reference the research design literature. Go beyond the required readings. 2B. Analytic Strategy and Internal Validity Assess the degree to which one can have confidence (internal validity) that one can have in the CONCLUSIONS reached by the author based on the analytic strategy and execution. Show that you have a good understanding the relationships between the research question, the author s hypotheses or propositions and the analytic strategy and procedures employed. Think about how all of these together, as a system, affect the confidence that we can have in the conclusions (or claims) that the author makes. A good approach to answering this question is to identify both the strengths and weaknesses in the strategy and execution. Start with the design. Does the analytic strategy fit the design? For example, any study with comparison groups as a feature of the design should use an analytic strategy that includes comparing the two groups in some way usually in terms of the outcome variables or concepts. Look at the specific steps in analysis. For example, qualitative data analysis typically involves four or five steps going from original level 1 coding (often hundreds of codes) to identifying themes, to creating models and finally to comparing models for two or more comparison groups. Does the author describe each step? Statistical analyses are much simpler and authors usually assume you understand what the procedure is and does and what the requirements for the procedure (sample size, distribution of the sample, level of data). They may often just literally mention the name of the procedure (stepwise regression or hierarchical regression) or not even state the name because you should be able to identify the test by looking at the output table (the results). They certainly are not going to provide a primer on statistical analysis. They may not state the specific statistical hypotheses. Again, you as a reader need to be able to identify the hypothesis whether or not the author states it. Each analysis provides a specific kind of statistical test. For example, a t-test compares the mean scores for two or more different groups. If you see a t-test, you should understand that it tests for differences between the comparison groups included in the study. Use Frey in your assessment of statistical procedures. Otherwise, you will not be able to answer questions about internal and external validity with regard to statistical analytic strategies well. Claims are warranted when the design of the study is adequate to answer the question, the procedures used in sampling and data analysis are rigorous, and the conclusions are not overdrawn. This is what you are judging here. Avoid the pitfall of Assignment 3, FYC 6800 Page 6

7 restating the author s question or objectives and therefore holding him/her accountable for conclusions that s/he never intended to make. Make extensive use of the data analysis literature as you answer this question. Consult, cite and reference the research design literature. Go beyond the required readings. 3. External validity Generalizing the conclusions 3A. Sampling and External Validity Focus on determining which, if any, of the conclusions the author can generalize in the way that s/he wanted to generalize them based on sampling decisions. Do not go beyond what the author claims. This is the author s study, not yours. Stick with analyzing the degree to which the author s sampling decisions were adequate for what s/he wanted to accomplish. Distinguish between theoretical and statistical generalization. An author may want to generalize theoretically, for example, but be relatively uninterested in statistical generalization or vice versa. Consider all aspects of the study as you answer this question. For example, even though we rarely get the perfect sample, there ways to offset the impact of a less than perfect sample like carefully defining the theoretical population to reduce inherent variance that would have to be taken into account in a sample of the general population of some city, state or nation. Refer to the research question as you think about this. Here is an example of one aspect of research design and the features you would have to consider in assessing external validity from the perspective of the adequacy of the sampling procedures and the sample itself. Remember that there are some instances in which the ability to generalize will depend greatly on having a probability sample. Do you think the researcher needed a probability sample? If the researcher needed and/or tried to get a probability sample, did the sample meet all requirements for a true probability sample? If you think the sample failed to meet all requirements, what aspects of the sampling procedure do you think violated the requirements of the specific procedures used? Sometimes, even though the author would like or needs a probability sample, it is not possible to get a probability sample and researchers use other kinds of samples, especially referral, quota and volunteer samples. Remember that there is no general rule or gold standard for sampling. If the researcher did use a non-probability sample even knowing that a probability sample would be better do you think s/he made a good choice? One way to think about this is to ask yourself if the sample is representative enough to achieve the kind of generalization the author wants to make. Assume the author is not interested in statistical generalization of any sort (inferential or descriptive). A probability sample may not be necessary or useful given the research question and planned generalization. However, it is still not OK to take any old haphazard (or convenience as we like to say) sample. You still need a representative sample just not statistically representative. Avoid the many blanket errors made in talking about samples: small n samples are inherently insufficient, only random samples are valuable, purposive sampling is always bad, a bigger sample is always better, etc. Think before you generalize about sampling. 3B. Analytic Strategy and External Validity Focus on determining which, if any, of the conclusions the author can generalize in the way that s/he wanted to generalize them based on the analytic strategy used. Do not go beyond what the author claims. This is the author s study, not yours. Stick with analyzing the degree to which the author s analytic strategy and procedures were adequate to meet his/her objectives. Distinguish between theoretical and statistical generalization. An author may want to generalize theoretically, for example, but be relatively uninterested in statistical generalization or vice versa. Consider all aspects of the study as you answer this question. Refer to the research question as you think about this. Here is an example of one aspect of research design and the features you would have to consider in assessing external validity from the perspective of the adequacy of the analytic strategy and procedures. An Assignment 3, FYC 6800 Page 7

8 author conducted in-depth interviews with parents of children participating in high school competitive sports. You have to understand what the author wanted to generalize. Did s/he want to generalize mostly about the topic and setting (specific population, specific group of people, specific sport)? Did s/he want to generalize about what we know and understand about parental involvement in sports? Or did s/he want to add to theory by adding components of the critical race theory to family systems theory to better explain the role of sport involvement on youth development? Were the analytic strategy and procedures good enough to achieve the kind of generalization the author wants to make? You are the reader. If you do not understand the strategy or the procedures generally do not know what a student t-test is and what the requirements are for level of data and sample size, look it up. That is your responsibility. Avoid the many blanket errors made in talking about analysis: failure to find a significant difference means the study was bad, the author couldn t make recommendations for practice, the analysis was qualitative and therefore bad. Think before you generalize about analysis. Please see the final (next) table in this document for a detailed set of criteria used to determine your score for this assignment. You need to examine the detailed table to understand how to perform well on an assignment. Do that before you try to complete the assignment and to save yourself time and effort. This is one of those cases where more detail SAVES you time. I assess your submission for each of the two articles separately and then use the average of the two scores for the grade on this assignment. Assessment Criteria Flow Chart Boxes 1-3 and Box 8 Stated and interpreted the researcher s question and intended contributions to the body of knowledge correctly Was able to distinguish between topical, explanatory and theoretical objectives Correctly identified the design used and was able to determine whether the design used comparison groups Identified the theoretical constructs and linkages between them used in the study Did not confuse theoretical constructs with variables specific to the study Distinguished between the theoretical or research hypotheses and the statistical hypotheses (if used) Defined the comparison groups if any were incorporated into the design Distinguished between outcome or dependent variables and predictor or independent variables that represent constructs in the theory and was able to associate the variables with the appropriate construct Was able to distinguish between a descriptive variable (typically demographics) and theory-based variables Identified the topical, explanatory and theoretical conclusions reached and did not confuse them with the results of the study Flow Chart - Box 4 Was able to distinguish between the theoretical and accessible population Distinguished between the accessible population and the sampling frame Explained how sample size was determined in your own words, based on the information in the article (avoid making assumptions) Possible Points Your Points LIST A Your Points LIST B Assignment 3, FYC 6800 Page 8

9 Distinguished between screening criteria based on the traits of importance to the research (based on the author s intended contributions to the body of knowledge and comparison groups of interest in the study) and routine reports of demographic traits Identified the specific type of sample Distinguished between response rate and using screening criteria to delimit the sample and was able to describe the replacement procedures used, if any Flow Chart Boxes 5-7 Distinguished between the strategy and the individual components of analysis and was able to identify and describe the components in data analysis Provided enough detail to show thorough understanding for example, did not just say statistical analysis, but rather identified the specific procedures used, listed every hypothesis represented by statistical tests Distinguished between the theoretical or research hypotheses, statistical hypotheses (if used), and the propositions (hypothetical statements, also sometimes called anticipated outcomes) typical in qualitative data analysis Even if the authors did not state statistical hypotheses, was able to identify and list them based on tables and other presentations of statistically derived results Even if the authors did not state their propositions or anticipated outcomes, was able to identify and state them at least in a general sense Did not confuse reports of demographic traits (description) and statistical analysis, comparison groups, or qualitative results Distinguished between planned analyses (t-test for two comparison groups) and ad hoc or unplanned analyses (testing for differences between age groups when no samples based on age were taken) Explained each step in a qualitative data analysis strategy from data entry to the final steps of generating conclusions, including both planned and ad hoc procedures employed Discussion Questions -- Sampling Was able to explain why (or why not) the sampling approach was appropriate for answering the research question based on the nature or type of questions the authors posed Identified and discussed specific aspects of the sampling approach and procedures that strengthened or weakened internal validity, external validity and explanatory power Demonstrated understanding of the differences between theoretical and statistical generalization and was able to assess the author s sampling strategy and procedures based on these key concepts without misrepresenting the author s intentions regarding generalization Discussed the choice of accessible population from a design perspective Discussed how the author used (or did not) use screening or purposive sampling, explicitly discussing the degree to which the author used these techniques to improve the quality of the sample based on the Assignment 3, FYC 6800 Page 9

10 nature of the research question, the author s objectives and the design used in the study Assessed the degree to which the sample is representative of the theoretical population, showing a thorough understanding of how the decisions made at each of the five steps in the sampling process affect representation of the theoretical population Used sampling terminology correctly (for example, not using the word survey to mean a random sample) even if the authors used terms incorrectly Assessed the author s approach to sample size determination whether for qualitative or statistical analysis drawing explicitly on the concepts of Malterud in both cases Provided an analysis of the potential effects, if any, of low response rate and replacement procedures on the quality of the final sample from a design perspective Discussion Questions: Analysis Was able to explain why (or why not) the analytic strategy was appropriate for answering the research question based on the nature or type of questions the authors posed Identified and discussed specific aspects of the analytic approach and procedures that strengthened or weakened internal validity, external validity and explanatory power Demonstrated understanding of the requirements for statistical procedures and could assess how appropriate the procedures used were, paying attention to things like failure to meet the assumptions of a procedure Demonstrated understanding of the different approaches to qualitative analysis and could distinguish between them Provided a sophisticated analysis of the rigor in qualitative analytic approaches, drawing on Malterud, but also on other materials like the reading about reliability and validity in qualitative research (week 4) Assessed the adequacy of the author s procedures used to eliminate, account for, or test alternative explanations other than the proposed (theoretical, hypothesized) explanation in the study and gave examples Distinguished between the results of this study and the overall conclusions reached and was able to identify both weaknesses and strengths that affected the author s ability to achieve his/her research objectives Research Design Literature Focus on Sampling Used extensive materials about research design to develop your responses to the discussion questions including materials about the types of designs and about sampling Includes materials other than the required readings Consistently explained how you used the information in each resource to reach conclusions Cited all materials you use in the responses to the discussion questions in APA format Included full references for all materials consulted When appropriate, cited materials with opposing or conflicting perspectives and explained which perspective you employed in your Assignment 3, FYC 6800 Page 10

11 responses and why you chose those perspectives Based your responses on a critical realist perspective of scientific knowledge and research Research Design Literature Focus on Analysis 50 Used extensive materials about research design to develop your responses to the discussion questions including materials about the types of designs and about analysis Includes materials other than the required readings Consistently explained how you used the information in each resource to reach conclusions Cited all materials you use in the responses to the discussion questions in APA format Included full references for all materials consulted When appropriate, cited materials with opposing or conflicting perspectives and explained which perspective you employed in your responses and why you chose those perspectives Based your responses on a critical realist perspective of scientific knowledge and research Total 300 Average Score Recorded in Grade Book Assignment 3, FYC 6800 Page 11

12 Performance Standards for Sampling Correctly identified all components and accurately described what the author(s) did, even components that were unclear or erroneously stated in the article Correctly stated and interpreted the researcher s intent and question Correctly distinguished between the theoretical or research hypotheses and the statistical hypotheses (if used) Correctly identified the components in the sampling procedure, the implementation of the study, and the analysis of the information (data) collected Correctly distinguished between results and conclusions and stated each accurately Assignment 3, FYC 6800 Page 12 Excellent Satisfactory Needs Improvement Identify & Describe the Components in the Article (Mostly Based on the Flow Chart) Identified most components correctly and only occasionally distorted or misunderstood what the author(s) did not explain unclear or confusing components well Correctly stated but failed to interpret the researcher s intent and question Identified some of the differences between the theoretical or research hypotheses and the statistical hypotheses (if used) Correctly identified major components in the sampling procedure, the implementation of the study, and the analysis of the information (data) collected, but lacked detail Did not fully distinguish between results and conclusions and tended to misstate them Clearly explained the degree to which this specific design depended on an intervention or external event (a poke), the temporal relationship of cause and effect, and/or comparison groups to warrant claims of causality and make comparisons Showed a sophisticated understanding of the concept of controlling for non-experimental (or non-study) factors in scientific explanation and could give specific examples in the study Correctly distinguished between causality and direct cause and effect Discussed in some detail the adequacy of the author s procedures used to eliminate, account for, or test alternative explanations other than the proposed (theoretical, hypothesized) explanation in the study and used examples Consistently misidentified components or misstated what the author(s) die and failed to explain any but the most straightforward and clear components of the article Stated the researcher s intent and question incorrectly Did not distinguish between the theoretical or research hypotheses and the statistical hypotheses (if used) Correctly identified few components in the sampling procedure, the implementation of the study, and the analysis of the information (data) collected, but lacked detail Did not distinguish between results and conclusions Apply Design Concepts to Assess Internal Validity, External Validity & Explanatory Power of the Conclusions (Mostly Based on Discussion Questions) Explained in broad terms how this general group or type of design uses an intervention or external event (a poke), the temporal relationship of cause and effect, and/or comparison groups to warrant claims of causality and make comparisons Showed an understanding of the concept of controlling for non-experimental (or non-study) factors in scientific explanation but did not give specific examples in the study Correctly distinguished between causality and direct cause and effect Discussed the adequacy of the author s procedures used to eliminate, account for, or test alternative explanations other than the proposed (theoretical, hypothesized) explanation in the study in general terms, with few or no examples Correctly identified & explained the key components of the sampling approach and procedures in detail Explained specifically why (or why not) the sampling approach was appropriate for answering the Correctly identified and explained the broad features of the sampling approach Stated a few specifics and some generalities about why (or why not) the sampling approach was Limited the discussion of causality to broad generalities about the role of an intervention or external event (a poke), the temporal relationship of cause and effect, and/or comparison groups to warrant claims of causality and make comparisons Could not identify the presence or absence of techniques used to control for non-experimental (or non-study) factors in scientific explanation Confused causality and direct cause and effect Did not analyze the adequacy of the author s procedures used to eliminate, account for, or test alternative explanations other than the proposed (theoretical, hypothesized) explanation in the study in general terms, with few or no examples Did not correctly identify the broad features of the sampling approach Stated generalities about the relationship between sampling approach and research question

13 question Made a reasoned assessment of the degree to which the sample is representative of the theoretical population Assessed the representativeness of the sample based on specific traits or characteristics of this specific sample that could affect the results of this study Identified specific aspects of the sampling approach and procedures that strengthened or weakened internal validity Used specifics and provided examples to show how the strengths and weaknesses of the sampling approach and procedures used affect the degree to which the conclusions can be generalized theoretically and statistically Distinguished correctly between results and conclusions Stated the authors conclusions accurately in your own words Made a fair and reasonable assessment of the responsiveness of the conclusions to the research question Assessed both advantages and disadvantages of the data analysis techniques for the research question posed Explanation of statistical data analyses were accurate and showed that the team understood the results, including providing examples of the different types of results produced The discussion of statistical analyses identified the logic of the relationship between research question, sampling, and data analysis decisions and was specific to this article (not generalities) If qualitative data analysis was used, assessed the rigor of the approach and was able to distinguish between descriptive analysis and analytic or explanatory analysis Correctly identified most or all of the relevant specific aspects of the design that enhance or weaken the internal validity of the conclusions Assignment 3, FYC 6800 Page 13 appropriate for answering the question Identified some relevant considerations with regard to the degree to which the sample is representative of the theoretical population Identified some specific traits of the procedures and sample that could affect the results of this study, but over-relied on generalizations about sampling Identified few specific aspects of the sampling approach and procedures that strengthened or weakened internal validity Explained largely in general terms how sampling approaches and procedures used could affect the degree to which the conclusions can be generalized theoretically and statistically and justified and explained your conclusions Drew on the some relevant key concepts about sampling that we have discussed to explain how decisions about sampling affected the adequacy of the sample in terms of the research question posed in the article, but some concepts were misstated or misapplied Some comments were specific to the sampling scheme and context in the article, but some were generalities about sampling Limited discussion largely to the general appropriateness of the data analysis techniques for the research question posed Explanation of statistical data analyses were accurate but lacked detail and use of examples that would demonstrate a thorough understanding The discussion of statistical analyses identified only the overall general logic of the relationship between research question, sampling, and data analysis decisions If qualitative data analysis was used, little assessment of the quality of and rigor of the process was provided with little distinction made between descriptive analysis and analytic or explanatory analysis Overall Consistency, Sophistication and Completeness of Your Analysis Correctly identified some of the specific aspects of the design that enhance or weaken the internal validity of the conclusions reached Misstated factors that could affect the degree to which the sample is representative of the theoretical population Repeated generalizations about how sampling can affect results rather than give specifics relevant to this study Misidentified or failed to identify specific aspects of the sampling approach and procedures that strengthened or weakened internal validity Drew broad, general conclusions not specific or relevant to this study about how the general approach to sampling can affect the degree to which conclusions can be generalized theoretically or statistically and justified and explained your conclusions Explanation of statistical data analyses were inaccurate in several ways and indicated only a broad, basic examination of the process Significant errors about the relationship of data analysis to question were stated Explanation of statistical data analyses were not accurate The discussion of statistical analyses identified incorrectly stated relationships between data analysis, sampling approach and nature of the research question If qualitative data analysis was used, there was no distinction made between descriptive analysis and analytic or explanatory analysis Relied almost completely on generalities about design features that strengthen or weaken internal validity and design in your discussion of internal

14 reached In each case, explained in your own words the reasons why you believe the specific design features you identified strengthened or weakened internal validity Correctly identified the most important specific features of the design that contributed to the explanatory power this contribution (study) makes to the body of knowledge Considered all three components of the body of knowledge in your assessment of the way design decisions were used to enhance explanatory power Formulated a well-balanced (not super-critical, not anything is fine ) assessment of the quality of the author s research question based on your considerations in Q7-9 Misidentified some specific design features and/or over-relied or focused on generalities about internal validity rather than specific components of this study Correctly identified overall features of the design that contributed directly to the explanatory power this contribution (study) makes to the body of knowledge Considered some of the components of the body of knowledge in your assessment of the way design decisions were used to enhance explanatory power Formulated a well-balanced(not super-critical, not anything is fine ) assessment of the quality of the author s research question, but did not provide evidence that your assessment grew out of your considerations in Q7-9 validity Did not offer explanations that were specific to the actual features of the design in your study Formulated an unrealistic (probably either supercritical, or anything is fine ) assessment of the quality of the author s research question Did not justify that your assessment grew out of your consideration of internal validity, external validity & explanatory power Responded to all aspects of this assignment in your own words, even the complex components Relied little on direct citations or paraphrased repetition of what the authors say Consulted and referenced extensive materials about research design in your responses, especially materials about sampling, design choice, and analysis, including materials other than the required readings Cited all references in the body of the document Consistently explained how you used the information in each resource to reach conclusions When appropriate, cited materials with opposing or conflicting perspectives and explained which perspective was used and why Other Responded to many aspects of this assignment in your own words, but had difficulty expressing or explaining more complex ideas in your own words Tended to rely on direct citations or paraphrased repetition of what the authors say Consulted and referenced some materials about research design in your responses, especially materials about sampling, design choice, and analysis, including materials other than the required readings Cited most, but not all, of the references in the body of the document Sometimes explained how you used the information in each resource to reach conclusions Rarely cited materials with opposing or conflicting perspectives Consistently relied upon direct quotes and paraphrases in your responses Consulted and referenced few materials about research design in your responses, especially materials about sampling, design choice, and analysis, and included very few materials other than the required readings Failed to cite several of the references in the body of the document Did not explain how you used the information in each resource to reach conclusions Never cited materials with opposing or conflicting perspectives Often seems to throw in citations or references not directly relevant to the discussion Assignment 3, FYC 6800 Page 14