Chapter 5 Analyzing Quantitative Research Literature

Transcription

1 Activity for Chapter 5 Directions: Locate an original report of a quantitative research, preferably on a topic you are reviewing, and answer the following questions. 1. What characteristics of the report you located led you to believe that it is an example of quantitative research? Although Gordon s study focuses on perceptions rather than cause and effect or direct comparisons of data, there are several clues to the qualitative nature of this paper. The first and foremost clue, and an important criterion in my search efforts, was the use of a numerical approach to the data (I searched for key terms such as mean and standard deviation along with the general topical search terms). The researcher gathered data through a survey and analyzed the results. The author was concerned with sample size, methodology, and other elements and included the mean and standard deviation information all data elements. From there, I looked at the format of the paper itself. While the paper doesn t include all the elements described in the text, it does follow the basic format beginning with an abstract followed by description of the conception framework and literature review. The next section was a key indicator identifying it as a quantitative paper; the method section describing the population sample, instrument, data collection and data analysis methodology. Finally, the researcher provided his findings followed by a discussion and recommendation. The appendix included the survey instrument. 2. Is the study experimental or nonexperimental? On what basis did you decide? Non-experimental. This is an example of a descriptive survey research effort. The author makes no attempt to control an independent variable. 3. If the study is experimental, were the participants assigned at random to treatment conditions? If not, how were they assigned? This is a not experimental study. 4. If it is nonexperimental, was the researcher attempting to examine cause-andeffect issues? If yes, did he or she use the causal-comparative method? Explain. No, this is an example of a descriptive research project. The author was not attempting to examine a cause and effect relationship. There is no attempt to reach a conclusion based on a correlation of data elements. In a casual-comparative method research project, the researcher would have established values for an independent variable even though she would not have direct control over it. 5. What type of measures (i.e. instruments) were used? Did the researcher provide enough information about them to allow you to make judgments on their adequacy for use in the research? If yes, do you believe they were adequate in

2 light of the information provided? If no, what types of additional information about the measures should have been reported? Gordon measured the perceptions of the participants through the use of a survey (a Psychometrics Group Instrument). The body of the survey asked the respondents to rate 29 items on a 1-5 Likert-type scale. Gordon included a section detailing the survey instrument which included information relating to the reliability coefficient, content and face validity, the pilot test reliability results, and the use of reverse wording to minimize response set influences. The entire survey instrument was also included in an appendix. I believe that the information presented provides enough information to substantiate the survey as an adequate tool. 6. How did the researcher obtain a sample of participants? Was it at random from a population? If the study is a mailed survey, what was the response rate? The sample was random from a population. The entire population consisted of the members of the American Vocational Education Research Association, AVERA (N- 160). Due to resource constraints, a probability sample was used (a simple random sample). The researcher described the formula he used to reach a 95% confidence level (sample size of 113 was required). Gordon also explained how he utilized the random number generator in Excel to select the study group. The use of a probability sample also left some AVERA members available for the pilot study. Had the researcher surveyed the entire population, the researcher would have had to obtain members of the pilot group by pulling them out of the set or looking outside of the member group for pilot study participants. The response rate was 35%. The author does discuss the limitations of the results based on the response rate. The author warns against generalizing based on the results but stresses that the results still provide valuable information for researchers. I also found it interesting that the Gordon also examined the statistical differences between early and late responders to control non-response error and maintain validity. I intend to explore the citations to learn more about how early and late respondents correlate to non-respondents. 7. Has the researcher described the demographics of the populations in sufficient detail? Explain. The researcher included several demographics in a background section of the survey instrument. For members of the group population, this would probably be sufficient detail. However, no demographics or other background related to the larger AVERA group was included.

3 As a casual reader seeking to evaluate the research itself, I am left with only a partial picture of the group. While the paper is an interesting examination of a small selfselected group s opinion toward statistical issues in quantitative methods of study within the field of vocational education, it does not provide enough information to generalize the results beyond the respondents (probably not even enough to generalize it to the 160 members of AVERA). 8. If the researcher reported statistically significant differences, did they discuss whether they were large differences? In your opinion are the differences large enough to be of practical importance? Explain. The differences are large enough to be of practical importance within the sample group (again, I would not generalize the results beyond the participants). The researcher did report statistically significant differences in several response areas, or in the terms of the author perception clusters. In the narrative accompanying the findings the author used terms such as general agreement or disagreement and slightly disagreed or agreed. Mean and standard deviation was also included in the findings. It is interesting that Gordon s research indicated vocational educators are in agreement that results should be reported as statistically significant, as in the question being asked above, rather than simply significant, believing that the term significant can be misleading. 9. Did the researcher critique their own research by describing its limitations? Yes, in several areas. Beginning in the data collection section, Gordon states that the results should not be generalized to the entire population due to the 35% response rate, citing research asking whether the respondents are still representative of the population. I believe that Gordon saw the small sample size to be a primary limitation of his research. Beyond his direct warnings, throughout the discussion Gordon cites additional research to support the results of his study. In the discussion section, Gordon begins with a statement describing how the group may have been self selected, with only the interested members responding. Gordon cites similar research that yielded similar results to strengthen the credibility of his research. The researcher also raises the question of whether the participants understood the statistical techniques being asked about. Gordon cites other research suggesting that 70% of the participant group needed additional training on emerging statistical techniques.

4 10. Briefly describe any major flaws in the research that you did not cover in your answers. I selected this paper, not because it is directly related to my research topic, but because I wanted to learn more about quantitative research techniques. What better way than studying a quantitative research paper on the perception of statistical significance tests in quantitative research within a group of vocational educators? Since the audience was a group interested in the statistical significance, the author of the paper went to great lengths to make sure that his research fit the expectations of the group. It was a very interesting example. However, in the big picture of things, I m not sure that this research study provided any results of practical importance (though they most certainly have statistical significance!). While the results may mean something to the small group of individuals who participated, the results cannot be generalized beyond the respondents. The author does warn of this, but the natural inclination of the reader is to accept the data presented as a generalization of the entire population. Other than being an excellent example of a descriptive survey based research project, the results are fairly meaningless outside of the context of the participant group. The author also wanders from his primary thesis, the examination of the respondents perceptions of several specific statistical techniques and methodologies. Throughout the paper the author introduces topics connected to the primary focus of the paper only by his personal rabbit trails; i.e. the second class image of vocational education, the entire discussion of the meaningfulness of statistics, discussion of practical significance even when results are not statistically significant, etc. Here s my favorite paragraph, which just sits there not directly tied to the preceding paragraph, the following paragraph, or the study objectives: The etiology of the propensity to conduct statistical significance tests can be traced to two dynamics. The first involves an unrecognized error in logic when consciously trying to be scientific, whereas the second dynamic occurs as a frankly irrational process. These two dynamics undergirding continued emphasis on statistical tests must be understood if reform efforts are to be effective. I kept waiting to find out what the frankly irrational process was. I never found out. After 23 paragraphs of wandering labeled conceptual framework and related literature, the author finally gets to his thesis; which takes two sentences and is not clearly defined anywhere else, including the abstract. The data collection section is well done. Since cover letter narrative can impact the data collected, it would have been helpful to have information related to the cover

5 letter that accompanied the survey. The letter itself could have been included in the appendix. The findings section is also well done, focusing on 29 survey questions related to very specific statistical analysis techniques. The conclusion does, with the exception of a couple of paragraphs near the end, focus on interpreting the data collected. The results themselves do support the objectives of the study by defining the perceptions of the participants relating to specific statistical issues and the appropriateness of statistical significance testing within the respondent group.

6 References Gordon, H. R. D. (2001). American Vocational Education Research Association Members' Perceptions of Statistical Significance Tests and Other Statistical Controversies. Journal of Vocational Education Research. 26(2). pp