Science students' critical examination of scientific information related to socioscientific issues Stein Dankert Kolstø 1, Berit Bungum 2, Terje Kristensen 2, Erik Arnesen 1, Anders Isnes 2, Ketil Mathiassen 2, Idar Mestad 1, Andreas Quale 2, Anne Sissel Vedvik Tonning 1, Marit Ulvik 1 1 University of Bergen, 2 University of Oslo 1
Two agendas 1. Present a study Theoretical framework Method Findings Discussion 2. Discuss and exemplify qualitative analysis of qualitative data 2
Focus for the study Science for all (scientific literacy) Political dimensions of science (STS) Intellectual independence of experts and authorities Socioscientific issues (SSI) Critical assessment of scientific claims (Science for further education - not in focus here!) 3
The study 89 science teacher education students Article on the Internet of their own choice Related to a socioscientific issue To assess the reliability of scientific claims Present their assessment in a short text. 4
Theoretical perspectives Socioscientific issues Include two main questions: 1 Political / ethical Should irradiation of food be legalised or not? 2 Scientific Do irradiated food have lower nutritional value? 5
Theoretical perspectives Constructivist epistemology Fromdisputed claims from the frontier of research To reliable consensual core science Thus degrees of reliability Not necessarily correct or wrong 6
Framework for the analysis Criteria defined Values, norms or ideas appealed to when making a judgment Examples of criteria Consistency Relating findings to existing theories 7
Framework for the analysis Criteria for judging scientific claims Scientific criteria? Contextual criteria? Possible to differentiate? Underderermination of theories by empirical evidence Focused on identifying the kind of criteria used by the students all kinds: focused on content and source! 8
From the literature: Criteria for judging scientific arguments Criteria referring to scientific values Empirical adequacy Internal and external theoretical consistency Contextual factors Competence, reputation, interests, personal qualities, Other? Completeness (publication), level of consensus 9
Research questions What criteria do the science teacher students use in their examinations? What knowledge do they draw upon in their evaluations? Which criteria might be relevant for inclusion in science education? Discussed in the paper 10
Method Data collection 89 science teacher education students Groups of 2 and 3 Search the Internet for articles Socioscientific issue of interest 11
Task Method Write an evaluation Focus in particular on the trustworthiness of science-related claims Twenty-eight articles Between a half and a full written page 12
Analysis Method Analyzed using qualitative methods Constant comparative method Code and retrieve Inter-coder reliability of 69 percent 13
(Back stage: Codes in three levels) 3: Main categories (4 groups of criteria) 2: Categories (the 13 criteria identified) Based on codes constantly compared for similarities and differences Judged to express similar judgments Categories and retrieved codes constantly compared to quotations 1: Codes Condensations of quotations which also included tentative indication of category (label for easy retrieve and inspection) 14
(Back stage: Memos) Possible useful observations Tentative hypothesis Judgments about strategies in the analysis Keep track of process, like new categories Impression from the different texts Definitions of the evolving categories 15
Findings part I Criteria focusing on empirical and theoretical adequacy Instances Groups - Quality of references 8 7 - Consistency of argumentation 22 13 - Face validity of argumentation 19 12 - Compatibility with own scientific knowledge 7 4 In total 56 16
Quality of references "relevant references "prominent scientific journals "trustworthy and respected sources", Gr.14. Because the article [...] makes references to a great many concrete scientific articles, we consider, as a starting point, the article to be trustworthy. Requires knowledge enabling differentiation between different kinds of sources 17
Consistency of argumentation Logical correctness and consistency of an argument Gr.21: The article says nothing about the possibility that increasing number of instances of cancer might have other causes, like for instance more frequent sunbathing. Requires the idea that evidence and logical consistency is needed 18
Face validity of argumentation Focuses on the quality of arguments presented, but does not involve any focus on consistency Gr.18: The factuality of the text seems to be great, due to the numerical statements and evaluations, [...]. "a scientific method, "sounds reasonable" "detailed explanations "professional argumentation" 19
Compatibility with own scientific knowledge Many have put forward this claim as one of the most risky. How stable new genes are in the new organism is difficult to predict. It is proved that naturally occurring bacteria can transfer genetic material from one plant to another. This way you can not make a guarantee saying that a gene installed in one plant to increase its resistance against disease, herbicides or something of the sort can not be transferred to weed. In this way we might attain a super weed that will be difficult to conquer. Moreover,... Presupposes relevant knowledge at a relatively sophisticated level 20
Findings part II Criteria focusing on completeness of information Instances Groups - Completeness of references 23 14 - Completeness of an argument 21 12 - One-sidedness in the presentation 14 10 In total 58 21
Completeness of references Complained that an article wanted references for claims Gr.22: We can not find any direct references to these [claimed research findings], and this is a weakness of this page. [...] We want more links that could make it easier for us to check facts and claims. Relevant for differentiating between documented claims and mere guesses 22
Completeness of an argument Wanting explanations, details, counterarguments or argumentation in general Gr.19: A discharge reduction of dioxin at 99,7% with new cleaning technology is unquestionably very good, but [the environmental organization] does not state any reasons for why 0.027g/MT [gram per megaton] incinerated waste is an acceptable value. Emphasis the importance of requiring disclosure of arguments for further inspection 23
One-sidedness in the presentation Commented explicitly on the lack or presence of counter-arguments Gr.9: The article has a one-sided focus. [The author] writes a great deal about advantages related to the use of nutritional supplement, but mentions little about advantages of [the alternative action of] changing eating habits. 24
Findings part III Criteria focusing on social aspects Instances Groups - Possible underlying interest 23 12 - Personal value-related qualities 3 3 - Author s or expert s competence 27 15 - Level of professional recognition 5 4 - Level of expert agreement 16 11 In total 74 25
Possible underlying interests Make inferences from what they read to what might have caused a possible bias Gr.23: It should not be concealed that the specialized competence that the gene technologists possess can be used as a "smoke screen" in order to secure their own work. Especially in relation to the fact that gene technologists within the industry in general need to get their products and ideas sold, in order to make a living of their expertise. % 26
Possible underlying interests economy and workplace professional prestige and position for experts loyalty to friends Requires knowledge about institutional characteristics 27
Personal value-related qualities Critical attitude towards "both sides" Gr.13: The critical attitude towards both homeopathy and scholastic medicine is one of the things which contribute to our impression of seriousness. Rare use of criteria like integrity, conscientiousness, honesty 28
Author s or cited expert s competence Relevant education and current occupation Gr.24: The article emerges as trustworthy as it is medical practitioner [name] that states this. He is director at the [name] [medical] clinic and therefore he has some credibility. + Place of publication to judge competence 29
Level of professional recognition Gr.25: It is made a reference to Dr. [name] as an acknowledged biochemist, but he has only published works about noni. It therefore emerges as improbable that he really is an acknowledged biochemist, and not a biochemist working on a contract for noni juice. Presupposes knowledge of different experts' prestige in science 30
Level of expert agreement About the standing of the sciencerelated claim Gr.8: According to [the researcher], this whale [Keiko] is not suited for a life outside captivity, and costs very much money. This corresponds to other researchers' utterances and experiences. Requires knowledge about the role of critique, argumentation and consensus in science 31
Findings part IV Criteria focusing on manipulative strategies Instances Groups - Manipulative strategies 6 4 32
Appeals to emotions Whether deliberately manipulating readers Gr.25: The history of the noni fruit is used to show that it is a "natural" and therefore healthy product. They are again appealing to popular feelings and to the popular opinion that everything that is old is good. Used surprisingly infrequent 33
Additional findings Most student-groups did not conclude on reliability Yang (2004): attribute their uncertainty to insufficient information Echoed in this study Rare use of strategies for crosschecking claims Even if they worked online! 34
Summary Number of critical comments identified per student group Number of critical comments identified 16 14 12 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 The 28 student groups Ma Em So Co Ma: Manipulative str. Em: Empirical and th. So: Social aspects Co: Completeness of inf. 35
Discussion Context-dependent findings Additional criteria in other contexts Certain criteria in the literature not used (internal consistency, simplicity, novelty, ) Epistemic independence? Critical examination: Yes! Cross-checking: No! Relevance of criteria used? Both relevant and legitimate 36
Consequences for science education Critical examination of science dimension of issues: Scientific content knowledge is not sufficient? The students draw upon their knowledge of: Methodological norms in science Science content Social processes in science Institutional aspects of science Need to be included in the science curricula? 37
Implications for science teacher education Science teacher students weaknesses Rare use of strategies to cope with incompleteness of information Students evaluations varied considerably in depth and number of criteria used Facione (1990): critical thinking should be taught explicitly Including criteria! Need to bring more science teachers up to a proper level for teaching critical examination 38
Thank you for your attention! The project was supported by a grant from the Norwegian Ministry of Education 39