Statistical interpretation of studies among doctors and medical students AUTHORS: AZEVEDO André; CARDOSO Ana; FIGUEIREDO Carlos; GONÇALVES Francisco; GONÇALVES Raquel; MOITA João; MONTEIRO Renata; NOGUEIRA Miguel; PINTO Sérgio; RODRIGUES Daniel; RODRIGUES Marta; TEIXEIRA Ana (01fmup@gmail.com) ADVISORS: Ricardo Santos and Alfredo Castro Class number 1 ABSTRACT: Biostatistical knowledge is of the uttermost importance in the interpretation of scientific literature. Nowadays, with the growing emphasis placed upon Evidence-Based Medicine, it becomes necessary for physicians and medical students to have a minimum grasp of biostatistical concepts and principles in order to understand and critically appraise the literature. Several studies have concluded that medical personnel still has a lack of preparation in this matter. Our study aims to evaluate doctors and medical students biostatistical knowledge. We applied a questionnaire to a randomly selected sample of 208 physicians and medical students from an University Hospital (Hospital São João). Data relative to the importance participants give to biostatistics, their confidence level regarding statistical interpretation and their actual knowledge in this area was collected. It was concluded that participants had a lack of the necessary knowledge to adequately interpret and evaluate results in medical research articles. KEY-WORDS: Biostatistics: Data interpretation, Statistical; Evidence-Based Medicine; Knowledge; Medical Students; Physicians; Questionnaire.
Title Statistical interpretation of studies among doctors and medical students Introduction Since the dawn of its existence, mankind has been plagued and haunted by malady, illness, disease and injury, brought on by time, the world that surrounded him, or simple misfortune. The consequences of these problems were severe, causing death or disability, and were an heavy burden to man and his society. Thus, Medicine or rather, its early ancestors came to be. Man sought solutions to these problems, cures for diseases, remedies for injuries. These first steps were, however, riddled with primitive thoughts and superstitions, and the role of the healer was a magical one, with little weight being given to actually testing and studying the treatments proposed. With the passage of time, Medicine became to analytically study and observe the phenomenon of disease with evidence being treated progressively with greater reverence than it had been. The advent of the scientific method propelled the systematic study of the organism, of the physiological and pathological mechanisms and studies capable of reaching to verifiable evidence appeared. Eventually, Medicine took on the shape it has today, extremely oriented by the literature that studies and describes the treatments, diseases and injuries that doctors deal with on a day to day basis, so that a more carefully thought-out approach may be taken toward the patient, supported by proven evidence. Evidence Based Medicine, or EBM, was established. It becomes self-evident that in a field with such a rapid evolution, like Medicine, doctors need to stay up to date with the literature that drives clinical practice. Moreover, a deep understanding of the literature is of the uttermost importance when one follows an academic career. However, scientific literature is not without its faults and subtleties and it not readily accessible should one lack core concepts in Biostatistics. Biostatistics is an applied science for biology, medical and health sciences and it plays an increasingly important role in the production of medical knowledge and clinical practice [1]. In consequence, doctors must have a minimum grasp of biostatistic concepts should they hope to adequately access evidence contained in the literature. However, there is a wide variety of statistical treatment said information receives [2]. Also, some studies concluded that medical students do not believe that biostatistics education provides an additional benefit for their occupational career and that this pattern is strengthened by the lack of interest in an academic career [1]. Furthermore, statistics is a very different subject from those on which medical students spend most of their time. [3] Accounting for these problems, many researchers have asked the question of whether or not
physicians and medical students do possess said knowledge. Medical residents in Connecticut were asked a series of twenty questions related to statistical interpretation, of which they were only able to correctly reply to eight, in average, despite stating a belief in the importance of said interpretation [2], indicating a lack of biostatistics knowledge. Cahan A et al reached conclusions that pointed towards the same direction as the former for the Israeli nation, specifically [4]. Yet another study found a low level of knowledge of basic principles of research methods and data analysis among doctors [5]. There is a seeming uniformity to the conclusions reached by these researchers doctors (and residents, where adequate) lack the necessary base skills in statistics to be able to comprehend the true depth of scientific literature. Other conclusions were also observed in some of these articles, regarding correlations between differences in the statistical knowledge of certain groups of individuals and the demographics of said groups. Given that no other literature was found regarding statistical knowledge of Portuguese physicians (either in general or in the Oporto region), our work was designed to analyze this parameter. Biostatistical knowledge and importance given to this subject by an University Hospital's physicians and medical students, as well as their confidence in evaluating statistical information will be evaluated by applying a questionnaire, always with the general amelioration of medical healthcare in mind. With the collected information, we will investigate the possible existence of correlations between academic qualifications and biostatistical knowledge of participants. Research Question: Do doctors and medical students from Portugal have the necessary knowledge to adequately interpret biostatistical information in scientific articles? Participants and Methods Study participants Our target population consists of a Portuguese University Hospital physicians and medical students. In order to study the defined population certain sampling methods were adopted: 1 class of 5 th year of MIMED ( Mestrado Integrado em Medicina at FMUP) was randomly selected to apply the pilot survey-class: class number 8. 4 classes of the 5 th year of MIMED ( Mestrado Integrado em Medicina at FMUP) were randomly selected out of a total of 7, using a random number generator classes: 1,5,6,7. 7 departments out of 43 at HSJ ( Hospital de São João ) were randomly selected; only 5 colaborated: Patologia Clínica, Pneumologia, Oftalmologia, Neonatologio, Estomatologia.
Inclusion and exclusion criteria Students: Participants must belong to Hospital's students community (excludes individuals studying at the hospital under mobility projects); Participants must be studying at the Hospital at the moment (excludes the students studying abroad under mobility projects http://erasmusfmup.blogspot.com); Participant must be matriculated at the 5 th year; Participants mustn't have participated in the pilot survey. Physicians: Participants must integrate the medical population of the Hospital. Study design This study is a cross sectional study. The unit of analysis is the individual. According to the sampling method, this study can be classified as a randomized trial. Data collection methods We applied a questionnaire (Appendix) based on the one used in Medicine Residents Understanding of the Biostatistics and Results in the Medical Literature [2]. In this study, the statistical methods more frequently represented in 239 original articles published from January to March of 2005 in each issue of 6 general medical journals (American Journal of Medicine, Annals of Internal Medicine, BMJ, JAMA, Lancet, and New England Journal of Medicine) were determined. The questions were built accordantly. The questionnaire, that was modified after applying the pilot survey, is divided in four parts. The first part meant to evaluate the participants attitude towards statistics. The second is about the participants confidence in evaluating statistical information. The third, containing 9 questions, meant to evaluate the participants actual knowledge of statistical information and interpretation of study results. And finally, the fourth, meant to evaluate parameters such as academic degree, years since the graduation, participation in research projects and scientific literature reading habits. There are 3 versions of the questionnaire (A, B and C). The order of the questions present at the third part is switched from version to version. This intends to contribute to asses biostatistical knowledge with more fidelity, as the influence of the order of the questions is attenuated. Variables description Attitudes were rated on a 5-point Likert Scale and confidence questions were rated using a 5- point scale (1 indicates no confidence and 5 indicate complete confidence). Questions that aim to access participants knowledge of statistical methods were classified as correct, incorrect or missing
values. As the questions have equal weight, the third part of the questionnaire will be rated using the percentage of correct answers, as in a score from 0 to 100. Statistical analysis Data (from the questionnaire) was processed using Statistical Package for Social Sciences (SPSS). Several demographic parameters (the fourth group of the questionnaire) were analyzed for students and physicians separately using frequency tables. For the attitude and confidence variables, answers 4 and 5 were considered Agree and Confident, respectively, and 1, 2 and 3 were considered Doesn't agree and Not confident. These variables were also analyzed using frequency tables. Relatively to the third group (biostatistics questions), the mean percentage of correct answers for each question was calculated for students and physicians. Also, the mean of correct answers for the whole third group was compared between those that answered differently to each fourth part question, using T-test for independent samples (results were considered statistically relevant for p < 0,05). This analysis was performed for students and physicians separately. Normality of distribution and homogeneity of variances were confirmed. Tables present in Results were constructed in order to present the results of the analysis. Results Fifty-two students from the 5 th year of MIMED answered the questionnaire. Among this students 61.5% affirmed to have biostatistics training and 46.2% EBM training; 50% claimed to read scientific literature frequently and 30.8% to have had the possibility to work in research projects. Thirty-five doctors from the 5 departments that collaborated answered the questionnaire and among them 71,4% had a degree, 14,3% were masters, and the other 8,6% had completed their doctorate degree. More than half of the physicians had concluded the Licentiate ten or less years ago. Among the doctors group, 54.3% were specialists and this was also the percentage of doctors that claimed to be trained in biostatistics, whereas only 31.4% claimed to have EBM training. More than 85% of the physicians read scientific literature frequently and more than 65% even had the chance of participating into in research projects.
Student (n=52) Doctor (n=35) % % Total 59,8 40,2 Academic Degree Is a specialised physician Years since completing Licenciature Student 100 /// Licenciate - 71,4 Master - 14,3 Doctorate - 8,6 54,3 Less then one 77 0 One to three 78,8 25,7 Four to ten 86,5 25,7 Eleven to twenty 13,5 11,4 More then twenty one 0 31,4 Trained in Biostatistics 61,5 54,3 Trained in EBM (Evidence Based 46,2 31,4 Medicine) Reads Scientific Literature frequently 50 85,7 Has had oportunity to work in research projects 30,8 65,7 Table 1. Characteristics of the Participants Relatively to the biostatistics knowledge test, the questions that had the higher and lower percentage of right answers (table 1) were common to students and doctors. The interpretation of the meaning of p<0.05 had the lowest percentage of right answers (2.9% of the doctors and 9.6% of the students) and the understanding of the purpose of double-blindness had the highest (74.3 % of the doctors and 92.3% of the students). The second and third questions with a lower percentage of right answers were the interpretation of a Kaplan-Meyer Analysis and the interpretation of the concept confidence intervals. Furthermore, there are 7 questions to which more than half of the doctors answered incorrectly and 5 for the students.
Interpreting Results in a Kaplan-Meyer Analysis Identifying a continuous variable Identifying an ordinal variable Identifying a nominative variable Determining which test is most specific Interpreting the meaning of P<0.05 Understanding the purpose of double-blindness Interpreting standard deviation Understanding the definion of bias Recognizing a case and control study Doctors (n=35) Students (n=52) Mean correct (%) Mean Correct (%) 17,1 5,8 60 78,8 40 51,9 51,4 69,2 42,9 67,3 2,9 9,6 74,3 92,3 31,4 46,2 71,4 84,6 48,6 28,8 Interpreting CI 95% and 14,3 11,5 statistical significance Table 2. Percentages of Correct Answers for the Knowledge-Based Questions Among the students, 38.5% were confident to interpret a study result with a statistical method but only 11.5% were confident to evaluate if a statistical procedure was correctly applied in a certain study. 59.6% of the students showed confidence in the interpretation of the P value for a given result. Among the doctors, 82.9% showed lack of confidence in the interpretation of studies results using a statistical method and only 2.9% had confidence in evaluating whether the correct statistical procedure was used in a certain study. Concerning the interpretation of P for a given value, 74.3% of the doctors showed lack of confidence.
Doctors (n=35) %* Students (n=52) %* 1 -"I have a desire to learn more about biostatistics." 85,7 50 2- "I can understand almost all statistical terms encountered in scientific literature." 17,1 34,6 3 -"Because it is easy to lie with statistics, I don't trust it." 17,1 1,9 4 -"I normally use statistics to form opinions or make clinical decisions." 57,1 55,8 5 -Interpreting a studies results with a statistical method 17,1 38,5 6 -Evaluating whether the correct statistical procedure was used in a certain study 2,9 11,5 7- Interpreting the value of P for a given result 25,7 59,6 Table 3. Atittude and confidence results by Doctors and Students' characteristics * percentages of doctors/students that agree with the displayed sentence (1 to 4) or that have confidence in performing the displayed task (5 to 7). Several factors were compared among doctors and students, such as the existence of previous training in biostatistics, the existence of previous training in evidence based medicine and the reading habits as far as medical journals are concerned. The amount of years passed since medical school graduation and the conclusion of the specialization were only taken into consideration for the doctors' group. Although some differences are visible in the values obtained, none of the differences was statistically significant.
Students (n=52) Doctors (n=35) Characteristic Mean correct P Mean correct P (%) value * (%) value * Has had previous training in biostatistics 52,8 0,133 45,5 0,573 Has had previous training in evidence based medicine 48,7 0,645 40,5 0,493 Regularly reads medical journals 49,3 0,875 45,2 0,207 Has previous participated in scientific research work 54,5 45,5 0,462 The amount of years passed since medical school graduation is... <10 47,0 >10 35,3 0,112 Is a specialist 42,6 0,678 Total 49,7 41,3 Table 4. Knowledge Scores by Doctors and Students' characteristics. * Obtained through a T-test compairing the mean score of the populations that answered yes and no to each question or >10 years and <10 years. Discussion As in previous studies [2], we concluded that physicians and students from the Hospital São João, have a lack of knowledge in biostatistics. For instance, more than 97% of the doctors and 90% of the students weren't able to interpret the p-value. More worrying, the percentage of correct answers clearly doesn't match with the confidence revealed in interpreting this biostatistical concept: almost 26% of the doctors and 60% of the students claimed to be able to interpret the p value. We also came to the conclusion that 64,4 % of the inquired had reading habits, half of them lacked confidence in understanding the results of a statistical method used in a research. Doctors have a bigger desire to learn more than students, and the mean percentages of correct answers were predominantly higher in the students' group. It is possibly that doctors understand the need to have a grasp of some biostatistical knowledge, as we concluded that they read more scientific literature than students (almost 86% of the doctors vs. 50% of the students claim to read medical journals frequently). This may also be due to the fact that the inquired students claim more frequently formation of Biostatistics (almost 62% of the students vs. less than 55% for the doctors' group). Study Limitations Our study has some limitations, such as the low answer rate: only 87 out of 208 questionnaires were answered. This caused serious problems in results interpretation, as it may
have introduced an error in the match between our sample's answers and our target population actual knowledge, confidence and attitudes towards biostatistics. Said answering rate was caused by the difficulty in contacting the heads of some departments and some lack of collaboration from students and doctors. Some inquired physicians answered only to the third part of the questionnaire (biostatistics test). If this choice was motivated by the lack of knowledge, there is a clear overestimate of doctors' biostatistical knowledge. Acknowledgements We gratefully thank our advisors Ricardo Santos and Alfredo Castro for their support and help in completing the task we have been assigned. We also direct our acknowledgements to departments Patologia Clínica, Pneumologia, Oftalmologia, Neonatologia and Estomatologia from Hospital de São João, and students of the classes: 1,5,6,7 and 8 of the fifth year of Mestrado Integrado em Medicina, at Faculdade de Medicina da Universidade do Porto, that cooperated with us. And finally, we would like to express gratitude for Prof. Altamiro Pereira's constructive criticism. References [1] -Sahai H, Ojeda MM. Problems and Challenges of Teaching Biostatistics to Medical Students and Professionals. Med Teach. 1999; 21(3):286 288. [2] - Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA : the journal of the American Medical Association. 2007;298(9):1010-22. Epub 2007/09/06. [3] - Altman DG, Bland JM. Improving Doctors Understanding of Statistics. J Roy Stat Soc A Sta. 1991; 154 (2):223-267. [4] - Cahan A, Gilon D, Manor O, Paltiel O. Probabilistic reasoning and clinical decision-making: do doctors overestimate diagnostic probabilities? QJM : monthly journal of the Association of Physicians. 2003;96(10):763-9. Epub 2003/09/23. [5] - Novack L, Jotkowitz A, Knyazer B, Novack V. Evidence-based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgraduate medical journal. 2006;82(974):817-22. Epub 2006/12/07. [6] Ghosh AK, Ghosh K, Erwin PJ. Do medical students and physicians understand probability? QJM : monthly journal of the Association of Physicians. 2004;97(1):53-5. Epub 2004/01/02.
Appendix ID Task Name Start Finish Fev 2012 Mar 2012 Abr 2012 Mai 2012 Jun 2012 19-2 26-2 4-3 11-3 18-3 25-3 1-4 8-4 15-4 22-4 29-4 6-5 13-5 20-5 27-5 3-6 1 Pilot Survey 27-02-2012 23-03-2012 2 Sample selection 23-03-2012 29-03-2012 3 Database development 30-03-2012 13-04-2012 4 Questionnaire delivery 16-04-2012 14-05-2012 5 Data collection 14-05-2012 18-05-2012 6 Data insertion 18-05-2012 25-05-2012 7 Results interpretation 25-05-2012 29-05-2012 8 Final presentation production 25-05-2012 30-05-2012 9 Article redaction 25-05-2012 06-06-2012 Table 1 - Gantt Chart describing a timetable for the proposed research the red numbers indicate the lective interruptions
Survey (version A)