Author s response to reviews Title: Comprehension of confidence intervals - development and piloting of patient information materials for people with multiple sclerosis: qualitative study and pilot randomised controlled trial. Authors: Anne Rahn (a.rahn@uke.de) Imke Backhus (i.backhus@uke.de) Franz Fuest (ranz_fuest@web.de) Karin Riemann-Lorenz (karin.riemann-lorenz@uksh.de) Sascha Köpke (sascha.koepke@uksh.de) Adrianus van de Roemer (adrian.vanderoemer@idm-didaktik.com) Ingrid Mühlhauser (Ingrid_Muehlhauser@uni-hamburg.de) Christoph Heesen (heesen@uke.de) Version: 1 Date: 30 Jun 2016 Author s response to reviews: Dear Ms Sacchi and dear reviewers, Thank you very much for giving us the opportunity to revise our manuscript according to the comments of the reviewer. We would like to take this opportunity thanking the reviewers for the constructive comments, which helped to improve our manuscript. We answered them point by point in the following. Reviewer #1: This article is well written and interesting, but verbiose. There are parts where the same content could be communicated with much fewer words, for example, where the qualitative interview results are described. The level of detail in these descriptions in unjustified given that the descriptions are not further analysed in the discussion or conclusion sections. Furthermore, some comment needs to be made about the generalisability of the findings given factors reported in the statistical analysis. The fact that the findings were statistically significant is not further discussed in terms of practical significance; does a mean question response increase of 1 point, although statistically significant, make any practical impact on patients' ability to interpret and utilise information such as confidence intervals? This is a key factor that this paper should
discuss, given you are going onto a broader RCT, applicability to other health conditions and the potential that resources are to be utilised in producing the kind of PI that has been tested here. Reply: We shortened the article, especially in the qualitative results section. In addition, we discussed the generalisability of the findings as suggested and addressed the relevance of a change of 1 point on the outcome scale. Changes: With a mean difference of one question between groups clinical and practical relevance is an open question. Nevertheless, with more than two thirds of the questionnaire answered correctly by the IG it could be assumed that this kind of information on treatment options is understandable for PwMS. However, results need to be confirmed in a larger sample. Further, other presentation formats as for example videos might be a more attractive format for the user to receive information on CI than written information. Lines 33-35: meaning could be put in a simpler manner by editing the structure of this sentence. I would suggest splitting the sentence so reference is made to developing materials and piloting questionnaires. Reply: We followed the suggestion and changed as follows: We aimed to develop and pilot-test different written patient information materials explaining confidence intervals in people with relapsing-remitting multiple sclerosis. Further, a questionnaire on comprehension of confidence intervals was developed and piloted. Line 52: what is 'mostly' (in terms of a number out of 16)? Reply: We changed the wording of the sentence which now reads: Feasibility of the patient information versions was tested with 16 people with multiple sclerosis. Lines 59-61: it is information which is well understood, as presented in this intervention. Reply: We changed the wording of the sentence. Pilot randomised controlled trial results indicate that the patient information intervention is well understood and that knowledge gain on confidence intervals can be assessed with a set of six questions. Line 85: I suggest replacing 'not possible' with 'impeded', or something similar. Reply: We replaced not possible as suggested with impeded. Section 2: methodology is well written and clear. Could have been said in fewer words. Reply: We shortened the content were possible. Line 158: convenience, not convenient. We corrected the word.
Line:174-175: was the questionnaire administered immediately after they read the PI? Reply: Yes, the questionnaire is administered directly after the intervention. We included the information: Directly after the intervention, they were asked to fill in the multiple-choice questionnaire. Line 234: why were the PIs shown in this order? We added: The order was chosen to give the PwMS with the short version a little break between the long and more complex versions. Line 274: remove the word 'mostly' Reply: We removed the word. Lines 285-296: how many participants answered which questions correctly is best summarised in a table. Reply: We added two tables to display the content in an additional file. Section 3.3: best presented in a tabular format, present or comment on response rates. Reply: We added a study flow chart (please see figure 3). Lines 395-402: another method could be randomising the order in which participants were exposed to PIs Reply: It was planned that participants only look on one patient information material on confidence intervals. Seeing further patient information was optional after completing of questionnaires. We revised the sentence in the manuscript and added information: Due to the length and dense of information and drop-out rates it is not scheduled that PwMS see more than one PI material. Lines 412-414: comment on how the low response rates might affect generalizability of the study. Comment on how the results in Table 1 (more men in control group, mostly RRMS & SPMS in intervention group) affect the generalizability of the PI tool selected. Reply: We discuss the response rates and the results presented in table one: Being aware that not all people read the newsletter, to us the response rate with 64 replies out of 115 who did login into the survey seemed sufficient for a pilot study and our recruitment target of 60 PwMS was fulfilled. However, a large study with a less biased sample is needed to evaluate the PI on CI. Caused by the small sample, the percentage of females in our pilot trial was imbalanced between the groups. However, we do not believe that this effected study results. Nevertheless, we will
investigate on the impact of sex on the outcomes in the larger study. Other minor and not statistically significant differences should not have impacted study results, but again a larger trial is needed to draw any conclusions about the impact of any demographic factor. Additional file 2: comment on criteria used by FF to discard/include articles. Detail on the checklist mentioned is also be relevant here. Reply: We added information on study selection and on the checklist. Please see additional file 2. Reviewer #2: It is a challenge to study how patients understand confidence intervals with respect to decision making about treatment options for their condition. I must admit, reading the manuscript was not easy because of the technical jargon and ample use of abbreviations despite the list of 16 (sic) of abbreviations. The study raises a number of questions. First, it assumes that patients in one way or another are rational thinkers, and believe that understanding of a statistical measure helps them to make assessment about the progress of their own conditions. The fact that well-educated patients have been recruited for the qualitative study reinforces this idea (such a sample is called a "convenience sample", page 6). But statistical information does not say much about the of the individual's response to a treatment. So it would be appropriate if the authors would carefully examine how PI (to use one of the abbreviations in the manuscript) impacts the patient's perspective on his or her disease (and treatment options do not cure MS, but only help to manage symptoms, and live expectancy is still reduced). As the authors write addressing uncertainties can help patients to cope with their condition, but the described study is not convincing about the role that is attached to knowing a CI. The issues are a limited sample size, and an apparent in selecting respondents. Otherwise, the study appears to done according to the book; the statistical analysis appears to plausible. However the emphasis in the manuscript is on the methodology, not on the underlying issues of patient information, shared decision making about treatments options and how patients deal with a chronic disease that influences them for the rest of their lives. I would like to see more of the foundational issues and how the study results should interpreted in this context, and not about some 'technical' outcomes how the respondents liked or not liked information about CI. Reply: We thank the reviewer for raising the issue of the relevance of communicating risk rates. And we totally agree that communication of confidence intervals only can be a clue to better health management. This patient information only aims to explain confidence intervals and is not expected to have an impact on the patient s disease perspective. But as described in the manuscript (page 8 and 15) this information will be embedded in an information platform on multiple sclerosis and ideally in a shared decision making approach (Rahn 2015). It is well known that people with multiple sclerosis want to participate in decision making (shared decision making) or even decide autonomous (Heesen 2004, Solari 2013). In addition, people with multiple sclerosis appreciate extensive information (Kasper 2006, Kasper 2008) and are able to do absolute risk calculations (Kasper 2006). Following the principles of evidence based medicine and shared decision making (Bunge 2010, Charles 1997), informed decision making on therapy options is not possible without information on treatment options in numbers.
Therefore, displaying the absolute risk reduction with the according confidence intervals is one way to present treatment effects of medications with the uncertainties. In fact we did not use a convenience sample but a purposeful sample for the pilot-test phase aiming to mirror different education levels and different disease durations. We now tried to better communicate the results concerning educational level of our qualitative sample: In total six of 13 PwMS received 12 years of education and thereof access to higher education Germany. As this is a pilot study we have evaluated the patient information in a limited sample. This issue has been better addressed in the discussion: Being aware that not all people read the newsletter, to us the response rate with 64 replies out of 115 who did login into the survey seemed sufficient for a pilot study and our recruitment target of 60 PwMS was fulfilled. However, a large study with a less biased sample is needed to evaluate the PI on CI. We reduced the number of abbreviations. Reviewer #3: The presented work Aimed to Develop and pilot-test different written patient information materials explaining confidence intervals and a questionnaire on comprehension of confidence intervals in people with relapsing-remitting multiple sclerosis. The article is well written and the idea is quite interesting, but some issues are important for a review. 1. I confess that I was confused about the number of people who were studied for the pilot randomised controlled trial. In Lines 52 and 53: "For the pilot randomised controlled trial, 64 people with multiple sclerosis were randomised (intervention group: n=38; control group: n=28)." Here 38+28= 66. In Line 151 to 154: "A sample size of 60 people was chosen, assuming that this would allow gaining sufficient information for the planned evaluation of the questionnaire as well as the PI in a larger sample. It was not aimed to reach a statistical significant difference between the two groups, yet to use the results for the sample size calculation of the main RCT." Finally in the line 308 and 309 the authors wirte: Therefore, 64 PwMS were included (36 IG/ 28 CG). Reply: We thank the reviewer for picking up these inconsistencies. Unfortunately there was in mistake in the abstract, which has been corrected. We aimed to include at least 60 people for the pilot and resulted in 64 people. We clarified the lines 151 to 154: A sample size of around 60 people was chosen, assuming that this would allow gaining sufficient information for the planned evaluation of the questionnaire and the PI in a larger sample. 2. In Line 118: "A systematic literature search" that is different of a Systematic Review, as pointed in figure 1. Reply: We changed the wording in figure 1 to systematic literature search.
3. In the line 339 "The difficulty of the six items ranged between 0.53 and 0.94 in the IG" The correct is "The difficulty of the six items ranged between 0.43 and 0.94 in the IG Reply: The number in line 339 has been corrected. 4. The line statement 332-334 is not at all correct, "PWMs in the IG Showed Significantly higher knowledge on CI" should be mitigated because a mean 4.8 (SD 1.3) and 3.8 (SD 1.2) which was statistically are not "Significantly higher" Reply: We added this issue to the discussion and changed the sentence: PwMS in the IG answered 4.8 (mean, SD 1.3) of six questions correctly, while PwMS in the CG answered 3.8 (SD 1.2) questions correctly (mean difference 1.1 (95% CI 0.42-1.69), p=0.002, two-tailed t- test). Discussion: With a mean difference of one question between groups clinical and practical relevance is an open question. Nevertheless, with more than two thirds of the questionnaire answered correctly by the IG it could be assumed that this kind of information on treatment options is understandable for PwMS. However, results need to be confirmed in a larger sample. Further, other presentation formats as for example videos might be a more attractive format for the user to receive information on CI than written information. References Bunge M, Mühlhauser I, Steckelberg A. What constitutes evidence-based patient information? Overview of discussed criteria. Patient Educ Couns. 2010;78:316 28. Charles C, Gafni A, Whelan T. Shared decision-making in the medical encounter: what does it mean? (Or it takes at least two to tango). Soc Sci Med. 1997;44:681 92. Heesen C, Kasper J, Segal J, Köpke S, Mühlhauser I. Decisional role preferences, risk knowledge and information interests in patients with multiple sclerosis. Mult Scler. 2004;10:643 50. Kasper J, Köpke S, Mühlhauser I, Heesen C. Evidence-based patient information about treatment of multiple sclerosis. A phase one study on comprehension and emotional responses. Patient Educ Couns. 2006; 62:56-63. Kasper J, Köpke S, Mühlhauser I, Nubling M, Heesen C. Informed shared decision making about immunotherapy for patients with multiple sclerosis (ISDIMS): a randomized controlled trial. Eur J Neurol. 2008;15:1345 52.
Rahn A, Köpke S, Kasper J, Vettorazzi E, Mühlhauser I, Heesen C. Evaluator-blinded trial evaluating nurse-led immunotherapy DEcision Coaching In persons with relapsing-remitting Multiple Sclerosis (DECIMS) and accompanying process evaluation: study protocol for a cluster randomised controlled trial. Trials. 2015;16:106. Solari A, Giordano A, Kasper J, Drulovic J, van Nunen A, Vahter L, et al. Role preferences of people with multiple sclerosis: image-revised, computerized self-administered version of the control preference scale. PLoS One. 2013;8:e66127.