Author s response to reviews. Title: The development of PubMed search strategies for patient preferences for treatment outcomes.

Transcription

1 Author s response to reviews Title: The development of PubMed search strategies for patient preferences for treatment outcomes Authors: Ralph van Hoorn (Ralph.vanHoorn@radboudumc.nl) Wietske Kievit (Wietske.Kievit@radboudumc.nl) Andrew Booth (a.booth@sheffield.ac.uk) Kati Mozygemba (kati.mozygemba@uni-bremen.de) Pietro Refolo (pietro.refolo@rm.unicatt.it) Dario Sacchini (dsacchini@rm.unicatt.it) Ansgar Gerhardus (ansgar.gerhardus@uni-bremen.de) Gert Jan van der Wilt (gertjan.vanderwilt@radboudumc.nl) Marcia Tummers (marcia.tummers@radboudumc.nl) Version: 1 Date: 04 Jun 2016 Author s response to reviews: Dear editor, dear dr. Pentti Nieminen, Thank you for your valuable comments and those of the reviewers on an earlier version of our manuscript entitled The development of PubMed search strategies for patient preferences for treatment outcomes (Ms. No. BMRM-D ). We have revised our paper considerably in response to those comments and we believe we have adequately addressed all of them. We think this has improved the paper substantially. Changes in de manuscript are marked with track

2 changes. We are pleased to submit the revised version of the manuscript and hope that it will be considered acceptable for publication in BMC Medical Research Methodology. On behalf of my co-authors, yours sincerely, Ralph van Hoorn, MSc. Nijmegen, the Netherlands (Reviewer/editor comments capitalised) In response to the comments and remarks of the editor: COMMENT 1: THE REFERENCE LIST NEEDS UPDATING. MOST OF THE REFERENCES ARE RATHER OLD. THERE ARE NO CITATIONS TO RELEVANT ARTICLES PUBLISHED DURING We have added a number of more recent references to substantiate some of our claims. The following references are added to the reference list: 5. Lindhiem O, Bennett CB, Trentacosta CJ, McLear C: Client preferences affect treatment satisfaction, completion, and clinical outcome: A meta-analysis. Clinical psychology review 2014, 34(6): Shingler SL, Bennett BM, Cramer JA, Towse A, Twelves C, Lloyd AJ: Treatment preference, adherence and outcomes in patients with cancer: literature review and development of a theoretical model. Current medical research and opinion 2014, 30(11):

3 10. Fraenkel L, Suter L, Cunningham CE, Hawker G: Understanding Preferences for Disease-Modifying Drugs in Osteoarthritis. Arthritis Care & Research 2014, 66(8): Abraham NS, Naik AD, Street RL, Jr., Castillo DL, Deswal A, Richardson PA, Hartman CM, Shelton G, Jr., Fraenkel L: Complex antithrombotic therapy: determinants of patient preference and impact on medication adherence. Patient preference and adherence 2015, 9: MacLean S, Mulla S, Akl EA, Jankowski M, Vandvik PO, Ebrahim S, McLeod S, Bhatnagar N, Guyatt GH, American College of Chest P: Patient values and preferences in decision making for antithrombotic therapy: a systematic review: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest 2012, 141(2 Suppl):e1S-23S. 15. Brooker A-S, Carcone S, Witteman W, Krahn M: Quantitative patient preference evidence for health Technology Assessment: a case study. International Journal of Technology Assessment in Health Care 2013, 29(03): Muhlbacher AC: Patient-centric HTA: different strokes for different folks. Expert review of pharmacoeconomics & outcomes research 2015, 15(4): Eiring Ø, Landmark BF, Aas E, Salkeld G, Nylenna M, Nytrøen K: What matters to patients? A systematic review of preferences for medication-associated outcomes in mental disorders. BMJ Open 2015, 5(4). 21. Rodriguez RW: Delay in indexing articles published in major pharmacy practice journals. American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists 2014, 71(4): In response to the comments and remarks of reviewer 1: THIS IS A RIGOROUS APPROACH TO SEARCH FILTER DEVELOPMENT; AND THE DESCRIPTION OF SENSITIVITY, SPECIFICITY ETC IS VERY CLEAR. COMMENT 2: HOWEVER, AS THE AUTHORS POINT OUT THIS MAY BE A VERY SMALL EVIDENCE BASE TO LOCATE. THE SEARCH TERMS HAVE THUS BEEN

4 DEVELOPED ON A VERY SMALL SAMPLE. IF PATIENT PREFERENCE IS A GROWING RESEARCH FIELD THEN THE TERMINOLOGY AND ANY FUTURE VALIDATED SEARCH MEASURES' COULD CHANGE SIGNIFICANTLY, INVALIDATING THE SEARCH STRATEGIES. IN RELATION TO THIS, I'M CONCERNED THAT A MORE RECENT YEAR COULD HAVE BEEN APPROPRIATE E.G SHOULD STILL HAVE ALLOWED FOR MEDLINE INDEXING OR A RANGE OF YEARS TO INCREASE THE SAMPLE OF RELEVANT PAPERS. COULD THE AUTHORS PROVIDE MORE DETAIL ON THEIR RATIONALE FOR THESE DECISIONS? AND HIGHLIGHT THE LIMITATIONS IN THE ABSTRACT? The development of the search strategies started with a selection of relevant papers based on abstracts of all papers published in a selection of journals at the end of For a proper filtercreation, we required all abstracts to be MeSH-indexed. After contacting information specialists we decided to include articles which were published at least one year earlier to be sure the articles were MeSH-indexed. The delay in MeSH-indexing (i.e. the difference in time between study publication and MeSH-indexing) was recently reported to be 52 days (median, interquartile range days) for medical journals published in Choosing an entire year of publications was to make a clear demarcation of article selection which would be easy to retrieve and yield a consistent set of papers for any journal. We added to the abstract a reference to the year: Methods: A total of 27 journals were hand-searched for articles on patient preferences for treatment outcomes published in We added to the discussion: Inevitably, the performance of the search strings presented in this paper reflects a particular terminology that was used by researchers who published findings of their work on patients preferences for treatment outcomes in It cannot be ruled out that certain changes take place in this terminology over time, which might affect the performance of the search strings presented

5 in this paper. For this reason, an update of the performance of these search strings in a couple of years may be warranted. Alternatively, researchers in this area might be encouraged to employ the terminology that resulted in efficient retrieval of relevant papers. This, then, would likely further enhance the performance of these search strings in the future. COMMENT 3: I'M A LITTLE UNCLEAR AS TO WHICH SEARCH STRATEGIES THE AUTHORS ARE RECOMMENDING TO USE. ASSUMING THIS IS THE INTENTION - THE PAPER COULD BENEFIT WITH SOME EDITING TO MAKE MORE CERTAIN RECOMMENDATIONS. OR AT LEAST TO PRESENT TWO OPTIONS IE HIGH SENSITIVITY, LOWER SPECIFICITY; AND HIGH SPECIFICITY VERSION. We understand the confusion. We have rephrased the explanation for each of the optimisation measures and how one should make a choice between them. We made adjustments to the methods: Each single search-term found by PubReMinder was tested individually to determine its sensitivity (Se), specificity (Sp), accuracy (Ac) and Number Needed to Read (NNR) (see Table 1). The Se is a measure of the proportion of relevant articles retrieved compared with all relevant articles. A search filter high in Se can be used when relevant literature is expected to be scarce or when the other filters do not return enough relevant literature. Specificity is a measure for the non-retrieval of non-relevant citations [24]. A search filter high in Sp may be used if the likely effect of missing relevant literature is not considered critical (e.g. given a large amount of relevant literature available). Accuracy is defined as the proportion of articles correctly handled by the search strategy [20], and the NNR is defined as the average number of articles one needs to screen to find one relevant article [25]. Filters high on Ac and low on NNR return few irrelevant papers while minimizing the number of missed relevant papers. We also made adjustments to the discussion:

6 Broad and narrow search filters were developed to allow for the efficient retrieval of scientific literature on patient preferences for treatment outcomes. The choice of filters may depend on the scope of the problem under investigation. A reasonable strategy might be to start with sensitivity-optimised filters, followed by specificity-optimised filters when the initial set of retrieved literature seems to vast and contaminated with marginally relevant papers. Clearly, the choice will also depend on the time-constraints and needs of the user. COMMENT 4: THERE ARE A FEW MINOR TYPO'S (E.G. LINE 42) We have gone through the manuscript once more and fixed any typo s we came across In response to the comments and remarks of reviewer 2: COMMENT 5: THE AUTHORS DESCRIBED THE SELECTION OF JOURNALS AS "ON THE BASIS OF EXPERT OPINION" (LINE 25, PAGE 4). IT WOULD BE HELPFUL IF THE AUTHORS DESCRIBED THE RATIONALE SINCE OF ALL THE 31 JOURNALS, MOST OF THEM ARE ON GENERAL MEDICINE TOPICS, WHILE SEVERAL OF THEM ARE ON ARTHRITIS AND RHEUMATISM. FOUR ARTICLES OF THE 26 ARTICLES INCLUDED WERE ARTHRITIS RELATED, WITH DEVELOPMENT SET AND VALIDATION SET EACH CONTAINING TWO OF THEM. THERE ARE CERTAINLY PREFERENCE STUDIES ON OTHER DISEASE SETTINGS BUT IF THEY ARE INDEXED DIFFERENTLY, THE SELECTION OF ARTHRITIS RELATED JOURNALS COULD BE A BIASED SELECTION. Following this comment, we have repeated our analyses while excluding the four Rheumatology journals. This resulted in slightly different combination of terms, with higher face validity and better sensitivity and specificity values in the validation set. We decided to present the results of both analyses, the one with and the one without the Rheumatology journals. In order to avoid confusion, we suggest to present the results of the analysis with the general journals in the main

7 article, and to present the results of the analysis including the Rheumatology journals in the appendix. We think this enhances the generalisability of our findings, and presents readers with a better understanding of how the performance of the search filters may be affected by including a specific medical specialty. Specifically, when the Rheumatology journals are added, search terms such as adheren* and choice* performed slightly better, possibly reflecting the importance of drug treatment in this area. We added to the methods a clarification of the journal selection process: A set of relevant papers was constructed by hand-searching 27 journals on papers reporting empirical evidence on patient preferences for treatment outcomes. The list of journals was selected on the basis of expert opinion from the authors of this paper, experts in patient preferences and information specialists (see table 2). Journals were selected on their likelihood of publishing relevant papers. We also adjusted the results on the basis of these new analyses throughout the manuscript (See also changes in Table 2-4). Furthermore, we added to the discussion: A second limitation of our study is that its focus is on general medical journals. Conceivably, slightly different terminology may be used in specific medical sub-specialties that could affect the performance of our search strings. In fact, when the literature source was extended to include the domain of Rheumatology (data not shown, but results are available in Appendix B), we found that search terms such as adheren* and choice* performed slightly better. Possibly, this reflects the relative importance of drug treatment in this area. For this reason, we suggest users determine whether the search filters identify key publications in the specific disease field. COMMENT 6: THE AUTHORS MENTIONED (LINE 49, PAGE 4) "STUDIES THAT ONLY DESCRIBED TREATMENT PREFERENCES (I.E. PREFERENCE FOR TREATMENT A

8 OVER B) FOR DECISION INVOLVEMENT OR INFORMATION, OR PREFERENCES CONCERNING DIAGNOSIS WERE NOT SELECTED UNLESS THEY ALSO DESCRIBED PREFERENCES FOR SPECIFIC OUTCOMES (E.G. FATIGUE, PAIN)". HOWEVER, THE CRITERIA OF PREFERENCES FOR TREATMENT OUTCOME MAY NEED MORE DETAILED DESCRIPTION TO MAKE IT IMPLEMENTABLE AND REPRODUCIBLE FOR READERS. WHAT IF THE PREFERENCE IS ABOUT TREATMENT ATTRIBUTE, FOR EXAMPLE THE RISK OF ADVERSE EVENTS FOR A TREATMENT, SHOULD IT BE INCLUDED OR EXCLUDED? The difference between treatment preferences and preferences for treatment outcomes is key in understanding the consequences of these findings. We tried to explain more clearly in the background section what is meant by preferences for treatment outcome. We added to the background: Patients preferences are usually described as a preference between one treatment or another, but such preferences are difficult to generalize as they are very context-dependent. Therefore, it would be more relevant to retrieve information on treatment outcomes which might explain such preferences, e.g. risks on adverse events, or specific outcomes such as functional status. COMMENT 7: THE MAJOR ISSUE THIS STUDY HAS THE SMALL NUMBER OF ARTICLES IN EITHER SET. ONLY 12 ARTICLES IN THE VALIDATION SET, MEANS INCLUDING ONE MORE OR LESS ARTICLE BY CHANCE, THE SENSITIVITY WOULD CHANGE BY ABOUT 8%, WHILE IN CONTRAST, THE SPECIFICITY WOULD BE RELATIVELY STABLE. We agree that the in- or exclusion of one article has an impact on the sensitivity of the search strategy. To make readers more aware of this we discuss the effect of the sample size on filter performance in the discussion section. We added to the discussion:

9 A limitation of our study is the relatively low number of relevant papers that were found in the literature, increasing the odds of overfitting (i.e. making the filters too specific for our gold standard set) during the creation of the search filters. Due to the limited set of relevant papers, missing a single article will result in a drop of around 9% in sensitivity, while specificity suffers much less due to its dependence on prevalence of relevant papers. Only 22 of all 8238 handsearched articles (0.27%) reported empirical evidence on patient preferences for treatment outcomes. COMMENT 8: THE MESH-TERMS IN THE TABLE 3 ARE NOT SPECIFIC FOR "PREFERENCES FOR TREATMENT OUTCOMES" (PAGE 8). THESE TERMS COULD ALSO BE USED FOR IDENTIFYING STUDIES ON "TREATMENT PREFERENCES". AGAIN THE AUTHORS NEED TO JUSTIFY THE EXCLUSION OF "TREATMENT PREFERENCES" STUDIES. IT MAY BE ALSO HELPFUL TO DISCUSS THE IMPACT ON THE SENSITIVITY AND SPECIFICITY IF THESE STUDIES ARE INCLUDED /EXCLUDED. I FEEL IT WOULD BE IMPORTANT FOR THE AUTHORS TO DISCUSS WITH MORE DETAILS (THAN WHAT HAS BEEN INCLUDED ON PAGE 10) THE IMPACT OF THE SMALL NUMBER ON THE DEVELOPMENT OF STRATEGY AND THE IMPACT ON THE ESTIMATES OF SENSITIVITY, SPECIFICITY, ETC. This issue was also addressed by reviewer 1: please see our response to the fifth comment of this reviewer, and our response to comment 7 above. COMMENT 9: THE SEARCH WAS CONDUCTED THROUGH PUBMED FOR THE SELECTED JOURNAL, IS THAT RIGHT? This is correct. COMMENT 10: IT WOULD BE HELPFUL TO PROVIDE THE AGREEMENT OF STUDY SELECTION BETWEEN REVIEWERS.

10 Eligibility of research papers for inclusion was independently assessed by two researchers. The results of both researchers were compared and any differences were resolved by consensus or discussion with a third researcher. Consensus between the two researchers was, in fact, complete (100%). The manuscript erroneously mentioned the use of majority rule ; this has been removed from the methods section. We altered the methods: Any disagreements were resolved by consensus with a third author We added to the results: A total of 262 relevant articles (0.27%) were selected with 100% agreement as reference set COMMENT 11: THE AUTHORS MENTIONED THE HAND-SEARCH WAS LIMITED IN THE YEAR 2011 SINCE... "ARTICLES FROM LATER YEARS MAY NOT ALL BE PROPERLY MESH-INDEXED" (LINE 27, PAGE 4), FOR THIS SAME REASON, THE SENSITIVITY AND SPECIFICITY ESTIMATED MAY NOT APPLY TO THE SEARCH RESULTS IN LATER YEARS IF OTHERS USE THIS STRATEGY. THE AUTHORS MAY DISCUSS THIS ISSUE IN THE APPLICABILITY OF THE SEARCH STRATEGY. This comment was also raised by reviewer 1: please see our response to the second comment of this reviewer. COMMENT 12: THE AUTHOR MENTIONED "THE AMOUNT OF RELEVANT ARTICLES WAS BALANCED BETWEEN THE TEST AND DEVELOPMENT SET FOR EACH JOURNAL" (LINE 10, PAGE 5). FOR THIS POINT, I AM CONFUSING, BECAUSE VERY

11 FEW JOURNALS CONTAIN MORE THAN ONE STUDIES, WHILE EVEN FOR THOSE WITH MORE THAN ONE, IT IS NOT "BALANCED". COULD YOU PLEASE EXPLAIN? We have rephrased the description of the randomisation process as follows: To allow for internal validation of the search queries, all articles were randomized (1:1) between a development set and a validation set using Microsoft Excel. This randomization process was done in such a way that each journal was equally represented in both sets and that the amount of relevant articles was balanced between the development set and the validation set. COMMENT 13: PERSONALLY I HAVE CONCERNS WITH THE FACE VALIDITY OF "LOGIT" AS A SEARCH TERM, COULD YOU PLEASE EXPLAIN? The occurrence of the term logit can be explained by the (5/22) papers using discrete choice experiments (DCEs) for eliciting patient preferences for interventions. DCEs are a standard method for this purpose, and frequently employ logit models (a specific type of regression model). We added to the results section a brief explanation of this term: The best Sp, Ac and NNR could be gained with the keyword Logit (a term related to a specific type of regression model that is often used in discrete choice experiments which, in turn, are used to elicit patients preferences for treatment outcomes), yielding Sp and Ac > 99% and an NNR of 2.8. OVERALL, DESPITE THE MANY POINTS I RAISE FOR THIS MANUSCRIPT, IT IS CRITICALLY IMPORTANT WORK THAT IS OF HIGH QUALITY.