**Report from The BMJ s manuscript committee meeting**

Transcription

1 **Report from The BMJ s manuscript committee meeting** These comments are an attempt to summarise the discussions at the manuscript meeting. They are not an exact transcript. Manuscript committee September 7, 2017 Wim Weber (chair), Rafael Perera (statistician), Elizabeth Loder, Jose Merino, Tiago Villanueva, Daoxin Yin Decision: Ask for revision The committee was interested in the topic of your research. The following concerns were mentioned: 1. It is a specialist area, even for neurologists, but it is a dramatic disorder, often involving difficult decisions in terms of withholding further treatment. In that regard a simple to use prediction tool might be very useful. We agree with the reviewer that while this is a specialized area, it is a dramatic disorder and a simple prediction tool may offer considerable utility in managing this devastating condition. The care of affected individuals involves a wide range of healthcare providers who play some role at different stages of the care spectrum. Reviewer number 3 highlighted this point succinctly by stating that, This is an acute condition that is of relevance to clinicians from various specialties, including neurosurgeons, neurologists, intensivists, and emergency physicians. The long-term outcomes from SAH are relevant to broader clinical disciplines such as family medicine, physical medicine and rehabilitation, and the allied health professions such as physiotherapy. 2. What do you mean by unfavourable outcomes? Isn t death an unfavourable outcome? We have included a new figure (See Figure 1) to provide more specific details on the definition of the study end points. Specifically, we have defined an unfavourable outcome as either death, persistent vegetative state, or severe disability and a favourable outcome as moderate disability or good recovery. 3. The calculator seems easy to use (what does it mean that it was based on the 6 models? Does it reflect the full model?), but we agree that it seems easier to interpret and act on a score / percentage than these bars / graphs. The SAHIT prediction models include two sets of 3 consecutive models. The calculator does reflect the full model. For simplicity, we have revised the statement on model presentation from We developed an online prognostic calculator based on the six models that to now read We developed an online prognostic calculator based on the SAHIT prediction models that We are revising the calculator to output only the percentages. 4. The study was well done and presented. The study connected various datasets. It is relevant and useful for the BMJ readership. Thank you. 5. We disagree with a reviewer on dropping the external validation, because, in the study, the authors used a completely different core of datasets and findings add, though we agree that proper 'independent external validation' should be done by a different group/data. We agree with the committee member s opinion.

2 6. You may want to consider providing a table with the actual coefficients for the different models (including intercept). Their table 5 is the closest to this. Please see supplementary Table 5. We have deleted all redundant tables and renumbered all tables and figures appropriately. 7. You provided a calculator as a tool in the web, but it would be helpful if you can provide the actual different models as supplementary. Please see supplementary Table SAH is a very challenging problem with a high mortality rate and many patients who survive have a bad outcome. There are risks of being overly aggressive despite a poor prognosis on the one hand, and of withdrawing care too soon on the other. You pointed out the difficulties in practice, and you could highlight more on its clinical impact in shared decision making. You could consider telling readers how it would be useful for patients. We have rewritten the Discussion subsection on the implications of the study to provide more specific details on how we imagine the models could be useful in clinical practice, in education and for research purposes. In summary, we discussed the potential use of the calculator in the outpatient clinic to inform discussions around SAH prognosis with patients who are being seen for unruptured intracranial aneurysm, to track the progress of recovery at follow up, and to inform rehabilitation plans at hospital discharge. We also described the potential application during acute care to inform treatment decision, to facilitate timely referral, and inform discussions around an alternative model of care that could potentially optimize hospital resource utilization without compromising the quality of care. 9. Table 2 and 3 are the same as supplemental tables 1 and 2. We have revised the tables and figures to delete redundant tables. 10. How did you identify the predictors? In the Methods subsection on variable selection, we stated that the predictor variables were selected on the basis of a previously completed systematic review to appraise clinical prediction models and identify predictors of outcome following SAH (Jaja et al., 2013). Our prior prognostic analyses further support the value of the included predictor items (see Jaja et al., 2014, 2015). 11. The committee was persuaded by reviewers that the topic is not too specialized It is our hope that the SAHIT prognostic tool will have positive impact on clinical practice, as it addresses an important practice need for the broad range of specialties that play one role or the other in the care of the individual with SAH. 12. We agree you should acknowledge other existing models. We have expanded the Discussion subsection titled, Comparison with previous studies to more fully acknowledge previous research and better situate our study in the literature of outcome prediction research in SAH. 13. The calculator is easy to use, but it seems easier to interpret and act on a score/percentage than these bars/graphs. This point is well taken. We are updating the web calculator to output only the percentages with the associated error margins. 14. Please define what is the unfavourable outcome as death could also be counted as an unfavourable outcome.

3 We have included a new figure (See Figure 1) to provide more specific details on the definition of the study end points. Specifically, we have defined an unfavourable outcome as either death, persistent vegetative state, or severe disability and a favourable outcome as moderate disability or good recovery. Please accept my apology for the delayed report from the manuscript meeting. I didn t take part because of my summer holiday. First, please revise your paper to respond to all of the comments by the reviewers. Their reports are available at the end of this letter, below. Please also respond to the additional comments by the committee. In your response please provide, point by point, your replies to the comments made by the reviewers and the editors, explaining how you have dealt with them in the paper. ** Comments from the external peer reviewers** Reviewer: 1 Recommendation: Comments: 1. On reflection, perhaps I should have declined to review this manuscript: I found the text difficult to follow, and the data presented were beyond my level of comprehension. I recognise the importance of SAH, however, and I accept that an outcome prediction model would be useful for those treating the condition. I imagine therefore that the paper is relevant to the specialists involved, but I cannot believe many patients or many non-specialists would regard the tool described as being particularly useful to them. The lack of patient involvement in the study is disappointing, but I'm unable to offer any suggestions as to how this might be improved. We have noted and are appreciative of the reviewer s opinion. Here There are a fewseveral ways we have attempted to improve the readability of the paper and the utility of the web calculator and app for nonexperts: we have added a lay summary to the manuscript; the study end points have been more concisely defined (Figure 1); the calculator and app is being revised taking the opinion of stakeholders into consideration, such as those of survivors of the condition. Additional Questions: Please enter your name: John Gooderham Job Title: patient/public Institution: none Reimbursement for attending a symposium?: No

4 A fee for speaking?: No A fee for organising education?: No Funds for research?: No Funds for a member of staff?: No Fees for consulting?: No Have you in the past five years been employed by an organisation that may in any way gain or lose financially from the publication of this paper?: No Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this paper?: No If you have any competing interests <A HREF=' (please see BMJ policy) </a>please declare them here: Reviewer: 2 Recommendation: Comments: 1. The authors need to be congratulated to this extensive and highly relevant work. The rationale behind the study, its design and results are well outlined and the data sufficiently discussed. Moreover, this collaborative group provides multinational and multi continental data, making the results even more conclusive and relevant. Thank you. 2. Concerning methodology, results and interpretation, I have nothing to add. From the view of a reader naive of the whole study, I would find it helpful to provide a flow sheet showing the number of trials which were considered to be included in the study and the reasons and quantity of studies why others were excluded. This might be important to rule out any potential selection bias of the data. But as said, this is just a suggestion. Thank you for this suggestion. We have provided a table of the studies which datasets were excluded and reasons for the exclusion (see Table 1 of the supplementary file). We have expanded the section on study limitation to indicate the potential for selection bias due to such exclusion (see study limitation number 7). Additional Questions: Please enter your name: Dr. Sandro Krieg Job Title: Attending Neurosurgeon

5 Institution: Klinikum rechts der Isar, TUM Reimbursement for attending a symposium?: Yes A fee for speaking?: Yes A fee for organising education?: No Funds for research?: Yes Funds for a member of staff?: No Fees for consulting?: Yes Have you in the past five years been employed by an organisation that may in any way gain or lose financially from the publication of this paper?: No Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this paper?: No If you have any competing interests <A HREF=' (please see BMJ policy) </a>please declare them here: I am consultant for Brainlab AG (Munich, Germany) and between 2015 and 2016 I was consultant for Nexstim Oyi (Helsinki, Finland) Reviewer: 3 Recommendation: Comments: 1. Originality - does the work add enough to what is already in the published literature? If so, what does it add? If not, please cite relevant references. Prediction models for SAH have been studied previously, however, poorly in the literature. So, this work is original. The authors present a strong case for how their models improve upon past work. The authors, however, do not give enough recognition to previous research. In the Introduction the authors state that clinicians presently rely on intuition in predicting SAH outcomes. This is a very general statement and is not the case at every centre. This statement should be revised. Various prediction models exist, although limited, for clinicians to help to direct clinicians in managing patients with subarachnoid haemorrhage. The authors should add a statement recognising past work. We have revised the introduction section to delete the statement which started as Though clinicians presently rely on intuition in predicting the outcomes, clinical intuition alone has been shown to be unreliable. We have added a new sentence that reads, Reliance on clinical intuition alone may be insufficient for accurate prognostication of SAH outcome. 3 We also have revised the Discussion subsection titled Comparison with previous studies to discuss in greater detail previous research and how our present work complements and advance prior effort on the development and validation of clinical prediction models and risk scores for SAH.

6 2. Importance of work to general readers - does this work matter to clinicians, patients, teachers, or policymakers? Is a general journal the right place for it? This work discusses a prognostic model for subarachnoid haemorrhage (SAH). This is an acute condition that is of relevance to clinicians from various specialties, including neurosurgeons, neurologists, intensivists, and emergency physicians among others. The long-term outcomes from SAH are relevant to broader clinical disciplines such as family medicine, physical medicine and rehabilitation, and the allied health professions such as physiotherapy and occupational therapy. This work would help to improve the care of patients with SAH and is of relevance to a general audience. We can see this prognostic tool being adopted by the American Heart Association in future guidelines to direct patient care. We thank the reviewer for the insight. We agree that healthcare providers across a broad range of specialties may find the SAHIT prognostic tool an easy-to-use practical tool to inform patient-provider conversations around outcome expectations and shared decision making. 3. Scientific reliability. Research Question - clearly defined and appropriately answered? This is a well written report with strong and meticulous methodology tackling an important clinical question. We have some suggestions which would help to further strengthen this report. A 2015 systematic review by Lo, et al. examined clinical prediction tools and prognostic factors in aneurysmal subarachnoid haemorrhage. It identified that most studies suffer from within study patient heterogeneity. How did the authors account for heterogeneity between their patient cohorts, given regional variability in treatment protocols, trial criteria, and investigator bias? We thank the reviewer for highlighting the issue of between-study heterogeneity. We adopted the following approach to account for heterogeneity. First, we studied the baseline characteristics of the pooled development cohort and the pooled validation cohort as well as the distribution of the variables in the different constituent studies, to gain insight into possible case-mix differences that could impact predictor effects. The predictor-outcome association was estimated in models adjusting for the fixed effect of study. We formally tested and quantified between-study heterogeneity at validation (internal-external and full external) using the I 2 statistic. Our analysis did demonstrate significant between-study heterogeneity for some models. Thus, we adopted a leave-one-study-out cross validation procedure for internal-external validation, and report the results for each study. For full external validation, we performed a two-stage meta-analysis validating the models in the constituent studies of the validation set and pooling the results using a random effects model. In the results section, we drew attention to the characteristics of the studies where the models performed poorly mostly in the case of the highly selected CONSCIOUS I trial. We also highlighted the studies where the models performed consistently well, mostly in the unselected hospital datasets. The approach we adopted to deal with heterogeneity was guided by recommended modern guidelines for studies reporting individual participant data meta-analysis (IPD-MA) for prognostic modeling (Ahmed et al., 2014; Debray et al., 2015) 4. The authors report that the modified Rankin Scale (mrs) was used to impute values for the Glasgow Outcome Score (GOS) when the GOS was unavailable. While the authors discuss that this is standard practice for prediction models, what differences did use of mrs create from the prediction model? A valuable sensitivity analysis would be to examine performance of the model when the mrs is used to impute the GOS for patients where GOS data is readily available and determine how reliable mrs truly is for this imputation. There are 6 studies in the SAHIT repository where both mrs and GOS outcomes are recorded (n = 6556). However, wbut we are a bit cautious about this kind of sensitivity analysis for the very fact that since any results from such sensitivity analysis of these datasets which were not collected primarily for to compareing the psychometric properties or predictive validity of the two outcome scales can only be taken at face value. For instance, a comparison of their AUC for functional outcome at apparent validation

7 indicated a difference of only 0.01, and a simple correlation analysis (Spearman) of the two outcomes gave a value of 0.93, which, on face value, could be interpreted to validate our choice to impute the mrs for the GOS score. But we doubt that these results provide true proof of the reliability of the imputation. For one, it is very unlikely both outcome measures were assessed at the same time point and by the same rater in each study. In the worst-case scenario, any potential differences between both scales will have contributed to the between-study heterogeneity that was noted in the present study, and relevant only in the case of the two datasets for which the mrs was imputed for the GOS score the Kurashiki and SWISS-SOS datasets. We had highlighted in an earlier response the recommended approach we took to account for heterogeneity, which, in our opinion, should suffice. 5. It is stated that the development and validation cohorts are to some extent similar; however, there are no statistics to support this claim. WFNS grade, aneurysm size, and Fisher grade appear to possibly be different between these cohorts. Please provide statistical support for this claim. We have deleted from the results section the statement, To some extent, both cohorts are similar. We did so to allow the reader make her/his judgement on the extent of comparability between the development and validation cohorts. 6. In the analysis aneurysm size was categorised as <12mm, 12 24mm, >25mm. However, the calculator requires input of a continuous variable. Please explain why it is continuous on the calculator. Perhaps input of a categorical variable may be quicker and easier. We felt the input of aneurysm size as a continuous variable may be easiest for most clinicians but could modify this as a categorical variable should the editors feel strongly about this comment. 7. Did any of the study cohort manage patients with revisions/rebleeds? If so, please list them. In most studies, a very small proportion of patients re-bled and some needed to have the aneurysm recoiled or surgically clipped, the exception being the CONSCIOUS 1 trial. We have added this fact information to the supplementary file. 8. This analysis is comprehensive and outlines covariates that are important in the prognosis of patients with subarachnoid haemorrhage. To make the study more complete, I would suggest including results of an analysis on interactions of prognostic factors and report them in the supplementary material. It was not our intention to perform this analysis a priori given the likelihood of type 1 error from the very large development set of over 10,000 patients, and our desire to design a relatively simple tool. However, motivated by the reviewer s comment, we have done so. We have added brief notes to the method and results sections, and deleted the study limitation on interaction terms to reflect the analysis. The results of the interaction analysis are presented in supplementary table 4. In brief, the interaction between age and WFNS grade and that between ruptured aneurysm location and treatment modality were significant at the one-percent significance level. But when we extended the main model ( full model) with the interaction terms, no improvement was noted in model performance at apparent validation, confirming our a priori expert opinion. 9. Other clinical tools, such as the ICH score, adopted by the American Heart Association, output a numerical percentage that is easy to use and interpret in rapidly changing clinical situations. The authors should consider revising the output of their tool to an easily interpreted score instead of the three graphs presently given. We are updating the Web calculator to output only the predicted probabilities in percentage with the associated error margins. Formatted: Font: 10 pt, English (Canada)

8 10. There are more missing values in the external validation cohort (table 3) compared to the internal validation cohort (table 2). Despite using multiple imputation method which is widely accepted, please consider this as an additional limitation. We have included this point as a potential limitation. See the sentence, Also, the proportion of missing data differed between the development and validation datasets which has been added to point 6 on study limitation. 11. Some other common covariates mentioned in the literature (e.g. your systematic review - reference 6) for subarachnoid haemorrhage include hyperglycaemia, smoking, presence of cardiac manifestations, and presence of neurogenic pulmonary oedema. Why were these not included as covariates for this model? We would suggest reporting only on covariates with the strongest prognostic effect in the Conclusion. We recognize that several predictors of outcome following SAH have been reported in the literature, all of which could potentially be included in a prediction model. However, for most predictors including the ones highlighted by the reviewer the evidence is low; we do not know how much added incremental value they contribute. (For a review of SAH prognostic biomarkers, see van der Bilt et al., 2009; Lanterna et. al., 2007; Kaneda et al., 2010). The predictors in the SAHIT models have been well studied and validated (see Jaja et al., 2013 for our review of SAH prognostic models; see Rosengart et al., 2007 for SAH outcome determinants). Also, our previous analysis confirmed their added incremental value for outcome prediction (Jaja et al., 2014, 2015). It is possible that future prognosis research will provide further evidence in support of those predictors that we did not consider for our models, or those that are less studied presently, or unravel hitherto latent prognostic factors of SAH. These may include metabolomics, proteomic, genetic or genomic biomarkers; or sociodemographic, clinical, physiologic or imaging parameters. When that happens, there is the possibility to improve the predictive accuracy of the SAHIT models by incorporating these covariates to update the models. 12. The authors published previously (Jaja, et al and Jaja, et al. 2015) prognostic models that did not evaluate all of the covariates studied in the present manuscript. While the present manuscript is more comprehensive than these past studies, we do not consider it appropriate to include an external validation within the same paper in which a tool is first reported. We recommend that the authors remove their external validation analyses as true external validation analyses would be those conducted by a separate group, following publication of the prognostic tool. We recognize that the external validation of a prediction model is a continuous process, as several researchers examine the applicability of the model to their setting. It is very likely the SAHIT models will not be spared that process. We therefore think that irrespective of our external validation data, researchers will still perform further analysis on the external validity of the SAHIT prediction models, and confirm the utility for their setting. To support the process of independent external validation, we have included the intercept and coefficients of the predictors of the different models in supplementary table 5. Given this scenario, we are unsure what useful purpose will be served by excluding from the manuscript the data of our extensive, highly powered, external validation analysis that was performed after the development of the models have been completed, and that was based on a cohort from a variety of settings of care. We are of the opinion such data exclusion will only weaken our study conclusion and create a publication bias. 13. We recommend publication of this study in BMJ once the authors have addressed the above concerns. This study is comprehensive, novel, and would be the first to provide a reliable numerical scale to predict patient outcomes following SAH. This would have far reaching implications across multiple clinical disciplines, and serve to greatly improve patient care. We thank the reviewer for the insight and recommendation.

9 Additional Questions: Please enter your name: Saleh Almenawer Job Title: Neurosurgeon Institution: Hamilton Health Sciences and McMaster University Reimbursement for attending a symposium?: No A fee for speaking?: No A fee for organising education?: No Funds for research?: No Funds for a member of staff?: No Fees for consulting?: No Have you in the past five years been employed by an organisation that may in any way gain or lose financially from the publication of this paper?: No Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this paper?: No If you have any competing interests <A HREF=' (please see BMJ policy) </a>please declare them here: Reviewer: 4 Recommendation: Comments: 1. The prediction model, based on the SAHIT data and externally validated using modern trials and series, is well written. The use of real life data from the repository is most relevant. The strengths and the limitations of the model are clearly demonstrated and addressed. The model would serve as a very useful and important tool for the emergency, acute medical, and neurosurgical clinicians. We thank the reviewer for the comment. Additional Questions: Please enter your name: William Lo

10 Job Title: Fellow Institution: Hospital for Sick Children, Toronto Reimbursement for attending a symposium?: No A fee for speaking?: No A fee for organising education?: No Funds for research?: No Funds for a member of staff?: No Fees for consulting?: No Have you in the past five years been employed by an organisation that may in any way gain or lose financially from the publication of this paper?: No Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this paper?: No If you have any competing interests <A HREF=' (please see BMJ policy) </a>please declare them here: No Reviewer: 5 Recommendation: Comments: 1. The authors report their collaborative effort of developing a prediction model for outcome after aneurysmal subarachnoid haemorrhage. Based on individual patient data from patients derived from known clinical trials of asah they have established a set of three consecutively staged prediction models for outcome (GOS at 3 months) and mortality. The predictive value was externally validated using data sets from 3355 patients with asah from later clinical studies not included in the initial development of the predictions models. The authors found a reasonable predictive value for outcome at 3 months (AUC 0.81) using their "core" model that includes age, premorbid hypertension and WFNS scale. Including further neuro-imaging or treatment parameters slightly improved the predictive value on a statistical level. However, whether this is clinically meaningful in the all-day-setting remains open. They found similar results for their prediction models for mortality (AUC 0.79). Given the natural limitations of such an effort in this disease (which have been correctly addressed and outlined throughout the manuscript) the results are sound and address a highly important issue in the management of asah. The strength and practical value of this model (which can be easily used with the internet based tool) is based on the simple clinical data of the individual patient that is usually readily available at admission. Overall this is a well-written manuscript and the scientific and clinical community would clearly benefit of its publication. Minor points: Abstract: the abbreviation of SAHIT should get introduced

11 Table 1 - bracket is missing (aneurysm size - line 24) We thank the reviewer for the comment. The highlighted minor points have been corrected. Additional Questions: Please enter your name: Nils O. Schmidt Job Title: Senior Neurosurgeon Institution: University Medical Center Hamburg-Eppendorf Reimbursement for attending a symposium?: No A fee for speaking?: No A fee for organising education?: No Funds for research?: No Funds for a member of staff?: No Fees for consulting?: No Have you in the past five years been employed by an organisation that may in any way gain or lose financially from the publication of this paper?: No Do you hold any stocks or shares in an organisation that may in any way gain or lose financially from the publication of this paper?: No If you have any competing interests <A HREF=' (please see BMJ policy) </a>please declare them here: REFERENCES Ahmed I, Debray TA, Moons KG, Riley RD. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Medical Research Methodology 2014; 14:3. Debray TPA, Riley RD, Rovers MM, Reitsma JB, Moons KGM, Cochrane IPD Metaanalysis Methods group. Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use. PLoS Med 2015; 12(10): e doi: /journal. Jaja BN, Cusimano MD, Etminan N, Hanggi D, et al. Clinical prediction models in aneurysmal subarachnoid hemorrhage: A systematic review. Neurocritical Care 2013; 18:

12 Jaja BN, Lingsma H, Thorpe KE, Schweizer TA, Steyerberg EW, Macdonald RL, for the SAHIT Collaboration (2014). Prognostic value of premorbid hypertension and baseline neurologic status in aneurysmal subarachnoid hemorrhage: pooled analysis of individual patient data in the SAHIT repository. Journal of Neurosurgery 2015; 122(3): Jaja BN, Lingsma H, Thorpe KE, Schweizer TA, Steyerberg EW, Macdonald RL, for the SAHIT Collaboration. Neuroimaging characteristics of ruptured aneurysm as predictors of outcome after aneurysmal subarachnoid hemorrhage: pooled analyses of the SAHIT cohort. Journal of Neurosurgery 2015; 124(6): Kaneda K, Fujita M, Yamashita S, Kaneko T, Kawamura Y, Izumi T, Tsuruta R, Kasaoka S, Maekawa T. Prognostic value of biochemical markers of brain damage and oxidative stress in post-surgical aneurysmal subarachnoid hemorrhage patients. Brain Res Bull 2010; 81: Lanterna LA, Ruigrok Y, Alexander S, Tang J, Biroli F, Dunn LT, Poon S. Meta-analysis of APOE genotype and subarachnoid hemorrhage - Clinical outcome and delayed ischemia. Neurology 2007; 69(8): Rosengart AJ, Schultheiss KE, Tolentino J, MacDonald RL. Prognostic factors for outcome in patients with aneurysmal subarachnoid hemorrhage. Stroke 2007;38(8): van der Bilt IAC, Hasan D, Vandertop WP, Wilde AAM, Algra A, Visser FC, Rinkel GJE. Impact of cardiac complications on outcome after aneurysmal subarachnoid hemorrhage A meta-analysis. Neurology 2009;72(7):