BMJ - Decision on Manuscript ID BMJ.2017.040049
Body: 14-Sep-2017 Dear Dr. Jaja Manuscript ID BMJ.2017.040049 entitled "Development and external validation of outcome prediction models for aneurysmal subarachnoid haemorrhage: The SAHIT multinational cohort study" Thank you for sending us your paper. We sent it for external peer review and discussed it at our manuscript committee meeting. We recognise its potential importance and relevance to general medical readers, but I am afraid that we have not yet been able to reach a final decision on it because several important aspects of the work still need clarifying. We hope very much that you will be willing and able to revise your paper as explained below in the report from the manuscript meeting, so that we will be in a better position to understand your study and decide whether the BMJ is the right journal for it. We are looking forward to reading the revised version and, we hope, reaching a decision. Yours sincerely, Georg Roeggla groggla@bmj.com **Report from The BMJ s manuscript committee meeting** These comments are an attempt to summarise the discussions at the manuscript meeting. They are not an exact transcript. Manuscript committee September 7, 2017 Wim Weber (chair), Rafael Perera (statistician), Elizabeth Loder, Jose Merino, Tiago Villanueva, Daoxin Yin Decision: Ask for revision The committee was interested in the topic of your research. The following concerns were mentioned: It is a specialist area, even for neurologists, but it is a dramatic disorder, often involving difficult decisions in terms of withholding further treatment. In that regard a simple to use prediction tool might be very useful. What do you mean by unfavourable outcomes? Isn t death an unfavourable outcome? The calculator seems easy to use (what does it mean that it was based on the 6 models? Does it reflect the full model?), but we agree that it seems easier to interpret and act on a score / percentage than these bars / graphs. The study was well done and presented. The study connected various datasets. It is relevant and useful for the BMJ readership. We disagree with a reviewer on dropping the external validation, because, in the study, the authors used a completely different core of datasets and findings add, though we agree that proper 'independent external validation' should be done by a different group/data.
You may want to consider providing a table with the actual coefficients for the different models (including intercept). Their table 4 is the closest to this. You provided a calculator as a tool in the web, but it would be helpful if you can provide the actual different models as supplementary. SAH is a very challenging problem with a high mortality rate and many patients who survive have a bad outcome. There are risks of being overly aggressive despite a poor prognosis on the one hand, and of withdrawing care too soon on the other. You pointed out the difficulties in practice, and you could highlight more on its clinical impact in shared decision making. You could consider telling readers how it would be useful for patients. Table 2 and 3 are the same as supplemental tables 1 and 2 How did you identify the predictors? The committee was persuaded by reviewers that the topic is not too specialized We agree you should acknowledge other existing models. The calculator is easy to use, but it seems easier to interpret and act on a score/percentage than these bars/graphs. Please define what is the unfavourable outcome as death could also be counted as an unfavourable outcome. Please accept my apology for the delayed report from the manuscript meeting. I didn t take part because of my summer holiday. First, please revise your paper to respond to all of the comments by the reviewers. Their reports are available at the end of this letter, below. Please also respond to the additional comments by the committee. In your response please provide, point by point, your replies to the comments made by the reviewers and the editors, explaining how you have dealt with them in the paper. ** Comments from the external peer reviewers** Reviewer: 1 On reflection, perhaps I should have declined to review this manuscript: I found the text difficult to follow, and the data presented were beyond my level of comprehension. I recognise the importance of SAH, however, and I accept that an outcome prediction model would be useful for those treating the condition. I imagine therefore that the paper is relevant to the specialists involved, but I cannot believe many patients or many non-specialists would regard the tool described as being particularly useful to them. The lack of patient involvement in the study is disappointing, but I'm unable to offer any suggestions as to how this might be improved. Please enter your name: John Gooderham Job Title: patient/public Institution: none Reimbursement for attending a symposium?: No
A fee for speaking?: No A fee for organising education?: No Funds for research?: No Fees for consulting?: No </a>please declare them here: Reviewer: 2 The authors need to be congratulated to this extensive and highly relevant work. The rationale behind the study, its design and results are well outlined and the data sufficiently discussed. Moreover, this collaborative group provides multinational and multi continental data, making the results even more conclusive and relevant. Concerning methodology, results and interpretation, I have nothing to add. From the view of a reader naiv of the whole study, I would find it helpful to provide a flow sheet showing the number of trials which were considered to be included in the study and the reasons and quantity of studies why others were excluded. This might be important to rule out any potential selection bias of the data. But as said, this is just a suggestion. Please enter your name: Dr. Sandro Krieg Job Title: Attending Neurosurgeon Institution: Klinikum rechts der Isar, TUM Reimbursement for attending a symposium?: Yes A fee for speaking?: Yes A fee for organising education?: No Funds for research?: Yes
Fees for consulting?: Yes </a>please declare them here: I am consultant for Brainlab AG (Munich, Germany) and between 2015 and 2016 I was consultant for Nexstim Oyi (Helsinki, Finland) Reviewer: 3 * Originality - does the work add enough to what is already in the published literature? If so, what does it add? If not, please cite relevant references. Prediction models for SAH have been studied previously, however, poorly in the literature. So, this work is original. The authors present a strong case for how their models improve upon past work. The authors, however, do not give enough recognition to previous research. In the Introduction the authors state that clinicians presently rely on intuition in predicting SAH outcomes. This is a very general statement and is not the case at every centre. This statement should be revised. Various prediction models exist, although limited, for clinicians to help to direct clinicians in managing patients with subarachnoid haemorrhage. The authors should add a statement recognising past work. * Importance of work to general readers - does this work matter to clinicians, patients, teachers, or policymakers? Is a general journal the right place for it? This work discusses a prognostic model for subarachnoid haemorrhage (SAH). This is an acute condition that is of relevance to clinicians from various specialties, including neurosurgeons, neurologists, intensivists, and emergency physicians. The long-term outcomes from SAH are relevant to broader clinical disciplines such as family medicine, physical medicine and rehabilitation, and the allied health professions such as physiotherapy. This work would help to improve the care of patients with SAH and is of relevance to a general audience. We can see this prognostic tool being adopted by the American Heart Association in future guidelines to direct patient care. * Scientific reliability Research Question - clearly defined and appropriately answered? This is a well written report with strong and meticulous methodology tackling an important clinical question. We have some suggestions which would help to further strengthen this report. A 2015 systematic review by Lo, et al. examined clinical prediction tools and prognostic factors in aneurysmal subarachnoid haemorrhage. It identified that most studies suffer from within study patient heterogeneity. How did the authors account
for heterogeneity between their patient cohorts, given regional variability in treatment protocols, trial criteria, and investigator bias? The authors report that the modified Rankin Scale (mrs) was used to impute values for the Glasgow Outcome Score (GOS) when the GOS was unavailable. While the authors discuss that this is standard practice for prediction models, what differences did use of mrs create from the prediction model? A valuable sensitivity analysis would be to examine performance of the model when the mrs is used to impute the GOS for patients where GOS data is readily available and determine how reliable mrs truly is for this imputation. It is stated that the development and validation cohorts are to some extent similar; however, there are no statistics to support this claim. WFNS grade, aneurysm size, and Fisher grade appear to possibly be different between these cohorts. Please provide statistical support for this claim. In the analysis aneurysm size was categorised as <12mm, 12 24mm, >25mm. However, the calculator http://sahitscore.com requires input of a continuous variable. Please explain why it is continuous on the calculator. Perhaps input of a categorical variable may be quicker and easier. Did any of the study cohort manage patients with revisions/rebleeds? If so, please list them. This analysis is comprehensive and outlines covariates that are important in the prognosis of patients with subarachnoid haemorrhage. To make the study more complete, I would suggest including results of an analysis on interactions of prognostic factors and report them in the supplementary material. Other clinical tools, such as the ICH score, adopted by the American Heart Association, output a numerical percentage that is easy to use and interpret in rapidly changing clinical situations. The authors should consider revising the output of their tool to an easily interpreted score instead of the three graphs presently given. There are more missing values in the external validation cohort (table 3) compared to the internal validation cohort (table 2). Despite using multiple imputation method which is widely accepted, please consider this as an additional limitation. Some other common covariates mentioned in the literature (e.g. your systematic review - reference 6) for subarachnoid haemorrhage include hyperglycaemia, smoking, presence of cardiac manifestations, and presence of neurogenic pulmonary oedema. Why were these not included as covariates for this model? We would suggest reporting only on covariates with the strongest prognostic effect in the Conclusion. The authors published previously (Jaja, et al. 2014 and Jaja, et al. 2015) prognostic models that did not evaluate all of the covariates studied in the present manuscript. While the present manuscript is more comprehensive than these past studies, we do not consider it appropriate to include an external validation within the same paper in which a tool is first reported. We recommend that the authors remove their external validation analyses as true external validation analyses would be those conducted by a separate group, following publication of the prognostic tool. We recommend publication of this study in BMJ once the authors have addressed the above concerns. This study is comprehensive, novel, and would be the first to
provide a reliable numerical scale to predict patient outcomes following SAH. This would have far reaching implications across multiple clinical disciplines, and serve to greatly improve patient care. Please enter your name: Saleh Almenawer Job Title: Neurosurgeon Institution: Hamilton Health Sciences and McMaster University Reimbursement for attending a symposium?: No A fee for speaking?: No A fee for organising education?: No Funds for research?: No Fees for consulting?: No </a>please declare them here: Reviewer: 4 The prediction model, based on the SAHIT data and externally validated using modern trials and series, is well written. The use of real life data from the repository is most relevant. The strengths and the limitations of the model are clearly demonstrated and addressed. The model would serve as a very useful and important tool for the emergency, acute medical, and neurosurgical clinicians. Please enter your name: William Lo Job Title: Fellow Institution: Hospital for Sick Children, Toronto Reimbursement for attending a symposium?: No A fee for speaking?: No
A fee for organising education?: No Funds for research?: No Fees for consulting?: No </a>please declare them here: No Reviewer: 5 The authors report their collaborative effort of developing a prediction model for outcome after aneurysmal subarachnoid haemorrhage. Based on individual patient data from 10936 patients derived from known clinical trials of asah they have established a set of three consecutively staged prediction models for outcome (GOS at 3 months) and mortality. The predictive value was externally validated using data sets from 3355 patients with asah from later clinical studies not included in the initial development of the predictions models. The authors found a reasonable predictive value for outcome at 3 months (AUC 0.81) using their "core" model that includes age, premorbid hypertension and WFNS scale. Including further neuro-imaging or treatment parameters slightly improved the predictive value on a statistical level. However, whether this is clinically meaningful in the all-day-setting remains open. They found similar results for their prediction models for mortality (AUC 0.79). Given the natural limitations of such an effort in this disease (which have been correctly addressed and outlined throughout the manuscript) the results are sound and address a highly important issue in the management of asah. The strength and practical value of this model (which can be easily used with the internet based tool) is based on the simple clinical data of the individual patient that is usually readily available at admission. Overall this is a well-written manuscript and the scientific and clinical community would clearly benefit of its publication. Minor points: Abstract: the abbreviation of SAHIT should get introduced Table 1 - bracket is missing (aneurysm size - line 24) Please enter your name: Nils O. Schmidt Job Title: Senior Neurosurgeon Institution: University Medical Center Hamburg-Eppendorf
Reimbursement for attending a symposium?: No A fee for speaking?: No A fee for organising education?: No Funds for research?: No Fees for consulting?: No </a>please declare them here: