SPARRA Mental Disorder: Scottish Patients at Risk of Readmission and Admission (to psychiatric hospitals or units)

SPARRA Mental Disorder: Scottish Patients at Risk of Readmission and Admission (to psychiatric hospitals or units) A report on the work to identify patients at greatest risk of readmission and admission to psychiatric hospitals or units in Scotland. Mental Health Programme Information Services Division October 2008 Version history and revisions: Version Date Revision Actioned 1.0 18 th August 2008 Final Report Scott Buchanan 1.1 30 th August 2008 Back ground added 1.2 30 th September Introduction 2008 added. Scott Buchanan Linda Reid/Scott Buchanan 1

Contents Executive Summary 3 Page 1. Introduction 4 2. Background 5 3. Methodology 6 4. Results 9 5. Future Developments 16 2

Executive Summary Scottish Patients at Risk of Readmission Mental Disorder (SPARRA MD) is a risk prediction algorithm, developed by the Information Services Division (ISD) to identify patients aged 15 years and over at risk of readmission and admission to a psychiatric hospital or unit. This meets the additional commitments laid out in Better Health, Better care. The algorithm uses information from patient histories from the SMR04 psychiatric admissions database to predict their future likelihood of admission. The algorithm will only select patients who have experienced at least one admission to a psychiatric hospital or unit in the previous 3 years. The feedback of the information generated by the algorithm will take two forms. Firstly, aggregate risk distribution information which shows at NHS Board or Community Health Partnership (CHP) how many patients are at given levels of risk of future psychiatric admission. This will help with local strategic planning in the healthcare provision. Secondly, information on individual patient records identified by SPARRA can be made available to review and access patients, in order to decide which patients may benefit from preventive coordinated care in the community. There were approx 36,500 patients admitted to psychiatric hospitals or units between 1 st April 2003 to 31 st March 2006. There were a number of independent variables calculated and tested for this cohort via logistic regression. Six independent variables were significant and included in the final predictive model (further details found within the report). ISD approached a number of NHS Boards to pilot these data. This involved considering how useful are the data, how many of the patients were known by local systems, how ISD could improve the information provided. Further developments of the SPARRA MD model will continue including investigating the feasibility of including prescribing information. 3

1 Introduction Published commitments and established targets introduced and endorsed by the Scottish Government combine to address the need for change and improvement across the spectrum of information; planning; commissioning; service delivery; quality of care; and all other relevant aspects of mental health in Scotland. Attention is focussed on 14 timetabled commitments and (currently) 4 Health Efficiency Access and Treatment (HEAT) targets for NHS Boards in Scotland but involving partnership approaches for delivery. The timetabled commitments include attention to improving equality, social inclusion, recovery and rights of people with mental health problems. There is attention also on improving responses to and earlier intervention for depression, crisis prevention and response and on psychological therapy services. A focus is also underway on improving mental health services for children and young people and on better and earlier attention to the physical health of people with mental health problems. Standards for integrated care pathways for 5 key mental health conditions have been published and compliance is progressing. Action on compliance for the key conditions covered is informing quality and other dimensions of care for all mental health services. The current HEAT targets, address desired reductions in antidepressant prescribing, readmissions to hospital and suicide. The targets also cover structured ambition for increase in the training on suicide prevention and other aspects for frontline care staff; and there is a defined target for improvements in early diagnosis and management of those with dementia. The readmissions target drives the attention and ambition set out in this report. In full the target states: We will reduce the number of readmissions (within one year) for those that have had a hospital admission of over 7 days by 10% by the end of December 2009. This high level target and the wider attention to improvements overall, including not least, attention to crisis prevention, care and recovery combine to support and complement the objectives set out in this document for identifying patients at high risk of readmission to a psychiatric hospital or unit. 4

2 Background Identifying patients at high risk of admission to an acute hospital due to a mental disorder: the complementary model. SPARRA Mental Disorder, the risk prediction algorithm described in this report, applies to admissions to psychiatric hospitals and units. Only such admissions (recorded on the SMR04 record) are involved either in providing predictive information or as outcomes. However this model was originally developed as one of a pair. The other algorithm involved - admissions to acute hospitals due to a mental disorder (as recorded on the SMR01 record). In this complementary model, the definition of mental disorder was deliberately kept broad. Any person with any mention in the three years prior to the outcome year of a mental or behavioural disorder as defined by ICD-10 was included. The outcome was any emergency admission with any mention of these diagnoses. There were two reasons for developing two models. The first was pragmatic to avoid the complexities and delays involved in merging the analysis of two different data sets with different conventions and structures. The second was a hypothesis that the two models would identify two different populations. The second supposition has proved to be the case. The mental disorder model identifies a pool of around 3,500 people in the community who are at high risk (greater than 50% in one year) of admission to an acute hospital due to a mental disorder. Only 174 of these individuals were also at high risk of inpatient admission to a psychiatric hospital or unit. The alternative model is thus identifying a sizeable cohort who do not require admission to a psychiatric hospital or unit but pose severe challenges to the system of general acute care and are not necessarily receiving the preventive care which they need. An individual picked at random from the high risk category had a 79% chance of admission, six or more admissions due to a mental disorder in the previous three years involving 19 bed days and diagnoses relating to self-harm, alcohol and substance misuse. Why did we not develop the alternative model further? At the same time as the two versions of SPARRA MD were being developed, the age range of classic SPARRA was being extended. Classic SPARRA calculates the risk of any emergency inpatient admission and originally applied only to people aged 65 and over. In recent months however, the scope of classic SPARRA has been extended to all ages it therefore identifies everyone who is at high risk of future acute hospital admission on the basis of previous hospital admissions. In order to identify patients at high risk of admission to an acute hospital due to mental disorder, a selection of the relevant patients should be made from the all ages SPARRA high risk group. The SPARRA team are setting up procedures to enable these lists to be provided on request. 5

3 Methodology The SPARRA Mental Disorder risk prediction algorithm was developed using logistic regression on recent SMR04 (psychiatric inpatients) data. The cohort of patients entering this model was anyone in the Scottish population who had experienced at least one admission to a psychiatric hospital or unit in the period 1 st April 2003 to 31 st March 2006, and was aged 15 and over. The outcome for the model was whether these patients were readmitted or admitted in the outcome year 1 st April 2006 to 31 st March 2007. This outcome year was chosen because the outcome for these patients would be known, and the model was being developed on the most complete data. 1 st April 2006 2003/04 2004/05 2005/06 2006/07 Predictor variables Outcome year The data used to develop the risk prediction algorithm were the linked set of SMR04 psychiatric admissions and death records held at ISD Scotland. These records are routinely linked using probability matching based on a range of personal identifiers such as name, date of birth and hospital case reference number. This allows us to identify patients that may have died following discharge from hospital; therefore any patient who had died before the index date of the prediction period was excluded from the model. Brief description of variables included in analysis Age The age of the patient was calculated as at the index data i.e. 1 st those patients aged 15 and over were included in the analysis. April 2006. Only Gender No description needed. Number of previous psychiatric admissions This is the total number of inpatient admissions to a psychiatric hospital or unit in the previous three years from the index data i.e. 1 st April 2003 to 31 st March 2006. 6

Time since most recent admissions This is the time calculated in days and months from the index date (1 st April 2006) to the date of their most recent psychiatric admission. Total number of bed days Total number of bed days accumulated in the previous three years to the index date. Scottish Index of Multiple Deprivation (SIMD) Each patient was assigned the Scottish Index of Multiple Deprivation based on their home postcode on their last psychiatric admission. The deprivation measure is based on deciles ranging from 1 (most affluent) to 10 (most deprived). NHS Board of Residence Patient s NHS Board of residence. Urban/Rural Patients were assigned an urban or rural flag based on their data zone on their last psychiatric admission. Diagnosis group This shows the principal diagnosis recorded at the most recent psychiatric admission. The diagnoses are grouped into twelve broad groups shown below in Table 1. These groups were recommended by NHS Lothian, where they are already in use. Number of different diagnosis groups This is a count of the number of different diagnostic groups each patient has recorded in their 3-year history. Each group is only counted once and is a good measure of the complexity of the patient s diagnosis history. Formal admissions The total number of formal admissions (mental health act) the patient has experienced in the 3-year period. Severe admissions The total number of severe admissions (patients with hospitals stays greater than 7 days) the patient has experienced in the 3-year period. Outcome Whether or not the patient is admitted to a psychiatric hospital or unit in the outcome year i.e. 1 st April 2006 to 31 st March 2007. 7

Logistic Regression The aim of the model is to calculate, using SPSS, the likelihood that a given patient is admitted to a psychiatric hospital or unit in a given time period. Logistic regression is a statistical technique that uses a range of independent variables (listed above) and identifies those variables that have a significant independent effect on the binary outcome of the model. Table 1 Diagnostic Groups Organic, including symptomatic, mental disorders Mental and behavioural disorders due to psychoactive substance use Schizophrenia, schizotypal and delusional disorders Mood (affective) Disorder Bipolar affective disorder Neurotic, stress-related and somatoform disorders Behavioural syndromes associated with physiological disturbances and physical factors Disorder of adult personality and behaviour Mental retardation Disorder of psychological development Behavioural and emotional disorders with onset usually occurring in childhood and adolescence Other ICD 10 codes F00-F09 & G30 F10 F20-F29 F30, F32-F39 F31 F40-F49 F50-F59 F60-F69 F70-F79 F80-F89 F90-F98 Non Mental or Behavioural disorder diagnoses Specifying the model Specifying the model is the phase of the analysis that identifies which of the variables has a significant independent effect on the outcome. Firstly, forward stepwise selection was used to identify independent variables to be included in the model. This method identifies which variable on their own has the greatest predictive power. The remaining variables are then tested and successively added until none of the remaining variables are adding any predictive power. The model was tested on a number of different outcome years before finalising the model on the 2006/07-outcome year. Specification of the model was also tested on multiple random 10% samples of the cohort to assess the stability and determine whether the same set of independent variables were being identified from the random samples. Interaction effects between the independent variables were also looked for. These occur when the effect of one variable depends on the effect or the value of another. There were a number of combinations tested, but there were no obvious effects occurring. 8

Results Effects of the independent variables Table 2 below shows the odds ratios for the main effects of the variables that emerged from the model. The odds ratios show how much the probability of a psychiatric admission for a patient in a given category of the independent variable, compares with probability for someone in the reference category of that variable. For example, we can look at the effect of the previous number of psychiatric admissions. The odds ratio for the reference category, one previous psychiatric admission, is set at 1.00. Patients experiencing 3 admissions are 1.83 times more likely to be admitted and patients experiencing 6 or more admissions are 4.20 times more likely to be admitted. Table 2 Independent Variable Cases Odds Ratio Age group 15-19 (Reference) 724 1.000 20-24 2162 1.066 25-29 2820 0.959 30-34 3566 0.924 35-39 4143 1.080 40-44 4406 1.033 45-49 3678 1.048 50-54 2810 0.971 55-59 2333 0.923 60-64 1620 0.960 65-69 1536 0.895 70-74 1537 0.866 75-79 1742 0.746 80-84 1676 0.599 85-89 1093 0.598 90+ 566 0.382 Previous Psychiatric admissions One (Reference) 23,975 1.000 Two 6750 1.629 Three 2579 1.830 Four 1255 2.159 Five 659 3.034 Six or more 1194 4.209 Total bed days Between 0-10 days (Reference) 9814 1.000 Between 11-31 days 9525 1.322 Between 32-50 days 4226 1.569 Between 51-100 days 5469 1.768 Greater than 100 days 7378 1.977 9

Time since most recent Psychiatric admission Less than 1 Month (Reference) 1776 1.000 Between 1-2 Months 1630 0.691 Between 2-3 Months 1544 0.567 Between 3-4 Months 1238 0.549 Between 4-5 Months 1425 0.529 Between 5-8 Months 3711 0.396 Between 8-12 Months 4334 0.319 Between 1-2 Years 12048 0.226 Between 2-3 Years 8706 0.155 Principal diagnosis Organic, including symptomatic, mental 4133 0.681 disorders Mental and behavioural disorders due to 7478 1.000 psychoactive substance use Schizophrenia, schizotypal and delusional 6956 1.134 disorders Mood (affective) Disorder 8526 0.756 Bipolar affective disorder 2671 1.095 Neurotic, stress-related and somatoform 2904 0.581 disorders Behavioural syndromes associated with 191 0.733 physiological disturbances and physical factors Disorder of adult personality and behaviour 1338 1.152 Mental retardation 495 0.605 Disorder of psychological development 72 0.585 Behavioural and emotional disorders with onset 40 0.151 usually occurring in childhood and adolescence Other 1602 0.743 Rurality Urban (Reference) 31,502 1.000 Rural 4910 0.895 Summary of effects of independent variables There are 6 main predictor variables. By far, the most powerful effect is that of the number of previous psychiatric admissions. The next influential variable is the time since most recent admission there is a clear linear decline in effect as the time gets further away from reference category. Other reasonable sized effects are diagnosis, with Schizophrenia and Bipolar Disorder standing out as the strongest predictors of future psychiatric admissions. Bed days, age and rurality have moderate effects. The variables that were not significant in predicting future psychiatric admissions were: Number of formal and severe admissions, gender, NHS Board of residence, Scottish Index of Multiple Deprivation and the number of different mental disorder diagnoses. 10

Cohort size and distribution of risk scores Figure 2 shows the distribution of predicted probabilities for the full SPARRA Mental Disorder cohort of approx 36,500 patients. 1,770 individuals or 4.9% had a predicted probability of psychiatric admission. A similar size of cohort, of 1,528 patients, or 4.1%, scored between 40-50%. Figure 2 Distribution of Predicted Probabilities for SPARRA cohort for 2006/07 outcome 16000 14000 13510 12000 11682 Number in category 10000 8000 6000 4000 2000 5256 2666 1528 911 567 292 Series1 0 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70%+ Risk score Examples Table 3 shows three examples selected at random to show the characteristics the individuals from different risk categories. Table 3 Very High prob Medium prob Low prob (%) (%) (%) Case A Case B Case C Predicted admission 73% 52% 20% probability Age 15-19 60-64 35-39 Number of Psychiatric 6 or more 3 admissions 1 admission admissions Time since last Between 1-2 Between 1-2 2-3 Months admission months months Diagnostic group Disorder of personality and Schizophrenia Mood affective Disorder behaviour Bed days Greater than 100 51-100 11-31 days Rural Urban Urban Urban 11

Performance of the model The simplest way to gauge the performance of the model is to compare the risk prediction probabilities against the outcome of the model. In other words, comparing what the model predicted against the actual outcome. Figure 3 shows the distribution of predicted probabilities against the outcome. This shows the numbers of patients in each of the predicted risk categories against whether they were admitted in 06/07 or not. The model performs well especially in the higher risk scores i.e. 50% and above. Figure 3 Risk categories split by whether patient is admitted in outcome year or not 14000 12000 12586 10000 9970 Number in category 8000 6000 Not admitted Admitted 4000 3973 2000 0 1712 1736 1283 924 930 917 611 399 512 191 376 72 220 Under 10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70% plus Risk score Table 3 shows for each risk category, the percentage of patients that were admitted in the outcome year. A similar pattern emerges across the three selected years with a strong correlation between the percentage admitted and the risk category. This suggests that the model is stable and performs well predicting those patients most likely to be readmitted, especially in the higher risk categories. The model was applied to these independent years to reduce the bias of the model being developed and its performance measured on the same year 06/07. Table 3 % admitted 2006/07 outcome year % admitted 2005/06 outcome year % admitted 2002/03 outcome year Under 10% 6.8 7.7 8.1 10-20% 14.7 15.1 16.1 20-30% 24.4 24.4 26.1 30-40% 34.9 35.5 36.7 40-50% 40.0 43.7 46.2 50-60% 56.2 52.1 56.8 60-70% 66.3 67.2 72.7 70 plus 75.3 77.5 77.4 12

Another common used technique to assess the performance of the model is to calculate the area under the Receiver Operator curve (ROC). Figure 4 shows the ROC curve for the full SPARRA Mental Disorder pilot. The area under the curve measures the accuracy of the predictive technique. In a perfect predictive model, the curve would run straight up the left hand axis and straight along the top of the graph and the area under the curve would be 1. In a model where there is no predictive power at all, the curve is equal to the diagonal (green line) and the area under the curve is equal to 0.5. Area Under Curve = 0.744 The SPARRA Mental disorder model has an area under the curve of 0.75. This shows that the model has an acceptable predictive power. It is important to recognise that the ROC is a method of describing and quantifying how the predictive technique performs across the entire range of predicted probabilities i.e. 0-100%. Therefore, the success in predicting patients with low probabilities of admission contributes to the shape of the ROC curve as much as the success in predicting which patients have a high risk of admission. For the purposes of helping to identify patients that could benefit most from intensive case management, we are primarily interested in how good the model is at predicting these high-risk patients. 13

Feedback of results NHS Boards and CHPs wishing to identify patients at risk of psychiatric admission can access data in two main forms: aggregate risk distributions or identifiable patient records. Aggregate risk distributions For strategic and planning services, aggregate data may be useful to show how many patients in a given NHS Board or CHP are at various levels of risk of future psychiatric admission. For example, Figure 5 and Figure 6 below show these data at Scotland level by age group and number of previous admissions. Figure 5 Age group make up of SPARRA risk categories 100% Scotland 90% 80% 70% Age Group Percentage (%) 60% 50% 40% 30% 65+ 45-64 15-44 20% 10% 0% 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90% and over Risk Probability Group Total 14

Figure 6 Number of Admissions by SPARRA Mental Disorders risk categories Scotland 100% 90% 80% Percentage (%) 70% 60% 50% 40% 30% Number of Admissions 6 or more admissions 5 admissions 4 admissions 3 admissions 2 admissions 1 admission 20% 10% 0% 0-10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% 70-80% 80-90% 90% and over Risk Probability Group These graphs enable NHS Boards or CHPs to build profiles of the type of patients who are at highest risk of psychiatric readmission. Identifiable details of patients ISD Scotland will send to local team s identifiable details of patients at risk of psychiatric admission. Details will include identifying fields such as names, date of birth, hospital case reference number, postcode and CHI. There are a few steps that need to be considered before the release of this data. Confidentiality procedures ISD Scotland is returning identifiable information to NHS Boards about their residents. These data essentially belong to NHS Boards. Release of data will require the completion of a confidentiality form by the NHS Board/CHP concerned. Completeness of historic SMR04 data The benefit of SPARRA Mental Disorder will only be realised if the data feeds from NHS boards are sufficiently accurate and timely. There have been significant improvements in data submission over recent months but there is still room for improvement. Currently only five mainland NHS Boards are estimated to be complete up until December 2007. However, SPARRA MD may act as a catalyst for improved data submission. 15

As the data used to calculate predicted risks in some NHS Boards are likely to be a few months out of date, there are a few issues that need to be addressed. For example, if the admissions data for a given NHS Board were complete up to the end of March 2008, then we could predict admissions for the period 1 st April 2008 to 31 st March 2009. This means that if a local team were to receive these data in July, a number of months of the prediction period will have passed. This means that the: 1. The patient may have died these records should always be checked against local systems. 2. The patient may have already been admitted. Further developments Monitoring of the benefits and use of SPARRA Mental Disorders will be crucial in maximising its potential. This monitoring will help assess whether the tool is useful in supporting the HEAT target to reduce psychiatric readmissions. The Scottish Government has provided a monitoring form to each NHS board s Director of Public Health, and this should be forwarded on to a designated contact within the board to receive the data. The form is in the style of a questionnaire, and we would require its return no later than the 12 th January 2009. ISD will also visit some participating NHS Boards in its evaluation of the SPARRA tool. In terms of future developments, the feasibility of including prescribing information in the algorithm for the SPARRA Mental Disorders model will be investigated. 16