A randomized controlled trial of league tables and control charts as aids to health service decision-making

Transcription

1 International Journal for Quality in Health Care 2004; Volume 16, Number 4: pp /intqhc/mzh054 A randomized controlled trial of league tables and control charts as aids to health service decision-making TOM MARSHALL, MOHAMMED A. MOHAMMED AND ANDREW ROUSE Public Health and Epidemiology, University of Birmingham, Birmingham, UK Abstract Objectives. Health service managers and other decision-makers are required to act on the basis of data. Little attention has been paid to the effects of data presentation on the decisions taken. This study uses a randomized controlled trial design to investigate the effects of two forms of data presentation league tables and control charts on health service decisionmakers. Methods. Directors of public health in 122 health authorities in the UK were mailed three case studies and a questionnaire. The case studies showed data on variations in mortality by health service provider. The questionnaire asked them to indicate whether they would take action as a result of the data and to identify the health service providers on whom they would take action. Participants were randomly allocated to receive the same data in the form of ranked histograms (league tables) or control charts. Main outcome measure. The percentage of participants who would take action on health service providers. Results. Fifty-seven questionnaires were returned. For each case study, respondents receiving data as league tables stated they would take action on significantly more health service providers than those receiving data as control charts: for the first case study, the percentages were 3.3% versus 1.8% (P < 0.001) for league tables and control charts, respectively; for the second case study, 15.9% versus 6.7% (P = 0.029), respectively; and for the third, 5.9% versus 0.7% (P = 0.002), respectively. Respondents receiving data as league tables were significantly more likely to request further information on case mix. Conclusions. Compared with league tables, health service decision-makers identify fewer outliers for further action when performance data are presented as control charts. They also reduce the tendency to request further information. Using control charts rather than league tables for the routine presentation of comparative data would reduce over-investigation of unusual performance. Keywords: decision-making, performance monitoring, statistical process control Health authorities, health service managers, and clinicians involved in clinical governance are responsible for maintaining and improving quality of care. One method of achieving this is through routine monitoring of health service performance. The aim of monitoring is to identify variations in performance between or within health service providers, and to use these variations in performance as a guide to appropriate improvement action. Action may be taken either to investigate isolated exceptional performance or to make improvements across the whole system. Data are often presented graphically to assist interpretation and to help guide action. Correct action depends on correct identification of the type of variation between health care providers. There is evidence that mode of presentation of results from trials influences decision-making of patients [1], physicians [2], and health service purchasers [3]. Health service performance data are often presented visually to aid interpretation. Nevertheless, little research has investigated the influence of the mode of presentation of health service performance indicators on decision-making. Despite concerns about their usefulness, league tables are the most commonly used graphical technique for presenting National Health Service (NHS) performance data [4]. League tables are a way of displaying comparative rankings of performance indicator scores of several similar providers. They are popular in consumer organizations and the popular press. League tables usually take the Address reprint requests to Tom Marshall, Public Health and Epidemiology, University of Birmingham, Birmingham, UK. t.p.marshall@bham.ac.uk International Journal for Quality in Health Care vol. 16 no. 4 International Society for Quality in Health Care and Oxford University Press 2004; all rights reserved 309

2 T. Marshall et al. form of ranked histograms of performance, with a separate bar for each provider. In some cases they include error bars representing 95% confidence intervals. For the NHS Executive, performance league tables have two purposes: firstly, to identify a few providers whose performance indicator scores are appreciably greater or lower than expected; and secondly, to show the range of variation between providers. The Department of Health s intention in publishing the league tables is to ensure that where there are large and unexplained variations in performance, every effort is made to find out why, and work is put in train to bring about an early improvement [5]. However, they are not the only way of graphically representing performance data. Control charts have also been advocated for performance monitoring in health care [6,7]. In the 1920s, a physicist and engineer Walter Shewhart studied variation from the perspective of a manager trying to improve quality. He concluded that two distinct kinds of variation should guide managers towards distinct kinds of action. The first kind of variation is that exhibited by a stable process. This variation is intrinsic to the process itself, and Shewhart called it common cause variation. In order to improve a stable process, the most efficient improvement action is to change the process fundamentally. The second kind of variation is caused by external factors acting on a process. Shewhart called this special cause variation. Special cause variation is a signal that something unusual is affecting performance. When we see special cause variation, we should investigate and identify the special cause. The special cause guides us as to what action is most likely to improve the process as a whole. Shewhart developed a simple graphical tool to help guide managers to take the appropriate action to improve quality: the control chart [8]. The control chart consists of three parallel lines: the average, an upper control limit, and a lower control limit. Data points between the control limits indicate common cause variation. Data points outside the control limits, and certain unusual patterns, indicate special cause variation. Control charts have found wide applications in the manufacturing industry [9,10] and have been shown to be potentially useful in the health service [6,11]. This study investigates the interpretation of health service performance data by Directors of Public Health in UK health authorities. Data are presented in two different forms: as traditional league tables and as Shewhart control charts. Directors of Public Health are usually physicians who have completed specialist training in public health. One of their roles is to advise health service managers on quality of services. There are many different types of control charts [12]. Two of the control charts used in this study are drawn using a methodology advocated by Deming for binomial data [9,13]. These charts are easily drawn on double square root paper (also known as probability paper), designed on the assumptions of the binomial distribution, first developed by Mostellor and Tukey [14]. They can also be easily produced using an Excel spreadsheet. The procedure for drawing the chart in Excel is relatively straightforward. Take the square root of the raw binomial data. Use the square root data to draw a scatter plot. The mean odds of the binomial outcome is represented by a straight line through (0,0) and (Σx,Σy). On double square root paper (or using square root transformed data) the standard deviation is usefully regarded as a constant 0.5 perpendicular distance from the mean. Three sigma control limits are therefore parallel lines 1.5 mm above and below the mean. Methods Participants were the public health directorates from each of the 122 health authorities (or health boards) in England, Scotland, and Wales. We sent each Director of Public Health a letter requesting their participation, performance data illustrated graphically, and a questionnaire asking them to comment on the enclosed performance data. Directors of Public Health were randomly allocated to receive performance data presented either as league tables or as control charts. We invited participants to pass the questionnaire to the most appropriate person in their department. The performance data consisted of three case studies showing outcome data from a number of service providers. These data were presented in one of two types of graph: league tables (ranked histograms with 95% confidence intervals) and control charts (Figure 1). For each type of graph we provided a brief explanation of the graphs and their interpretation (Box 1). The first case study apparently reported the results of an audit of 30-day mortality rates in 50 consecutive cases admitted for fractured hip under each of 26 surgeons. Participants were informed that nationally, the average 30-day mortality was 20%. In fact these data were generated from a simple experiment. Using a paddle designed to select 50 beads, T.M. drew 26 consecutive samples (with replacement) from a population consisting of 3000 white beads and 750 red beads. Red beads were reported as deaths and white beads as survivors. Any variation in mortality between the 26 surgeons was therefore a product of the mechanical sampling process, i.e. common cause variation. The reported 30-day mortality rate is approximately accurate [15]. The second case study reported 30-day mortality following admission for myocardial infarction at each of 37 very large acute-care hospitals. These data were obtained from NHS performance indicators for the years [16]. The third case study reported 30-day mortality following admission for myocardial infarction at each of 28 small- or medium-sized acute-care hospitals. These data were also obtained from the NHS performance indicators for the years For each case study, participants were asked whether they would take action as a result of the data, and if so to identify the service providers towards whom action would be directed. Supplementary questions asked participants to indicate what type of action they would take and to indicate on a four-point rating scale how easy or difficult they found this exercise. Space was provided for participants to add any further comments as free text. The primary outcome was the number of health service providers singled out for investigation with each mode of data presentation in each of the three case studies. We also 310

3 Data presentation and decision-making Case study Case study 1 Mortality rate Mortality in 50 consecutive cases Surgeon 0 Surgeon Case study 2 Case study 2 Case study 3 Square root (number of deaths) Mortality rate Square root (number of deaths) Hospital Square root (number alive) Case study 3 Mortality rate Hospital Square root (number alive) Actual graphs were larger and all bars and data points were clearly identified by number. Figure 1 Data presentations used in the survey: respondents randomized to league tables (ranked histograms) received graphs on the left, while those randomized to control charts received the data on the right (actual graphs were larger and all bars and data points were clearly identified by number). Box 1 Explanations of data presentations used in the survey: respondents randomized to league tables (ranked histograms) received the explanation on the left, while those randomized to control charts received the explanation on the right Explanation of ranked histograms The graph shows the percentage 30-day mortality rate. The bars are the 95% confidence interval for this result. Explanation of control charts The control chart has three lines: the central line is the mean and the upper and lower lines are termed control limits. Data points within the control limits suggest common cause variation (intrinsic to the process). No action should be taken on these individuals: improvement can only be achieved by improving the process as a whole. Data outside the control limits suggest special cause variation and should be investigated. report on the ease of interpretation of data with each mode of data presentation. Finally, we counted the number of free-text requests for additional information and categorized them by the main type of request. We then carried out a semi-quantitative analysis of the main number and type of information requests. We used SPSS 11.0 software for all statistical analysis. Results In the first case study, using a control chart analysis, no provider showed special cause variation. In the second case study, five out of 37 service providers showed special cause variation: mortality rates for data points 37 and 34 were 311

4 T. Marshall et al. Table 1 Numbers of health service providers singled out for investigation by respondents receiving performance data as league tables and control charts Case study Providers singled out with league tables (28 responses)... Providers singled out with control charts (29 responses)... Mann Whitney U-test (two-tailed) Median Mean (%) Total Median Mean (%) Total (3.3%) (1.8%) 26 P < (15.9%) (6.7%) 37 P = (5.9%) (0.7%) 28 P = higher than expected, and mortality rates for data points 1, 4, and 6 were lower than expected. In the third case study, one service provider showed special cause variation: data point 1 showed a lower than expected mortality rate. One hundred and twenty-two questionnaires were sent out, comprising 60 control charts and 62 league tables. Fiftyseven usable questionnaires were returned 29 control charts and 28 league tables to give an overall response rate of 47%. Some respondents did not answer all the questions. Primary outcome Respondents receiving data as league tables identified more of the health service providers for further investigation than those receiving data as control charts. In the first case study, respondents receiving league tables singled out a median of 1 (mean 0.9) and those receiving control charts a median of 0 (mean 0.5) of the 26 providers (P < using the Mann Whitney U-test). In the second case study, a median of 4 (mean 4.5) and 2 (mean 1.9) of 37 providers were singled out with league tables and control charts, respectively (P = using the Mann Whitney U-test). In the third case study, a median of 0 (mean 1.6) and 0 (mean 0.2) of 28 providers were singled out with league tables and control charts, respectively (P = using the Mann Whitney U-test) (Table 1). Secondary outcomes The great majority of respondents indicated that they found it easy or very easy to take action on the basis of league tables, whereas fewer than half indicated that they found it easy to take action on the basis of control charts. The mean ease of interpretation score in the league table group for the first, second, and third case studies was 1.9, and in the control chart group it was 2.4, 2.6, and 2.6, respectively (a higher score indicates more difficulty in interpretation). For each case study, the groups were significantly different according to the Mann Whitney U-test (Table 2). Free-text comments Overall, respondents randomized to league tables were more likely to add free-text comments requesting additional information. For the first case study, 24 (86%) and 12 (41%) respondents who received league tables and control charts, respectively, requested additional information (P = using the Mann Whitney U-test). For the second case study, 25 (89%) and 22 (76%) of those who received league tables and control charts requested additional information, respectively (P = using the Mann Whitney U-test). For the third case study, 21 (75%) and four (14%) of those who received league tables and control charts requested additional information, respectively (P < using the Mann Whitney U-test). This compared with 12, 22, and four of the 29 who received control charts. For the first and third case studies, these differences were significant according to the Kruskal Wallis test. The most common type of additional information requested was on case mix. This was also more common among those receiving league tables. For the first, second, and third case studies, 13, 10, and five of the 28 who received league tables requested additional information. This compared with two, four, and one of the 29 who received control charts. For all three case studies, these differences were significant using the Mann Whitney U-test (Table 2). Free-text comments centred on three major themes: elaboration of the kind of action to be taken on individual provider units; requests to check the accuracy of data; and requests for further data or further analysis. Some free-text comments were very non-specific, for example seek obvious reasons for low and high mortality and explore reasons for variation. Free-text comments indicated that respondents were more likely to identify and suggest taking action on outliers with high than with low mortality rates. Only two respondents, both commenting on case study 2, mentioned learning from providers with low mortality rates. A summary of the comments is provided in Table 3. Discussion This study demonstrates that compared with league tables, data presented in the form of control charts reduce the tendency to identify and take action on outliers ; however, this interpretation needs to be treated with some caution. In some respects, interpretation of performance data serves as a kind of diagnostic process. Like any diagnostic process or test, it is subject to error in the form of false 312

5 Data presentation and decision-making Table 2 Ease of interpretation score 1 per case study with control charts and league tables, and free-text comments Case study Ease of interpretation scores... League tables Control charts... 1 Higher score indicates more difficulty interpreting data.... Mann Whitney U-test (two-tailed) Mean score Total responses Mean score Total responses P = P < P < Case study Proportion of respondents adding a free-text comment... League tables Control charts Number (%) with Total responses Number (%) with Total responses a comment a comment Mann Whitney U-test (two-tailed) 1 24 (86%) (41%) 29 P = (89%) (76%) 29 P = (75%) 28 4 (14%) 29 P < Case study Proportion of respondents requesting data on case mix... League tables Control charts Number (%) Total responses Number (%) Total responses requesting case mix requesting case mix Mann Whitney U-test (two-tailed) 1 13 (46%) 28 2 (7%) 29 P < (36%) 28 4 (14%) 29 P = (18%) 28 1 (3%) 29 P = positives and false negatives. Diagnostic tests are typically assessed against a gold standard. However, there is no gold standard in the interpretation of performance data. Indeed the only true test of this is empirical do the resulting decisions result in improved quality of care? Nevertheless, the data used in the first case study were generated from a quasi-random process. It is reasonable to assume that under these circumstances, a health service decision-maker should not take action on any individual provider. But the results demonstrate that when quasi-random data are presented as a control chart it leads to more appropriate action than when they are presented as a league table. The latter mode of data presentation led respondents to spuriously identify outliers when there were none, i.e. false positives were generated. The finding that league tables increase the tendency to identify and take action on outliers is not entirely surprising. An implicit assumption of a league table is that there are performance differences between providers. The rankings generated by league tables are therefore a reflection of differences in performance: those at the bottom are in some way inferior to those at the top. Control charts make the opposite assumption. Unless there is evidence to the contrary, all providers are part of a single process, and variations in performance are attributable to the same process. The secondary findings are paradoxical. Decision-makers reported that it was easier to interpret league tables and yet were more inclined to ask for further information, in particular, information on case mix. They also (in the first case study) made worse decisions. The fact that decision-makers reported that it was easier to interpret league tables may simply reflect the fact that respondents are more familiar with them. The unusual format (square root scales) of two of the control charts chosen may also have made users less comfortable with their use. Training in the use of control charts would probably change this finding. Requests for additional information can be a means of delaying or avoiding difficult decisions. When patients are asked to choose between risk information presented in a variety of formats, it has been observed that they sometimes prefer formats that also lead them to less accurate perceptions of risk [17]. It may be that this finding has a parallel among health service decision-makers, with decision-makers preferring familiar modes of data presentation, even though they may not lead to better decision-making. It is of interest that respondents receiving both league tables and control charts tended to interpret their task in terms of identifying poor performance (bad apples) rather than exemplary performance. This study has two principal limitations. Firstly, the response rate was not high. Because the response rate was not 313

6 T. Marshall et al. Table 3 Summary of free-text comments made by respondents Comment League tables... Case Case Case study 1 study 2 study 3 Control charts... Case study 1 Case study 2 Case study 3. Any written comment Request information on case mix Investigate data quality Feedback to all providers Refer to clinical guidelines Discuss with individual provider and/or clinicians Ask management or clinical governance team to investigate Confidence intervals for mean Investigate care pathway Investigate training of health professionals Collect more data Learn from low mortality providers Reconsider whether provider should offer the service Respondent cannot understand graphs Investigate facilities and resources of provider Determine whether outlier is in my geographical area Includes any requests for information on age, sex, and socioeconomic characteristics of patients, severity of condition, and the presence of co-morbidities. 2 Includes any requests for information on length of stay, door to needle time, local protocols, or review of case notes. high, we cannot be certain that our respondents were typical of health service decision-makers. As the survey was completely anonymous we were unable to identify the characteristics of responders and non-responders. However, despite this, the findings are not biased because the response rate does not seem to have systematically favoured either the league table or the control chart groups. Secondly, the study reports the actions decision-makers say they would take rather than their actual action. This is a problem with any survey research. However, we can at least infer that different modes of data presentation result in different intentions. Conclusion Mode of data presentation affects health service decisionmaking. Compared with league tables, health service decisionmakers identify fewer outliers for further action when performance data are presented as control charts. This seems to be the case even without previous training in their interpretation. Further research should investigate the influence of mode of data presentation on managerial decision-making. References 1. Edwards AGK, Elwyn G, Mathews E, Pill R. Presenting risk information a review of the effects of framing and other manipulations on patient outcomes. J Health Commun 2001; 6: Cranney M, Walley T. Same information, different decisions: the influence of evidence on the management of hypertension in the elderly. Br J Gen Pract 1996; 46: Fahey T, Griffiths S, Peters TJ. Evidence based purchasing: understanding results of clinical trials and systematic reviews. Br Med J 1995; 311: Adab P, Rouse AM, Mohammed MA, Marshall T. Performance league tables: the NHS deserves better. Br Med J 2002; 324: NHS Executive. HSC 2000/023. Quality and performance in the NHS. Performance indicators: July Leeds: Department of Health, Tekkis PP, McCulloch P, Steger AC, Benjamin IS, Poloniecki JD. Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. Br Med J 2003; 326: Lim TO. Statistical process control tools for monitoring clinical performance. Int J Qual Health Care 2003; 15: Shewhart WA. Economic Control of Quality of Manufactured Product. New York: D. Van Nostrand Company, 1931 (reprint in 1980 by ASQC Quality Press). 9. Deming WE. Out of the Crisis. Massachusetts Institute of Technology, Deming WE. The New Economics, 2nd edition. Massachusetts Institute of Technology,

7 Data presentation and decision-making 11. Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart s forgotten lessons. Lancet 2001; 357: Wheeler D. Advanced Topics in Statistical Process Control. The Power of Shewhart s Charts. United States: SPC press, Inc., Deming WE. Sample Design in Business Research. Wiley Classics Library Edition, Mostellor F, Tukey J. The uses and usefulness of probability paper. J Am Stat Assoc 1949; 44: Todd CJ, Palmer C, Camilleri-Ferrante C et al. Differences in mortality after fracture of hip: the East Anglian audit. Br Med J 1995; 310: Department of Health. Hospital Episode Statistics. England: Department of Health, 1999/ Lipkus IM, Hollands JG. The visual communication of risk. J Natl Cancer Inst Monogr 1999; 25: Accepted for publication 25 March

8