Performance Dashboard for the Substance Abuse Treatment System in Los Angeles Abdullah Alibrahim, Shinyi Wu, PhD and Erick Guerrero PhD Epstein Industrial and Systems Engineering University of Southern California Los Angeles, CA, 90089 Abstract In the United States, alcohol and substance abuse ensued costs are estimated to be $328 billion each year from numerous inflictions including illness, deaths, and accidents. The effectiveness of substance abuse treatment programs to reduce substance use plays a critical role in achieving treatment goals, achieving successful completion and controlling annual costs. Survey data on treatment episodes (2006-2011) from 423 programs partially funded by the Los Angeles County Department of Public Health, Substance Abuse Prevention and Control (SAPC) treatment episodes was analyzed. The availability of client data from a large number of programs called for the need to develop a multidimensional data visualization tool to represent the data meaningfully and readily for providers. A performance dashboard was created that utilized data visualization approaches to facilitate strategic performance monitoring, analysis and management using survey data. The dashboard was designed in accordance to human factors principles to allow for visual monitoring of program key performance indicators and program attributes. It enables decision makers to perceptively spot high and low performers in the county treatment system and compare programs and groups for a better understanding of program standing and yearly performance. The dashboard reports on an aggregated LA County level, a predetermined influential program group and individual levels. Key performance indicators included treatment episode duration and completion rates. Program attributes included patient referral source, race, patients suffering from homeless, mental health conditions patients. Key performance indicators were aggregated for each attribute on the aforementioned levels. Initial feedback from county officials on the acceptability and readability was provided and included in the discussion and implications section. Keywords Performance Dashboard, Data Visualization, Substance Abuse, Alcohol Abuse 1. Introduction In the United States, alcohol and substance abuse costs an estimated $328 billion each year from numerous inflictions including illness, deaths, and accidents. It is estimated that substance and alcohol abuse adversely affect an estimated 23 million people throughout the United States [1]. Regretfully, drug overdose is the fourth leading cause of premature death and the 17th leading cause of death overall [2]. Substance Abuse Prevention and Control (SAPC), a division of the Los Angeles County Department of Public Health, plays a critical role in controlling annual costs and improving case outcomes through regulation and monitoring service delivery at substance abuse treatment (SAT) programs. It is believed that substance abuse is a chronic condition that requires constant maintenance and prevention. Therefore, SAPC provides and array of Alcohol and Other Drugs prevention, treatment and recovery programs and services for County residents with their $200 million annual budget. More than 60,000 patients were admitted to publically funded Los Angeles County SAPC programs during the fiscal year 2009-10. The SAPC department has evolved over decades relying on federal and State funding. One of the main
challenges to improve standards of care for a treatment system is to have the proper data drive monitoring systems. This paper describes the developmental process of a performance dashboard to assist in the monitoring of performance of approximately 408 publicly funded treatment programs in Los Angeles County. In Performance Dashboards: Measuring, Monitoring, and Managing Your Business, Eckerson describes a performance dashboard as a tool that provides timely information and insights that enable business users to improve decisions, optimize processes and plans, and work proactively [3]. Essentially, a performance dashboard is data visualization tool that allows users to infer the status of an operational entity, in this case a substance abuse treatment program, at a glance based on predetermined performance indicators. Substance abuse treatment programs main goal is to support their clients recovery from substances. When clients fulfill their treatment plan of recovery they successfully complete that portion of their treatment. Although addictions are considered a chronic disease requiring maintenance treatment, using treatment episodes is an adequate way to measure incremental success. Prior research has indicated that the length of the treatment episode is directly related to improved outcomes after completion, and the optimal length of treatment episode was around 3 months, or 90 days [4]. So it is critical to add these performance indicators in a performance dashboard tool to inform exposure to treatment interventions and likelihood to complete treatment goals successfully. 2. Substance Abuse Prevention and Control Treatment Programs LA county SAPC provides a myriad of treatment programs intended to offer a continuum of preventative, treatment and recovery services for all different cases. The ethnic, age and socioeconomic diversity in the population demand a wide range of treatment programs to offer equal access and improved outcomes throughout. LA county SAPC treatment programs work with diverse populations, varying resources, and ultimately have different expected outcomes. The treatment programs offered by SAPC include Youth and Family programs, Youth and Family Criminal Justice programs, and Adult Criminal Justice Probation Programs. From a systems perspective, treatment programs vary in inputs; populations, service types, and resources, and therefore have different expected outputs. It is therefore necessary to examine data based on different type of categories (e.g. population, service type, resources) to be able to increase comparability in terms of performance indicators and develop tailored interventions. The current study relies on data from 2006 to 2011. All clients entering programs funded by LA County SAPC are required to fill an intake and discharge survey with demographic and clinical information. The survey contains 140 questions ranging from name and personal info to type of substance abuse and usage. The data set contains patient level data from 2006 to 2011 for each treatment episode in SAPC treatment programs. Of the 140 variables, we pilot tested first with a smaller but clinically meaningful set of variables. See table 1 below. Table 1. Variables in Initial Subset Facility ID Treatment Completion Patient ID Discharge Status Race Mental Status Referral Source Treatment Duration Homeless Status Year completed
The data was aggregated from 147,531 treatment episodes into treatment programs and yielded a total of 406 treatment programs. Of the 423 treatment programs, the largest 200 were selected because preliminary analysis showed that the smaller 223 treatment programs had negligible influence on the performance of the county, as they serve less than 6% of the total county treatment episodes. After consulting with the project s advisor, the remaining 200 treatment programs were categories based on the total number of treatment episodes into 3 categories; small, medium and large, each containing 66 treatment programs. The largest 5 and largest 10 were grouped into two additional categories accordingly, as they had the most influence on the overall county performance and are intuitively of great strategic importance. The largest 5 facilities account for 28% of the total treatment episodes, and the largest 10 facilities account for 37% of the total treatment episodes. For each treatment program group mentioned above, the following key performance measures were aggregated in total for all patients within a group of treatment programs as well as each risk factor within a treatment program. Table 2 shows the categorization of each aggregated variable. Table 2. Aggregate Key Performance Indicators and Risk Factors Variable Type Completion Rate Performance Indicator Treatment Episode Duration Performance Indicator Discharge Status Performance Indicator Race Composition Risk Factor Referral Source Composition Risk Factor Homeless Composition Risk Factor Mental Illness Composition Risk Factor 3. Dashboard Development The objective of this project is to develop an intuitive data visualization tool to meaningfully represent survey data that will aid in effective monitoring, analysis, and management of treatment programs. Monitoring would require decision makers to track key metrics in determining the overall performance of treatment programs. The dashboard would allow for viewing and comparing the performance of treatment programs based on their service type and size. Analysis of treatment programs would require stakeholders to view performance of past years to identify trends or possible outliers and their causes. Management of treatment programs would require a user to view overall performance and the implications of population composition of treatment programs patients and aid in the relevant decision-making. 3.1 Design Principles The design of the proposed dashboard design was based on the following display principles for visual monitoring [5]: Keep boundaries of dashboard to a single screen. Supply adequate context for the data. Choose appropriate display media. Highlight what is important. Avoid clutter with useless decoration.
Avoid misuse or overuse color. 3.2 Functional Components Table 1 shows the six main components around which the dashboard design was proposed Table 1 Dashboard Functional Components Input Component Output User s Needs and Interest Personalization Customized grouping of treatment programs based on size and service type Low and High Level Data Charting and Reporting Optional detailed view of population composition and performance User s Needs and Program Performance Requirements Alerts Visual out-of-range notification Personalized Alerts Exception Management Past performance visible to identify trends and exceptions Pertinent Material & Information Resources List best performers Strategic Objectives Goal Setting Allow users to compare current performance to user set goals and performance limits The dashboard is designed to allow the user to extract the desired available information and aid in informed decision-making. To allow for such uses, the dashboard is capable of comparing treatment program groups based on pre-determined grouping for size and two different service types; Outpatient and Inpatient programs. The dashboard is also capable of producing year-specific aggregates for all performance measures. The type of treatment service influences the expected performance in that outpatient episodes are expected to have longer treatment episode duration and lower treatment completion rates. Furthermore, treatment programs with more treatment episodes are expected to have more resources and higher completion rates. The proposed dashboard design allows for performing test to confirm the previous hypotheses. Next, the dashboard would have the ability to show lower level metrics of the user-selected treatment program group and service type. The lower level data would include performance measures visuals with respect to the specific population group and their composition. For example, the user will be able to see racial composition of the population and completion rates based on each race category; 28% of the episodes are white patients and they have a 35% completion rate. Lower level metrics should be visible and comparable to other user-selected groups on the same screen. The dashboard would accept user-determined goals and acceptable levels for completion rates and average treatment episode duration. The resulting key performance measures of the selected group and service type will be represented in the context of the user-set critical levels and goals. Treatment program groups performing outside the critical performance level would be visually identifiable. It also allows for a dashboard that adapts to changing strategic goals. In addition, best performing treatment program within the selected group will be shown in the dashboard to facilitate the process of learning from best performers.
3.3 Capabilities and Uses According to Eckerson s framework [3], there are three types of dashboards: operational, tactical and strategic [source]. Operational dashboards are used by frontline workers to monitor and control operational processes. Managers use tactical dashboards to overlook an organizational unit s performance. Strategic dashboards are used by higher executives to make sure strategic objectives are met. The components and capabilities of the proposed dashboard design allow it to be both a strategic and tactical dashboard. Executives and county decision makers can use the dashboard to make sure the performance and yearly trends align with the strategic objectives. Strategically, decision makers will also observe the effectiveness and results of decisions they make. It would also be possible to compare different program sizes and service types to accommodate more patients and to aid in resource allocation. Furthermore, local program managers can utilize the dashboard to have a benchmark tool to see how they stand in comparison to similar treatment programs. It would allow them to learn from best practices and identify potential areas of improvement with respect to gearing their services to their population composition and referral sources. So in a strategic and tactical sense, the dashboard would allow managers and county executives to monitor, analyze and manage their treatment programs. 4. From Concept to Actualization The dashboard uses data visualization to aid in strategic and tactical monitoring, analysis and management of treatment programs in Los Angeles County Substance Abuse Prevention and Control. The performance measures listed in section 2 were represented in the dashboard design in accordance with the customization and design principles mentioned in section 3. The framework in figure 1 shows the approach taken for visual management of treatment programs. Figure 1. Framework for Managing Treatment Programs in SAPC The framework explains how the dashboard would fit in the treatment program management system. The dashboard will provide the means to visually represent the end components of the above framework. Treatment programs are operating entities within SAPC s treatment program management system. In that system, performance goals are critical to ensure that the system performing above desirable levels to achieve strategic goals. Strategic goals involve increasing treatment effectiveness, quality and completion within risk factors. Aggregated outcomes of treatment programs placed in context of strategic performance goals makes the dashboard a tool to infer the status of the treatment programs with regards to achieving strategic goals. Strategic goals also encompass delivering more effective treatment episodes with to patients with risk factors. For example, SAPC would likely aim to deliver higher completion rates among patients with a mental health condition. Operational management involves the performance of individual treatment programs. A benchmarking approach would allow a treatment program or a group of programs to compare its performance to others of similar resources or itself in another year.
Benchmarking is a critical tool that allows for learning from best practices, developing standards, and identifying areas of improvement. With such capabilities, the dashboard was designed to fit within SAPC s treatment program management system. Microsoft Excel was used to make the proposed dashboard design. The dashboard automatically populates the charts and graphics based on the user-selected options in the dropdown list at the top of the dashboard. 4.1 Basic View Viewing the performance of a performing entity is one of the basic functions of this dashboard. The meaning of the values of the performance indicators, however, is in comparing it with relevant groups. Therefore, the dashboard is separated into two symmetric regions, shaded differently, that will allow for the comparison of two different treatment program groups. See Figure 2 for a screenshot of the layout. The first step of using the dashboard involves selecting the type treatment programs: Outpatient only or All. Then the user selects a group of programs to view (County, largest 5, largest 10, large, medium and small programs) and the year. The selections are to be made from preset drop down lists with appropriate labeling. Then the user can choose the comparison group from another identical dropdown list. The dropdown lists are labeled 1 in figure 2 below. Table 3 below lists the set of possible inputs the user can sets for the dashboard to populate the relevant graphics. Table 3. Dashboard User Inputs Label Possible Selections Year 2006 2007 2008 2009 2010 2011 All Years Service Type Outpatient All Services Group 1 Largest 5 Treatment Programs Largest 10 Treatment Programs Large Treatment Programs Medium Treatment Programs Small Treatment Programs Group 2 Largest 5 Treatment Programs Largest 10 Treatment Programs Large Treatment Programs Medium Treatment Programs Small Treatment Programs The basic view shown in figure 2 view allows for viewing the main metrics: selected completion rate speedometer (#2), overview of yearly completion rates (#4), selected discharge category rates (#5) and treatment duration speedometer (#6). The same metrics are also represented for the comparison groups. The dashboard in figure 2 has been set to show data from the year 2009, for the largest 10 treatment programs and all County treatment programs.
The basic view of the dashboard uses a speedometer to depict the completion rate for a treatment program on the context of the user-set performance levels (#2). Intuitively, the green region represents good performance, the yellow region represents performance that can be improved, and the red region indicates critical performance that requires further assessment. Below the speedometer, the user will be able to spot the best performer in that group (#3). Also, a line graph of the performance of the selected group over all the years can be seen to allow the user to spot trends and outliers (#4). Then the discharge statuses percentages can be seen to compare the different groups are performing in each specific outcome (#5). The color scheme for the discharge status composition refers to the desirability of that component. So for example, completed episodes are shaded in green (more is better), and incomplete statuses are shaded red (less is better). Finally, a speedometer is used again to put the average treatment duration in context with user-set levels of acceptable average treatment durations (#6). The screenshot in figure 2 was achieved by selecting all treatment services category, largest 10 facilities and County comparison groups, and the year 2009. The selections are made in the dropdown lists labeled #1 in figure 2. Based on the first performance indicator (#2 in figure 2), the largest 10 facilities have a successful completion percentage in the green region, meaning it is within the desired performance level. On the other hand, it is noticeable that the county is performing in the yellow region. Therefore, the viewer will infer that the county did not perform within desired performance levels. It also possible to infer that, during 2009, the largest 10 facilities pulled the County s performance up during that year, since they have higher completion rates and serves around 35% of the county treatment episodes. Facility #40 was the best performing treatment program in the largest 10 programs during 2009, while facility #6754 was the best in the county during that year, as shown near label #3. Also, it is notable that the largest 10 facilities had their peak completion percentage during the year 2009, and that it dropped in the years after. While the County s completion percentage dropped after 2009 and then returned back up in 2011, as shown near label #4. Near label #5, the user will identify the composition of treatment episodes discharge statuses for that year. It is notable that the county had more incomplete episodes where patients leave with unsatisfactory progress compared to the largest 10 facilities. The average treatment duration, near label #6 in figure 2, are in the critical levels for the largest 10 facilities, and at a cautionary level for the whole county. Based on the previous results, the user can then perform additional analyses, perhaps comparing smaller programs to the county or different years, to monitor, analyze and manage treatment programs in SAPC.
Figure 2. Dashboard Layout
4.1 Extended View An extended view allow for a breakdown of the risk factors, and the performance of the group of programs with respect to the present amount of that risk factor in its population. The risk factors include homelessness, mental conditions, race, and referral sources. For each of those risk factors, pie charts showing the amount of patients with that risk factor, and the average completion rates for those patients are visible. The same charts are also populated for the comparison group and symmetrically placed to allow for a quick comparison at a glance. The extended view in the dashboard, as shown in figure 3 below, breaks down the completion rates for individual risk factors. Users of the dashboard will first see graphics to represent the homeless composition of the selected group s patient population in a pie chart, (#1 in figure 3) and a bar graph comparing completion rates for homeless and non-homeless episodes in the group (#2 in figure 3). A small line graph is placed under each risk factor to identify noticeable trends (#3 in figure 3). An identical layout is used to represent the data for mentally ill patients and their completion rates. This would allow users to identify how treatment programs perform with each risk factor and how prevalent is this risk factor. Small icons are used to identify areas of the graphs to allow for faster comprehension of the representations. A similar layout is used to represent the composition of the patient population with respect to race and referral source: composition and corresponding completion rates. The dashboard in figure 3 has been set to show data from the year 2009, for the largest 10 treatment programs and all County treatment programs. The extended view offers all the visual in the basic view, along with more insights on risk factors for the selected categories. The extended view in figure 3 is achieved with the same inputs in the basic view from figure 2 and therefore shows the same information. The pie charts labeled #1 refer to the composition of the patients in the selected groups. So for example, based on the first pie chart, about 27.8% of the treatment episodes were for homeless patients in the largest 10 facilities during the year 2009, as opposed to 26.8% homeless treatment episodes in the county for that year. The bar charts, labeled #2 in figure 3, show that 56.7% of the homeless patients completed their treatment episodes in the largest 10 facilities, while only 36.7% of the homeless patients completed their treatment in the county during 2009. On the other hand, the largest 10 facilities underperformed in comparison to the county with mentally ill patients, by having 50% completion when 68.5% of mentally ill patients completed their treatment in the county during the year 2009. The two other categories, referral source and race, also have a similar layout with pie charts representing the population, and bar charts representing their completion rates.
Figure 3. Extended View of the Dashboard
5. Challenges Early feedback on the proposed dashboard design is positive. However, there are several concerns and challenge. A main concern is the flexibility of the design and its capability of importing updated raw data. It is important for the design to be flexible and capable of accepting new raw data to ensure its sustainability. Challenges to adopting such a tool include obtaining organizational support for using and improving the proposed design as an aid in strategic and tactical monitoring, management and analysis of treatment programs in LA County. It would also require investing in aggregating new yearly data into the current dashboard to allow for continual use and improving the current design. The proposed design can be improved in several ways. The reliability of the data can be assessed to ensure the data presented in the dashboard is usable and representable of real performance. Also, additional risk factors can be included in the design, including the age of the patient population, employment, and criminal status. Furthermore, more detailed breakdowns can be available, such as an optional monthly performance summary to address short-term strategic and tactical goals and performance. Finally, a design that allows for more flexible grouping of treatment programs would be considered more desirable, as it would allow the user to aggregate data based on their needs and strategic objectives. 6. Conclusion The positive feedback obtained from this proposed dashboard design for LA County SAPC treatment programs shows promise in the usability and effectiveness of such an accessible tool. The dashboard tool puts survey data to meaningful use by using data visualization approaches and human factors concepts to design a dashboard that would be easy to use and readily provides information based on the users needs. The design can be potentially improved and used by executives and program managers to allow for comparisons with similar treatment programs and strategic goals, and learning from best practices. Acknowledgements We would like to thank Yinfei Kong for his input and ISE 564 students for their participation in this project. References [1] The Annual Catastrophe of Alcohol in California: Los Angeles County. Marin Institute, July 2008. http://www.marininstitute.org/site/images/stories/pdfs/coststudylafinal.pdf [2] Mortality in Los Angeles County 2007: Death and Premature Death with Trends for 1998-2007, County of Los Angeles Department of Public Health. [3] Eckerson, W.W., 2010, Performance Dashboards: Measuring, Monitoring, and Managing Your Business Wiley, New Jersey. [4] Simpson, D. D. (1981). Treatment for drug abuse: Follow-up outcomes and length of time spent. Archives of General Psychiatry, 38(8), 875. [5] Few, S., 2006, Information Dashboard Design: The Effective Visual Communication of Data, O'Reilly.