Assessing Representativeness of the California Department of Mental Health Consumer Perception Surveys

Transcription

1 Assessing Representativeness of the California Department of Mental Health Consumer Perception Surveys by: Ernest L. Cowles, Ph.D., Director & Principal Investigator Kristine Harris, M.A., Research Analyst Cristina Larsen, Ph.D., Research Analyst Angela Prince, M.A., Research Analyst with Michael A. Small, M.A., Research Analyst 3/22/2010 Institute for Social Research Sacramento State University

2 Assessment of CDMH Consumer Perception Survey Representativeness Page 2

3 Table of Contents INTRODUCTION... 1 Background and Purpose... 1 METHODLOGY... 2 Data Sources... 2 Data Cleaning and Structure... 2 Testing Representativeness... 3 STUDY FINDINGS... 4 Key Findings... 4 Child/Family File November Reviewing CPS Data and Process for Omitting Cases... 5 Table 1: Number of Cases for Individuals Over 18 Years of Age... 5 Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data... 5 Table 3a: Number of times a Client ID appeared in data file... 6 Table 3b: Number of individuals represented by duplicate Client ID s... 6 Table 4: Gender discrepancies within CPS duplicate data... 7 Table 5: Ethnicity discrepancies within CPS duplicate data... 7 Table 6: Process for identifying CPS cases with data discrepancies... 8 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist between the Data:... 9 Table 7: Gender discrepancies between CPS Child/Family November 2006 and CSI data... 9 Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values*... 9 Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist between the Data: Table 10: Ethnicity Discrepancies between CPS and CSI data Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Assessment of CDMH Consumer Perception Survey Representativeness Page iii

4 Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data Table 14: Age Distributions across CPS and CSI Population for Children in Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Child/Family File May Reviewing CPS Data and Process for Omitting Cases: Table 1: Number of Cases for Individuals Over 18 Years of Age Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Table 3a: Number of times a Client ID appeared in data file Table 3b: Number of individuals represented by duplicate Client ID s Table 4: Gender discrepancies within CPS duplicate data Table 5: Ethnicity discrepancies within CPS duplicate data Table 6: Process for identifying CPS cases with data discrepancies Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 7: Gender discrepancies between CPS and CSI data Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 10: Ethnicity discrepancies between CPS and CSI data Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 14: Age Distributions across CPS and CSI Population for Children in Assessment of CDMH Consumer Perception Survey Representativeness Page iv

5 Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Adult/Older File November Reviewing CPS Data and Process for Omitting Cases: Table 1: Number of Cases for Individuals Under 18 Years of Age Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Table 3a: Number of times a Client ID appeared in data file Table 3b: Number of individuals represented by duplicate Client ID s Table 4: Gender discrepancies within CPS duplicate data Table 5: Ethnicity discrepancies within CPS duplicate data Table 6: Date of Birth (DOB) discrepancies within CPS duplicate data Table 7: Process for identifying CPS cases with data discrepancies Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 8: Gender discrepancies between CPS and CSI data Table 9: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Table 10: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 11: Ethnicity discrepancies between CPS and CSI data Table 12: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Table 13: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 14: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 15: Age Distributions across CPS and CSI Population for Adults in Table 16: Difference between CPS and CSI Age Data: Expected versus Observed Values* Adult/Older May Reviewing CPS Data and Process for Omitting Cases: Assessment of CDMH Consumer Perception Survey Representativeness Page v

6 Table 1: Number of Cases for Individuals Under 18 Years of Age Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Table 3a: Number of times a Client ID appeared in data file Table 3b: Number of individuals represented by duplicate Client ID s Table 4: Gender discrepancies within CPS duplicate data Table 5: Ethnicity discrepancies within CPS duplicate data Table 6: Date of Birth (DOB) discrepancies within CPS duplicate data Table 7: Process for identifying CPS cases with data discrepancies Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 8: Gender discrepancies between CPS and CSI data Table 9: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Table 10: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 11: Ethnicity discrepancies between CPS and CSI data Table 12: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Table 13: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 14: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 15: Age Distributions across CPS and CSI Population for Adults in Table 16: Difference between CPS and CSI Age Data: Expected versus Observed Values* Comparison of Demographic Data across Surveys Table 1: Comparison of Gender across Surveys* Table 2a: Comparison of Ethnicity across Child/Family Surveys* Table 2b: Comparison of Ethnicity across Adult/Older Adult Surveys* Table 3: Comparison of LA versus Non LA across Survey* Assessment of CDMH Consumer Perception Survey Representativeness Page vi

7 APPENDIX A: Child/Family File November Benchmark #1 Data Table A1: Ethnicity discrepancies between CPS and CSI data Benchmark # Table A2: Gender discrepancies between CPS and CSI data Benchmark # APPENDIX B: Child/Family File May Benchmark #1 Data Table B1: Ethnicity discrepancies between CPS and CSI data Benchmark # Table B2: Gender discrepancies between CPS and CSI data Benchmark # APPENDIX C: Adult/Older Adult File November Benchmark #1 Data Table C1: Ethnicity discrepancies between CPS and CSI data Benchmark # Table C2: Gender discrepancies between CPS and CSI data Benchmark # APPENDIX D: Adult/Older Adult File May Benchmark #1 Data Table D1: Ethnicity discrepancies between CPS and CSI data Benchmark # Table D2: Gender discrepancies between CPS and CSI data Benchmark # APPENDIX E: Discussion of Data Inconsistencies Methodological Issues Relating To Variables in CPS datasets Over Arching Methodological Issues Assessment of CDMH Consumer Perception Survey Representativeness Page vii

8 INTRODUCTION Background and Purpose The Consumer Perception Surveys are administered to consumers who receive face to face community mental health services from county operated and contract providers. Prior to FY , the surveys were administered twice a year for a two week period, in early May and November by mental health service providers at the county level. This form of survey administration utilizes a convenience sampling approach in which consumers who come into the mental health service organization during the twoweek sampling period receive a Consumer Perception Survey to complete and return on the premises. This affordable and easily administered sampling method can result in large sample sizes. However, it has limitations in that it leads to the exclusion of consumers who do not, for whatever reason, come in for services during the survey administration timeframe. Recognizing the limitations of this sampling approach, CDMH has enlisted this Institute for Social Research to conduct an analysis of the representativeness of the existing convenience sample on key characteristics such as age, gender and race 1, as detailed in this report. The purpose of this analysis was to 1) determine whether survey samples are representative of their respective populations; 2) provide California Department of Mental Health (CDMH) with recommendations for increasing representativeness and decreasing county survey administration burden, and 3) identify data quality issues and suggest strategies for addressing data quality concerns. The nationally developed Mental Health Statistics Improvement Program (MHSIP) Consumer Perception Survey (CPS) measures consumer perception of satisfaction of various facets of mental health service delivery. For the purposes of assessing representativeness, the Institute for Social Research (ISR) worked with key client demographic data, specifically gender, age, ethnicity, and county. The target populations were youth and adults/older adults receiving services through California s community mental health system, specifically, those receiving face to face services, case management, day treatment and medication services from county operated and contract organization providers in California (irrespective of funding sources). The four surveys administered are the Youth Services Survey (YSS), Youth Services Survey for Families (YSS F), Adult Survey, and Older Adult Survey. Youths of sufficient age to reliably complete a survey (minimum age is 13 years old) complete the YSS survey; family members/caregivers of youth under the age of 18 complete the YSS F; adults ages 18 to 59 years complete the Adult Survey, and older adults ages 60 years or more complete the Older Adult Survey. CDMH obtained data over three fiscal years, FY , FY , and FY (November 2006, May 2007, November 2007, May 2008, and November 2009). These surveys were available in English, 1 Note: Race and ethnicity have been collapsed for reporting purposes. For the remainder of the report, ethnicity is used to refer to the race/ethnicity variable. Assessment of CDMH Consumer Perception Survey Representativeness Page 1

9 Spanish, Russian, Chinese, Tagalog and Vietnamese. CDMH obtains approximately 500,000 completed surveys every survey administration period. For the purposes of this representativeness analysis, the ISR used survey data from November 2006 and May 2007 as they represented the most complete datasets at the time of the onset of this project. METHODLOGY Data Sources Client level data were provided to the ISR by the CDMH in July, The data is contained in the following files: 1) Client and Service Information (CSI) population data for the time period of 7/1/06 through 6/30/07, and CPS survey data corresponding to two survey administration periods: first 2 weeks in November 2006 and the first two weeks in May 2007 for 2) youth (YSS, YSS F) and 3) adults and older adults. In order to assess representativeness, it was necessary to compare the survey samples to their populations on key demographic variables. The following demographic variables appeared in both the survey and population datasets: ethnicity, gender, age and county. Selection of demographic variables which appeared in both files allowed for comparisons of data between the datasets and for significance testing which helped determine whether or not the survey data was representative of the population. Los Angeles contains a large percentage of mental health consumers; therefore the county variable was recoded into the dichotomous variable, LA versus Non LA County. CSI population data for each of the demographic variables was matched into the CPS survey data files based on Client ID. This data is referred to as matched CSI Data. In other words, the matched CSI data is a subset of CSI cases that have a one to one correspondence with CPS data matched on Client ID. One benefit of matching the CSI data was that if, for any reason, the CPS data was unusable it could be overridden by CSI data. A second benefit of matching the CSI data was that comparisons could be made between the matched CSI data and the population data to determine if the matched data was representative of the population. One limitation of the matching approach is that it is subject to sampling error or any flaws that may have been introduced during the sampling process. Essentially, where sampling is concerned the CSI matched data is susceptible to the same limitations as the raw survey data itself. Analyses were performed on CPS survey data, CSI population data, and CSI matched data. Data Cleaning and Structure Data cleaning included the following processes. First, an Age variable was constructed using the survey date and consumer date of birth (DOB). DOBs in the survey population were inconsistent in both format and date in each of the files. In order to maintain a consistent methodological approach when calculating the Age variable, DOBs from CSI were used to calculate Age across all files. Once Age was determined, clients who had a reported age that did not fit their respective categories were removed. Assessment of CDMH Consumer Perception Survey Representativeness Page 2

10 Therefore, cases in the Adult/Older Adult survey files that contained individuals 17 years and younger were omitted from the analyses. Similarly, individuals 18 years and older were removed from the Child/Family files. It should be noted that possible Transition Age Youth (TAY) may have been removed from the Child/Family files and not included in this analysis. The TAY population is comprised of mental health consumers ages 18 to 25 years who receive services through the children s system of care. The Child/Family databases did not contain an indicator identifying TAY membership and therefore these individuals could not be distinguished from non TAY individuals (ages 18+) in the Child/Family datasets. Second, cases containing Client ID numbers which appeared in the CPS survey data but not in the CSI population data were removed from the files. Third, it was decided that cases with Client IDs that appeared three or more times would be excluded from analysis. These criteria resulted in approximately 12 to 15 percent of cases omitted from each data file. Frequencies were then run on the CPS survey and matched CSI data to create distributions for each of the demographic variables and compare them with those of the CSI population data. This first comparison is referred to as Benchmark #1 and can be found in Appendices A through D. Next, we determined whether duplicate Client IDs (2 appearances) contained uniform responses for each of the demographic variables within the CPS datasets. Duplicate Client IDs that did not contain uniform responses or had missing values were excluded from analysis. This process of exclusion was applied on a rolling basis with replacement. For example, when testing the gender variable, cases satisfying the above criteria were excluded before frequencies were run. The gender data was then fully replaced into the dataset and the same procedure was conducted for the next demographic variable. In this way, each variable could be analyzed separately without affecting outcomes of the remaining demographic variables. The frequencies reported at this phase of the analysis are referred to as Benchmark #2 and are reported in the main body of the report. Testing Representativeness Chi Square tests were run to see if there were statistically significant differences between: 1) CPS survey and CSI population data, and 2) matched CSI data and CSI population data. Chi Square is a test used to determine whether there is a significant difference between the expected frequencies and the observed frequencies in one or more categories. Expected values were derived from the population data (CSI database) for each demographic variable. These expected values were run against observed values for both CPS and matched CSI data. For all chi square tests we selected the.01 level of significance. Essentially, this would mean that if random samples were repeatedly drawn from the CSI population, we would expect them to differ by chance (error) only 1 time in 100 (1% of the time) or less (p.01).. Obtained p values at or below this level indicate statistical significance and warrants the subsequent interpretation of lack of representativeness. Although the.01 significant level was selected as the threshold level for significant differences, results reported in this report revealed consistent differences at the p.001 level. Assessment of CDMH Consumer Perception Survey Representativeness Page 3

11 STUDY FINDINGS Key Findings Representativeness of Convenience Sample Surveys Two different methods of analyses of four primary demographic variables revealed that neither the November or May administrations of the current Child/Family and Adult/Older Adult Consumer Surveys were representative of the larger CSI population. They significantly differed in: Gender, Ethnicity, Age, and LA versus Non LA Counties. Females are significantly overrepresented in all four surveys and males are significantly underrepresented. Blacks, Asian, Pacific Islanders, Other and Unknown are significantly underrepresented in both surveys. Whites and Hispanics are significantly overrepresented in both surveys. American Indians and Multiple Ethnicities are overrepresented in one survey and underrepresented in another. Whites, American Indian, and Other are underrepresented in both surveys. Blacks and Multiple Ethnicities are overrepresented in both surveys. Hispanics and Asian Pacific Islanders are overrepresented in one survey and underrepresented in another. Children below the age of 13 are underrepresented in both surveys, while youth between are overrepresented. Ages 18 29, 30 39, and 70 year olds and above were underrepresented, while ages were overrepresented. LA County was overrepresented in the November survey, but underrepresented in the May survey for the Child/Family survey, but was consistently overrepresented on the Adult/Older Adult survey. Data Quality Issues There were 9 major areas where data quality impedes the ability of CDMH to truly know what Consumers think of services or where strengths and weaknesses are found: 1. Lack of congruity between how variables are defined in CSI versus CPS 2. Multiple iterations of files and variables within the agency 3. Non standardized application of methodology to variables across files 4. Duplicate cases contain discrepancies across demographic variables 5. Data in CPS files do not reflect the response categories on surveys 6. Backfilling CPS data with CSI data 7. TAY versus Non TAY 8. Duplicate Client Id s in CPS 9. Inaccuracies in CPS data (e.g., DOB entries) Assessment of CDMH Consumer Perception Survey Representativeness Page 4

12 Child/Family File November 2006 Reviewing CPS Data and Process for Omitting Cases Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Child/Family target population consists of individuals 0 through 17 years of age. The CPS Child Family data file contained cases of individuals whose age was calculated as over 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Child Family dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the CPS Child Family dataset as TAY or as misclassified data (adults present in children dataset). Therefore, all individuals over the age of 18 years (3%) were omitted from this CPS Child Family dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Over 18 Years of Age Frequency Percentage Original number of cases 26,750 Deleted cases for individuals over % Total 25, % An additional data cleaning procedure involved excluding CPS Child Family cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 88% of CPS Child Family cases had a corresponding Client ID in the CSI population dataset. The 12% of cases that did not match up with a CSI Client ID were excluded from the CPS Child Family dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 22, % Non Matched 3, % Total 25, % Assessment of CDMH Consumer Perception Survey Representativeness Page 5

13 Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Child Family data set. Duplicate client ID s could have occurred because clients actually took more than one survey or because of data error. As can be seen, Client ID emerged twice in approximately 34% of the cases and 3 to 5 times in a little over 2.5% of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding IDs rows of data Percentage of cases 1 14, % 2 7, % % % % Total 22, % Table 3b contains the number of times a Client ID appeared in the CPS Child Family data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 78% of individuals had their Client ID appear only once and 21% of individuals had their Client ID appear twice. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals Percentage of individuals 1 14, % 2 3, % % % 5 2.0% Total 18, % Assessment of CDMH Consumer Perception Survey Representativeness Page 6

14 As can be seen in Table 4, gender matched for one or more duplicate cases or there was missing gender data for one of the duplicate cases 96% of the time. The rest of the duplicate cases (4%) did not match on gender or had a value coded as system missing. As can be seen in Table 5, ethnicity matched on approximately 78% of the duplicate cases. Table 4: Gender discrepancies within CPS duplicate data Frequency Percent Gender matches or missing gender data for one of the duplicate cases 7, % Duplicate cases that are a non match or have a double sysmis value % Total 7, % Table 5: Ethnicity discrepancies within CPS duplicate data Frequency Percent Ethnicity matches 5, % Ethnicity non matches 1, % Total 7, % Assessment of CDMH Consumer Perception Survey Representativeness Page 7

15 Table 6 contains various data discrepancies that led to the exclusion of 5,802 cases from the dataset based on gender discrepancies and 6,337 cases based on ethnicity discrepancies for the purposes of the current analysis. Table 6: Process for identifying CPS cases with data discrepancies CPS Gender Variable CPS Ethnicity Variable Original number of CPS cases 26,750 26,750 Cases of children over CPS cases that did not match back to CSI 3,206 3,206 Client Id s that had 3 or more duplicates 618 Duplicate cases that are a nonmatch or have a double sysmis value 302 Cases with one sysmis value 877 New number of CPS cases 20,948 *Represents the number of cases with one sysmis value or a non match on DOB ,714 0* 20,413 Benchmark #1 See Appendix A for data Benchmark #2 Assessment of CDMH Consumer Perception Survey Representativeness Page 8

16 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist between the Data: Table 7 contains gender data across three datasets: the CPS Child Family dataset in which 4% of the gender data was missing, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and finally, the CSI population data for the time period of July 1, 2006 to June 30, Table 7: Gender discrepancies between CPS Child/Family November 2006 and CSI data CPS data Matched CSI data CSI population data N % N % N % Female 8, % 8, % 77, % Male 12, % 13, % 121, % Other 20.1% 23.1% 557.3% Total 20, % 21, % 199, % Chi square tests were performed to determine if there were significant differences between actual or observed values for gender in the CPS data and expected values based on the CSI population data. As can be seen in Table 8, there is a statistically significant difference between what CPS Data reports for Gender and the CSI Population. Where Gender is concerned, CPS data is not representative of the CSI Population. Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8, % 8, % 318 Male 12, % 12, % 275 Other 20.1% 63.3% 43 Total 20, % Assessment of CDMH Consumer Perception Survey Representativeness Page 9

17 * significant at p =.001, df = 2. Table 9 contains Chi Square results for Matched CSI and CSI population gender data. There is a statistically significant difference between what the Matched CSI data reports for Gender and the CSI Population. Where Gender is concerned, Matched CSI data is not representative of the CSI Population. Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8, % 8, % 254 Male 13, % 13, % 211 Other 23.1% 66.3% 43 Total 21, % * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 10

18 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist between the Data: Table 10 contains ethnicity data across three datasets: the CPS Child Family data, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and the CSI population data for the time period of July 1, 2006 to June 30, Table 10: Ethnicity Discrepancies between CPS and CSI data CPS data Matched CSI data CSI Population data N % N % N % White 5, % 5, % 50, % Hispanic 9, % 7, % 74, % Black 2, % 2, % 30, % Asian or Pacific Islander % % 5, % American Indian 124.6% 127.6% 1,388.7% Other % % 7, % Multiple Ethnicities 1, % 1, % 12, % Unknown % 1, % 18, % Total 20, % 20, % 199, % Assessment of CDMH Consumer Perception Survey Representativeness Page 11

19 Table 11 contains Chi Square results for CPS Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the CPS data reports for Ethnicity and the CSI Population. Where Ethnicity is concerned, CPS data is not representative of the CSI Population. Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5, % 5, % 91 Hispanic 9, % 7, % 1,873 Black 2, % 3, % 527 Asian or Pacific Islander % % 58 American Indian 124.6% 143.7% 19 Other % % 481 Multiple Ethnicities 1, % 1, % 19 Unknown % 1, % 897 Total 20, % * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 12

20 Table 12 contains Chi Square results for Matched CSI Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the Matched CSI data reports for Ethnicity the CSI Population. Where ethnicity is concerned, Matched CSI data is not representative of the CSI Population. Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5, % 5, % 774 Hispanic 7, % 7, % 124 Black 2, % 3, % 285 Asian or Pacific Islander % % 38 American Indian 127.6% 143.7% 16 Other % % 220 Multiple Ethnicities 1, % 1, % 208 Unknown 1, % 1, % 546 Total 20, % * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 13

21 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 13 contains Chi Square results for CPS Ethnicity Data and CSI Population Data on county: Los Angeles (LA County) versus all other counties (non LA County). There is no statistically significant difference between what the CPS data reports for LA versus Non LA and the CSI Population. Where county dichotomy is concerned, CPS data is representative of the CSI Population. Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N LA County 5, % 5, % 64 Non LA County 16, % 16, % 64 Total 22, % * not significant Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data Table 14 contains the ages of cases in the CPS Child Family data file grouped into 3 categories, 0 to 5 years, 6 to 12 years and 13 to 17 years. One can see the under and over representativeness of the CPS dataset across age categories by comparing the CPS percentages with the CSI percentages. Table 14: Age Distributions across CPS and CSI Population for Children in 2006 Age Categories in Years CPS Data CSI Population 2006 N % N % 0 5 1, % 20, % , % 79, % , % 100, % Total 22, % 199, % Assessment of CDMH Consumer Perception Survey Representativeness Page 14

22 As can be seen in Table 15, there is a statistically significant difference between what the CPS data reports for Age and the CSI population. Where Age is concerned, CPS data is not representative of the CSI Population. Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Age Categories in Years Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N 0 5 1, % 2, % , % 8, % , % 11, % 1,332 Total 22, % * significant at p =.001, df = 1. Assessment of CDMH Consumer Perception Survey Representativeness Page 15

23 Child/Family File May 2007 Reviewing CPS Data and Process for Omitting Cases: Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Child/Family target population consists of individuals 0 through 17 years of age. The CPS Child Family data file contained cases of individuals whose age was calculated as over 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Child Family dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the Child Family CPS dataset as TAY or as misclassified data (adults present in children dataset). Therefore, all individuals over the age of 18 years (4%) were omitted from this CPS Child Family dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Over 18 Years of Age Frequency Percentage Original number of cases 26,407 Deleted cases for individuals over % Total 25, % An additional data cleaning procedure involved excluding Child Family CPS cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 87% of CPS Child Family cases had a corresponding Client ID in the CSI population dataset. The 13% of cases that did not match up with a CSI Client ID were excluded from the CPS Child Family dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 21, % Non Matched* 3, % Total 25, % * Non matched cases were subsequently removed from the CPS file. Assessment of CDMH Consumer Perception Survey Representativeness Page 16

24 Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Child Family data set. As can be seen, Client ID emerged twice in approximately 34% of the cases and 3 to 6 times in 3 % of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding rows of data Percentage of cases 1 13, % 2 7, % % % % % Total 21, % Table 3b contains the number of times a Client ID appeared in the CPS Child Family data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 78% of individuals had their Client ID appear only once and 21% of individuals had their Client ID appear twice. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals Percentage of individuals 1 13, % 2 3, % % % 5 3 0% 6 3 0% Total 17, % Assessment of CDMH Consumer Perception Survey Representativeness Page 17

25 As can be seen in Table 4, gender matched for one or more duplicate cases or there was missing gender data for one of the duplicate cases 96% of the time. The rest of the duplicate cases (4%) did not match on gender or had a value coded as system missing. As can be seen in Table 5, ethnicity matched on approximately 77% of the duplicate cases. Table 4: Gender discrepancies within CPS duplicate data Frequency Percent Gender matches or missing gender data for one of the duplicate cases 7,080 96% Duplicate cases that are a non match or have a double sysmis value 292 4% Total 7, % Table 5: Ethnicity discrepancies within CPS duplicate data Frequency Percent Ethnicity matches 5, % Ethnicity non matches 1, % Total 7, % Assessment of CDMH Consumer Perception Survey Representativeness Page 18

26 Table 6 contains various data discrepancies that led to the exclusion of 6,540 cases based on gender discrepancies and 6,846 cases based on ethnicity discrepancies from the dataset for the purposes of the current analysis. Table 6: Process for identifying CPS cases with data discrepancies CPS Gender Variable CPS Ethnicity Variable Original number of CPS cases 26,407 26,407 Adults >18 yrs CPS cases that did not match back to CSI 3,529 3,529 Client Id s that had 3 or more duplicates Duplicate cases that are a non match or have a double sysmis 1,734 value 292 Cases with one sysmis value 1,136 0* New number of CPS cases 19,867 *Represents the number of cases with one sysmis value or a non match on DOB. 19,561 Benchmark #1 See Appendix A for data Benchmark #2 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 7 contains gender data across three datasets: the CPS Child Family dataset in which 5.4% of the gender data was missing, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and finally, the CSI population data for the time period of July 1, 2006 to June 30, Table 7: Gender discrepancies between CPS and CSI data CPS data* Matched CSI data** CSI population data*** N % N % N % Female 8, % 8, % 74, % Male 11, % 12, % 116, % Other 19.1% 21.1% 542.3% Total 19, % 21, % 191, % Assessment of CDMH Consumer Perception Survey Representativeness Page 19

27 Chi square tests were performed to determine if there were significant differences between actual or observed values for gender in the CPS data and expected values based on the CSI population data. As can be seen in Table 8, there is a statistically significant difference between what CPS Data reports for Gender and the CSI Population. Where Gender is concerned, CPS data is not representative of the CSI Population. Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8, % 7, % 328 Male 11, % 12, % 287 Other 19.1% 60.3% 41 Total 19, % * significant at p =.001, df = 2. Table 9 contains chi square results for Matched CSI and CSI population gender data. There is a statistically significant difference between what the Matched CSI data reports for Gender and the CSI Population. Where Gender is concerned, Matched CSI data is not representative of the CSI Population. Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8, % 8, % 241 Male 12, % 12, % 199 Other 21.1% 63.3% 42 Total 21, % * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 20

28 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 10 contains ethnicity data across three datasets: the CPS Child Family data, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and the CSI population data for the time period of July 1, 2006 to June 30, Table 10: Ethnicity discrepancies between CPS and CSI data CPS data Matched CSI data CSI Population data N % N % N % White 4, % 5, % 48, % Hispanic 8, % 7, % 71, % Black 2, % 2, % 29, % Asian or Pacific Islander % % 4, % American Indian 163.8% 108.6% 1,337.7% Other % % 7, % Multiple Ethnicities 1, % 1, % 11, % Unknown 1, % 1, % 17, % Total 19, % 19, % 191, % Assessment of CDMH Consumer Perception Survey Representativeness Page 21

29 Table 11 contains Chi Square results for CPS Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the CPS data reports for Ethnicity and the CSI Population. Where Ethnicity is concerned, CPS data is not representative of the CSI Population. Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 4, % 4, % 21 Hispanic 8, % 7, % 1659 Black 2, % 2, % 679 Asian or Pacific Islander % % 42 American Indian 163.8% 137.7% 26 Other % % 426 Multiple Ethnicities 1, % 1, % 45 Unknown 1, % 1, % 513 Total 19, % * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 22

30 Table 12 contains Chi Square results for Matched CSI Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the Matched CSI data reports for Ethnicity the CSI Population. Where ethnicity is concerned, Matched CSI data is not representative of the CSI Population. Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5, % 4, % 769 Hispanic 7, % 7, % 367 Black 2, % 2, % 456 Asian or Pacific Islander % % 23 American Indian 108.6% 137.7% 29 Other % % 202 Multiple Ethnicities 1, % 1, % 261 Unknown 1, % 1, % 732 Total 19, % * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 23

31 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 13 contains Chi Square results for CPS Ethnicity Data and CSI Population Data on county: Los Angeles (LA County) versus all other counties (non LA County). There is a statistically significant difference between what the CPS data reports for LA versus Non LA and the CSI population. Where county dichotomy is concerned, CPS data is not representative of the CSI population. Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N LA County 5, % 5, % 362 Non LA County 16, % 15, % 362 Total 21, % * significant at p =.001, df = 1. Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 14 contains the ages of cases in the CPS Child Family data file grouped into 3 categories, 0 to 5 years, 6 to 12 years and 13 to 17 years. Table 14: Age Distributions across CPS and CSI Population for Children in 2007 Age Categories in CPS Data CSI Population 2007 Years N % N % 0 5 1, % 16, % , % 75, % , % 98, % Total 21, % 191, % Assessment of CDMH Consumer Perception Survey Representativeness Page 24

32 As can be seen in Table 15, there is a statistically significant difference between what the CPS data reports for Age and the CSI population. Where Age is concerned, CPS data is not representative of the CSI Population. Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Age Categories in Years Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N 0 5 1, % 1, % , % 8, % , % 11, % 1,429 Total 21, % 21,295(6) 100% * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 25

33 Adult/Older File November 2006 Reviewing CPS Data and Process for Omitting Cases: Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Adult/Older Adult target population consists of individuals years of age and older. The CPS Adult Older Adult data file contained cases of individuals whose age was calculated as under 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Adult Older Adult dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the CPS Adult Older Adult dataset as TAY or not. All individuals under the age of 18 years (46) were omitted from this CPS Adult Older Adult dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Under 18 Years of Age Frequency Percentage Original number of cases 26,232 Deleted cases for children under <0% Total 26, % An additional data cleaning procedure involved excluding CPS Adult Older Adult cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 86% of CPS Adult Older Adult cases had a corresponding Client ID in the CSI population dataset. The 15% of cases that did not match up with a CSI Client ID were excluded from the CPS Adult Older Adult dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 22, % Non Matched 3, % Total 26, % Assessment of CDMH CPS Survey Representativeness Page 26

34 Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Adult Older Adult data set. As can be seen, Client ID emerged only once in 94% of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding rows of data Percentage of cases 1 20, % 2 1, % % % % 6 6.0% Total 22, % Table 3b contains the number of times a Client ID appeared in the CPS Adult Older Adult data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 97% individuals had their Client ID appear only once. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals* Percentage of individuals 1 20, % % % 4 4.0% 5 3.0% 6 1.0% Total 21, % Assessment of CDMH CPS Survey Representativeness Page 27