Assessing Representativeness of the California Department of Mental Health Consumer Perception Surveys by: Ernest L. Cowles, Ph.D., Director & Principal Investigator Kristine Harris, M.A., Research Analyst Cristina Larsen, Ph.D., Research Analyst Angela Prince, M.A., Research Analyst with Michael A. Small, M.A., Research Analyst 3/22/2010 Institute for Social Research Sacramento State University
Assessment of CDMH Consumer Perception Survey Representativeness Page 2
Table of Contents INTRODUCTION... 1 Background and Purpose... 1 METHODLOGY... 2 Data Sources... 2 Data Cleaning and Structure... 2 Testing Representativeness... 3 STUDY FINDINGS... 4 Key Findings... 4 Child/Family File November 2006... 5 Reviewing CPS Data and Process for Omitting Cases... 5 Table 1: Number of Cases for Individuals Over 18 Years of Age... 5 Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data... 5 Table 3a: Number of times a Client ID appeared in data file... 6 Table 3b: Number of individuals represented by duplicate Client ID s... 6 Table 4: Gender discrepancies within CPS duplicate data... 7 Table 5: Ethnicity discrepancies within CPS duplicate data... 7 Table 6: Process for identifying CPS cases with data discrepancies... 8 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist between the Data:... 9 Table 7: Gender discrepancies between CPS Child/Family November 2006 and CSI data... 9 Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values*... 9 Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values*... 10 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist between the Data:... 11 Table 10: Ethnicity Discrepancies between CPS and CSI data... 11 Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values*... 12 Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values*... 13 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 14 Assessment of CDMH Consumer Perception Survey Representativeness Page iii
Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values*... 14 Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data... 14 Table 14: Age Distributions across CPS and CSI Population for Children in 2006... 14 Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values*... 15 Child/Family File May 2007... 16 Reviewing CPS Data and Process for Omitting Cases:... 16 Table 1: Number of Cases for Individuals Over 18 Years of Age... 16 Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data... 16 Table 3a: Number of times a Client ID appeared in data file... 17 Table 3b: Number of individuals represented by duplicate Client ID s... 17 Table 4: Gender discrepancies within CPS duplicate data... 18 Table 5: Ethnicity discrepancies within CPS duplicate data... 18 Table 6: Process for identifying CPS cases with data discrepancies... 19 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 19 Table 7: Gender discrepancies between CPS and CSI data... 19 Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values*... 20 Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values*... 20 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 21 Table 10: Ethnicity discrepancies between CPS and CSI data... 21 Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values*... 22 Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values*... 23 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 24 Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values*... 24 Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 24 Table 14: Age Distributions across CPS and CSI Population for Children in 2007... 24 Assessment of CDMH Consumer Perception Survey Representativeness Page iv
Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values*... 25 Adult/Older File November 2006... 26 Reviewing CPS Data and Process for Omitting Cases:... 26 Table 1: Number of Cases for Individuals Under 18 Years of Age... 26 Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data... 26 Table 3a: Number of times a Client ID appeared in data file... 27 Table 3b: Number of individuals represented by duplicate Client ID s... 27 Table 4: Gender discrepancies within CPS duplicate data... 28 Table 5: Ethnicity discrepancies within CPS duplicate data... 28 Table 6: Date of Birth (DOB) discrepancies within CPS duplicate data... 28 Table 7: Process for identifying CPS cases with data discrepancies... 29 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 30 Table 8: Gender discrepancies between CPS and CSI data... 30 Table 9: Difference between CPS and CSI Gender Data: Expected versus Observed Values*... 30 Table 10: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values*... 31 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 31 Table 11: Ethnicity discrepancies between CPS and CSI data... 31 Table 12: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values*... 32 Table 13: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values*... 33 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 34 Table 14: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values*... 34 Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 34 Table 15: Age Distributions across CPS and CSI Population for Adults in 2006... 34 Table 16: Difference between CPS and CSI Age Data: Expected versus Observed Values*... 35 Adult/Older May 2007... 36 Reviewing CPS Data and Process for Omitting Cases:... 36 Assessment of CDMH Consumer Perception Survey Representativeness Page v
Table 1: Number of Cases for Individuals Under 18 Years of Age... 36 Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data... 36 Table 3a: Number of times a Client ID appeared in data file... 37 Table 3b: Number of individuals represented by duplicate Client ID s... 37 Table 4: Gender discrepancies within CPS duplicate data... 38 Table 5: Ethnicity discrepancies within CPS duplicate data... 38 Table 6: Date of Birth (DOB) discrepancies within CPS duplicate data... 38 Table 7: Process for identifying CPS cases with data discrepancies... 39 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 40 Table 8: Gender discrepancies between CPS and CSI data... 40 Table 9: Difference between CPS and CSI Gender Data: Expected versus Observed Values*... 40 Table 10: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values*... 41 Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 41 Table 11: Ethnicity discrepancies between CPS and CSI data... 41 Table 12: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values*... 42 Table 13: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values*... 43 Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 44 Table 14: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values*... 44 Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data:... 45 Table 15: Age Distributions across CPS and CSI Population for Adults in 2006... 45 Table 16: Difference between CPS and CSI Age Data: Expected versus Observed Values*... 46 Comparison of Demographic Data across Surveys... 47 Table 1: Comparison of Gender across Surveys*... 47 Table 2a: Comparison of Ethnicity across Child/Family Surveys*... 48 Table 2b: Comparison of Ethnicity across Adult/Older Adult Surveys*... 49 Table 3: Comparison of LA versus Non LA across Survey*... 50 Assessment of CDMH Consumer Perception Survey Representativeness Page vi
APPENDIX A:... 51 Child/Family File November 2006... 51 Benchmark #1 Data... 51 Table A1: Ethnicity discrepancies between CPS and CSI data Benchmark #1... 51 Table A2: Gender discrepancies between CPS and CSI data Benchmark #1... 51 APPENDIX B:... 52 Child/Family File May 2007... 52 Benchmark #1 Data... 52 Table B1: Ethnicity discrepancies between CPS and CSI data Benchmark #1... 52 Table B2: Gender discrepancies between CPS and CSI data Benchmark #1... 52 APPENDIX C:... 53 Adult/Older Adult File November 2006... 53 Benchmark #1 Data... 53 Table C1: Ethnicity discrepancies between CPS and CSI data Benchmark #1... 53 Table C2: Gender discrepancies between CPS and CSI data Benchmark #1... 53 APPENDIX D:... 54 Adult/Older Adult File May 2007... 54 Benchmark #1 Data... 54 Table D1: Ethnicity discrepancies between CPS and CSI data Benchmark #1... 54 Table D2: Gender discrepancies between CPS and CSI data Benchmark #1... 54 APPENDIX E:... 55 Discussion of Data Inconsistencies... 55 Methodological Issues Relating To Variables in CPS datasets... 55 Over Arching Methodological Issues... 57 Assessment of CDMH Consumer Perception Survey Representativeness Page vii
INTRODUCTION Background and Purpose The Consumer Perception Surveys are administered to consumers who receive face to face community mental health services from county operated and contract providers. Prior to FY 2009 10, the surveys were administered twice a year for a two week period, in early May and November by mental health service providers at the county level. This form of survey administration utilizes a convenience sampling approach in which consumers who come into the mental health service organization during the twoweek sampling period receive a Consumer Perception Survey to complete and return on the premises. This affordable and easily administered sampling method can result in large sample sizes. However, it has limitations in that it leads to the exclusion of consumers who do not, for whatever reason, come in for services during the survey administration timeframe. Recognizing the limitations of this sampling approach, CDMH has enlisted this Institute for Social Research to conduct an analysis of the representativeness of the existing convenience sample on key characteristics such as age, gender and race 1, as detailed in this report. The purpose of this analysis was to 1) determine whether survey samples are representative of their respective populations; 2) provide California Department of Mental Health (CDMH) with recommendations for increasing representativeness and decreasing county survey administration burden, and 3) identify data quality issues and suggest strategies for addressing data quality concerns. The nationally developed Mental Health Statistics Improvement Program (MHSIP) Consumer Perception Survey (CPS) measures consumer perception of satisfaction of various facets of mental health service delivery. For the purposes of assessing representativeness, the Institute for Social Research (ISR) worked with key client demographic data, specifically gender, age, ethnicity, and county. The target populations were youth and adults/older adults receiving services through California s community mental health system, specifically, those receiving face to face services, case management, day treatment and medication services from county operated and contract organization providers in California (irrespective of funding sources). The four surveys administered are the Youth Services Survey (YSS), Youth Services Survey for Families (YSS F), Adult Survey, and Older Adult Survey. Youths of sufficient age to reliably complete a survey (minimum age is 13 years old) complete the YSS survey; family members/caregivers of youth under the age of 18 complete the YSS F; adults ages 18 to 59 years complete the Adult Survey, and older adults ages 60 years or more complete the Older Adult Survey. CDMH obtained data over three fiscal years, FY 2006 07, FY 2007 08, and FY 2008 09 (November 2006, May 2007, November 2007, May 2008, and November 2009). These surveys were available in English, 1 Note: Race and ethnicity have been collapsed for reporting purposes. For the remainder of the report, ethnicity is used to refer to the race/ethnicity variable. Assessment of CDMH Consumer Perception Survey Representativeness Page 1
Spanish, Russian, Chinese, Tagalog and Vietnamese. CDMH obtains approximately 500,000 completed surveys every survey administration period. For the purposes of this representativeness analysis, the ISR used survey data from November 2006 and May 2007 as they represented the most complete datasets at the time of the onset of this project. METHODLOGY Data Sources Client level data were provided to the ISR by the CDMH in July, 2009. The data is contained in the following files: 1) Client and Service Information (CSI) population data for the time period of 7/1/06 through 6/30/07, and CPS survey data corresponding to two survey administration periods: first 2 weeks in November 2006 and the first two weeks in May 2007 for 2) youth (YSS, YSS F) and 3) adults and older adults. In order to assess representativeness, it was necessary to compare the survey samples to their populations on key demographic variables. The following demographic variables appeared in both the survey and population datasets: ethnicity, gender, age and county. Selection of demographic variables which appeared in both files allowed for comparisons of data between the datasets and for significance testing which helped determine whether or not the survey data was representative of the population. Los Angeles contains a large percentage of mental health consumers; therefore the county variable was recoded into the dichotomous variable, LA versus Non LA County. CSI population data for each of the demographic variables was matched into the CPS survey data files based on Client ID. This data is referred to as matched CSI Data. In other words, the matched CSI data is a subset of CSI cases that have a one to one correspondence with CPS data matched on Client ID. One benefit of matching the CSI data was that if, for any reason, the CPS data was unusable it could be overridden by CSI data. A second benefit of matching the CSI data was that comparisons could be made between the matched CSI data and the population data to determine if the matched data was representative of the population. One limitation of the matching approach is that it is subject to sampling error or any flaws that may have been introduced during the sampling process. Essentially, where sampling is concerned the CSI matched data is susceptible to the same limitations as the raw survey data itself. Analyses were performed on CPS survey data, CSI population data, and CSI matched data. Data Cleaning and Structure Data cleaning included the following processes. First, an Age variable was constructed using the survey date and consumer date of birth (DOB). DOBs in the survey population were inconsistent in both format and date in each of the files. In order to maintain a consistent methodological approach when calculating the Age variable, DOBs from CSI were used to calculate Age across all files. Once Age was determined, clients who had a reported age that did not fit their respective categories were removed. Assessment of CDMH Consumer Perception Survey Representativeness Page 2
Therefore, cases in the Adult/Older Adult survey files that contained individuals 17 years and younger were omitted from the analyses. Similarly, individuals 18 years and older were removed from the Child/Family files. It should be noted that possible Transition Age Youth (TAY) may have been removed from the Child/Family files and not included in this analysis. The TAY population is comprised of mental health consumers ages 18 to 25 years who receive services through the children s system of care. The Child/Family databases did not contain an indicator identifying TAY membership and therefore these individuals could not be distinguished from non TAY individuals (ages 18+) in the Child/Family datasets. Second, cases containing Client ID numbers which appeared in the CPS survey data but not in the CSI population data were removed from the files. Third, it was decided that cases with Client IDs that appeared three or more times would be excluded from analysis. These criteria resulted in approximately 12 to 15 percent of cases omitted from each data file. Frequencies were then run on the CPS survey and matched CSI data to create distributions for each of the demographic variables and compare them with those of the CSI population data. This first comparison is referred to as Benchmark #1 and can be found in Appendices A through D. Next, we determined whether duplicate Client IDs (2 appearances) contained uniform responses for each of the demographic variables within the CPS datasets. Duplicate Client IDs that did not contain uniform responses or had missing values were excluded from analysis. This process of exclusion was applied on a rolling basis with replacement. For example, when testing the gender variable, cases satisfying the above criteria were excluded before frequencies were run. The gender data was then fully replaced into the dataset and the same procedure was conducted for the next demographic variable. In this way, each variable could be analyzed separately without affecting outcomes of the remaining demographic variables. The frequencies reported at this phase of the analysis are referred to as Benchmark #2 and are reported in the main body of the report. Testing Representativeness Chi Square tests were run to see if there were statistically significant differences between: 1) CPS survey and CSI population data, and 2) matched CSI data and CSI population data. Chi Square is a test used to determine whether there is a significant difference between the expected frequencies and the observed frequencies in one or more categories. Expected values were derived from the population data (CSI database) for each demographic variable. These expected values were run against observed values for both CPS and matched CSI data. For all chi square tests we selected the.01 level of significance. Essentially, this would mean that if random samples were repeatedly drawn from the CSI population, we would expect them to differ by chance (error) only 1 time in 100 (1% of the time) or less (p.01).. Obtained p values at or below this level indicate statistical significance and warrants the subsequent interpretation of lack of representativeness. Although the.01 significant level was selected as the threshold level for significant differences, results reported in this report revealed consistent differences at the p.001 level. Assessment of CDMH Consumer Perception Survey Representativeness Page 3
STUDY FINDINGS Key Findings Representativeness of Convenience Sample Surveys Two different methods of analyses of four primary demographic variables revealed that neither the November or May administrations of the current Child/Family and Adult/Older Adult Consumer Surveys were representative of the larger CSI population. They significantly differed in: Gender, Ethnicity, Age, and LA versus Non LA Counties. Females are significantly overrepresented in all four surveys and males are significantly underrepresented. Blacks, Asian, Pacific Islanders, Other and Unknown are significantly underrepresented in both surveys. Whites and Hispanics are significantly overrepresented in both surveys. American Indians and Multiple Ethnicities are overrepresented in one survey and underrepresented in another. Whites, American Indian, and Other are underrepresented in both surveys. Blacks and Multiple Ethnicities are overrepresented in both surveys. Hispanics and Asian Pacific Islanders are overrepresented in one survey and underrepresented in another. Children below the age of 13 are underrepresented in both surveys, while youth between 13 17 are overrepresented. Ages 18 29, 30 39, and 70 year olds and above were underrepresented, while ages 40 69 were overrepresented. LA County was overrepresented in the November survey, but underrepresented in the May survey for the Child/Family survey, but was consistently overrepresented on the Adult/Older Adult survey. Data Quality Issues There were 9 major areas where data quality impedes the ability of CDMH to truly know what Consumers think of services or where strengths and weaknesses are found: 1. Lack of congruity between how variables are defined in CSI versus CPS 2. Multiple iterations of files and variables within the agency 3. Non standardized application of methodology to variables across files 4. Duplicate cases contain discrepancies across demographic variables 5. Data in CPS files do not reflect the response categories on surveys 6. Backfilling CPS data with CSI data 7. TAY versus Non TAY 8. Duplicate Client Id s in CPS 9. Inaccuracies in CPS data (e.g., DOB entries) Assessment of CDMH Consumer Perception Survey Representativeness Page 4
Child/Family File November 2006 Reviewing CPS Data and Process for Omitting Cases Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Child/Family target population consists of individuals 0 through 17 years of age. The CPS Child Family data file contained cases of individuals whose age was calculated as over 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Child Family dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the CPS Child Family dataset as TAY or as misclassified data (adults present in children dataset). Therefore, all individuals over the age of 18 years (3%) were omitted from this CPS Child Family dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Over 18 Years of Age Frequency Percentage Original number of cases 26,750 Deleted cases for individuals over 18 799 3.0% Total 25,951 100% An additional data cleaning procedure involved excluding CPS Child Family cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 88% of CPS Child Family cases had a corresponding Client ID in the CSI population dataset. The 12% of cases that did not match up with a CSI Client ID were excluded from the CPS Child Family dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 22,745 88.0% Non Matched 3,206 12.0% Total 25,951 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 5
Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Child Family data set. Duplicate client ID s could have occurred because clients actually took more than one survey or because of data error. As can be seen, Client ID emerged twice in approximately 34% of the cases and 3 to 5 times in a little over 2.5% of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding IDs rows of data Percentage of cases 1 14,471 63.6% 2 7,656 33.7% 3 408 1.8% 4 200.9% 5 10.0% Total 22,745 100% Table 3b contains the number of times a Client ID appeared in the CPS Child Family data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 78% of individuals had their Client ID appear only once and 21% of individuals had their Client ID appear twice. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals Percentage of individuals 1 14,471 78.3% 2 3,828 20.7% 3 136.7% 4 50.3% 5 2.0% Total 18,487 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 6
As can be seen in Table 4, gender matched for one or more duplicate cases or there was missing gender data for one of the duplicate cases 96% of the time. The rest of the duplicate cases (4%) did not match on gender or had a value coded as system missing. As can be seen in Table 5, ethnicity matched on approximately 78% of the duplicate cases. Table 4: Gender discrepancies within CPS duplicate data Frequency Percent Gender matches or missing gender data for one of the duplicate cases 7,354 96.1% Duplicate cases that are a non match or have a double sysmis value 302 3.9% Total 7,656 100% Table 5: Ethnicity discrepancies within CPS duplicate data Frequency Percent Ethnicity matches 5,942 77.6% Ethnicity non matches 1,714 22.4% Total 7,656 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 7
Table 6 contains various data discrepancies that led to the exclusion of 5,802 cases from the dataset based on gender discrepancies and 6,337 cases based on ethnicity discrepancies for the purposes of the current analysis. Table 6: Process for identifying CPS cases with data discrepancies CPS Gender Variable CPS Ethnicity Variable Original number of CPS cases 26,750 26,750 Cases of children over 18 799 799 CPS cases that did not match back to CSI 3,206 3,206 Client Id s that had 3 or more duplicates 618 Duplicate cases that are a nonmatch or have a double sysmis value 302 Cases with one sysmis value 877 New number of CPS cases 20,948 *Represents the number of cases with one sysmis value or a non match on DOB. 618 1,714 0* 20,413 Benchmark #1 See Appendix A for data Benchmark #2 Assessment of CDMH Consumer Perception Survey Representativeness Page 8
Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist between the Data: Table 7 contains gender data across three datasets: the CPS Child Family dataset in which 4% of the gender data was missing, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and finally, the CSI population data for the time period of July 1, 2006 to June 30, 2007. Table 7: Gender discrepancies between CPS Child/Family November 2006 and CSI data CPS data Matched CSI data CSI population data N % N % N % Female 8,446 40.3% 8,722 40.0% 77,479 38.8% Male 12,482 59.6% 13,080 59.9% 121,623 60.9% Other 20.1% 23.1% 557.3% Total 20,948 100% 21,825 100% 199,659 100% Chi square tests were performed to determine if there were significant differences between actual or observed values for gender in the CPS data and expected values based on the CSI population data. As can be seen in Table 8, there is a statistically significant difference between what CPS Data reports for Gender and the CSI Population. Where Gender is concerned, CPS data is not representative of the CSI Population. Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8,446 40.3% 8,128 38.8% 318 Male 12,482 59.6% 12,757 60.9% 275 Other 20.1% 63.3% 43 Total 20,948 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 9
* significant at p =.001, df = 2. Table 9 contains Chi Square results for Matched CSI and CSI population gender data. There is a statistically significant difference between what the Matched CSI data reports for Gender and the CSI Population. Where Gender is concerned, Matched CSI data is not representative of the CSI Population. Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8,722 40.0% 8,468 38.8% 254 Male 13,080 59.9% 13,291 60.9% 211 Other 23.1% 66.3% 43 Total 21,825 100% * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 10
Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist between the Data: Table 10 contains ethnicity data across three datasets: the CPS Child Family data, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and the CSI population data for the time period of July 1, 2006 to June 30, 2007. Table 10: Ethnicity Discrepancies between CPS and CSI data CPS data Matched CSI data CSI Population data N % N % N % White 5,240 25.7% 5,923 29.0% 50,406 25.2% Hispanic 9,454 46.3% 7,705 37.7% 74,118 37.1% Black 2,599 12.7% 2,841 13.9% 30,606 15.3% Asian or Pacific Islander 473 2.3% 493 2.4% 5,221 2.6% American Indian 124.6% 127.6% 1,388.7% Other 316 1.5% 577 2.8% 7,754 3.9% Multiple Ethnicities 1,245 6.1% 1,434 7.0% 12,007 6.0% Unknown 962 4.7% 1,313 6.4% 18,159 9.1% Total 20,413 100% 20,413 100% 199,659 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 11
Table 11 contains Chi Square results for CPS Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the CPS data reports for Ethnicity and the CSI Population. Where Ethnicity is concerned, CPS data is not representative of the CSI Population. Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5,240 25.7% 5,149 25.2% 91 Hispanic 9,454 46.3% 7,581 37.1% 1,873 Black 2,599 12.7% 3,126 15.3% 527 Asian or Pacific Islander 473 2.3% 531 2.6% 58 American Indian 124.6% 143.7% 19 Other 316 1.5% 797 3.9% 481 Multiple Ethnicities 1,245 6.1% 1,226 6.0% 19 Unknown 962 4.7% 1,859 9.1% 897 Total 20,413 100% * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 12
Table 12 contains Chi Square results for Matched CSI Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the Matched CSI data reports for Ethnicity the CSI Population. Where ethnicity is concerned, Matched CSI data is not representative of the CSI Population. Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5,923 29.0% 5,149 25.2% 774 Hispanic 7,705 37.7% 7,581 37.1% 124 Black 2,841 13.9% 3,126 15.3% 285 Asian or Pacific Islander 493 2.4% 531 2.6% 38 American Indian 127.6% 143.7% 16 Other 577 2.8% 797 3.9% 220 Multiple Ethnicities 1,434 7.0% 1,226 6.0% 208 Unknown 1,313 6.4% 1,859 9.1% 546 Total 20,413 100% * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 13
Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 13 contains Chi Square results for CPS Ethnicity Data and CSI Population Data on county: Los Angeles (LA County) versus all other counties (non LA County). There is no statistically significant difference between what the CPS data reports for LA versus Non LA and the CSI Population. Where county dichotomy is concerned, CPS data is representative of the CSI Population. Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N LA County 5,728 25.9% 5,665 25.6% 64 Non LA County 16,399 74.1% 16,463 74.4% 64 Total 22,127 100% * not significant Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data Table 14 contains the ages of cases in the CPS Child Family data file grouped into 3 categories, 0 to 5 years, 6 to 12 years and 13 to 17 years. One can see the under and over representativeness of the CPS dataset across age categories by comparing the CPS percentages with the CSI percentages. Table 14: Age Distributions across CPS and CSI Population for Children in 2006 Age Categories in Years CPS Data CSI Population 2006 N % N % 0 5 1,318 6.0% 20,091 10.1% 6 12 8,380 37.9% 79,387 39.8% 13 17 12,429 56.2% 100,181 50.2% Total 22,127 100% 199,659 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 14
As can be seen in Table 15, there is a statistically significant difference between what the CPS data reports for Age and the CSI population. Where Age is concerned, CPS data is not representative of the CSI Population. Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Age Categories in Years Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N 0 5 1,318 6.0% 2,233 10.1% 915 6 12 8,380 37.9% 8,798 39.8% 418 13 17 12,429 56.2% 11,097 50.2% 1,332 Total 22,127 100% * significant at p =.001, df = 1. Assessment of CDMH Consumer Perception Survey Representativeness Page 15
Child/Family File May 2007 Reviewing CPS Data and Process for Omitting Cases: Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Child/Family target population consists of individuals 0 through 17 years of age. The CPS Child Family data file contained cases of individuals whose age was calculated as over 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Child Family dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the Child Family CPS dataset as TAY or as misclassified data (adults present in children dataset). Therefore, all individuals over the age of 18 years (4%) were omitted from this CPS Child Family dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Over 18 Years of Age Frequency Percentage Original number of cases 26,407 Deleted cases for individuals over 18 946 3.6% Total 25,461 100% An additional data cleaning procedure involved excluding Child Family CPS cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 87% of CPS Child Family cases had a corresponding Client ID in the CSI population dataset. The 13% of cases that did not match up with a CSI Client ID were excluded from the CPS Child Family dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 21,932 87.0% Non Matched* 3,529 13.0% Total 25,461 100% * Non matched cases were subsequently removed from the CPS file. Assessment of CDMH Consumer Perception Survey Representativeness Page 16
Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Child Family data set. As can be seen, Client ID emerged twice in approximately 34% of the cases and 3 to 6 times in 3 % of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding rows of data Percentage of cases 1 13,923 63.5% 2 7,372 33.6% 3 372 1.7% 4 232 1.1% 5 15.1% 6 18.1% Total 21,932 100% Table 3b contains the number of times a Client ID appeared in the CPS Child Family data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 78% of individuals had their Client ID appear only once and 21% of individuals had their Client ID appear twice. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals Percentage of individuals 1 13,923 78.3% 2 3,686 20.7% 3 124.7% 4 58.3% 5 3 0% 6 3 0% Total 17,797 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 17
As can be seen in Table 4, gender matched for one or more duplicate cases or there was missing gender data for one of the duplicate cases 96% of the time. The rest of the duplicate cases (4%) did not match on gender or had a value coded as system missing. As can be seen in Table 5, ethnicity matched on approximately 77% of the duplicate cases. Table 4: Gender discrepancies within CPS duplicate data Frequency Percent Gender matches or missing gender data for one of the duplicate cases 7,080 96% Duplicate cases that are a non match or have a double sysmis value 292 4% Total 7,372 100% Table 5: Ethnicity discrepancies within CPS duplicate data Frequency Percent Ethnicity matches 5,638 76.5% Ethnicity non matches 1,734 23.5% Total 7,372 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 18
Table 6 contains various data discrepancies that led to the exclusion of 6,540 cases based on gender discrepancies and 6,846 cases based on ethnicity discrepancies from the dataset for the purposes of the current analysis. Table 6: Process for identifying CPS cases with data discrepancies CPS Gender Variable CPS Ethnicity Variable Original number of CPS cases 26,407 26,407 Adults >18 yrs. 946 946 CPS cases that did not match back to CSI 3,529 3,529 Client Id s that had 3 or more duplicates 637 637 Duplicate cases that are a non match or have a double sysmis 1,734 value 292 Cases with one sysmis value 1,136 0* New number of CPS cases 19,867 *Represents the number of cases with one sysmis value or a non match on DOB. 19,561 Benchmark #1 See Appendix A for data Benchmark #2 Comparing Gender Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 7 contains gender data across three datasets: the CPS Child Family dataset in which 5.4% of the gender data was missing, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and finally, the CSI population data for the time period of July 1, 2006 to June 30, 2007. Table 7: Gender discrepancies between CPS and CSI data CPS data* Matched CSI data** CSI population data*** N % N % N % Female 8,036 40.4% 8,390 39.9% 74,125 38.8% Male 11,812 59.5% 12,592 60.0% 116,415 60.9% Other 19.1% 21.1% 542.3% Total 19,867 100% 21,003 100% 191,082 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 19
Chi square tests were performed to determine if there were significant differences between actual or observed values for gender in the CPS data and expected values based on the CSI population data. As can be seen in Table 8, there is a statistically significant difference between what CPS Data reports for Gender and the CSI Population. Where Gender is concerned, CPS data is not representative of the CSI Population. Table 8: Difference between CPS and CSI Gender Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8,036 40.4% 7,708 38.8% 328 Male 11,812 59.5% 12,099 60.9% 287 Other 19.1% 60.3% 41 Total 19,867 100% * significant at p =.001, df = 2. Table 9 contains chi square results for Matched CSI and CSI population gender data. There is a statistically significant difference between what the Matched CSI data reports for Gender and the CSI Population. Where Gender is concerned, Matched CSI data is not representative of the CSI Population. Table 9: Difference between Matched CSI Subset and Total CSI Population Gender Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N Female 8,390 39.9% 8,149 38.8% 241 Male 12,592 60.0% 12,791 60.9% 199 Other 21.1% 63.3% 42 Total 21,003 100% * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 20
Comparing Ethnicity Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 10 contains ethnicity data across three datasets: the CPS Child Family data, the Matched CSI data set which is CSI data that was matched back to the CPS Child Family dataset based on Client ID, and the CSI population data for the time period of July 1, 2006 to June 30, 2007. Table 10: Ethnicity discrepancies between CPS and CSI data CPS data Matched CSI data CSI Population data N % N % N % White 4,945 25.3% 5,693 29.1% 48,096 25.2% Hispanic 8,948 45.7% 7,656 39.1% 71,239 37.3% Black 2,291 11.7% 2,514 12.9% 29,109 15.2% Asian or Pacific Islander 466 2.4% 531 2.7% 4,893 2.6% American Indian 163.8% 108.6% 1,337.7% Other 336 1.7% 560 2.9% 7,444 3.9% Multiple Ethnicities 1,147 5.9% 1,453 7.4% 11,660 6.1% Unknown 1,265 6.5% 1,046 5.3% 17,304 9.1% Total 19,561 100% 19,561 100% 191,082 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 21
Table 11 contains Chi Square results for CPS Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the CPS data reports for Ethnicity and the CSI Population. Where Ethnicity is concerned, CPS data is not representative of the CSI Population. Table 11: Difference between CPS and CSI Ethnicity Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 4,945 25.3% 4,924 25.2% 21 Hispanic 8,948 45.7% 7,289 37.3% 1659 Black 2,291 11.7% 2,970 15.2% 679 Asian or Pacific Islander 466 2.4% 508 2.6% 42 American Indian 163.8% 137.7% 26 Other 336 1.7% 762 3.9% 426 Multiple Ethnicities 1,147 5.9% 1,192 6.1% 45 Unknown 1,265 6.5% 1,778 9.1% 513 Total 19,561 100% * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 22
Table 12 contains Chi Square results for Matched CSI Ethnicity Data and CSI Population Data. There is a statistically significant difference between what the Matched CSI data reports for Ethnicity the CSI Population. Where ethnicity is concerned, Matched CSI data is not representative of the CSI Population. Table 12: Difference between Matched CSI Subset and Total CSI Population Ethnicity Data: Expected versus Observed Values* Observed N for Matched CSI Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N White 5,693 29.1% 4,924 25.2% 769 Hispanic 7,656 39.1% 7,289 37.3% 367 Black 2,514 12.9% 2,970 15.2% 456 Asian or Pacific Islander 531 2.7% 508 2.6% 23 American Indian 108.6% 137.7% 29 Other 560 2.9% 762 3.9% 202 Multiple Ethnicities 1,453 7.4% 1,192 6.1% 261 Unknown 1,046 5.3% 1,778 9.1% 732 Total 19,561 100% * significant at p =.001, df = 7. Assessment of CDMH Consumer Perception Survey Representativeness Page 23
Comparing LA versus Non LA Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 13 contains Chi Square results for CPS Ethnicity Data and CSI Population Data on county: Los Angeles (LA County) versus all other counties (non LA County). There is a statistically significant difference between what the CPS data reports for LA versus Non LA and the CSI population. Where county dichotomy is concerned, CPS data is not representative of the CSI population. Table 13: Difference between CPS and CSI Dichotomous County Data: Expected versus Observed Values* Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N LA County 5,068 23.8% 5,430 25.5% 362 Non LA County 16,227 76.2% 15,865 74.5% 362 Total 21,295 100% * significant at p =.001, df = 1. Comparing Age Distributions across Files and Determining Whether Significant Differences Exist Between the Data: Table 14 contains the ages of cases in the CPS Child Family data file grouped into 3 categories, 0 to 5 years, 6 to 12 years and 13 to 17 years. Table 14: Age Distributions across CPS and CSI Population for Children in 2007 Age Categories in CPS Data CSI Population 2007 Years N % N % 0 5 1,212 5.7% 16,581 8.7% 6 12 7,644 35.9% 75,700 39.6% 13 17 12,439 58.4% 98,801 51.7% Total 21,295 100% 191,082 100% Assessment of CDMH Consumer Perception Survey Representativeness Page 24
As can be seen in Table 15, there is a statistically significant difference between what the CPS data reports for Age and the CSI population. Where Age is concerned, CPS data is not representative of the CSI Population. Table 15: Difference between CPS and CSI Age Data: Expected versus Observed Values* Age Categories in Years Observed N for CPS Data Expected N Based on CSI Population Distribution Difference Between Observed & Expected N s N % N % N 0 5 1,212 5.7% 1,853 8.7% 641 6 12 7,644 35.9% 8,433 39.6% 789 13 17 12,439 58.4% 11,010 51.7% 1,429 Total 21,295 100% 21,295(6) 100% * significant at p =.001, df = 2. Assessment of CDMH Consumer Perception Survey Representativeness Page 25
Adult/Older File November 2006 Reviewing CPS Data and Process for Omitting Cases: Tables 1 through 6 reflect stages of data cleaning that preceded descriptive and chi square data analyses. The Adult/Older Adult target population consists of individuals years of age and older. The CPS Adult Older Adult data file contained cases of individuals whose age was calculated as under 18 years old (Table 1). Transition age youth (TAY) are classified as 18 to 25 year old individuals in the Child Services system. The CPS Adult Older Adult dataset did not contain a TAY indicator and therefore it was impossible to identify those 18 years and over in the CPS Adult Older Adult dataset as TAY or not. All individuals under the age of 18 years (46) were omitted from this CPS Adult Older Adult dataset for the purposes of the current analyses. Table 1: Number of Cases for Individuals Under 18 Years of Age Frequency Percentage Original number of cases 26,232 Deleted cases for children under 18 46 <0% Total 26,186 100% An additional data cleaning procedure involved excluding CPS Adult Older Adult cases that did not contain a matching Client ID in the CSI population dataset. As the overall purpose of the current analyses was to assess the representativeness of the CPS samples, it was critical to eliminate data that was not present in the population. As can be seen from Table 2, 86% of CPS Adult Older Adult cases had a corresponding Client ID in the CSI population dataset. The 15% of cases that did not match up with a CSI Client ID were excluded from the CPS Adult Older Adult dataset for the purposes of the current analyses. Table 2: Number of Cases in CPS that Matched a Client ID in CSI Population Data Frequency Percentage Matched 22,400 85.5% Non Matched 3,786 14.5% Total 26,186 100% Assessment of CDMH CPS Survey Representativeness Page 26
Duplicate records or cases are captured in Table 3a which presents the number of times a Client ID appeared in the CPS Adult Older Adult data set. As can be seen, Client ID emerged only once in 94% of the cases. Table 3a: Number of times a Client ID appeared in data file Number of times a Client ID appeared in data file Number of corresponding rows of data Percentage of cases 1 20,949 93.5% 2 1,314 5.9% 3 96.4% 4 20.1% 5 15.1% 6 6.0% Total 22,400 100% Table 3b contains the number of times a Client ID appeared in the CPS Adult Older Adult data file and the number of individuals for whom this was the case. In order to compute the number of individuals present in the data file, the number of rows of data (see Table 2a) was divided by the corresponding number of times a single Client ID emerged in the data file (see Table 2a). As can be seen in Table 3b, 97% individuals had their Client ID appear only once. Table 3b: Number of individuals represented by duplicate Client ID s Number of times a Client ID appeared in data file Number of individuals* Percentage of individuals 1 20,949 96.8% 2 657 3.0% 3 32.1% 4 4.0% 5 3.0% 6 1.0% Total 21,646 100% Assessment of CDMH CPS Survey Representativeness Page 27