UPDATE ON THE ARIZONA EVALUATION PROJECT. Wayne D. Parker, PhD May 2, 2006

Transcription

1 UPDATE ON THE ARIZONA EVALUATION PROJECT Wayne D. Parker, PhD May 2, 2006

2 Elements of Project 1) Measurement Model 2) Predictive Model

3 Measurement Model Answers the question: Is this assessment technique valid and reliable?

4 Measurement Model Using factor analysis, the instrument was found to measure overall self-sufficiency and 2 subscales: Level of Independence Skills Level of Dysfunction

5 Measurement Model Reliability is essentially a measure of the amount of error in the scale The 3 scores (Independence, Dysfunction, and Total Score) were all found to have acceptable reliability

6 Measurement Model While the Arizona Self Sufficiency Matrix was found to be reliable and valid within all three Continua of Care, the typical client in each Continuum was found to score differently

7 CONTINUA SELF SUFFICIENCY SCORES Maricopa DYSFUNCTION Pima Rural INDEPENDENCE

8 Predictive Model Equations are generated from the data to determine the predictors of change in Independence, Dysfunction, and Total Self Sufficiency

9 Predictive Model These equations are then used to predict the amount of change in each individual client if randomly assigned to a homeless program. Each individual s predicted change is uniquely determined based upon the client s individual characteristics These predicted changes constitute the expected change

10 Predictive Model The predicted change for each individual is then compared with the actual change Agencies whose clients consistently do better than predicted are successful Agencies whose clients consistently underperform the predicted change are unsuccessful Agencies whose clients perform consistently with predictions are typical

11 Predictive Model All these steps have been taken with the data gathered between Oct 2005 and March 2006 Powerful results have been obtained and as increased data is collected going forward the accuracy will increase with each quarterly iteration

12 Response Rate For the predictive model to be truly accurate a high response rate/compliance is critical. A compliance rate (the percent of clients from each agency for whom full data is obtained) of 80 % is fair, 90 % is good, 95 % is excellent.

13 Response Rate Within the Maricopa Continuum of Care more than 10,000 new clients were entered into HMIS during the last 6 months Of these clients about 1900 have had an entry matrix completed a rate of less than 19%

14 Response Rate The response within each agency has ranged from 100 % to less than 1 % Of 21 agencies to have new clients entered into HMIS only 4 have had a compliance rate greater than 50 % 100 % at Phoenix Shanti 78 % at Mesa CAN 63 % at SW Leadership Foundation 60 % at META

15 Response Rate 12 agencies have a compliance rate of less than 25 % Without an adequate compliance rate the project cannot be successful

16 Preliminary Overall Findings The distinction between emergency and transitional programs appears to be an arbitrary definition there is no difference between the two types of programs in who they serve, the types and extent of problems their residents exhibit, and the expected change is the same from both programs.

17 Preliminary Overall Findings There is a window between 22 days and 47 days when programs have strong impact. Less than 22 days is insufficient to effect change and clients staying more than 47 days tend to regress. (Difference for families) This suggests that medium term stabilization between 3 and 7 weeks followed by placement in long term housing is likely to maximize client impact.

18 Additional Variables Maricopa Continuum of Care supplements the data set with measures of history of homelessness and primary reason for homelessness These additional variables considerably strengthen the ability to identify clients particularly appropriate or inappropriate for referral to a specific agency

19 Referral Profiles Agencies can be ranked from most to least effective from the results obtained thus far Profiles can be generated describing the clients most likely and least likely to benefit from a given agency

20 Profile A This agency is an exception to the 22 to 47 day window. Continues to positively impact longer term placements. Particularly effective with those homeless because of natural disasters and families. The following clients are poor referrals: Victims of domestic violence Parents with high potential for child abuse Those homeless primarily because of financial problems

21 Profile B This agency is the most successful in Maricopa County in having positive impact with those recently released from jail or prison. Agency is among the least effective in working with homeless Hispanics

22 Profile C This agency is very effective in meeting the needs of homeless families who were relatively high functioning prior to homelessness. Least effective agency in having positive impact with homeless Indians. While somewhat effective with mentally ill homeless, ineffective with homeless substance abusers.

23 Profile D This agency primarily serves older and/or disabled homeless. Highly effective with older clients. Much less effective with disabled. Needs to determine if they should further develop the older niche or make programmatic changes to be more effective with the disabled