CONSTITUENT VOICE FISCAL YEAR 2016 RESULTS REPORT

Transcription

1 CONSTITUENT VOICE FISCAL YEAR 2016 RESULTS REPORT April 2015 to May 2016 INTRODUCTION At LIFT, we strongly believe our members are the experts on their own lives. And, when working with LIFT staff, they drive the agenda in setting their goals and the path to achieve them. In 2013, this commitment to member-driven work led LIFT to implement Constituent Voice, a process for systematically gathering input from those we serve. Through Constituent Voice, we ask our members to complete short surveys about the quality of LIFT services as well as their own social connections and personal well-being (e.g., resilience and self-efficacy). We analyze the responses to each question to learn what we re doing well and what we need to improve, and to gauge members perceptions of their strengths and weaknesses. In addition, we match members survey responses against measures of their progress at LIFT (like the number of goals they ve completed) to explore whether or not survey responses are systematically related to member progress (spoiler alert: they are!). WHAT IS CONSTITUENT VOICE? Constituent Voice is an approach for systematically collecting member feedback and looping that information back into program design and implementation. Constituent Voice encompasses a member survey, member interviews, focus groups and the process of making changes based on member input. This report primarily focuses on the results from our survey. This report presents key findings from an in-depth, third party analysis of survey responses collected between April 2015 and May All analysis was conducted by an independent researcher to ensure high-quality, unbiased results. In total, 3,380 members completed the survey over this time period; 62 percent of respondents were women and 38 percent were men. The members were spread across six regions Boston, Chicago, Los Angeles, New York, Philadelphia and Washington, D.C. 1 The overall survey response rate across all regions was 60 percent. 1 Responses in Boston, Philadelphia and Chicago were collected through July 2015, when those offices paused or ceased operations. Regional breakdowns in this report focus on the three regions that remained open for the entirety of the reporting period Washington, D.C., Los Angeles and New York. 1

2 HOW DOES LIFT S CONSTITUENT VOICE SURVEY WORK? LIFT s Constituent Voice survey acts as a real-time vehicle for capturing member input. Members complete short surveys at the end of each in-person meeting on one of several ipads in each LIFT office. Survey questions cover five broad categories: (1) service quality, (2) relationship quality, (3) loyalty and engagement, (4) social connections and (5) personal well-being. Each survey contains five to eight questions that have been selected from a larger pool, and members answer a different set of questions each time they respond. This approach allows LIFT to collect information on a diverse set of topics while limiting the burden on individual members, as each survey only takes one to two minutes to complete. The survey questions are each phrased as a sentence. For example, Today at LIFT, I was treated with courtesy, dignity and respect. Members are then asked to rate the extent to which they agree with the statement using an 11-point scale, where 0 denotes strongly disagree and 10 denotes strongly agree. To analyze the data, we split members into three groups depending on their responses to each question: 1. High Scorers gave a score of 9 or 10 on the 0-10 scale 2. Neutral Scorers gave a score of 7 or 8 on the 0-10 scale 3. Low Scorers gave a score of 0 through 6 on the 0-10 scale This approach is drawn from the Net Promoter Score (NPS) methodology which is widely used in the corporate sector to effectively measure customer engagement. Note that the labels used in NPS are typically Promoters, Neutrals and Detractors, but LIFT adapted the language for the purposes of this analysis. 2

3 KEY FINDINGS This report presents several key findings from an in-depth, third party analysis of LIFT s Constituent Voice survey responses. 1. LIFT members are engaged and view the organization positively. Overall, members rated LIFT very highly on questions related to service quality, relationship quality and loyalty and engagement; 3 out of 4 members were high scorers (score of 9 or 10 on 0-10 scale) across almost every question related to LIFT services. 2. Engaged members made up to three times as much progress on their goals (e.g., accessing employment and benefits). Members who rated LIFT services highly completed more goals than those who gave lower ratings; high scorers completed up to 2.7 times as many goals as low scorers during their time at LIFT. This suggests higher engagement supports progress on goals. 3. Members who felt they made progress during their first meeting were more likely to return for subsequent visits. Members who reported making progress on their goals in their first meeting were more than 20 percent more likely to return for subsequent meetings an early predictive signal of retention. 4. Women were less likely to report having strong social connections and personal well-being. Women were less likely than men to be high scorers on questions related to social connections and personal well-being. The difference was especially stark when members were asked if they have family, friends, or neighbors in their life that support them; on this question, women were 15 percent less likely to be high scorers than men. 5. Members social connections and personal well-being showed improvement while working with LIFT. Members reported having greater social connections and personal well-being after completing meetings at LIFT. Half of members reported increased social connections and about 40 percent reported increased personal well-being (e.g., resilience and self-efficacy) after engaging with LIFT. 6. Stronger social connections and personal well-being translated to more progress on goals. Members who reported having strong social connections and personal wellbeing completed more goals than members who did not; high scorers completed up to 1.6 times as many goals as low scorers during their time at LIFT. 3

4 2 LIFT S CONSTITUENT VOICE SURVEY QUESTIONS 2 Service Quality 4. When I need a meeting, I can get one. 5. I get new and useful information when I come to LIFT. 6. I got a lot of work done today because my advocate was knowledgeable. (Advocate is the volunteer that you met with today.) 7. My advocate encourages me when I doubt myself. (Advocate is the volunteer that you met with today.) 8. When I have a hard problem, my advocate finds a way to fix it. (Advocate is the volunteer that you met with today.) Relationship Quality 9. Today at LIFT, I was treated with courtesy, dignity and respect. 10. Today, LIFT helped me with what is most important to me. 11. I think that LIFT will use my answers to this survey to improve its services. 12. With LIFT s help, I feel like I am making progress on my goals. Loyalty & Engagement 13. I plan to come back to LIFT again. 14. I would recommend LIFT to a friend or relative. Social Connections 15. Thanks to LIFT, I am more connected to community resources that can help me. 16. I know people who have overcome the same problems. 17. I have family, friends or neighbors in my life that support me. 18. In a time of need, I feel like I can come to LIFT for help. Personal Well-Being 19. I know I ll be okay, even if something doesn t go my way. 20. If I want to reach a goal, I try hard to achieve it and I never give up. 21. I believe I can achieve the goals I worked on at LIFT today. 2 Questions 1-3 asked for basic member demographic information, like gender and race/ethnicity. 4

5 RESULTS FINDING 1: LIFT members are engaged and view the organization positively. More than three out of four members were high scorers across almost every question related to LIFT services. Fewer than 13 percent of members were low scorers on any question. Among all questions on LIFT services, the average percent of members who were high scorers was 82 percent and the average percent who were low scorers was 7 percent. Members provided the highest scores on Q9 (Today at LIFT, I was treated with courtesy, dignity and respect) and Q13 (I plan to come back to LIFT again). For both questions, about 9 in 10 members were high scorers. Members provided the lowest scores for Q4 (When I need a meeting, I can get one) and Q12 (With LIFT s help, I feel like I am making progress on my goals). Figure 1: Percent of respondents who were high scorers on questions on service quality, relationship quality and loyalty and engagement Members rated LIFT services highly across all questions High Scorer (Score of 9 or 10 on 0-10 scale) 73% 82% 76% 82% 82% 91% 84% 84% 75% 88% 84% Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Service Quality Relationship Quality Loyalty & Engagement 5

6 Women gave lower ratings on some questions On average, men were slightly more likely to be high scorers than women across all questions but the difference was small: 83 percent vs. 82 percent. This trend was mainly driven by differences on two specific questions, shown below in Table 1. Table 1: Differences between women and men in percent of respondents who were high scorers Question Women Men Diff. Q5: I get new and useful information when I come to LIFT. 81% 83% 2% Q12: With LIFT s help, I feel like I am making progress on my goals. 72% 78% 6% Note: These differences were statistically significant at the p<.05 level when looking at the breakdown between high scorers, neutral scorers, and low scorers. The above trends held across all regions with the exception of Q5 (I get new and useful information when I come to LIFT) in Washington, D.C., where slightly more women were high scorers than men (84 percent vs. 80 percent). Parents felt they were making less progress on their goals Parents of children age 18 and younger for the most part rated LIFT services similarly to members without children, except on three questions. 3 On these three questions all related to attending meetings and making progress parents were less likely than non-parents to be high scorers, as shown below in Table 2. Table 2: Differences between parents and non-parents in percent of respondents who were high scorers Question Parent Non-parent Diff. Q4: When I need a meeting, I can get one. 71% 74% 3% Q6: I got a lot of work done today because my advocate was knowledgeable. Q12: With LIFT s help, I feel like I am making progress on my goals. 74% 78% 4% 71% 75% 4% Note: These differences were statistically significant at the p<.05 level when looking at the breakdown between high scorers, neutral scorers, and low scorers. The above trends held across all regions with the exception of Q12 in New York and Q6 in Los Angeles; for those questions and regions, parents were slightly more likely to be high scorers than non-parents (71 percent vs. 69 percent for Q12 in New York and 77 percent vs. 76 percent for Q6 in Los Angeles). 3 While LIFT now primarily serves parents and caregivers of children age 8 and younger, during the time period of this report (FY 2016), LIFT served both parents and non-parents. This section explores the differences between members who were parents and caregivers of children age 18 and younger versus members who were not. 6

7 Member feedback varied by region Members at LIFT-Los Angeles were the most likely to be high scorers, followed by LIFT-DC, and then LIFT-New York. The average percent of members who were high scorers was 85 percent for LIFT-Los Angeles, 82 percent for LIFT-DC and 75 percent for LIFT-New York. Differences were particularly large for Q12 (With LIFT s help, I feel like I am making progress on my goals), Q7 (My advocate encourages me when I doubt myself), and Q8 (When I have a hard problem, my advocate finds a way to fix it). Figure 2: Differences between regions in percent of respondents who were high scorers, by question type (service quality, relationship quality and loyalty and engagement) Members were most likely to be high scorers in Los Angeles LA DC NY 83% 85% 87% 80% 83% 87% 78% 78% 72% Service Quality (Q4-Q8) Relationship Quality (Q9-Q12) Loyalty & Engagement (Q13-Q14) Note: These differences were statistically significant at the p<.05 level when looking at the breakdown between high scorers, neutral scorers, and low scorers. 7

8 FINDING 2: Engaged members made up to three times as much progress on their goals (e.g., accessing employment and benefits). Members who gave high scores on questions related to LIFT services completed up to 2.7 times as many goals as members who gave low scores. Table 3 below shows the difference in the average number of completed goals between high scorers and low scorers for each question on LIFT services. As seen below, high scorers on Q13 (I plan to come back to LIFT again) completed 2.7 times as many goals as low scorers. High scorers on Q9 (Today at LIFT I was treated with courtesy, dignity and respect) completed more than twice as many goals as low scorers. To produce the below results, we employed a negative binomial regression model that appropriately fit the characteristics of our data count data that was highly skewed (since some members didn t complete any long-term goals in the time period). The model controlled for the number of meetings that members completed, the number of days over which members attended meetings and the total number of goals that members set for themselves. The analysis did not take into account the timing of survey responses relative to the timing of member progress. Table 3: Differences between high scorers and low scorers in the average numbers of completed goals 4 Question Strong confidence that these results apply to members generally High scorer progress (compared to low scorers) Q13: I plan to come back to LIFT again. 2.7 times Q11: I think that LIFT will use my answers to this survey to improve its services. HOW DOES LIFT MEASURE PROGRESS? At LIFT, our members set big goals and then outline the smaller steps needed to achieve them. To measure our members progress, we keep track of their big, long-term successes (such as securing a full-time job or enrolling in higher education) as well as their smaller, shorter-term successes (such as completing a resume or submitting applications). For this analysis, completed goal means the goal was fully completed or the member achieved an interim outcome (such as securing a part-time job on the way to a full-time job). 1.8 times Q5: I get new and useful information when I come to LIFT. 1.6 times Q12: With LIFT s help, I feel like I am making progress on my goals. 1.5 times Q4: When I need a meeting, I can get one. 1.4 times Moderate confidence that these results apply to members generally 8

9 4 Q9: Today at LIFT, I was treated with courtesy, dignity and respect. 2.1 times Q14: I would recommend LIFT to a friend or relative. 1.4 times Q10: Today, LIFT helped me with what is most important to me. 1.4 times Q6: I got a lot of work done today because my advocate was knowledgeable. No confidence that these differences apply to members generally 1.3 times Q7: My advocate encourages me when I doubt myself. no difference Q8: When I have a hard problem, my advocate finds a way to fix it. no difference When looking region-by-region, there were typically not enough data to accurately measure the difference between high scorers and low scorers within just one region. However, in the few instances where there were enough data, the trends above mostly held. In Washington, D.C., the difference between high scorers and low scorers on Q13 (I plan to come back to LIFT again) and Q11 (I think that LIFT will use my answers to this survey to improve its services) were even more pronounced than when looking at all regions combined; high scorers on Q13 completed 4 times as many goals and high scorers on Q11 completed 3 times as many goals compared to low scorers on those questions. High scorers also completed more action steps Overall, high scorers completed more action steps than low scorers. However, compared to differences in the number of completed goals (shown above), there were fewer questions that displayed a convincing difference between high scorers and low scorers on the number of completed action steps. 4 Our confidence that differences apply to members generally (as opposed to being specific to this group of members) is based on the following cut-offs for p-values: strong: <.05, moderate: , no confidence: >=.25. 9

10 FINDING 3: Members who felt they made progress during their first meeting were more likely to return for subsequent visits. Members who were high scorers on Q12 (With LIFT s help, I feel that I am making progress on my goals) in a survey taken right after their first meeting were more than 20 percent more likely than low scorers to return for subsequent meetings an early predictive signal of retention. Among high scorers on this question, 80 percent returned for a second visit. Among low scorers, only 57 percent returned for a second visit. This analysis was based on 239 members who were given the survey right after their first meeting instead of waiting until their second meeting. This below trend typically held among other questions asked right after the first meeting; however, the differences were smaller than for Q12 or there was not enough data to have confidence in the results. Furthermore, the number of respondents per region was too small to accurately measure the difference between high scorers and low scorers by individual region. Figure 3: Difference between high scorers and low scorers in percent of members returning for a second meeting on the question With LIFT s help, I feel that I am making progress on my goals Members who felt they made progress during their first meeting were 23% more likely to return 80% Percent returning for a second meeting 57% High Scorers Low Scorers Note: This difference was statistically significant at the p<.05 level. 10

11 FINDING 4: Women were less likely to report having strong social connections and personal well-being Women were less likely than men to be high scorers on most questions related to social connections and personal well-being. Differences were particularly stark for Q17 (I have family, friends or neighbors in my life that support me), Q16 (I know people who have overcome the same problems) and Q19 (I know I ll be okay, even if something doesn t go my way), as shown in Table 4 below. Women and men who took the survey were equally likely to be married and to be employed. Furthermore, men were actually more likely than women to be homeless. This suggests that women s lower levels of social connections and personal well-being were not caused by differences in their marital status, employment or housing as compared to men (i.e. the differences were not due to the fact that women were more likely to be single or unemployed). However, women who took the survey were more likely to be undocumented and less likely to speak English than men who took the survey, which could have led to some of the differences. Table 4: Differences between women and men in percent of respondents who were high scorers Question Women Men Diff. Social Connections Q15: Thanks to LIFT, I am more connected to community resources that can help me. 73% 78% 5% Q16: I know people who have overcome the same problems. 42% 50% 8% Q17: I have family, friends or neighbors in my life that support me. 40% 54% 14% Q18: In a time of need, I feel like I can come to LIFT for help. 86% 86% 0% Personal Well-Being Q19: I know I ll be okay, even if something doesn t go my way. 56% 64% 8% Q20: If I want to reach a goal, I try hard to achieve it and I never give up. Q21: I believe I can achieve the goals I worked on at LIFT today. 74% 78% 4% 80% 80% 0% Note: All differences (excluding Q18 and Q21) were statistically significant at the p<.05 level when looking at the breakdown between high scorers, neutral scorers, and low scorers. The trends above were less prominent when looking at individual regions, likely due to a smaller amount of data when split out by region. 11

12 FINDING 5: Members social connections and personal well-being showed improvement while working with LIFT. Among members who answered both Q16 (I know people who have overcome the same problems) and Q17 (I have family, friends or neighbors in my life that support me) more than once, 51 percent increased their scores on one or both questions from their first response to their later responses, indicating a rise in their social connections. For members who answered both Q19 (I know I ll be okay, even if something doesn t go my way) and Q20 (If I want to reach a goal, I try hard to achieve it and I never give up) more than once, 39 percent increased their scores on one or both questions from their first response to their later responses, indicating an increase in their personal well-being. Table 5. Percent of members with an increase on one or both questions from first response to later responses, by question type Percent with Questions increase Social Connections Q16: I know people who have overcome the same problems. 51% Q17: I have family, friends or neighbors in my life that support me. Personal Well-Being Q19: I know I ll be okay, even if something doesn t go my way. 39% Q20: If I want to reach a goal, I try hard to achieve it and I never give up. When you look only at members who were originally low scorers on questions related to their social connections (Q16 and Q17), 52 percent moved from being low scorers in their first response to being high scorers in their later responses on one or both questions. For members who were originally low scorers on personal well-being (Q19 and Q20), 57 percent moved from being low scorers to being high scorers. This provides additional evidence that many members experienced an increase in their social connections and personal well-being while working with LIFT. Table 6. Percent of members who were low scorers on their first response but high scorers on later responses on one or both questions, by question type Percent moving Questions low high Social Connections Q16: I know people who have overcome the same problems. 52% Q17: I have family, friends or neighbors in my life that support me. Personal Well-Being Q19: I know I ll be okay, even if something doesn t go my way. 57% Q20: If I want to reach a goal, I try hard to achieve it and I never give up. 12

13 FINDING 6: Stronger social connections and personal well-being translated to more progress on goals. Members who were high scorers on social connections and personal well-being questions tended to make more progress on their goals in comparison to those who were low scorers. Depending on the question, members completed anywhere from 1.3 to 1.6 times as many goals as low scorers. FOCUS GROUPS CONFIRM FINDINGS In addition to our survey, LIFT regularly conducts interviews and focus groups with members to confirm or expand what we re learning. Through these in-person conversations, The biggest differences were for the three members have reinforced again and questions related to personal well-being: Q19 (I again their strong desire to connect with know I ll be okay, even if something doesn t go other LIFT members and grow their my way), Q20 (If I want to reach a goal, I try social support networks. hard to achieve it and I never give up) and Q21 (I believe I can achieve the goals I worked on at LIFT today). The differences for questions related to social connections were smaller. Table 7: Differences between high scorers and low scorers in the average numbers of completed goals Question 13 High scorer progress (compared to low scorers) Q19: I know I ll be okay, even if something doesn t go my way. 1.6 times Q20: If I want to reach a goal, I try hard to achieve it and I never give up. 1.5 times Q21: I believe I can achieve the goals I worked on at LIFT today. 1.5 times Q16: I know people who have overcome the same problems. 1.3 times Q15: Thanks to LIFT, I am more connected to community resources that can help me. 1.3 times Q17: I have family, friends or neighbors in my life that support me. 1.3 times Note: All differences were statistically significant at the p<.05 level. When looking region-by-region, there were typically not enough data to accurately measure the difference between high scorers and low scorers within just one region. However, in the few instances where there were enough data, the trends above held. High scorers on Q19 (I know I ll be okay, even if something doesn t go my way) made more progress than low scorers across the board. They completed 1.6 times as many goals in Washington, D.C., 2.1 times as many goals in New York, and 1.7 times as many goals in Los Angeles. High scorers on Q21 (I believe I can achieve the goals I worked on at LIFT today) completed 1.9 times as many goals in Washington, D.C., and high scorers on Q17 (I have family, friends or neighbors in my life that support me) completed 2.3 times as many goals in New York.

14 CONCLUSION Over the last three years, LIFT s Constituent Voice process has empowered us to systematically listen to our members as we think critically about what we re doing well and what we need to improve. The results from our survey have provided invaluable insights into how we can better meet the needs of our members and we have listened. In response to feedback on the accessibility of meetings, LIFT began piloting flexible hours and virtual meetings to better meet the demands of busy members. Our New York office added evening and weekend hours and our Los Angeles office began offering phone meetings to returning members. Similarly, based on member input on the need for stronger social connections, LIFT has begun to create more opportunities for members to connect with each other in safe, supportive environments. These have included LIFT-facilitated peer groups as well as informal opportunities like our member lounge in New York that features free coffee and internet. LEARN MORE! Based on more than three years of experience implementing Constituent Voice across LIFT s national network, we have learned a lot about what works, and what doesn t, when gathering member input. Read more about this learning process in our recent report: Listening Better: 10 Lessons from LIFT s Member Feedback Survey In addition to providing actionable insights into how we can improve our services, Constituent Voice has generated reliable evidence that our intense focus on engaging members and building strong relationships has been vital in achieving critical change for our members. The survey results speak loud and clear: by putting members first in all that we do, we empower them to achieve transformational outcomes for themselves and their families. ABOUT LIFT Founded in 1998, LIFT is a national nonprofit dedicated to ending intergenerational poverty. Since then, we have helped 100,000 low-income individuals achieve their goals. Today we connect hardworking parents and caregivers of young children to the people, tools and resources they need. LIFT operates in Chicago, Los Angeles, New York and Washington, D.C., communities with some of the highest rates of concentrated poverty. By fostering relationships between lowincome parents and caregivers of young children (members) and dedicated volunteers (advocates), we help families build the personal well-being, social connections and financial strength to secure basic needs and achieve long-term goals and aspirations, like a safe home, living wages or a better education. 14