YJB THEORY OF CHANGE SEMINAR 2

Transcription

1 YJB THEORY OF CHANGE SEMINAR 2 James Noble v London, June 2014

2 AGENDA Recap of what theory of change is all about Case study triage and restorative justice Other issues / questions that have been raised

3 Recap

4 INTRODUCTION 4

5 ESSENCE OF THEORY OF CHANGE Description of how a programme is intended to work 5

6 1) PROGRAMME Shorthand way to refer to the full range of youth justice interventions including all approaches, practices, interventions and programmes Can be used to plan and evaluate internal change processes (links to change management) 1-1 work with individuals (?) 6

7 2) WORK Every programme has, and should clearly state, its long-term social goal Crucially, social change occurs through individuals (young people) Giving young people the resources, capabilities and motivations to change their own lives for that better Hence a focus on engagement and intermediate outcomes Descriptions of programmes (inputs, activities, outputs) should focus on how the programme is intended to achieve engagement and intermediate outcomes 7

8 3) DESCRIPTION : THERE S NO ONE WAY A THEORY OF CHANGE SHOULD LOOK Planning Triangle Logic Model Outcomes Chain - Each approach has pros and cons - However you represent your theory of change, it should be supported by a written description 8

9 THEORY OF CHANGE IS A PROCESS AND A DOCUMENT Process should involve others to; Strengthen the document Create a consensus Build your team Strengthen the intervention and delivery Document should; Communicate a programme's aims, relevance and credibility Show a hypothesis of how it will work Help you determine what the programme s evaluation should include 9

10 HOW TO CREATE A THEORY OF CHANGE To create a theory of change you start with: Agree what you are doing a theory of change on (project, organisation etc.) A consideration of your services users and what their needs are The final outcomes you want your service users to achieve (reduced reoffending, success at school) Then you move on to: What needs to happen to achieve this: Beginning with the change that needs to happen in service users (intermediate outcomes) and moving on to what you need to do in your programme to help them achieve this. Deploying the existing evidence base and the views of your colleagues at each stage Only consider measurability at the end 10

11 BENEFITS Help teams work together towards a shared understanding Help with partnership working, training, motivation Bring the process of real change to the forefront and identifies and open up black boxes in thinking. This should makes work more effective Quickly and effectively communicate aims Help determine what needs to be measured Structure reporting and make claims about impact 11

12 USE SO FAR BY YJ PRACTITIONERS Base: 39 practitioners who attended recent training 12

13 PERCEIVED VALUE AMONGST YJ PRACTITIONERS 13

14 NESTA STANDARDS OF EVIDENCE Level 1: You can describe what you do and why it matters logically, coherently, and convincingly Level 2: You capture data that shows positive change but you cannot confirm you caused this Level 3: You can demonstrate causality using a control or comparison group Level 4: You have one + independent replication evaluations that confirm these conclusions Level 5: You have manuals, systems and procedures to ensure consistent replication and positive impact (fidelity) 14

15 ANY QUESTIONS? 15

16 Case study

17 CASE STUDY: TRIAGE FOR TIER 1 YOUNG OFFENDERS Youth Justice Triage is a service to identify young people aged between 10 and 17 committing low (matrix) level offences, to engage them in restorative justice and supportive programmes to divert them from the formal criminal justice system and to reduce the risk of re-offending. Reduction in offending behaviour Reduced reoffending rate Activity Engagement Intermediate outcomes Long-term goal 17

18 INTERMEDIATE OUTCOMES a) Knowledge or skills b) Attitudes c) Behaviours - Attainment of qualifications - Awareness/understanding of consequences of antisocial/criminal behaviour - Awareness/understanding of long-term options (eg employment, careers) and awareness of how to achieve these options. - Awareness/understanding of short-term options such as training, other support. Able to make appropriate choices (lower drop-out) - Fewer experiences of being victim of crime/anti-social behaviour - Social skills, social attachments and material resources to enable refusal strategies - Aspirations/ambition/motivation towards achieving positive longerterm outcomes. - Efficacy (belief by YP that they can chose/options future) - Emotional resilience, strengths and difficulties, self-esteem & confidence. - Attitudes to anti-social & criminal behaviour and drug use. - Positive experiences of and attitudes towards schooling and employment. - Positive perceptions and trust of local services and staff members, willingness to engage with them - Access/usage of universal services (eg. young people s participation in positive activities, engagement in the arts - Contact/engagement with local services/voluntary sector. - Attendance/engagement at school & other school behavioural measures. - Reduced use of illicit substances, alcohol and volatile substances (inc smoking) - Quality of peer/family/community relationships and relationships with staff/other adults ('having people to talk about problems') - Reduced association with negative peer groups - Reduced low level anti-social behaviour (eg bullying) 18 Reduce offending behaviour

19 CASE STUDY: TRIAGE FOR TIER 1 YOUNG OFFENDERS Youth Justice Triage is a service to identify young people aged between 10 and 17 committing low (matrix) level offences, to engage them in restorative justice and supportive programmes to divert them from the formal criminal justice system and to reduce the risk of re-offending.? Good collaborative working with police custody officers & other partners Trained / experienced / credible staff Right procedures, systems, arrangements Assessment of all young people who are arrested Process which identifies all needs & risks Work to understand, relate to and engage young people Right practice skills Reduction in police time spent on minor offences Appropriate referrals (based on circumstances and levels of risk) Young people understand why they are being referred and appreciate how they might benefit Avoid unnecessary criminalisation of YP on fringes of criminal activity? Appropriate and effective restorative justice interventions Ongoing support from triage worker Positive perceptions and trust of local services and staff members More negative attitudes to anti-social & criminal behaviour and drug use More positive social attachments Attendance/ engagement at school Reduction in anti-social behaviour Reduced reoffending rate Activity Engagement Intermediate outcomes Long-term goal 19

20 THE DANGERS OF INCLUDING TOO MUCH 20

21 LOGIC MODEL FOR COLLABORATION WITH POLICE TO ESTABLISH EFFECTIVE TRIAGE SYSTEMS Enabling factors Inputs Engagement Intermediate outcomes Final outcomes Organisational outcomes Pre-existing partnership working between police / YOS / local services Low turnover of police custody staff Co-location of triage workers in custody suite Assign triage champion within local police Promote potential benefits through training and literature Clear legal guidance and protocols Mandate from ACPO Spend time with custody staff to explain the scheme Good communication between custody staff and triage workers Earlier experiences of collaboration are positive Give feedback to officers about the young people they have referred to triage High levels of police awareness of triage and its aims (knowledge) Police custody staff support the use of triage (attitudes) Effective joint-working between triage and police custody staff (behaviours) Appropriate referral to restorative justice interventions > Reduced reoffending Reduction in police time spent on minor offences Improved information sharing & collaboration between the police, CPS and YOS Culture change: "more humanitarian feeling" 21

22 THEORY OF CHANGE FOR A RESTORATIVE JUSTICE INTERVENTIONS - Engage and talk to young people and agree how they might make amends - Restorative conversations focussing on what happened, why it happened and what could be done to put it right - Mediation between victim and offender - Reparation activities (panels, apology letters) - Involvement of parents and carers Referral to other support services (substance misuse, training, counselling, anger management etc.) Meeting and celebration to mark the end of the process Positive perceptions and trust of local services and staff members Trust staff and see them as credible Listen to new ideas and ways of thinking New ideas seen as personally relevant & generate emotional response Take responsibility for past actions and for the need to change Address other barriers other lives and increase capacity for change Graduate with feeling of achievement & affirming their new outlook More negative attitudes to anti-social & criminal behaviour and drug use Ability to relate to others Trust in adults Understand consequence of actions (both positive and negative) Better understand / feel regret for impact of crime on victims Decision to move away from negative behaviour Positive feelings about the future Increasing self-worth wellbeing Change in identity from criminal to positive member of society More positive social attachments Attendance/en gagement at school Activity Engagement Intermediate outcomes Long-term goal 22

23 ROLE OF EVIDENCE: LOOKING IN TWO DIRECTIONS Look backwards: What do we already know that is relevant to your theory of change? Research has found that youth justice programmes based on theory were, on average, five times more effective than those not based on tested theoretical principles Look forwards What evidence do we need to collect to; a)assess whether we achieved what we set out to do? b)what we have learned for the future? Izzo, R.L. and Ross, R.R. (1990) Meta-Analysis of Rehabilitation Programs for Juvenile Delinquents, Criminal Justice and Behaviour See

24 LOOKING BACKWARDS 24

25 LOOKING FORWARDS - Engage and talk to young people and agree how they might make amends - Restorative conversations focussing on what happened, why it happened and what could be done to put it right - Mediation between victim and offender - Reparation activities (panels, apology letters) - Involvement of parents and carers Referral to other support services (substance misuse, training, counselling, anger management etc.) Meeting and celebration to mark the end of the process Positive perceptions and trust of local services and staff members Trust staff and see them as credible Listen to new ideas and ways of thinking New ideas seen as personally relevant & generate emotional response Take responsibility for past actions and for the need to change Address other barriers and increase capacity for change Graduate with feeling of achievement & affirming new outlook More negative attitudes to anti-social & criminal behaviour and drug use Ability to relate to others Trust in adults Understand consequence of actions (both positive and negative) Better understand / feel regret for impact of crime on victims Decision to move away from negative behaviour Positive feelings about the future Increasing self-worth wellbeing Change in identity from criminal to positive member of society More positive social attachments Attendance/en gagement at school Activity Engagement Intermediate outcomes Long-term goal 25

26 LOOKING FORWARDS (1) Developing evaluation questions What did we do? Number of young people completing the course Number of assessments Number of direct/indirect apology letters How well did we do it? Did the young people trust staff and see them as credible? Did the young people listen? Did they see the issues as personally relevant? And have an emotional response? Is anyone better off? Did young peoples ability to relate to others improve? Did feelings of self-worth improve? Has their attendance and engagement at school improved? 26

27 LOOKING FORWARDS (2) Deciding on evaluation methods Good collaborative working with police custody officers & other partners -Qualitative interviews - Descriptions of staff track record / experience Trained / experienced / credible staff Assessment of all young people who are arrested -Monitoring data Process which identifies all needs & risks Work to understand, relate to and engage young people Reduction in police time spent on minor offences - Existing research - Analysis of official data - Feedback from young people - Views of staff Appropriate referrals (based on circumstances and levels of risk) Young people understand why they are being referred and appreciate how they might benefit Avoid unnecessary criminalisation of YP on fringes of criminal activity - Existing research - Analysis of official data Appropriate and effective restorative justice interventions -Monitoring dataongoing support from - Views triage worker of staff / young people Positive perceptions and trust of local services and staff members -Longitudinal interviews More negative with attitudes to young anti-social people & (qual criminal & quant) behaviour and - Parent / drug use teacher perspectives More positive social attachments Attendance/en gagement at school - Existing research - Analysis of official data Reduction in offending behaviour Reduced reoffending rate Activity Engagement Intermediate outcomes Long-term goal 27

28 CONTRIBUTION ANALYSIS 1. Comparing evidence to the theory of change 2. Assessing congruence: do the results match the theory? 3. Disaggregating results (i.e. looking at patterns of results and outcomes amongst difference users). 4. Considering other explanations 5. Asking participants and facilitators 6. Making counterfactual comparisons (i.e. trying to estimate what would have happened if the programme had not been run) 7. Analysis of costs and benefits 8. Consider what has been learned? 28

29 ANY QUESTIONS? 29

30 Other issues / questions

31 A TOOL FOR WORKING WITH INDIVIDUALS? Yes, but it needs to be approached differently. Hypothesising what will work with individuals Backwards mapping with individuals from goals to actions 31

32 A TOOL FOR WORKING WITH INDIVIDUALS? 1) Long-term goals What are your long-term goals? What changes in your life need to happen for you to achieve your long-term goals? 2) Short-term goals 3) How this will be achieved What skills or knowledge will you need to achieve your long-term goals? How will you need to think differently to achieve your long-term goals? How will you need to behave differently to achieve your long-term goals? For each short-term goal, ask; i) how will you achieve this? ii) What will you need to do achieve this? 4) Further support needs What other support might you need to make this work? Who are the people around you that can give you support to reach your long term goals? Who do you feel might create the most obstacles and how will you manage these? Is there anything that frightens you about taking this journey forward? What can be done to help? 32

33 IS THIS ANYTHING NEW? The ideas behind theory of change are not new, but still too few programmes go through the process (or something similar) For example, a review of evaluations and the Effective Practice library suggests: A few evaluations are still focused on activities and outputs rather than outcomes (number of courses delivered, orders completed, young people signposted etc.) More commonly there will be a statement of long-term outcomes, but limited description of the intermediate steps service users need to go through for these to be achieved. There are descriptions of how services work, and the most important factors, but there is still scope to present these in a more organised way. Every evaluation is different, impeding our ability to look across and draw general conclusions. Design and delivery of interventions does make enough use of the existing evidence base 33

34 OTHER SYSTEMS: SMART TARGETS Smart targets (Specific, measureable, achievable, relevant, time bound) Engage and talk to young people and agree how they might make amends Increasing self-worth Why not? Partly because there s limited space Successfully engage 10 young people per month - as assessed by YOS worker - each producing agreed statements about how they will make amends. 50% of the young people we engage with will record increases in self-worth within 1 month, as measured by a Strengths and Difficulties Questionnaire More importantly, SMART targets are useful for intervention design but less for evaluation For evaluation ToC helps you to know what you need to measure and how, but it s not necessary to set targets. 34

35 OTHER SYSTEMS: RESULTS BASED ACCOUNTABILITY The RBA/OBA classification of performance measures generally fits well with theory of change: Quantity Quality Input / Effort How much service did we deliver? How well did we deliver it? = Inputs/ activities = Engagement = Outcomes Output / Effect How much change / effect did we produce? What quality of change / effect did we produce? 35

36 OTHER SYSTEMS: RESULTS BASED ACCOUNTABILITY RBA s Turning the Curve workshop can also be used as a way to ensure a team agrees on a common objective and can be used to support the development of a theory of change Quantity Quality Stage Inputs Outputs Intermediate outcomes Final outcomes Logicmodel What are the key design features of our intervention? Did we manage to deliver this? Did this deliver the outputs we expected? Did our target audience react in the way we wanted? What has this directly changed for the people involved? Have the changes we hope will have an impact on eventual outcomes taken place? Maths results amongst targeted pupils show improvement } Effect Effort How much How well did we do? did we do it? Is anyone better off? # % Based on data, encourages group buy-in, generates creative solutions Helps in detailing initiative design, identifies how each stage builds on the previous, helps to identify where things might have gone wrong Keeps the focus on the Final outcomes of the community not just the outputs observed in users 36

37 OTHER SYSTEMS: SOCIAL RETURN ON INVESTMENT (SROI) Theory of change is at the heart of SROI But it requires you to develop outcomes and a theory of change for all stakeholders. Also assigns monetary values to outcomes, including proxy monetary values for social impact, to provide a ratio of pounds saved per pounds spent. 37

38 ANY QUESTIONS? 38

39 London, June 2014 v