Meta-SEP Evaluating Service Evaluation Projects

Transcription

1 Meta-SEP Evaluating Service Evaluation Projects Matthew James Price Commissioned by: Dr. Jan Hughes, Dr. Gary Latchford, and Anita Dorsett, University of Leeds, Clinical Psychology Programme

2 Acknowledgements I would like to thank the commissioners who kindly took the time to share their experiences of commissioning service evaluation projects. Table of Contents Executive Summary - page 3 Background - page 4 Methods - page 6 Results - page 7 Discussion - page 15 References - page 18 Appendices - page 20 Prepared on the Leeds Doctorate of Clinical Psychology Programme,

3 Executive Summary This evaluation investigated the characteristics, impact and dissemination of Service Evaluation Projects (SEPs) completed by Trainee Clinical Psychologists at the University of Leeds. Methods Descriptive statistics were used to analyse the characteristics of SEPs completed over the past 13 years. Semi-structured interviews with commissioners were held to explore the impact that SEPs may have had on services, and how findings were disseminated. Thematic analysis was used to identify key themes relating to the impact of SEPs. Results 230 SEPs have been completed in the past 13 years, by a total of 141 commissioners. SEPs are commissioned in the Yorkshire and Humber region. Interviews with 15 commissioners about a total of 38 SEPs found that the majority had an impact (N=33), most of which brought about change independently of other initiatives / research within the service (N=20). Interviews with commissioners identified three key themes in terms of impact; improving processes, improving knowledge and improving resources. SEPs are often disseminated (N=32), usually through internal presentation to teams (N=13) and sharing the written report internally (N=10). Some SEPs have also been written for publication (N=6) and presented at conferences both nationally and internationally (N=6). Conclusions The SEP process often has a positive impact on services, the findings of which are disseminated through internal and external channels. This suggests that SEPs completed by Trainees are a helpful asset for commissioners who wish to evaluate their services. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

4 Background Service Evaluation and the NHS Service evaluation has long been identified as a central aspect of delivering care across all health and mental health services in the UK. Service evaluation refers to the systematic assessment of the implementation and impact of a project, programme or initiative (p. 6 NHS Institute for Innovation and Improvement, 2011). Recent changes in the NHS have increased demand for high quality service evaluations; for example, the Government has set a target for productivity improvements of 22 billion by 2020/21. This is in addition to a commitment to growing services; such as a commitment to a seven-day service. In this context, evaluating whether services are delivering effective outcomes is crucial. A recent paper by the King s Fund has made the case for placing quality improvement at the heart of how the NHS can respond to these unprecedented pressures and deliver the changes needed to meet new targets (Ham, Berwick, & Dixon, 2016). This rhetoric is not just limited to think tanks indulging in blue sky thinking. The NHS newly formed Improving Quality unit within the Department of Health has also recently emphasised the current pressure on services against a backdrop of austerity and the growing need for change and improvement (Fairman, 2015). Service Evaluation and Clinical Psychology Evaluating the effectiveness of services is one of the key roles of a clinical psychologist, and a cornerstone of the profession s scientist-practitioner ethos. Indeed, the Division of Clinical Psychology (DCP) within the British Psychological Society (BPS) has written widely about the importance of service evaluation. For example, their Leadership Framework outlines service evaluation as a task that clinical psychologists at every level should be carrying out, from Trainee Clinical Psychologists (Trainees) to Consultants and Clinical Directors (DCP, 2010). To equip the future workforce of clinical psychologists with the skills needed to conduct service evaluations, the BPS requires all doctoral training programmes to ensure that trainees have the capacity to conduct service evaluation, small N, pilot and feasibility studies and other research that is consistent with the values of both evidence based practice and practice based evidence (p.13; Partnership & Accreditation, 2014). Prepared on the Leeds Doctorate of Clinical Psychology Programme,

5 The Clinical Psychology doctorate programme at the University of Leeds meets the BPS requirement by assessing Trainees on their completion of a Service Evaluation Project (SEP). SEPs are commissioned by clinicians working in local services who identify the need to evaluate an aspect of their service. They must complete and submit a commissioning form to the course Research Director (see Appendix A). Trainees then select the SEP that they are most interested in, and liaise directly with the commissioner to complete the project. Trainees can also seek supervision from their Academic Tutor, part of the Clinical Psychology programme, on aspects of the research design, methodology and analysis. The Clinical Psychology programme at Leeds mandates that the service evaluation projects must be: practical and useful. It should relate to some aspect of a current service or planned service development. It should always be driven by the host service that commissions the project, providing valuable information that they will be able to use. (p.2 SEP Manual, 2016) Despite this mandate, no evaluation has been completed to identify whether SEPs provide valuable information that services can use to help bring about meaningful change. Current Service Evaluation Project This evaluation investigated whether the service evaluation projects completed by Trainees at Leeds meet the course requirement that they are practical and useful. To do this, the characteristics of completed SEPs were explored, looking at their impact and how they were disseminated. Exploring the impact of SEPs is important because this will allow better understanding of how information is used to improve practice. Investigating dissemination is important because to bring about lasting change, information on best practice must be shared. As the King s Fund explain Only by moving from pockets of innovation to system-wide improvement will the NHS deliver changes that are needed (p.9; Ham, Berwick, & Dixon, 2016). Thus, the present evaluation aims to: 1. Explore the characteristics of SEPs completed by Trainees at the University of Leeds. 2. Investigate the ways that completed SEPs are disseminated. 3. Investigate the impact of SEPs on services. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

6 Methods Ethical Considerations As this project involved analysing data already saved as part of the University of Leeds Doctorate of Clinical Psychology programme and speaking with colleagues already collaborating with the Programme, ethical approval was not required. Procedure and Analysis There were two parts to this evaluation. Firstly, the characteristics of completed SEPs were explored, and secondly, the impact and dissemination of SEPs was investigated by interviews with commissioners. This section describes the procedures and analyses of both parts. Part 1: Characteristics of SEPs Sampling A database with comprehensive information about the characteristics of completed SEPs was available from the Clinical Psychology Programme. Thirteen years of data was available. This database held information about the nature of the SEP, the commissioner, the location and specialty of the service (e.g. Adult Mental Health), and when the SEP was completed. Data Generation Data was cleaned in preparation for analysis. For example, the name given to the same clinical specialty sometimes varied by geography and so these were amended to ensure consistency. Information about characteristics was converted into count data. Analysis Inferential statistics were not used as this evaluation did not aim to test any experimental hypotheses. Thus, the data was analysed using descriptive statistics to summarise data and identify any patterns of commissioning. Part 2: Impact and Dissemination Sampling Prepared on the Leeds Doctorate of Clinical Psychology Programme,

7 Commissioners were approached by telephone or and asked whether they would be willing to share their experiences of commissioning a SEP. All commissioners approached were willing to share their experiences. The recruitment strategy ensured that the sample was as representative as possible. For example, the database of completed SEPs showed that there were some clinicians who had commissioned many SEPs, while others had completed only one. With this in mind, it was important to speak with clinicians who had commissioned multiple SEPs and those who had commissioned just one. Additionally, to account for potential recency bias in recall clinicians were interviews who had commissioned SEPs more recently as well as those who have not commissioned one for many years. Data Generation Semi-structured interviews were held with commissioners about their SEPs (see Appendix B for interview schedule). A combination of open and closed questions were used to generate a combination of quantitative data and qualitative. When interviewing commissioners about SEPs completed several years ago, reassured was given that the full details of the SEP did not need to be recalled and that we were working on the assumption that if there was something significant (or not) then it would be remembered. Analysis Quantitative data was analysed using descriptive statistics. Qualitative was analysed using thematic analysis, an approach for identifying, analysing and reporting themes within data (p. 79, Braun & Clarke, 2008). An alternative qualitative approach might be Grounded Theory, however, a central goal of Grounded Theory is to direct analyses towards theory development (Holloway & Todres, 2003). As the aims of the present study were not related to the theory development, a descriptive approach was preferable. Thus, the thematic analysis represented a simple, yet robust approach to analysis and was conducted in-line with the steps outlined by Braun and Clarke (2008). Results This section presents the results in two parts; Part 1 explores the SEP characteristics, and Part 2 explores the dissemination and impact of SEPs. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

8 Part 1: Characteristics Commissioners A total of 230 SEPs have been commissioned and completed since 2002, these have been commissioned by 141 different commissioners. Co-commissioning was relatively common, with 65 SEPs being commissioned by two or more commissioners. The modal number of SEPs commissioned per commissioner was one (n=88), with 32 commissioners commissioning two SEPs, and 21 commissioners commissioning three or more SEPs. Nearly half of those who commissioned three or more SEPs are (or were) staff of the Doctorate of Clinical Psychology programme. Indeed, of the ten commissioners who have commissioned (or co-commissioned) the highest number of SEPs, seven are staff of the Doctorate programme. This is likely to be understood in part by the frequent collaboration between the Doctorate programme and the local commissioners. Often, staff of the Doctorate programme provide supervision of the academic components of SEPs outside the field of Clinical Training and a collaborator will provide field supervision of the specialty within which the SEP is being conducted. Clinical Specialties Figure 1 demonstrates the wide variety of clinical specialties that have commissioned SEPs. Adult and child services are the most frequent commissioners of SEPs, closely followed by clinical training. Within the category of Adult, nearly half of these are Adult Health Psychology (14%) and the others are Adult Psychological Therapies. The child category is primarily made up of Paediatrics (10%), and Child and Adolescent Mental Health Services (CAMHS; 7%). Other child services rarely commissioned SEPs, for example, child development accounted for 3% of all SEPs. The category of Other captures specialties that have commissioned very few SEPs, including neuropsychology and multi-specialism SEPs; for example, one SEP evaluated both Adult and Older Adult services. The Other category also includes instances where it was unclear from the database which clinical specialty the SEP studied, and where the original SEP was not accessible to check. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

9 35% SEPs by Clinical Specialty SEPs Commissioned (%) 30% 25% 20% 15% 10% 5% 0% Adult Child Clinical Training Older Adult Learning Disabilities Other Forensic Clinical Specialty Figure 1: Percentage of SEPs commissioned by clinical specialties Geographical Spread Figure 2 shows that SEPs are almost exclusively commissioned within the West Yorkshire region, with each marker representing a NHS Trust that has commissioned a SEP. In practice, there is greater regional spread than this figure demonstrates. For example, there is one mark that represents South West Yorkshire Partnership NHS Foundation Trust, however, there have been projects commissioned from this Trust across Wakefield, Huddersfield and Dewsbury. Historically, there were instances where SEPs were commissioned out of area. However, changes in terms of commissioning guidelines for SEPs has meant that in recent years all SEPs have been commissioned in the Yorkshire and Humber region. Figure 2: SEPs by geographical location Prepared on the Leeds Doctorate of Clinical Psychology Programme,

10 Commissioning Trusts Five NHS Trusts commissioned 75% of SEPs. Figure 3 shows that nearly a quarter of all SEPs were commissioned by Leeds Partnership NHS Foundation Trust (23%). With Leeds Teaching Hospitals NHS Foundation Trust accounting for 21% of all SEPs. SEPs by Commissioning Trust SEPs Commissioned (%) Leeds Partnership Foundation Trust Leeds Teaching Hospitals Trust Clinical Psychology Programme, University of Leeds South West Yorkshire Partnership Foundation Trust Leeds Community Healthcare Trust Figure 3: Percentage of SEPs by commissioning NHS Trust Commissioning Trends Over Time Investigation of which specialties are commissioning SEPs over time did not show any patterns of change. Rather, it revealed that there was variability in the number of SEPs commissioned by each specialty each year. See Appendix C for a table of the number of SEPs commissioned by specialty over time. Part 2: Dissemination and Impact Participants Interviews were held with 15 commissioners, who were able to share their experiences of 38 different SEPs. The proportion of commissioners interviewed from each specialty represented the frequency with which each specialty had commissioned SEPs in total. Table 1 shows that these proportions are broadly similar. See Appendix D for an anonymised demographic breakdown of which commissioners were interviewed. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

11 Specialty Sample Interviewed Total Sample Adult 27% 30% Child 22% 21% Older Adults 17% 10% Learning Disability 15% 7% Neuropsychology 3% 2% Table 1: Comparison of percentage of SEPs by specialty for the sample interviewed and total sample Would projects have been completed without the SEP process? More than three quarters of projects would not have been completed without the SEP process to enable evaluation (n=29). Of the remaining nine projects that would have been completed even if the opportunity to commission a SEP was not available, all commissioners spontaneously added that the project would not have been completed to the same high standard. Examples from commissioners include: The project would have been done but clinicians are over-stretched and under-resourced so this was more thorough and additional supervision through the course staff if helpful. (Commissioner 7) It would have been completed but the SEP provided a more systematic approach, more rigour and more depth. (Commissioner 9) Moreover, five commissioners explained that the SEP process was particularly important because it afforded an element of impartiality that would not have been obtained otherwise. The commissioners who found this valuable had commissioned SEPs that required seeking feedback from staff or clients: It was important to have an independent researcher come into the Department because the staff were able to be more honest in their feedback than if we had gone in [as clinicians already working there]. (Commissioner 15) What organisational level do SEPs target? Prepared on the Leeds Doctorate of Clinical Psychology Programme,

12 Organisational level refers to the part of the system that the SEP aims to investigate. Most SEPs were either aimed at the staff level (N=16) or patient level (N=14). The remaining eight SEPs were targeted at a wider aspect of the organisation, such as tracking numbers through service pathways. How are SEPs disseminated? Commissioners were asked about any ways in which the SEP was disseminated. They were reminded that all SEPs are disseminated within the Clinical Psychology Programme s Annual SEP Conference but that for the purposes of this project, we were interested in how the SEP s findings were shared beyond this event. The most common method of disseminating the findings of SEPs was via internal presentation to the team within services (n=13). Internally sharing a copy of the report was also a common method of dissemination (n=10). Six SEPs led to publications, three of these were in the Clinical Psychology Forum, others include the Journal of Cystic Fibrosis, and the Journal of Nursing Children and Young People. Six SEPs have also been presented nationally or internationally at conferences. Do SEPs have an impact on services? The majority of SEPs were reported as having an impact on services (n=33). Of the five projects that were found to not have had any impact, there were no patterns in terms of SEP characteristics. Feedback from commissioners about the reasons for the lack of impact was varied. One commissioner explained that the service was closed shortly after the SEP was completed due to lack of funding. Another commissioner cited that the long time between data-collection and having a final copy of the report was a barrier to implementing changes; this was particularly elongated as the trainee had failed the academic component of the SEP. Other feedback from a commissioner who had commissioned multiple SEPs described their sense that the biggest factor relating to whether a SEP would have an impact was if the trainee was autonomously motivated to help bring about a change rather than completing the SEP as a tick box. This suggests that Trainees should be advised to choose a SEP topic that enthuses them, such that they are motivated to ensure that the SEP has an impact. How do SEPs impact services? Prepared on the Leeds Doctorate of Clinical Psychology Programme,

13 Of the SEPs that had an impact, nearly two thirds brought about an impact on services independently of other evaluations (n=20). Six projects were linked with other SEPs, with an additional three SEPs linked to research outside the SEP process. Finally, four SEPs brought about a change in conjunction with other initiatives that were ongoing within services. What is the impact of SEPs? Table 2 shows the themes and sub-themes found within the descriptions of impact that commissioners provided. There were three main themes: 1. Improved processes - either through changes to direct clinical work with patients, indirect working with staff, or though changes to broader service pathways. 2. Improved knowledge - either through the development of an evidence-base, as one might be looking for an evaluation of a group, or through better understanding of issues relating to services or patients. 3. Improved resources either through improving the amount of subsequent research and analysis completed by the trust or by improving the clinical services available. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

14 Table 2: Themes identified about the impact of SEPs Theme Sub-theme Examples Improved Processes Improved Knowledge Improved Resources Clinical work with patients Working with staff Service pathways Building an evidence-base Understanding of issues Developing research programmes Developing clinical services "Some participants struggled to remember what had happened in previous sessions so we gave them more handouts with more detailed information so that they could act as memory aids" (Commissioner 4) "We have changed how we assess for our therapy group - giving more consideration to demographics like gender" (Commissioner 6) "We changed our approach to supporting staff. We developed more events and training" (Commissioner 14) "We have made sure that there is a greater presence of psychology at ward rounds for staff" (Commissioner 15) "It helped us to decide which intake model the service should use" (Commissioner 9) "We have changed our approach to follow-up care - e.g. calling patients to reduce drop-out / non-engagement" (Commissioner 12) "The evaluation showed that the new group was effective, which enabled it to continue to run" (Commissioner 11) "The data was positive so it encouraged us to continue to run the groups" (Commissioner 1) "It has made the service become more aware of the needs of a certain group of clients" (Commissioner 2) "It helped us to understand that there was a disparity between psychologists in terms of what they were offering" (Commissioner 7) "The service has commissioned another SEP to further explore the findings of the first two SEPs" (Commissioner 2) "This SEP was the foundation of more extensive subsequent evaluation two years later" (Commissioner 11) "Using the SEP, the Head of Service was able to find funding for additional training of staff" (Commissioner 4) "A new Clinical Psychologist was in post within a year" (Commissioner 1) Prepared on the Leeds Doctorate of Clinical Psychology Programme,

15 Any further plans to implement changes in the future? Some commissioners interviewed had only recently had their SEP completed (i.e. within the past 6 months). To ensure that any impact that these SEPs were going to have was not missed, commissioners were asked about whether there were any future changes planned but not yet implemented. There were five projects that had plans that were not yet implemented. Of these, four related to the Improved processes theme and one to the improved resources theme. Discussion This service evaluation aimed to explore the characteristics of SEPs that had been completed, and better understand the impact and dissemination of SEPs through interviews with clinicians who had commissioned them. Trainees have completed 230 SEPs over the past 13 years. These have been commissioned across the Yorkshire and Humber region, and supported a wide range of clinical specialties. Interviews with 15 commissioners about a total of 38 SEPs found that the majority have had an impact on the commissioning service in some way. The impact of these SEPs can be summarised as improving process, knowledge or resource. Moreover, the SEPs have been disseminated across many channels. Predominantly, this has been internally through presentations to teams and sharing of the SEP report. In addition to internal dissemination SEPs have led to publications, and presentations at national and international conferences. This service evaluation has found that the SEP process can enable services to bring about meaningful change and share it widely, in a way that would not have been possible without the availability of Trainees to complete SEPs. In an ever-changing NHS landscape, where there is constant pressure to prove value for every cost and every initiative, it will be more important than ever to have ways for services to evaluate their practices. It is clear from the findings presented here that Trainees are a valuable asset in supporting services to evaluate their activity in a robust way. Having the skills to bring research methods into clinical practice is not a strength of all health professionals in the NHS. For example, one year following the Berwick (2013) report, a survey found that support for staff to improve the processes of care was the area where the least progress had been made (Health Foundation, 2014). It may well be the case that service evaluation and using this to Prepared on the Leeds Doctorate of Clinical Psychology Programme,

16 lead improvements is an area of added value that the field of clinical psychology brings to the healthcare sector. Limitations There are four key limitations of this evaluation that need to be considered. Firstly, as with many forms of qualitative research, the sample size is small. This has implications for the generalisability of the findings from this evaluation. To mitigate this issue as much as possible, the representativeness of the commissioners that were interviewed was given careful consideration. For example, the proportion of specialties interviewed is broadly in line with the proportion of SEPs commissioned by each specialties (see Table 1). Moreover, sampling could have been improved with the inclusion of a Trainee perspective on the impact of SEPs.. This would have added particular value if Trainee-Commissioner dyads could have been selected. Unfortunately, this was not possible due to the restricted timeframes of the evaluation. Secondly, the approach to data generation via semi-structured interviews with commissioners may have resulted in positivity bias. For example, the context of the interviews was a Trainee currently completing a SEP and asking for feedback on the impact of SEPs. If commissioners thought that describing a lack of impact may be disheartening or de-valuing, they may have withheld some information, or been more inclined to share the positives. To overcome this, an alternative method would have been to circulate a questionnaire. Arguably, an anonymised questionnaire may have made it easier for commissioners to share more critical views about the impact of SEPs. The decision to hold semi-structured interviews was made to enable more depth of information to be gathered. Indeed, the spontaneous comments that commissioners gave provided a greater level of understanding than may have been achieved through questionnaire feedback. For example, the description of Trainees as adding more depth than would have been achieved if the commissioners had completed service evaluation themselves, was not specifically asked about but repeatedly emerged through discussion. Thirdly, one must be cautious when relying on individual recall about the impact and dissemination of SEPs. It is possible that SEPs that had an impact were brought to mind in more detail than those that did not have an impact. If so, the findings of this evaluation may be considered an inflation of the true impact of SEPs. To limit the influence of biased recall, Prepared on the Leeds Doctorate of Clinical Psychology Programme,

17 commissioners who had completed SEPs more recently were also interviewed. Moreover, a spread of the magnitude of impact was described by commissioners; including some relatively small changes that had occurred as a result of the SEP. This suggests that recall bias did not overly impact the feedback regarding SEPs. Finally, the limitation of using of thematic analysis and its reliance on the frequency of themes as a proxy of importance merits consideration. While this is a common method to infer the importance of a theme, it is a measure of frequency and typicality, which may or may not relate to intensity or importance (Sandelowski, 2001). For example, if a SEP had led to one extremely negative impact on a service then we might consider this to be atypical but important. Thematic analysis is not sensitive to such cases. To mitigate this and promote transparency, negative case (i.e. instances where commissioners described no positive impact on their services) were explicitly presented in the results section. Reflections on Present Service Evaluation As a meta-sep, this service evaluation sought to understand the impact of previous service evaluations. With this in mind, it is worth reflecting on how this evaluation might fit with the themes identified from discussions with commissioners. I think that this project will have an impact in terms of Improving Knowledge, particularly in-line with the sub-theme of building an evidence-base. The findings of this project are the first to establish that the SEP process is one that has impact for services. One could consider this project as a baseline to which one could return in the future and explore how things may have changed over time. I also wonder whether a secondary impact of this SEP may be in terms of Improved Process ; with a focus on the sub-theme of working with staff. The insights produced by this project may be of interest to Trainees who are about to embark on the completion of a SEP. For example, thinking in advance about how they might disseminate, or thinking about the features of a SEP that have not had an impact. Recommendations There are three recommendations that are important from this evaluation: 1. SEPs are a valuable resource for local services and should continue to be utilised by clinicians seeking to evaluate activity in an ever-pressurised NHS environment. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

18 2. There are many ways in which SEPs can be disseminated, trainees and commissioners should discuss from the outset of the project how they will disseminate findings. 3. If clinicians are looking for evaluation with the aim of improving processes, knowledge or resources, they should consider the SEP process as a solution to meet this need. Dissemination of this SEP This evaluation will be disseminated in three ways. First, presentation of findings at a conference for all Clinical Psychology Doctorate Programmes, second, presentation and poster presentation at the Leeds and York Partnership NHS Foundation Trust Research and Development Conference, and third, an article has been submitted to the International Journal of Practice-Based Learning in Health and Social Care. Conclusion In sum, SEPs are able to add value by: Providing an impartial approach to evaluation - particularly when asking clients, service users, or staff for feedback. Bringing a vast amount of additional evaluation capacity to the Yorkshire and Humber region. Being able to work across different clinical specialties. Targeting all levels of the healthcare system including patients, staff and wider service pathways. Contributing to wider research or clinical initiatives as well as bringing about change independently of other service activities. Improving process, improving knowledge, and improving resources. Prepared on the Leeds Doctorate of Clinical Psychology Programme,

19 References Braun, V., & Clarke, V. (2008). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), Berwick, D., (2013) A promise to learn a commitment to act: Improving the safety of patients in England. London: Department of Health. Division of Clinical Psychology (2010) Clinical Psychology Leadership Development Framework. Leicester: British Psychological Society. Fairman, S., (2015) NHS Improving Quality Annual Report. Foreword. Ham, C., Berwick, D., & Dixon, J., (2016) Improving quality in the English NHS: A strategy for action. The King s Fund. NHS Institute for Innovation and Improvement (2011) Improvement Leaders Guide Evaluating improvement: General improvement skills. British Psychological Society (2014) Partnership and Accreditation: Standards for Doctoral Programmes in Clinical Psychology. Leicester: British Psychological Society. Sandelowski, M. (2001). Real qualitative researchers do not count: The use of numbers in qualitative research. Research in nursing & health, 24(3), SEP Manual (2016) Doctorate of Clinical Psychology Programme, University of Leeds. The Health Foundation (2014) A Commitment to Act? Online publication: Holloway, I., & Todres, L. (2003). The status of method: flexibility, consistency and coherence. Qualitative research, 3(3), Prepared on the Leeds Doctorate of Clinical Psychology Programme,

20 Appendix A SEP Commissioning Form Prepared on the Leeds Doctorate of Clinical Psychology Programme,

21 Appendix B Interview Schedule Prepared on the Leeds Doctorate of Clinical Psychology Programme,

22 Appendix C Number of SEPs Commissioned Over Time (by specialty) Prepared on the Leeds Doctorate of Clinical Psychology Programme,

23 Adult Health APTS CAMHS Forensic LD Clinical Training N.B. APTS=Adult Psychological Therapies, CAMHS=Child and Adolescent Mental Health Services, LD=Learning disabilities. Older Adults Paediatrics Appendix D Anonymised Characteristics of Commissioners Interviewed Prepared on the Leeds Doctorate of Clinical Psychology Programme,

24 SEP ID Commissioner ID Date Completed Location Specialty Leeds Clinical Training Leeds Adult Health Psychology Leeds Adult Health Psychology Leeds Clinical Training Leeds Adult Health Psychology Leeds Clinical Training Leeds Adult Health Psychology Leeds Adult Psychological Therapies Leeds Adult Psychological Therapies Leeds Adult Psychological Therapies Leeds Adult Health Psychology Wakefield Neuropsychology Wakefield Neuropsychology Leeds Adult Health Psychology Leeds Adult Health Psychology Leeds Paediatrics Leeds Paediatrics Leeds Paediatrics Ossett Adult Psychological Therapies Leeds Learning Disability Leeds Learning Disability Leeds Learning Disability Leeds Learning Disability York Older Adults Leeds Child and Adolescent Mental Health Leeds Child and Adolescent Mental Health Leeds Child and Adolescent Mental Health Dewsbury Learning Disability Huddersfield Adult Psychological Therapies Leeds Adult Health Psychology Wakefield Child Development Leeds Older Adults Leeds Older Adults Leeds Older Adults Leeds Clinical Training Leeds Clinical Training Prepared on the Leeds Doctorate of Clinical Psychology Programme,

25 Leeds Clinical Training Leeds Adult Health Psychology Prepared on the Leeds Doctorate of Clinical Psychology Programme,