Chapter 4 Section 4.01 Ministry of Children and Youth Services Autism Services and Supports for Children Follow-up to VFM Section 3.01, 2013 Annual Report RECOMMENDATION STATUS OVERVIEW # of Status of Actions Recommended Actions Fully In Process of Little or No Will Not Be Recommended Implemented Being Implemented Progress Implemented Recommendation 1 5 1 4 Recommendation 2 1 1 Recommendation 3 1 1 Recommendation 4 3 2 1 Recommendation 5 1 1 Recommendation 6 3 3 Recommendation 7 2 1 1 Recommendation 8 3 2 1 Total 19 3 13 3 0 % 100 16 68 16 0 Background The prevalence of autism spectrum disorder (commonly known as autism) has been increasing. In Ontario, children diagnosed with autism may access general services and supports including speech therapy, occupational therapy and mental health services, funded by the Ministry of Education and the Ministry of Health and Long-Term Care, among others. Our 2013 audit focused primarily on services and supports funded by the Ministry of Children and Youth Services (Ministry) and provided exclusively to children with autism. The Ministry funds two types of autism intervention services or therapies Intensive Behaviour Intervention (IBI) and Applied Behaviour Analysis (ABA)-based services. These services and other supports exclusively for children with autism are delivered through approximately 90 community or hospital-based agencies that are usually not-forprofit organizations. In 2014/15, transfer payments for autism services and supports totalled $188 million ($182 million in 2012/13). Chapter 4 Follow-up Section 4.01 583
584 2015 Annual Report of the Office of the Auditor General of Ontario Chapter 4 Follow-up Section 4.01 In December 2012, the Ministry convened an expert panel to give it advice on up-to-date research on effective clinical practice and autism programs, and it introduced a mechanism through which families can request an independent review of service provider decisions about their child s eligibility for, or discharge from, IBI services. In our 2013 Annual Report, we noted the following key observations: Over the decade ending in 2013, the Ministry had quadrupled autism funding. Despite this, there were more children with autism waiting for government-funded services than there were children receiving them. We estimated that children with autism were diagnosed in Ontario at a median age of a little over 3 years, later than the 18-to-24-month screening period endorsed by the Canadian Pediatric Society for children with risk factors. Although scientific research showed that children with milder forms of autism had better outcomes with IBI, the program was available only to children assessed with more severe autism at the time of our 2013 audit. Research has shown that children who started IBI before age 4 had better outcomes than those who started later. However, due to long wait lists, children in Ontario did not typically start IBI until almost age 7. The median wait time for IBI services in the three regions we visited was almost four years. It was up to the lead service agency to decide how to allocate ministry funding between two IBI service-delivery options: direct service, where the child receives service directly from a service provider at no cost; or direct funding, where the family gets funds from the lead service agency to purchase private services. Wait times for IBI services could differ significantly between the two options and among regions. In one region in 2012, the average wait for IBI services under the direct-funding option was five months longer than the average wait under the direct-service option. In another region, the situation was reversed. Of the children discharged from IBI services in 2012/13 province-wide, those under the direct-funding option received on average almost one year more of services than those under the direct-service option (35 months versus 25 months). In general, children receiving IBI under the direct-service option received fewer hours of therapy than they were approved for. One of the key reasons that this arises is because missed appointments cannot be rescheduled. From 2006 to 2013, the Ministry had reimbursed up to 60 people for a total of $21 million for the ongoing cost of IBI outside of the regular service program. Per person, this represents more than twice the amount that a child in the regular service system typically received. This practice of special treatment continued while others were on a wait list for services. ABA-based services allow a child to work on only one goal at a time, and therefore might not be sufficient for children who have many behavioural problems or goals to achieve. After achieving one goal, the child returns to the end of the wait list for the next available spot. More work was necessary to ensure that ABA methods were being effectively used in schools to educate children with autism. The Ministry had not assessed whether resources were being distributed equitably across the province. Ontario did not have a provincial autism strategy. However, in May 2013, the provincial legislature passed a motion creating a select committee to work on a comprehensive developmental services strategy that was planned to include autism.
Autism Services and Supports for Children 585 Status of Actions Taken on Recommendations The Ministry provided us with information in the spring and summer of 2015 on the current status of our recommendations. According to this information, the Ministry had fully implemented our recommendations in the areas of obtaining from service providers the reasons children are discharged from the IBI program, and, in conjunction with the Ministry of Education, defining minimum training requirements for educators and facilitating access to online resource tools for teachers. The Ministry was in the process of implementing over 60% of our recommendations, mainly in the areas of ensuring that children with autism have earlier access to evidence-based supports and services, assessing whether children and youth made successful transitions to adult services, and developing performance targets and conducting program evaluations. Overall, the Ministry has either fully implemented or was in the process of implementing over 80% of our recommended actions. However, the Ministry had made little or no progress in the following areas: verifying data submitted by service providers through site visits; modifying the IBI and ABA programs (evaluations have not yet been completed); and addressing the inequities of continuing to reimburse 38 people (over 40 people in 2013) for expenses that are outside of the regular IBI program. Regarding the latter, the Ministry reported it was still reimbursing expenses to these individuals because no transition planning work has been done as of yet with the families. The Ministry is considering options with respect to the transition work with the objective of ensuring children with autism receive an equitable level of service and support. The status of each of our recommendations is as follows. Access to Intervention Services and Provision of Intervention Services Intensity and Duration of Service, and Discharge Decision Recommendation 1 To help ensure that children with autism and their families have earlier access to autism supports and services, the Ministry of Children and Youth Services (Ministry) should: work with the Ministry of Health and Long-Term Care and the medical community to facilitate the identification and diagnosis of autism in children before age 3, in accordance with the original objective of the Ministry s Intensive Behaviour Intervention (IBI) program; March 2016. Between September 2013 and November 2014, the Ministry held several meetings with representatives from the health sector, including the Ministry of Health and Long-Term Care, primary health-care professionals and the Ontario Medical Association, to discuss ways autism might be identified earlier in children and to seek input on improving access to diagnosis and assessment. The Ministry was also told by families that access to health-care professionals who diagnose autism is limited. In partnership with the Ministry of Health and Long-Term Care, the Ministry expected to launch two to five multi-disciplinary diagnostic teams in November 2015 in up to six of the province s 14 Local Health Integration Networks (LHINs) to help clinicians recognize the early signs of autism and to increase capacity to diagnose autism. These teams will be based within existing primary health-care providers, including family health teams, community health centres, nurse-practitioner-led clinics and aboriginal health access centres. The Ministry expects to spend $500,000 over 12 months to increase providers capacity to diagnose autism. Chapter 4 Follow-up Section 4.01
586 2015 Annual Report of the Office of the Auditor General of Ontario Chapter 4 Follow-up Section 4.01 monitor wait times as well as wait-list data across the province for both IBI services and Applied Behaviour Analysis based services. February 2016. In April 2014, the Ministry implemented a new quarterly report that tracks wait-list and wait-time data for both IBI and ABA-based services. The Ministry s data shows that the number of children on wait lists for both services has grown since our 2013 audit. For IBI, the wait list grew from 1,748 children to 2,192 children from December 2012 to March 2015; for ABA-based services, the wait list grew from 7,986 children to 13,966 children from March 2013 to March 2015. The Ministry reported that, as of March 2015, children waited an average of 27 months to access IBI services under the direct-service option and 29 months under the direct-funding option. We could not compare this information to what we reported in our audit in 2013 because at that time the Ministry was not tracking provincial wait times for IBI services. However, according to the wait-time information we obtained in three of the Ministry s nine regions, the median wait time for IBI services in 2012 was about 46 months. In September 2014, the Ministry engaged an external consultant to provide advice to three IBI service providers on improving operational efficiencies by following LEAN principles (a business-operation methodology aimed at creating more value for customers with minimal waste). The service providers modified their practices in January 2015 and started to measure the impact on IBI wait times. The Ministry anticipated that the providers would start seeing positive results by the summer and fall of 2015. The Ministry plans to analyze the results and consider next steps by February 2016. In our 2013 Annual Report, we noted that children waited an average of 2.4 months to begin ABA-based services in the 2012/13 fiscal year. As of March 2015, children had to wait 13 months on average. The Ministry received approval to increase spending on ABA-based services by $4.5 million in the fiscal year ending March 31, 2016, to reduce wait lists and wait times. The Ministry told us it expects the funding increase to allow 9,400 children to receive ABA-based services, compared to 7,601 in 2013/14. To help improve program transparency and ensure equity of service in the best interests of the child, the Ministry should: ensure that clear eligibility, continuation and discharge criteria for IBI services are developed and are applied consistently, so that children with similar needs can access a similar level of services; Status: In the process of being implemented; Ministry has committed to report back to the Office of the Auditor General by July 2016. The Ministry s guidelines for IBI services have not changed. They still state that the clinical directors who oversee the provision of IBI services are responsible for making decisions on eligibility and discharge. At the time of our follow-up, the Ministry still did not mandate standard intake criteria or the assessment tools that clinicians should use to determine IBI eligibility, or standard criteria for when a child receiving IBI services should be discharged. In January 2014, the Autism Spectrum Disorder Clinical Expert Committee, comprising researchers, academics and clinicians, recommended the development of clinical criteria for decision-making regarding the progress of children in IBI. At the time of our follow-up, the Ministry was considering options for implementing the Clinical Expert Committee s recommendations regarding the IBI and ABA programs, and would be in a position to respond more fully to the Office of the Auditor General by July 2016. ensure that service providers clearly specify, for every child, the reason that the child is
Autism Services and Supports for Children 587 discharged from the IBI program and report this information to the Ministry for analysis; Status: Fully implemented. it could further monitor whether discharge trends are related to clinical decisions or to how the program is delivered. At the time of our 2013 audit, the Ministry tracked the following reasons for discharging a child from the IBI program: no longer needed IBI; declined services; moved out of region; transferred; deceased; reached age limit; and other. In April 2013, the Ministry changed the list to: met goals; did not meet goals; and other. The latter is now to be used to capture specific reasons for discharge other than those that are related to meeting goals. These reasons must be reported. For the year ending March 31, 2015, of the approximately 700 children discharged from IBI, 61% had reached their goals (down from 74% in the prior year); 23% had not met their goals (up from 14% in the prior year); and 16% were discharged for other reasons (up from 12% in the prior year), such as the family voluntarily withdrew them from IBI, the family moved, or the child turned 18. Service providers across the nine regions reported varying rates of children who were discharged because they had reached their service goals from 27% to 89%. The Ministry concluded that these vast discrepancies were likely due to differences in the ways that each agency interprets whether children have met their goals. Specifically, the Ministry found that the agency that reported 89% of discharged children having reached their goals had modified the individual goals of some children over time to make the goals more attainable. Meanwhile, the agency that reported 27% of its discharged children having reached service goals had set very high clinical goals for all the children in its programs. The Ministry told us that as of April 2015, it planned to further define met goals to ensure service providers practices and interpretations of the term match the intent of the program. In April 2015 the Ministry also began tracking the reasons for discharge under the two service delivery options direct service and direct funding so review the reasons for significant regional differences in the use of the direct service option and the direct funding option, and ensure that decisions on the capacity to provide each service are being made objectively. November 2015. At the time of the 2013 audit, we noted that on average, children under the direct funding option received IBI services for 35 months, whereas children under the direct service option received IBI services for 25 months a difference of almost one year. We also found that the average length of time in IBI varied across the province for both service delivery options: for the direct service option, the length of time ranged from 15 months in one region to 34 months in another; for the direct delivery option, the length of time ranged from 11 months in one region to 49 months in another. At the time of the follow-up, the Ministry informed us that regional differences in wait times are still significant, depending on which service delivery option a family chooses. In response to these differences, the Ministry is considering options for providing direction to service providers to address the regional differences in the use of the direct service option and the direct funding option. The Ministry also identified that one region had a much higher number of children receiving IBI services under the direct funding option and for a much longer period of time. The Ministry had supported this agency in its efforts to reduce the number of months of service in the direct funding option. The agency would work with private IBI providers that it funds to amend aspects of service delivery, including setting discharge timelines and revising individual service plan requirements. Chapter 4 Follow-up Section 4.01
588 2015 Annual Report of the Office of the Auditor General of Ontario Chapter 4 Follow-up Section 4.01 Appropriateness of Intervention Methods Recommendation 2 To help ensure that children with autism have access to evidence-based interventions appropriate to their needs, the Ministry should consider the costs and benefits of offering additional types of therapies and interventions not currently provided, and existing interventions at various degrees of intensity. Status: In the process of being implemented; Ministry has committed to report back to the Office of the Auditor General by July 2016. In January 2014, the Autism Spectrum Disorder Clinical Expert Committee (Committee) provided a series of recommendations relating to improving the continuum of care for children with autism. At the time of our follow-up, the Committee was preparing follow-up reports on intervention directed at children showing early signs of autism and best practices in the design and delivery of ABA. The Ministry received the first report in May 2015, and expects the second report to become available by fall 2015. Based on the input from families and other experts, including the Committee, the Ministry is developing options for ensuring that children with autism have access to evidence-based interventions appropriate to their needs. Intervention Services Funded Outside the Regular Program Recommendation 3 To ensure that children with autism and their families receive an equitable level of service and support and to address existing inequities, the Ministry of Children and Youth Services should apply the same program guidelines to all those who meet the eligibility criteria. Status: Little or no progress. In our 2013 audit, we reported that more than 40 people aged 14 to 25 were still actively claiming IBI therapy and other costs as of March 31, 2013. These people had been receiving services for at least twice as long as children in the regular IBI program. The Ministry does not typically provide IBI services to children aged 14 or older; as at March 31, 2015, only five of the other over 1,400 children in the Ministry s regular IBI program were in that age range. We also found instances where the Ministry reimbursed expenses to some of these individuals to which children in the regular program were not entitled, and that it reimbursed for therapy beyond the 40 hours per week maximum allowed under the regular program. At the time of our follow-up, the Ministry reported it was still reimbursing expenses to these individuals because no transition planning work has been done as of yet with the families. The Ministry is considering options with respect to the transition work with the objective of ensuring children with autism receive an equitable level of service and support. In the year ending March 31, 2015, the Ministry paid in total $2.6 million to 38 people, similar to the annual amount paid at the time of our 2013 audit. Autism Services and Supports in Schools Autism Training for Educators and Transitioning from Community-based Intervention to Schools Recommendation 4 To better ensure that children with autism receive cost-effective supports while in school, the Ministry of Children and Youth Services, in conjunction with the Ministry of Education, should: review the need for the use of autism spectrum disorder (ASD) consultants at many school boards that already employ people to provide similar services, and ensure that all ASD consultants are effectively utilized; June 2016.
Autism Services and Supports for Children 589 As noted in our 2013 audit, since the Ministry of Children and Youth Services (Ministry) launched the school support program in 2004, the Ministry of Education has supported the incorporation of ABA methods in the school system and the building of the capacity of schools to support students with autism, including funding for ABA experts and training activities for educators. In the year ending March 31, 2014, the Ministry transferred $5 million in funding from the School Support Program to the IBI program to create more spaces. As of March 31, 2015, there were 115 full-time-equivalent school support program staff employed in service agencies, 17% fewer than two years prior. As well, in May 2015, the Ministry and the Ministry of Education began examining what resources including the Ministry of Education s ABA expert resources, the Ministry s school support program staff resources, and teaching materials used in schools were available to schools compared to what students with autism and their educators need. The Ministry expected to complete this work by June 2016. define minimum training requirements to assist existing and future educators to use Applied Behaviour Analysis (ABA) principles in the classrooms, and monitor uptake of these education programs; Status: Fully implemented. In April 2014, the Ministry of Education notified all school boards that training for educators who work with students with autism (or who may work with them in the future) should include five areas: behaviour; functions of behaviour; ABA instructional methods; assessments and data collection to inform development, implementation and monitoring of effective individual education plans and transition plans that incorporate ABA methods in a variety of education settings; and principles of ABA with a focus on ABA instructional methods. In addition, the Ministry of Education partnered with the Geneva Centre for Autism, an organization in Ontario that provides clinical intervention services and training programs, to offer a new online certificate course that incorporates these principles. The course was launched in September 2014. As of March 2015, about 1,100 educators and school board personnel had started the course, about 500 of whom had completed it. The Ministry of Education told us it relies on school boards to report whether the training opportunities they offer to their staff comply with the new training requirement. For the 2014/15 school year, 57% of school boards reported that they complied all of the time, 18% reported they complied most of the time, and 25% reported they complied some of the time. assess the usefulness of various online and other resource tools available to assist teachers with effective educational practices for students with autism, and facilitate cost-effective access to the best tools available. Status: Fully implemented. In April 2014, the Ministry of Education launched an online forum for ABA expertise professionals in school boards. This forum contains resources that a provincial working group of school board ABA expertise professionals has identified as being effective in improving the outcomes for students with autism. As well, in January 2015, the Ministry of Education launched a dedicated page on Edu- GAINS, a website with classroom-ready resources for educators. This website includes resources to support educators in programming for students with autism. In addition, in partnership with the Geneva Centre for Autism, the Ministry of Education will monitor the use of the online certificate course mentioned above. Chapter 4 Follow-up Section 4.01
590 2015 Annual Report of the Office of the Auditor General of Ontario Chapter 4 Follow-up Section 4.01 Transition Services for Older Children Recommendation 5 To help ensure that appropriate services and supports are available to persons with autism as they prepare to leave the children and youth system, the Ministry of Children and Youth Services, in conjunction with the Ministry of Community and Social Services and the Ministry of Education, should develop processes to assess whether individuals with autism made successful transitions, including surveys to gauge satisfaction for those who made the transitions and their families. February 2016. In September 2014, the three ministries involved in transition-planning for people with developmental disabilities began to implement a framework establishing processes for collecting information to be used in evaluating the effectiveness of their transition-planning protocols. For example, one process was for conducting a survey to gauge client satisfaction with the transition-planning process. At the time of our follow-up, the three ministries were still developing the survey, and expected to implement it in February 2016. After collecting the responses, the ministries plan to analyze the information and make necessary adjustments to transition-planning protocols. The Ministry reported that between October and December 2014, a total of 1,112 unique requests for an integrated transition plan were made. The number increased to 1,629 between January and March 2015, suggesting that more people are aware of the opportunity to integrate transition plans. Autism Funding Recommendation 6 To ensure that all regions use autism funding costeffectively to meet local demands, the Ministry of Children and Youth Services should: ensure that all lead service agencies place children on the wait list for IBI services only after determining their eligibility, and review whether its funding allocation is aligned with service demand; November 2015 for the first recommended action and by December 2016 for the second recommended action. In the fall of 2015, the Ministry planned to revise the Waitlist Management Directive after consulting with the Regional Autism Providers of Ontario Network on the best way to bring consistency to the use of the referral date (the date to be used when placing a child on a wait list for IBI services) by all lead IBI service providers. In the case of the service agency that we noted in our 2013 audit that was placing children on the wait list before a diagnosis of autism was confirmed, the Ministry informed us that the agency was implementing a policy to stop accepting referrals for children who did not have a confirmed autism diagnosis. As for reviewing whether its funding allocation is aligned with service demand, the Ministry indicated it would wait until it had more information on IBI wait times in both the direct service and direct funding service delivery options, which it began collecting in April 2013. The Ministry said it intends to use that information to support future policy and program design activities and potential funding decisions. periodically compare and analyze agency costs for similar programs across the province, and investigate significant variances; December 2015. In June 2014, the Ministry collected staffing and financial data from service providers so it could
Autism Services and Supports for Children 591 compare and analyze costs for similar programs across the province. The Ministry completed analysis of the cost information in July 2015 and will complete a final report on the analysis at the end of 2015. According to the Ministry, this analysis will help identify cost drivers for the IBI program and will allow it to investigate significant variances across similar-sized agencies providing IBI services. review the reasonableness of the hourly rate under the direct funding option, which was set in 2006. December 2016. In our 2013 audit, we noted that the hourly reimbursement of $39 for the IBI program delivered under the direct funding option had not been updated since it was set in 2006. The Ministry has said it planned to determine the next steps in setting a new hourly rate based on the analysis of staffing and financial data from service providers and the salary levels of current therapist postings. Oversight of Service Providers Recommendation 7 To better monitor whether service agencies are meeting key program guidelines and providing quality services, the Ministry of Children and Youth Services should: review the type of data that agencies are required to submit, and ensure key information is received and analyzed; December 2016. Since April 2013, the Ministry has streamlined the quarterly reporting template that it requires IBI service agencies to complete, and has reduced the frequency that it requires certain data (such as the age of children receiving services) to be reported. In September 2014, the Ministry began producing new quarterly reports on autism services that compare current and historical information on IBI and ABA, including regional and provincial data on the following: wait times; wait lists; the number and age of children receiving services; the proportion of children who met program goals; and the overall average cost per space or instance of service. The Ministry started using a new data verification process. Corporate office staff are to verify that all required autism service information has been reported. Then they are to identify variances, follow up on the variances with regional offices and ensure that service agencies interpret the data correctly. In our 2013 audit, we suggested that the Ministry collect additional information, such as the percentage of families on the wait list that received support services, as well as the number and cause of lost hours of service. At the time of our followup, the Ministry had not collected this information, but plans to have conversations with the lead IBI service providers in fall 2015 about the possibility of tracking the lost hours of service by cause. Following these conversations, the Ministry will determine whether the lost hours of service can be tracked through the quarterly reporting mechanism or other means, and will implement any proposed action regarding this issue by the end of 2016. periodically verified through site visits. Status: Little or no progress. The Ministry has no plans to verify information through site visits as we suggested in our 2013 audit. The Ministry said its new data verification process would enable it to detect inaccurate information. However, it further stated that it might reconsider site visits if it determines that the improvements it made to data verification and analysis did not achieve their intended purposes. Chapter 4 Follow-up Section 4.01
592 2015 Annual Report of the Office of the Auditor General of Ontario Effectiveness of Autism Services and Supports Recommendation 8 To help ensure that services and supports for children with autism are meeting their needs, the Ministry of Children and Youth Services should: develop performance measures and targets for each of its autism services and supports to assess their effectiveness in improving children s outcomes; May 2016. the time of our follow-up. It indicated that it would develop performance measures after it establishes a resource inventory with the Ministry of Education, which it expects to complete by May 2016. conduct periodic program evaluations [on IBI and ABA], including parent satisfaction surveys, and consider conducting a long-term effectiveness study of children who received IBI services and children who were denied IBI services; May 2016. Chapter 4 Follow-up Section 4.01 The Ministry has developed a framework for performance indicators to measure accessibility, effectiveness and efficiency of ABA-based services. It had begun collecting data in areas such as the average time children had to wait to receive ABA-based services, the proportion of children who had met their service goals at discharge, and the average cost per instance of service. However, at the time of our follow-up the Ministry had not yet established any targets for these indicators. The Ministry rolled out two client outcome measurement tools to all ABA service providers in April 2015: the Child and Adolescent Needs and Strengths Autism Spectrum Profile (CANS-ASP); and the Measure of Processes of Care-20 (MPOC-20). The Ministry expects that the data collected with these tools will provide it with information on whether children, youth and their families experienced positive change, and whether families felt satisfied with the services they received. We reported in our 2013 audit that the Ministry did not have outcome measures for the IBI program and the School Support Program. At the time of our follow-up, the Ministry had begun collecting data on how many children had met their goals in the IBI program, but still had not established a target. The Ministry plans to evaluate whether MPOC-20 could help it to determine the effectiveness of IBI. Further, the Ministry had not yet established outcome measures for the school support program at The Ministry noted that it had conducted an evaluation on whether decisions on IBI eligibility and discharge had been made appropriately. It also implemented an independent review mechanism in December 2012; families can request an independent reviewer to rule on whether decisions made by original IBI service providers were consistent with information in the child s file. In the year ending March 31, 2015, 87.5% of the independent reviews on eligibility were consistent with the original decision, and 94% of the independent reviews on discharge were consistent with the original decision. At the time of our follow-up, the Ministry was not considering a long-term effectiveness study of children who received IBI services and children who were denied IBI services. The Ministry said that it expected to complete a comprehensive evaluation of the ABA program s implementation and outcome by May 2016. modify services and supports as required. Status: Little or no progress. As the evaluation results were not yet available at the time of our follow-up, the Ministry has not yet modified services and supports as recommended.