Designing a framework for the evaluation of paediatric telepsychiatry: a participatory approach

Transcription

1 Original article... " Designing a framework for the evaluation of paediatric telepsychiatry: a participatory approach Katherine M Boydell* {{, Natasha Greenberg* { and Tiziana Volpe* { *Community Health Systems Resource Group, The Hospital for Sick Children, Toronto; { Department of Public Health Sciences, University of Toronto; { Department of Psychiatry, University of Toronto, Toronto, Canada Summary While there is a great deal of interest in evaluating participants experiences of teleconsultation programmes, specific frameworks for such evaluations are scarce. We have conducted a multi-stage consultation to develop a framework for the study of a paediatric telepsychiatry programme. Emphasis was placed on ensuring the participation of stakeholders in the design and response stage of the evaluation. A three-part approach was taken that comprised an opinion scan, focus groups and individual interviews. This resulted in the identification of specific areas of enquiry for the evaluation. One of the key points to emerge was that attending to context is vital. In the case of telepsychiatry, it is critical to understand the nuances of the local community for whom consultations are being provided. This involves considering the social ecology of each evaluation site. The evaluation should take the form of a dialogue between the evaluators and those being evaluated, in order to maximize the uptake and integration of its findings. The framework we have developed should be viewed as a guide that is general enough to be used in the design of many different types of telepsychiatry programme. Introduction Although there has been an increase in evaluation research in telepsychiatry, comprehensive, participatory evaluative approaches remain scarce 1,2. Participatory evaluations that focus specifically on paediatric telepsychiatry are rarer still. Most evaluation research in the field has concentrated on descriptive data and user satisfaction with telepsychiatry 3. There is also a paucity of qualitative work in the evaluation of telepsychiatry 3 9. While evaluation studies have been conducted using generic frameworks 10, there is currently no framework specifically designed for a participatory approach to the evaluation of paediatric telepsychiatry. Traditional methods of evaluating health services are designed and controlled by consultants under the direction of programme managers or directors. Accepted 12 February 2004 Correspondence: Dr Katherine M Boydell, Population Health Sciences, The Hospital for Sick Children, 555 University Avenue, Toronto, Ontario M5G 1X9, Canada (Fax: ; katherine.boydell@sickkids.ca) Consequently, less powerful people, such as clients and front-line workers, are often overlooked 10. Such individuals rarely participate in the design of the evaluation itself, but rather provide information at a later stage of the study. Recent evaluations of health services have incorporated multiple perspectives at the outset, in order to include stakeholders in the design of the evaluation and to harmonize with the values of the local organizational environment 11. These newer approaches are built on strong theoretical and conceptual foundations, emphasize shared interests, employ multiple methods and perspectives, and meet rigorous methodological standards We have used a participatory process to design a framework for the evaluation of a paediatric telepsychiatry programme. Methods There were 10 rural and remote communities associated with the 10 videoconference sites in the Journal of Telemedicine and Telecare 2004; 10:

2 telepsychiatry programme of the division of child psychiatry at a large urban children s hospital. A threepart approach was taken to participant involvement: an opinion scan, focus groups and individual interviews. Opinion scan Telepsychiatry managers at each site worked with the research team to develop a brief opinion scan, which listed potential areas of focus for the evaluation, derived from the telepsychiatry literature as well as from informal discussions with members of the steering and advisory committees. Focus groups Following the opinion scan, 10 focus groups were conducted via videoconference, one with each of the telepsychiatry sites. Agency staff, familiar with their communities, collaborated with the research team by inviting appropriate local stakeholders to participate in the focus groups. Agency collaborators understood that the goal was to obtain a mixture of individuals, to represent a range of those front-line and managerial personnel who were involved in the provision of children s mental health services, as well as families whose children were using local mental health services 14. Although family members were invited to participate in the focus groups, none did so. Individual interviews Individual, face-to-face, semistructured interviews were conducted with seven consultant psychiatrists, who were purposively sampled according to their subspecialty and degree of involvement in the provision of telepsychiatry services. The telepsychiatry programme manager and medical director were also interviewed. Data analysis A seven-step method was used to analyse the qualitative data 15. The textual data comprised observational and verbatim field notes taken during the focus groups and interviews. In order to develop analytical codes and theoretical propositions, each transcript was read and re-read by the research team to obtain an overall understanding. Each team member then examined transcripts for possible themes, and a codebook was developed to reflect these themes. The transcripts were then systematically coded using the codebook. Within the context of qualitative research, issues such as the reliability of the data and the validity of research findings are included under the umbrella of trustworthiness Team-based peer debriefing was conducted about emerging trends and observations. An audit trail of field notes, meeting minutes and analytic decisions were available to members of the research team 17. Results Individuals who participated in the process included staff of the children s mental health agency, both frontline and management, staff of the telepsychiatry programme, the funder, community-based general practitioners and paediatricians, and other rural community support workers. The last included a faith healer, a trauma specialist, social workers and an independent counsellor. Stakeholder participants identified two aspects to be considered in the evaluation, namely, what information should be collected (content) and how it should be collected (structure). Content The focus groups and individual interviews elicited eight thematic categories of areas to be included in the evaluation (Table 1). (1) Contextual sensitivity A major theme that emerged was the importance of sensitivity to cultural and geographical contexts in the delivery of telepsychiatry services. Community context included factors such as location, local availability of services and resources, language and culture, and the skill level of service providers. It was acknowledged that cultural sensitivity might vary according to the individual consultant psychiatrist overseeing a particular case, who might be more or less aware of the challenges facing families and children living in isolated communities. (2) Education The telepsychiatry programme offered educational seminars to all sites via videoconferencing. Agency staff identified the need to address the barriers to attendance at these seminars. The positive response to seminars, reflected in the number of staff in attendance, was felt be an indicator of the value of the education component, and was therefore identified as an area for further study. (3) Access Most sites stated that telepsychiatry programmes should offer quick and convenient access to services, although extra time might be required when 166 Journal of Telemedicine and Telecare Volume 10 Number

3 Table 1 Evaluation framework for paediatric telepsychiatry Main area for evaluation Contextual sensitivity Education Access Utilization Technology and administration Communication Programme delivery Overall satisfaction Evaluation questions Do the users of the service feel that the programme is sensitive to their local cultural and geographical situation? Do recommendations deriving from the consultation reflect knowledge of the community context? What are the existing barriers to attendance at telepsychiatry educational seminars? How are educational seminars useful? How could the format and delivery of educational seminars be improved? How can increased capacity resulting from educational seminars be measured? How does increased knowledge translate into daily practice? What barriers and facilitators determine access to the telepsychiatry programme? Is the matching process (psychiatric consultant to client) satisfactory? Are telepsychiatry services delivered and written reports received in a timely manner? What administrative support does each host site require to run the telepsychiatry programme? How much staff time is dedicated to the telepsychiatry programme? Who is accountable at each site to ensure the programme runs smoothly? Does the host site feel accountable to the hub site or to the funder? Does this affect usage of the service? Are the objectives of the telepsychiatry programme clear? How are outcome assessed and success measured? How can local doctors be better included in telepsychiatry consultations? How comfortable are participants with use of the technology? Is the physical setting comfortable? Do participants feel that the consultation process is adequately explained to them? How prepared do participants feel when they attend a teleconsultation? Do participants feel that their voices are heard and their questions are answered? What are the key elements of a satisfying and successful consultation? What are the elements of a good report? If a diagnosis is made in a consultation, how is it useful? Are recommendations for treatment felt to be appropriate and feasible? How are the consultation reports used, in the short and long term? How is follow-up handled and by whom? How much preparation time is required by psychiatrists and agency workers in advance of a consultation? How much time is necessary and appropriate for post-consultation activities? How can the usefulness of the steering committee be improved? What role should the hub site play in the management of the programme? Do staff at the remote sites feel a sense of ownership of the programme? Are there areas of consulting expertise that are missing? How satisfied are consultants, service providers and families with the overall consultation process? What is the effect of the programme on:. Access to mental health services for rural family members?. Diagnostic, investigative and management decisions?. Patient and physician travel times?. Feelings of professional and personal isolation?. Educational opportunities? scheduling cases that demanded a specialized match between client and psychiatrist. There was a desire to explore this matching process. The struggles of individuals living in isolated areas in attempting to obtain paediatric mental health services were also highlighted as an issue. (4) Utilization Host sites indicated that there was a need to explore how, and the extent to which, the telepsychiatry programme was promoted in the local communities, as well as people s awareness of it. Utilization was described as being affected by a number of factors, including the remoteness of participating communities, discomfort with technology, lack of time or resources at particular sites, and the reluctance of local doctors to refer to the programme. There was an identified need to address the extent to which the lack of equipment at branch locations as well as distance from the host site made it impractical for workers and families to access the service. (5) Technology and administration Stakeholders identified the importance of examining the technological dimension of the telepsychiatry programme. Most participants felt that the evaluation Journal of Telemedicine and Telecare Volume 10 Number

4 should determine whether or not staff members were adequately trained to use the technology, or whether they had to rely on others to manage the equipment. They felt it was important to identify any barriers to the delivery of services via videoconferencing. (6) Communication Participants believed there was a need to address any lack of clarity regarding the consultation process and the information required to ensure a consultation was effective. The preparedness of the consultant psychiatrist was identified as requiring examination, as was the need to clarify who should be attending the session. Psychiatrists expressed an interest in knowing whether their recommendations were being followed. (7) Programme delivery Stakeholders identified the process of programme delivery as an important component of the evaluation. This included, for example, time and scheduling, the matching of client and psychiatrist, and the mandates of the steering and advisory committees. (8) Overall satisfaction It was strongly suggested that all users, including family members, clinical staff and other stakeholders, be surveyed for their levels of satisfaction. The type of information that was of interest included physical and psychological comfort, the technical aspects of the service, the quality of communication before, during and after consultations, the duration and timeliness of the service, and the helpfulness of the service. Structure In addition to being asked about the content of the evaluation (the selection of topics and questions to be addressed), study participants were asked about the structure of an evaluation, that is, how they would like data to be collected. Overwhelmingly they emphasized the importance of evaluating the programme but cautioned that the process needed to be concise. Staff in rural locations felt that they were already seriously under-resourced and therefore that their involvement in an evaluation study should not demand too much time. In order to ensure that all stakeholders had an opportunity to participate in the research, participants in the evaluation design insisted that potential resource, language and literacy barriers be addressed. Face-to-face or telephone interviews and focus groups rather than written questionnaires were suggested as data-collection methods. Participants were unanimous in advising that multiple and diverse perspectives be included in the evaluation. Maximizing the depth of understanding was the primary objective and, consequently, qualitative methods were proposed for the evaluation. Such methods allow researchers to examine the social and cultural context, and provide insight into the effect of the telepsychiatry programme on participants lives. Discussion There is evidence that the active involvement of stakeholders in the evaluation design and study processes improves the validity, utilization and integration of results with the decision process, and also empowers stakeholders 13. This type of research is highly personal and situational it emerges through a dialogue with intended users about their objectives, as well as the most meaningful (i.e. useful) indicators and means of data collection and measurement 19. It is necessary to capture the diversity of programme experiences and perspectives in order to build a holistic understanding of programme meaning and context 20. One of the key points to emerge from our design process was that attending to context is vital. In the case of telepsychiatry, it is crucial to gain an understanding of the particular nuances of the local community for whom consultations are being provided. This involves considering the social ecology of each evaluation site. By involving local stakeholders in the design of the evaluation, the evaluation team was able to gain information about the resources, barriers and facilitators that influence access to mental health care for children in rural and remote communities. The design and conduct of evaluations should take the form of a dialogue between the evaluators and those being evaluated. This approach recognizes the contribution of stakeholders to both the evaluation and the operation of mental health programmes 21. The participatory framework for evaluation described in this paper suggests rethinking not only what questions we ask, but also what we believe to be important outcomes and appropriate methods and analytical strategies to draw upon in seeking to answer our questions. The framework can be thought of as a map for participatory evaluation research in paediatric telepsychiatry (see Table 1). It is not intended to dictate where to begin or what questions are most important in every evaluation. Instead, the goal is to encourage researchers to seek out participation and feedback from relevant stakeholders at each stage of an evaluation, including the design stage. 168 Journal of Telemedicine and Telecare Volume 10 Number

5 Involving stakeholders in an evaluation from the earliest design phase opens up areas for exploration that are otherwise often overlooked, such as the importance of addressing contextual issues, including geographical and cultural differences. The framework for participatory evaluation design and study presented here should be viewed as a guide that is general enough to be used in the design process of many different types of telepsychiatry programmes. Acknowledgement: This study was made possible through funding from the Ontario Ministry of Children and Youth Services. References 1 McDonald I, Hill S, Daly J, Crowe B. Evaluating Telemedicine in Victoria: A Generic Framework. Melbourne: Victoria Department of Human Services, Yellowlees P. Practical evaluation of telemedicine systems in the real world. Journal of Telemedicine and Telecare 1998;4 (suppl. 1): Boydell KM, Volpe T, Brown N. An Evaluation Design for Paediatric Telepsychiatry. Toronto: Division of Child Psychiatry, University of Toronto, Gelber H, Alexander M. An evaluation of an Australian videoconferencing project for child and adolescent telepsychiatry. Journal of Telemedicine and Telecare 1999;5 (suppl.1): Simpson J, Doze S, Urness D, Hailey D, Jacobs P. Evaluation of a routine telepsychiatry service. Journal of Telemedicine and Telecare 2001;7: Kopel H, Nunn K, Dossetor D. Evaluating satisfaction with a child and adolescent psychological telemedicine outreach service. Journal of Telemedicine and Telecare 2001;7 (suppl. 2): Ermer DJ. Experience with a rural telepsychiatry clinic for children and adolescents. Psychiatric Services 1999;50: Dossetor DR, Nunn KP, Fairley M, Eggleton D. A child and adolescent psychiatric outreach service for rural New South Wales: a telemedicine pilot study. Journal of Paediatrics and Child Health 1999;35: Farrell SP, McKinnon CR. Technology and rural mental health. Archives of Psychiatric Nursing 2003;17: Milstein RL, Wetterhall SF. Framework for Program Evaluation in Public Health. Washington, DC: Centre for Disease Control, Government Printing Office, Schorr L. Common Purpose: Strengthening Families and Neighborhoods to Rebuild America. New York: Anchor Books, Doubleday, Papineau D, Kiely MC. Participatory evaluation in a community organization: fostering stakeholder empowerment and utilization. Evaluation and Program Planning 1996;19: Mercier C. Participation in stakeholder-based evaluation: a case study. Evaluation and Program Planning 1997;20: Patton M. Utilization-Focused Evaluation: The New Century Text. 3rd edn. Thousand Oaks, CA: Sage, Diekelmann NL. Learning-as-testing: a Heideggerian hermeneutical analysis of the lived experiences of students and teachers in nursing. Advances in Nursing Science 1992;14: Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, CA: Sage, Erlandson DA, Harris EL, Skipper BL, Allen SD. Doing Naturalistic Inquiry: A Guide to Methods. Newbury Park, CA: Sage, Lincoln YS, Guba EG. Naturalistic Inquiry. Beverly Hills, CA: Sage, Brinkerhoff JM. Assessing and improving partnership relationships and outcomes: a proposed framework. Evaluation and Program Planning 2002;25: Mathie A, Greene JC. Stakeholder participation in evaluation: how important is diversity? Evaluation and Program Planning 1997;20: Cracknell BE. Knowing is all: or is it? Some reflections on why the acquisition of knowledge, focusing particularly on evaluation activities, does not always lead to action. Public Administration and Development 2001;21:371 9 Journal of Telemedicine and Telecare Volume 10 Number