Evidence-based Consumer Health Information: Developing Teaching in Critical Appraisal Skills

Transcription

1 Pergamon International Journalfor Quality in Health Care, VoL 8, No. 5, pp , 1996 Copyright 1996 EUcvier Science Ltd. All right! rejerved Printed in Great Britain PH: S13S3-4505(96) /96 $ Evidence-based Consumer Health Information: Developing Teaching in Critical Appraisal Skills RUAIRIDH MILNE* and SANDY OLIVER Critical Appraisal Skills Programme, Oxford, UK Objective". To help people who give health information to the public develop the skills they need to make sense of evidence about effectiveness. Design: Educational approach, preceded by careful planning with representatives of possible participants. Setting and study of participants: Staff in consumer health information services and members of maternity self-help groups in the UK in summer Interventions: Pairs of half-day workshops introducing participants to randomised controlled trials and systematic reviews and to their critical appraisal. The workshops were run partidpatively and had at their centre a critical appraisal session hi small groups. Main outcome measurer. Attendance at workshops; satisfaction and enjoyment of workshops; comments at a follow-on event Resuhr. Four pairs of workshops were held (three for consumer health information services, one for maternity self-help groups), 54 people attended a pair of workshops and a further 34 attended individual workshops. The workshops were enjoyed and found to be a good use of time. Conclusions: It is feasible to introduce critical appraisal skills to people whose primary role is to give health information to the public There is a need for comparative evaluation of different approaches. Copyright 1996 Elsevier Science Ltd. INTRODUCTION In a "quality health service", health service users should be able to make informed choices about whether they are treated or cared for, by whom, where and for how long. But where are they to get the information on which to make an informed choice? One source in the UK that has been of increasing importance over the last decade is Consumer Health Information Services (CHIS) [1]. Their remit is to provide the public with information on all aspects of health, illness, treatments and health services, either directly or through health professionals. CHIS share a common remit, but in formal terms little else. They are extremely varied in their funding, their organisation and even the settings in which they work: they operate from bases in health centres or hospitals, from stand-alone "health shops" or from telephone information lines. Whereas initially CHIS dealt mainly with enquiries about the provision of services, they are increasingly being asked about the benefits and harms of services as well. A recent report funded by the King's Fund Centre (an independent service development centre) commented that "treatment outcome information is an important part of the services' work"[2]. The NHS Executive in England says that from April 1996 CHIS will be required to provide "researched/evidencebased information... in appropriate formats to enable users to make informed choices between treatment options" [3]. This requirement highlights two themes that are becoming increasingly prominent in the central management of the NHS. The first is a concern to enhance the role of the consumers, a concern which has, amongst other things, led to increased financial support for CHIS. The Received 29 February 1996; accepted 22 May Author to whom correspondence should be addressed at Critical Appraisal Skills Programme, PO Box 777, Oxford Quality Assurance in Health Care, Oxford OX3 7LF, UK. ruairidh@cix.compulink.co. 439

2 440 R. Milne and S. Oliver second is the wish to promote evidence-based health care and clinical effectiveness [4], one consequence of which has been the support given by the NHS Research and Development (R & D) programme to the UK Cochrane Centre and the NHS Centre for Reviews and Dissemination [5]. The UK Cochrane Centre, is of course, only one player in the Cochrane Collaboration, an international network of individuals helping to prepare, maintain and disseminate systematic reviews of the effects of health care [6]. If CHIS are to meet this challenge of providing clients with evidence-based information, they need to be able to do four things [7]: (i) turn clients' problems into questions which research-based evidence may address; (ii) find the best evidence relating to those questions; (iii) appraise (make sense of) that evidence; and (iv) report it back appropriately to their clients. Experience suggests that this may be difficult. The King's Fund report found, for instance, that CHIS made little use of Medline or of medical libraries (likely to be valuable of sources of relevant evidence) and that overall 48% of local and 31% of regional CHIS did not feel able to answer queries about treatment outcomes satisfactorily [2]. CHIS are generalist, with a remit that is very broad. Consumer self-help groups fulfil a complementary role, providing their members with specialist information about a particular disease, disability or aspect of the health service. Ideally, CHIS should be working in close co-operation with consumer self-help groups although, in England at least, this seems rarely to be the case [2]. The Critical Appraisal Skills Programme (CASP) is a UK project that seeks to help health service decision makers develop skills in the critical appraisal of evidence about clinical effectiveness, in order to promote the delivery of evidence-based health care [8], It does this through half-day workshops that introduce participants interactively to the ideas of evidence-based health care and those of the Cochrane Collaboration; and that take them through a participative critical appraisal session in small groups. This uses an approach to critical appraisal that focuses on the needs of users (rather than doers) of research, and so asks three key questions of a paper: is it trustworthy? how important are the results it presents? and, how relevant is it to your situation [9]? Most workshops are problem-based and multi-disciplinary. Since December 1993, CASP has supported more than 100 workshops, attended by over people from a wide range of backgrounds (clinical and non-clinical; some trained in research but most not; medical, nursing, therapist, quality assurance and so on) and settings (primary, secondary and tertiary care; purchasing/financing as well as health care providing) [10]. Two particularly successful workshops were run in the autumn of 1994 for members of the National Childbirth Trust [11], a prominent self-help group in maternity services. Building on this experience, at the beginning of 1995 CASP successfully bid to the King's Fund Centre for funding for a pilot project to run workshops for consumer health information services and self-help groups. The project involved employing a project worker with considerable experience in consumer health information to work a third-time for nine months, supported by the CASP Project Director. The aim and objectives of the project (Box 1) make clear that we planned a development and not a Box 1. Aim and objectives of the CASP for CHIS project Aim To help people who give health information to the public develop the skills they need to make sense of evidence about clinical effectiveness. Objectives (1) Identify with the target groups how these skills can best be developed. (2) Run workshops for target groups. (3) Support target groups in setting their own objectives. (4) Support the target groups in taking this work forward. (5) Produce reports and other follow-up materials for the King's Fund. (6) Evaluate the delivery of the workshops and their impact on target groups' knowledge and practice. (7) Make recommendations for future work in this area.

3 Evidence-based consumer information 441 research project and the format of the rest of this paper reflects this. PLANNING THE WORKSHOPS Much more time was spent planning the workshops than delivering them. This involved detailed work by the project worker in the two main areas: identifying and inviting participants; and defining the content and planning the delivery of the workshops. Participants We were asked by our funders to concentrate on CHIS but to reserve one pair of workshops for maternity self-help groups. In each case, assembling a list of people to be invited proved difficult and time consuming. Those working in CHIS in the UK are part of an extended and loose network with no central register, so an ad hoc register was collated from a variety of sources. We identified more than 30 maternity self-help groups in the UK and, from these, compiled a list of people who might be interested in attending the workshops. In all, we sent invitations to 238 members of CHIS and maternity self-help groups. Invitations were accompanied by an informative flier about the project and by an article about recent developments in finding evidence about effectiveness [12]. Content and delivery of workshops Typical half-day CASP workshops are structured around four sessions; an introduction, critical appraisal in small groups, feedback, and evaluation. Within this, we sought to involve representatives of possible participants as much as possible in planning the workshops' content and delivery. For each workshop, we identified a planning team of two or three people and had several meetings or telephone calls with them before the workshop. This allowed us to tailor the workshops more closely to participants' needs, both educationally and administratively. Educational tailoring included agreeing objectives for a workshop (typical ones are shown in Box 2), choosing learning topics of greatest relevance to participants, producing realistic scenarios and modifying the programme where appropriate. A crucial part of administrative Box 2. Typical objectives of a CASP for CHIS workshop Objectives for both workshops The two workshops will both: (1) give you confidence to understand and explain the importance of evidence-based consumer information; and (2) show how Consumer Health Information Services and librarians can share their interests, skills and resources to find and explain information for the public. Objectives for thefirstworkshop By the end of this workshop, you will: (1) understand different ways of testing whether health treatments work; (2) have critically appraised a published randomised controlled trial; (3) have the confidence to reply to consumer enquiries about treatment outcomes; (4) know who to contact to discuss critical appraisal within Consumer Health Information and library services; and (5) be able to explain why critical appraisal is important for informing consumers. Objectives for the second workshop By the end of this workshop, you will: (1) be able to understand the meaning, potential usefulness and pitfalls of published reviews, overviews and meta-analysis; (2) know more about the Cochrane Database of Systematic Reviews (CDSR) and the Centre for Reviews and Dissemination database and be able to use them; (3) have critically appraised a published review article; and (4) feel confident in offering the best evidence available about the outcomes of a treatment. tailoring involved choosing the time, date and venue convenient for the workshop. One way of increasing participants' ownership of a workshop is to involve opinion-leaders in

4 442 R. Milne and S. Oliver Box 3. Topics discussed in workshops Workshop 1 Workshop 2 Topics discussed at workshops Fitness programme for arthritis (Kovar [20]) Drug treatment for Alzheimer's disease (McLachlan [2 ID Aspirin for migraine prevention (Buring [22D Social support and pregnancy (Oakley [23D running it. While the "stand-up" roles at each workshop were taken by members of the project team, the workshop chair and those facilitating small groups were drawn from expected participants in each workshop. This was only possible by involving potential participants in the planning. DELIVERING THE WORKSHOPS Workshops were run in pairs, with participants strongly encouraged to come to both. The first workshop in each pair focused on how evidence about effectiveness is produced and where it can be found. Participants worked through the critical appraisal of a randomised controlled trial in small groups relating the topic (Box 3) to previously discussed questions. For instance, two questions related to the trial about fitness programmes for people with arthritis: does regular walking help people who suffer with osteoarthritis of the knee? Also, what advice should be given to a practice nurse (who wanted to know what advice she should give to her older clients suffering with arthritis)? The second workshop in each pair focused on how evidence is summarised, on the strengths (and weaknesses) of systematic reviews and introduced participants to the ideas of the Cochrane Collaboration. Again, participants worked through a critical appraisal exercise in small groups, this time looking at a systematic review (Box 3). We ran four pairs of workshops (the first and second of each pair separated by at least two weeks) between July and September Each Stroke prevention (Koudstaal [24]) Family interventions for schizophrenia (de Jesus Man [25]) Iron supplementation in pregnancy (Mahomed [26]) workshop lasted 4-5 hr (excluding breaks) and was in four parts: (1) a plenary session introduced the key concepts of the workshop. In the first workshop, these covered ideas of clinical effectiveness and its measurement, randomised controlled trials, and ACP Journal Club and Evidence Based Medicine as useful sources of randomised controlled trials. In the second workshop, the potential and pitfalls of reviews of evidence were discussed, and the work of the Cochrane Collaboration and the NHS Centre for Reviews and Dissemination cited as useful sources of systematic reviews; (2) participants critically appraised a research paper (of a randomised controlled trial in the first workshop, a systematicreviewin the second) in small groups; (3) participants fed the discussions and conclusions of their small groups back to a plenary session; and (4) a short plenary evaluation session, both verbal and written, concluded the workshop. Each workshop was delivered by a team consisting of both experienced CASP educators and CHIS opinion leaders. The workshops were run in a genuinely interactive way and so participants and those on the workshop teams learnt a lot about each others' ideas and expertise. For instance, participants were asked at one point to give examples of questions they had been asked about treatment outcomes: a selection of those mentioned at the maternity workshop is shown in Box 4. Overall, 84 people attended the workshops, with the number attending each occasion ranging from 9 to 26. Fifty four people attended a pair of workshops and 30 attended individual events only.

5 Evidence-based consumer information 443 Box 4. Selection of questions asked of participants about treatment outcomes Can I have a vaginal childbirth after Caesarean section? Will formula milk help my baby's blood sugar level? Is a stand-alone maternity unit unsafe? Should breast fed babies have extra Vitamin K? Is ultrasound safe? THE IMPACT OF THE WORKSHOPS We report here on the workshops' impact, as assessed by the project team using several complementary approaches. Discussions with planning team before and after workshop. It was clear that those on the planning teams valued this involvement, despite the time it took (typically 8-10 hr, spread over two months). Their involvement was especially important in choosing the topics for discussion at workshops, developing suitable scenarios as a focus for the critical appraisal exercise, and helping identify further participants and identifying suitable venues. Verbal feedback from participants at the end of the workshop. Most feedback about the workshops was positive. Those who attended, however, were clearly a selected group, since it can be difficult for people working on their own or in small teams to get away for training days. Similarly, some parents of small or school-age children had difficulties (we did not provide a creche). Completion by participants of a satisfaction questionnaire at the end of the workshop. At the end of each workshop, participants completed an anonymous questionnaire that included two summary evaluation questions. Replies were received from 105/139 (76%) participants and the results were encouraging. All respondents said they enjoyed the workshop Box 5. Some comments on what some participants learnedfrom the workshops "I feel much more confident about using research literature." "This was a very good afternoon and made me much more aware of what was and wasn't being said in papers. It made me much less willing to accept what was said at face value." "Very informative. I now feel much better able to search for information in future." "Do not feel fully confident to evaluate yet but this was very helpful." "I find it interesting and relevant but still rather difficult." "Interesting but I am still unsure about relevance." "We now need to cascade this training down to front line helpline workers." "We need to think how we can create this awareness in others in our organisation." "quite a lot" or "very much" and nearly all (93%) said they found it a good or excellent use of their time. More illuminating, perhaps, were some of the comments in response to open questions. Those in Box 5 illustrate the widespread sense of empowerment and demystification reported, as well as some continuing uncertainty. Extensive discussion at a follow-on event in October 1995 All 84 people who had attended a workshop were invited to a follow-on "Taking It Forward" day in October There was a positive response from 36 (43%), with 12 attending and a further 24 expressing interest but unable to attend. The event proved a valuable forum in which the importance of CHIS and self-help groups moving into the area of evidence-based health care could be more fully explored. The central challenge was acknowledged to be the growing expectation that CHIS and self-help groups would give out evidence-based outcomes information: an expectation shared by funders of the services, by their users and by those providing them. At the same time, it was recognised that meeting this challenge required a coordinated approach in three areas. First,

6 444 R. Milne and S. Oliver services that give health information to the public need better access to good evidence. Second, providers of such services will need to develop new sorts of skills (including skills in finding and making sense of evidence, and skills in sharing this information with clients). Third, such services will need to change the way they work (for instance, providing more time for training and more time for work with clients, developing quality standards for providing information about outcomes of treatment, and securing funding to support all these changes). Underpinning all these changes was the need for a debate with the research community about the information required. RECOMMENDATIONS In developing and delivering a programme of CASP workshops for people in consumer health information services and maternity self help groups, we were privileged to gain a clearer view of the interface between evidence-based health care and consumer choice. This view informed the recommendations we made in our final report to our funders, targeted at three groups: the staff of Consumer Health Information Services; those who commission them and those who are involved in their training; and the NHS R&D programme. The recommendations covered four areas. We highlighted a need for: (a) CHIS to give clients much high quality information about outcomes; (b) CHIS to have training to support this developing role; (c) formal support and guidance for the provision of outcomes information, in the form of co-ordination, funding, standards and structures for CHIS; and (d) a streamlined dissemination of outcomes information to the public by NHS R&D, working proactively with CHIS; DISCUSSION This project worked in the area of evidencebased consumer choice, a concept which brings together two important modem movements within health care [13]. These are evidencebased medicine (which has been described as a "paradigm shift" in the practice of medicine [14]) and consumer choice (putting the consumer, rather than the clinician, at the centre of health care has been described as "Copernican revolution" [15]). It has been suggested that there may be a perceived conflict between the two [16], but we agree with those who maintain that there can and should be a synthesis [17]. Indeed, part of the project's success came from working with a selfhelp group that has for some years asserted the importance of research based care and informed choice, while defending the rights of patients to base decisions about their own care on other beliefs, wishes or priorities [18]. The experience of the project shows that it is feasible to run CASP workshops for people who give health information to the public. Clearly, however, while such workshops are enjoyed and found useful, they cannot turn critical appraisal novices into experts. The workshops last only half a day and they deal with critical appraisal only in the wider context of evidence-based medicine and the Cochrane Collaboration. From anecdotal evidence, what we believe they can do is to help participants improve their knowledge and skills in this area; more formal evaluation of this is under way. At the same time, the impact of the project should have been more than the impact of the workshops, including as it did a range of activities designated to raise the profile of evidence-based consumer choice: informative invitations, thorough planning, participative evaluation and the "Taking It Forward" event. A number of important issues that arose during the workshops should be mentioned here. These include difficulties in discussing risk and uncertainty; the misunderstandings which may arise from differences in lay and professional use of language; issues of who is involved in decision making; and the dividing lines between health information, education, counselling and advice (our use of the term "advice" in the questions relating to the topic on a fitness programme for arthritis caused considerable disquiet). Most important, if consumers are to make informed choices, they must have access to treatment outcomes information that is reliable, expressed in terms that are meaningful [19] and that applies to their situation. CHIS and selfhelp groups are likely to have an important role in promoting such access and we have found that people in CHIS and self-help groups can learn these skills and are keen to share them with their

7 Evidence-based consumer information 445 colleagues. The challenge now is to develop and evaluate different approaches to helping them do this. Acknowledgements: We are grateful to the King's Fund Centre for funding the project; to the very many people who helped make the project a success and in particular to Gaire Spittlehouse and Samantha Chapman for administering the workshops; to all participants in the workshops for their hard work, good humour and constructive criticisms; and to-. Andrea Wilkinson for taking the paper through many drafts. REFERENCES 1. Dunning M and Needham G. But will it work doctor? Consumer Health Information Consortium, Milton Keynes, Gann R and Buckland S. Dissemination of information on treatment outcomes by consumer health information services. King's Fund, London, Health Service Guidelines. Provision of the national freephone information service. NHS Executive HSG(95)44. NHS Executive, Leeds, UK, NHS Executive. Promoting clinical effectiveness. NHS Executive, Leeds, UK, Sheldon T and Chalmers I, The UK Cochrane Centre and the NHS Centre for Reviews and Dissemination. Health Economics 1994; 3: How the Collaboration works, in About the Cochrane Library. The Cochrane Library, issue 1. London: BMJ Publications, Sackett D L and Rosenberg W M C, On the need for evidence-based medicine. Journal of Public Health Medicine 1995; 17: Critical Appraisal Skills Programme. Orientation Guide. Oxford: Institute of Health Sciences, Oxman A D, Sackett D L and Guyatt G H, Users' guides to the medical literature: I. how to get started. JAMA 1993; 270: Critical Appraisal Skills Programme. CASP Update, March Institute of Health Sciences, Oxford, Oliver S, Does health care do us good? How do we know? New Generation Digest. 6 September Vines G, Is there a database in the house? New Scientist (21 January) 14, Hope T, Evidence-based Patient Choice. Institute of Health Sciences, Oxford, UK, Evidence-Based Medicine Working Group. Evidence-based medicine. JAMA, 268: , Battista R N, Practice guidelines for preventive care: the Canadian experience. British Journal of General Practice 1993; 43: Pickering W G, Does medical treatment mean patient benefit? Lancet 1996; 347: Oliver S, How can health service users contribute to the NHS research and development programme? British Medical Journal 1995; 310: Chalmers I, NNT or LBHH? Bandolier 1995; 3: Milne R and Sackett D L, The message is in the medium. BMJ 1995; 311: Kovar P A, Allegrandte J P and MacKenzie R, Supervised fitness walking in patients with osteoarthritis of the knee: a randomised controlled trial. Ann Intern Med 1992; 116: Kruck TP«/ al., Intramuscular desferrioxamine in patients with Alzheimer's disease. Lancet 1991; 337: Buring J E, Peto R and Hennekens C H, Low dose aspirin for migraine prophylaxis. JAMA 1990; 264: Oakley A, Rajan L and Grant A, Social Support and Pregnancy outcome. Br Journal of Obstetrics and Gynaecology 1990; 97: Koudstaal P, Secondary prevention following stroke or transient ischemic attack in patients with nonrheumatic atrial fibrillation: anticoagulant versus antiplatelet therapy. In: Warlow C, Van Gijn J, Sandercock P (eds) Stroke Module of The Cochrane Database of Systematic Reviews, 1995 (updated 9 March 1995). Available from BMJ Publishing Group, London. 25. de Jesus Man Jair and David Streiner. The effects of family intervention for those with schizophrenia. In: Adams C, Anderson J, De Jesus Mari J (eds) Schizophrenia Module of The Cochrane Database of Systematic Reviews, 1995 (updated 22 February 1995). Available from BMJ Publishing Group, London. 26. Mahomed K, Routine iron supplementation in pregnancy. In: Enkin M W, Keirse M J N C, Renfrew M J, Neilson J P (eds.) Pregnancy and Childbirth Module of The Cochrane Database of Systematic Reviews, 1995 (updated 24 February 1995). Available from BMJ Publishing Group, London.