The Adoption of Evidence Khaled El Emam University of Ottawa Background Software engineering research for many years Health informatics in 2001 TrialStat SRS v1.2-2
Do We Really Need Evidence? Some things are just too obvious need to use common sense Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials v1.2-3 EBM Can We Generalize? Some characteristics of medicine: Regulation (eg, colleges, FDA) CME Long tradition of doing empirical work A very large number of journals and many subdisciplines We have to be careful how much generalization we make to software engineering and software engineering research v1.2-4
Some Motivations for EBM Improve practice (eg, outcomes and safety) Control costs v1.2-5 Goals of EBM Save the practice of medicine from wide variations in clinical practice, the use of unproven intervention, and the failure to apply consistent practice guidelines v1.2-6
EBM Process Convert information needs into answerable questions Track down the best evidence with which to answer these questions Critically appraise the evidence for its validity and importance Integrate this appraisal with clinical expertise and patient values to apply the results in clinical practie Evaluate performance v1.2-7 Difficulties Adopting EDM - I Can effect the income of physicians Redistributes power away from physicians (reduces discretion and autonomy) Resistance to change individual practices Quality of systematic reviews is not always high v1.2-8
Difficulties Adopting EDM - II There may be no, inconsistent, or poor quality evidence therefore have to reply on experience anyway Training practitioners on how to interpret evidence (and uncertainty around that), and how to incorporate evidence into their practice v1.2-9 Difficulties Adopting EDM - III Best practices/treatments have costs and therefore it is not always feasible (time and $) to apply them in the real world (eg, physician time, patient insurance) v1.2-10
Application of EBM - I A practitioner-researcher can apply all of the EBM steps Many practitioners just want to use EBM and therefore do not want to appraise evidence; they just use evidence that has undergone appraisal by others, such as evidence-based practice guidelines and evidence summaries v1.2-11 Application of EBM - II In many cases tracking down evidence and appraising evidence is skipped; practitioners just follow the recommendations/decisions of respected opinion leaders The approach taken will depend on many factors, including whether the condition is encountered frequently in the clinic v1.2-12
Hierarchies of Evidence Randomized controlled trials (RCTs) are considered the gold standard for evidence One has to take the quality of each study into account when summarizing the evidence, and there are quality assessment checklists for many methodologies v1.2-13 Criticisms of EBM - I Over-empiricism: tendency to ignore the inter-play between theory development and evidence development Rare diseases and these with small patient populations are not ideal for RCTs (low power) v1.2-14
Criticisms of EBM - II Many RCTs conflict with each other Evidence from well conducted quasiexperiments and observational studies can come to the same conclusions as RCTs N s too small in RCTs to detect safety problems v1.2-15 Criticisms of EBM - III The behavioral element to treatment, which can be very important not always ideal for RCTS (eg, compliance to protocol, and people change their behavior when they are being watched) Generalizability of findings from RCTs (representativeness of patients, quality of care, availabiliyt of drugs) v1.2-16
Criticisms of EBM - IV Role of sponsors Role of media Publication bias Taking costs into account, not just the benefits Some RCTs would be unethical to perform v1.2-17 Systematic Reviews Calibration of reviewers Inclusion of abstracts Grey literature Updating Incorporation of qualitative evidence Non-English literature Which databases to use v1.2-18
The (Ultimate) Objective of EBSE....is to have an impact on actual software development practice: Adopt new practices and tools Improve existing practices and tools Eliminate ineffective practices and tools v1.2-19 The SE Context - I Largely unregulated: adoption of technology is voluntary Multiple levels of decision making: organizational, project, individual Not that many empirical studies Few replications Few journals to search v1.2-20
The SE Context - II Many SE innovations are preventative, which are more difficult to adopt Many outcome variables are correlated Technology moves quickly, and therefore the evidence may come too late to have an influence on the adoption decisions v1.2-21 The SE Context - III Technology is not standardized, therefore difficult to know if studies are really evaluating the same thing Funding is poor for the development of evidence and non-existence for summarizing it Unit of observation is often projects v1.2-22
Evidence and Adoption The appropriate evidence needed to influence the adoption of a technology is dependent upon: State of evolution of organization State of evolution of the technology Evidence has to be produced in time to match the adoption stage v1.2-23 Technology Adoption - I Process Management Defect Detection Effectiveness (Testing) Productivity Management Size (and productivity evaluation) Defect Management Code Defects Planning and Tracking Resource and Timescale Measurement Objectives Product Support Start Post-Implementation Problem Reports logged Evolutionary Path Example Measurement Performed v1.2-24
Technology Adoption - II v1.2-25 Case Studies - I Provide peer references that help convince adopters in the early stages Experiments and strong evidence is most useful at the later stages of adoption Sometimes a handful of case studies can be packaged to seem like a large scale adoption is going on v1.2-26
Case Studies - II To be able to do larger scale quantitative studies in real settings it is necessary to have a broad adoption base anyway The utility of student experiments SW-CMM is a good example of relationship between evidence and adoption v1.2-27 Adoption Considerations Size matters: Availability of skills Impact on delivery dates Availability of funds for training and consulting Need evidence and adoption packages tailored to specific contexts v1.2-28
Recommendations - I An authoritative consortium of researchers and industry to develop evidence-based practice guidelines Professional societies need to play a bigger role (more than just publishing houses) Lobby for more funding for empirical research in SE v1.2-29 Recommendations - II Incorporate the development and use of evidence in education and training Adopt more multi-method approaches (primary studies and systematic reviews) Promote the publication of case studies (we need the rigor) Promote the publication of negative experiences and failures v1.2-30
www.ehealthinformation.ca v1.2-31