When Passion Obscures the Facts: The Case for Evidence-Based Testing

Size: px

Start display at page:

Download "When Passion Obscures the Facts: The Case for Evidence-Based Testing"

Lynn Hunt
6 years ago
Views:

1 When Passion Obscures the Facts: The Case for Evidence-Based Testing Stuart Reid, 2010 Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: Fax:

2 How not to write an abstract? For many years we have enjoyed (and sometimes endured) the debates between rival factions in the software testing community. These factions can be labelled into schools of testing, such as the context-driven, standard, analytic, and quality schools, although many simplify this to the context-driven versus the rest. Proponents are equally passionate both for and against, but passion is often used to conceal a lack of real evidence supporting the opposing positions. We see evangelists on both sides making claims and it is often those who speak the loudest, those who speak most persuasively, and those who appear the most passionate swaying opinion mainly by sheer force of personality. This presentation will identify which claims are supported by valid evidence, which claims disagree with the available evidence, and those claims where there is currently insufficient evidence to reasonably support a claim one way or the other.

3 Adult Testing Blogs? That is, let s cry havoc and let slip the dogs of testing. There are bad people in the world I won't be put out if one of you bigger dogs gets on him first."

4 Passion, charisma facts? Have you believed everything you ve heard so far at this conference? who do you trust? Are you part of a testing school? does it need defending from outsiders? or does your school defend the industry? does it make your views more extreme? How easily do you change your mind? Don t believe me try using the evidence

5 Evidence-based medicine (EBM) EBM aims for the ideal that healthcare professionals should make "conscientious, explicit, and judicious use of current best evidence" in their everyday practice Wikipedia Gathering the evidence Using the evidence researchers providers

6 Evidence-based practice what do you need to know? finding the evidence appraising the evidence applying the results

Medical Evidence - sources 1 2 3 4 5 6 meta-analysis (of randomized controlled trials) randomized controlled trials controlled

7 Medical Evidence - sources meta-analysis (of randomized controlled trials) randomized controlled trials controlled studies without randomisation well-designed quasi-experimental studies case studies, etc. expert reports, opinions and/or clinical experience

8 Gathering evidence confidence in evidence fact testing is too immature to wait for facts we don t need 100% confidence opinion # studies

9 1 2 combined result

10 Meta-analysis.. saves >10,000 babies from dying in the last 20 years

11 Evidence-based testing EBT aims for the ideal that testing professionals should make "conscientious, explicit, and judicious use of current best evidence" in their everyday practice Stuart Gathering the evidence Using the evidence researchers testers

12 Gathering the testing evidence 1 meta-analyses 2 academic research 3 case studies 4 literature searches 5 surveys expert opinion 6 7 vendor reports

13 Providing the testing evidence Academic evidence vs Practitioner evidence We need to know its: context - where is this evidence valid? level of confidence We need to be able to find it searchable tags In a standard format?

14 Evidence-based testing practice what do you need to know? finding the evidence appraising the evidence applying the results

15 Finding & appraising the evidence peer journals academic conferences text books / education personal experience commercial conferences magazines training courses / expert opinions anecdotes

16 Confidence in the evidence most confident peer journals academic conferences text books / education commercial conferences / magazines least confident training courses / expert opinions / anecdotes

17 Complexity of evidence highly complex peer journals academic conferences text books / education commercial conferences / magazines easy to digest training courses / expert opinions / anecdotes

18 US Pre-schooler (2-5) diet advice

19 US Kids (6-11) diet advice

20 US Adult diet advice?

21 More adult dietary advice Best Antioxidants for Skin Cochrane Review Meta-Analysis: antioxidant supplements--vitamins A, C, and E; beta-carotene; and selenium--do not prevent heart disease or cancer, do not forestall death, and may even increase mortality risk slightly. The first signs of aging usually occur when there are wrinkles on the skin. There are various antioxidants for skin that can help in decreasing the aging process. The effects of antioxidants are evident after a long period of time. Consulting a skin doctor before you incorporate any of the antioxidant supplements in your diet is always the safest option you can take. Here are some best antioxidants for skin. Read more on antioxidants: how they work. Vitamins Sun skin creams and lotions are rich in vitamin E and this is so, because the vitamin E in these cosmetics prevents the damage of cell membranes. It is believed that when vitamin E is applied before sun exposure, it helps in lessening the skin redness, swelling and sun burn. A vitamin supplement that suits your skin is a good choice, as an antioxidant. Wheat germ, safflower, corn, soybean oil, mangoes, nuts Nice and broccoli are some skin, foods that are rich in vitamin E. Similarly, there are Vitamin C (ascorbic acid) and Vitamin A (1st antioxidant used for anti-aging) antioxidants, that one can use to control aging process. shame she s dead! While vitamin C rich foods are cereals, beef, poultry, and fish vitamin A is found abundantly in potatoes, carrots, milk, egg yolks, and cheese. Read more on best antioxidant supplements. From

22 A naïve view of the profession number of testers capability

23 Actual IT Professionals number of IT professionals Process Maturity Profile of the Software Community 2001 Year End Update, SEI, capability

24 Are testers more capable? number of testers/developers capability

25 The profile for testers? number of testers capability

26 Testers are not all the same number of testers 20% 60% 20% capability

27 Complexity of evidence highly complex peer journals academic conferences text books / education commercial conferences / magazines easy to digest training courses / expert opinions / anecdotes

28 Usability of evidence complexity of evidence the boundary of understanding capability

evidence Should already be competently finding and appraising

29 Presenting the evidence to testers Need to be provided with simple rules or guidelines May need to be trained to find and appraise evidence Should already be competently finding and appraising evidence PLUS Should consider contributing to the guidelines on current best evidence

30 So, is there any evidence already out there.?

31 Useful Questions/Evidence? Are formal reviews an efficient way of finding defects or are there better alternatives? Do testing tools save you money, cost you money or are they simply a distraction? Does using standards reduce the probability of project failure? Is a certified tester more effective than those without industry certification? Will exploratory testing detect more bugs than scripted approaches?

32 For each Question confidence 35% defect removal efficiency against for

33 Formal reviews? confidence 85% max. defect removal efficiency Capers Jones Software Engineering Best Practices Lessons from Successful Projects in the Top Companies, McGraw Hill, Based on data from more than 13,000 projects 600 corporations and more than 30 government organizations. against for

34 System Testing? confidence 40% max. defect removal efficiency Capers Jones against for

35 Automated Static Analysis? confidence 87% max. defect removal efficiency Capers Jones against for

36 Testing tools? EuroSTAR 2007 survey approx. 600 respondents 42% achieved significant benefit 33% had reasonable benefit 16% had some benefit 10% had no benefit from the tools

37 Synergistic Approach? confidence Formal Reviews System Testing Static Analysis 99% max. defect removal efficiency Capers Jones against for

38 Standards? confidence 8.33% average defect removal Capers Jones against for

39 Certified testers? confidence 8.5% average defect removal Capers Jones against for

40 Certified testers? EuroSTAR 2007 survey approx. 600 respondents 35% said that they knew more about testing 26% got a better job or contract 17% were better recognised for their testing skills 1% reported higher pay 23% said it had no effect

41 Exploratory testing? Will exploratory testing detect more bugs than scripted approaches? Yes, if there s no time to script and run tests Not a useful question the two approaches should ideally be used as complementary approaches

42 Evidence-based testing what do you need to know? finding the evidence appraising the evidence applying the results

43 Conclusions We should base our decisions on all the available evidence not restrict ourselves to one or two favourite sources (such as ourselves) More research is needed and the results need to be made accessible to all Testers are not a homogeneous group different capabilities have different needs It doesn t help the discipline if those with different views spend too much of their time arguing amongst themselves

44 An (academic) EBT Bibliography Procedures for Performing Systematic Reviews, Barbara Kitchenham, NICTA Technical Report T.1, July, Reviewing 25 Years of Testing Technique Experiments, N Juristo et al, Empirical Software Engineering, 9, 7 44, Evidence-Based Software Engineering for Practitioners T Dyba et al, IEEE Software, January / February In Search of What We Experimentally Know about Unit Testing, N Juristo et al, IEEE Software, November/December What Do We Know about Defect Detection Methods?, P Runeson et al, IEEE Software, May/June Software for Dependable Systems: Sufficient Evidence?, D Jackson, MIT, NSA HCSS, Baltimore, MD, May 9, Systematic literature reviews in software engineering A systematic literature review, Information and Software Technology 51, 7 15, A systematic review on regression test selection techniques, E Engström et al, Information and Software Technology, 52, 14 30, 2010.

ABLE TO READ THE LABEL?

ARE ABLE TO READ THE LABEL? A Consumer s Guide to Navigating Food Labels Food and Drug Safety Program Consumer and Environmental Health Services Phone: 609-826-4935 Fax: 609-826-4990 http://nj.gov/health/eoh/foodweb/