Defect Density Prediction with Six Sigma. Software Measurement European Forum Roma, 28-May-2009

Transcription

1 Defect Density Prediction with Six Sigma Software Measurement European Forum Roma, 28-May-2009

2 Dr. Thomas Fehlmann 1981: Dr. Math. ETHZ Mathematical Logic, Combinatory Logic : Manager Software Development Datacolor AG Color Quality Management : Senior Consultant Project Office DEC CH Six Sigma Black Belt for Systems Integration : Sales Support Manager Proposal Center Unisys Schweiz and Italy 1999ff: Euro Project Office AG, Zürich Project Management, Coaching & Support SwiSMA: Software Metrics, Function Points, COSMIC FFP Effort & Defect Prediction for SW Project Akao Price 2001 for original contributions to QFD Member of the Board of QFD Institute Germany QFD Architect Six Sigma Black Belt for GMC Software AG Master Black Belt for Siemens Building Technologies Defect Density Prediction, Slide 2

3 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 3

4 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 4

5 What is Six Sigma? 1. Focus on Customer s Needs 2. Make Processes defect free 3. Manage quantitatively σ Six Sigma Defect Density Prediction, Slide 5

6 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 8

7 Simple Defect Counting DPMO (Defects Per Million Opportunities): Every click is one opportunity for a defect! A click triggers entries, outputs, and browser transactions, as well as transactions to internal data store, and external data sources Expert users count! Defect Density = Defect Density Prediction, Slide 12 # of Defects # of Clicks for each SW module Transactions!

8 Mistakes, Bugs, and Defects Developers find Mistakes and eliminate them Testers find Bugs Developers eliminate them Customers find Defects Supporters assess them, Developers eliminate them, and Testers repeat testing! Defect Density Prediction, Slide 14

9 Inception Planning and Design Implementation Testing and Acceptance The Origins Of Defects Customer s Needs (VoC) Defects introduced: 11 Defects Business Objectives (RD) 63 Defects Technical Solution (TS) 52 Defects Coding & Unit Tests (VER) 124 Defects Integration Testing (PI) 14 Defects Application Testing (VAL) 2 Defects Customer Operations (Ops) Defects removed: Detected in Mgmt Reviews: 7 VoC defects Detected in Peer Reviews: 2 VoC defects 49 RD defects 0 VoC defects Detected in Tests: 7 RD defects 30 TS defects 1 VoC defect 1 RD defect 16 TS defects 92 VER defects 1 VoC defect 6 RD defects 6 TS defects 22 VER defects 13 PI defects Detected during productive use: Defect Density Prediction, Slide 15 0 VoC defects 0 RD defects 0 TS defects 10 VER defects 1 PI defect 2 VAL defects

10 Simplified Defect Density Count Released Defects: VoC RD TS VER PI VAL Ops Total Defects Detected: VoC RD TS VER PI VAL 2 2 Total Count Defects Identify Origin Calculate Density Defect Density Defect Density VoC RD TS VER PI VAL Ops Total VoC RD TS VER PI VAL Total Number of Defects VAL PI VER TS RD VoC 0 VoC RD TS VER PI VAL Ops Defect Density Prediction, Slide 16

11 Defect Density Strategies Defect Density Defect Density Number of Defects VAL PI VER TS RD VoC Number of Defects VAL PI VER TS RD VoC 0 VoC RD TS VER PI VAL Ops 0 VoC RD TS VER PI VAL Ops Defect Density Defect Density Number of Defects VAL PI VER TS RD VoC VAL PI VER TS RD VoC 0 0 VoC RD TS VER PI VAL Ops VoC RD TS VER PI VAL Ops Defect Density Prediction, Slide 17

12 Two Problems 1. What is one (1) defect? How to account for dependencies between defects? How to rate defects that can range from software unfriendliness to malfunction? 2. How significant is it to count clicks? Will two experts use the same amount of clicks? Isn t there any better scale for functionality? Defect Density Prediction, Slide 18

13 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 19

14 Functional Size of Business Requirements Data Browser EQ 2 / 2 ComboBox EQ 1 / 2 Function Click with Parameters EI 2 / 6 Report EO 2 / 12 Data Modification Click EI 2 / 8 IFPUG Functional Size (ISO/IEC 20926:2003) is easily derived from Use Case Analysis, or from the User Manual EIF 1 / 2 External Clock EI EO EQ ILF EIF Boundary Six Sigma Sample FP Count, Folie 1 ILF 2 / 12 Data Store UFP=31 Good choice for understanding and sizing user requirements Business Requirements Sizing Unit (UFP = Unadjusted FP) Defect Density Prediction, Slide 20

15 User Device, or Engineered Device Data Group Functional Size of Technical Requirements The COSMIC Full Function Points metrics (ISO/IEC 19761:2003) size requirements from different viewpoints in COSMIC functional size units (Cfsu) No. of Entry/Exits to/from Functional Processes No. of Read and Writes to/from Data Groups (Storage) Entry (E) Exit (X) Entry (E) Exit (X) Functional Process Write (W) Read (R) Defect Density Prediction, Slide 21

16 Learning Opportunity Ratio (LeOR) You can best express the LeOR value by its Sigma Sigma 1: initial Sigma 2: basic Sigma 3: acceptable Sigma 4: mature Sigma 5: excellent Sigma 6: a dream! Learning Opportunity Ratio Success Rate Sigma Value 93.0% % % % % % % % % % % % % % % % % % % % % % % % % % 6.0 Defect Density Prediction, Slide 22

17 Measuring LeOR Time is of Essence: Time Functional Size Measure time by two categories: Time spent (PD) per Work Packages Percentage of time used per Work Package for Learnings Refactoring Tracing Debugging Rework Bug Fixes not including reviews and tests Ratio in Percent (%) or 5 PD out of 12 PD Defect Density Prediction, Slide 23

18 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 25

19 Different Views? Defect Density Prediction, Slide 26

20 y = f(x) Transfer Functions Use Case Solution (x ) Customer s Needs (y) Defect Density Prediction, Slide 27 x y = T(x): The Use Case Solution x yields the Customer s Needs y = T(x) Response to Customer s Needs is a function of Use Case Solution T(x) Select critical parameters Representative profiles for business and technical requirements Given technical requirements profile x = <x 1,,x n >, response profile to business needs is y = T(x) = < 1 (x),, m (x)> T -1 is the inverse transfer function T -1 predicts the solution x that yields y = T(x), given goal y: x = T -1 (y) For a matrix representation of T, T T is the transposed matrix T T approximates T -1 if T has an Eigenvalue

21 The Convergence Gap The Convergence Gap measures the gap between the importance of solution requirements, and the response they effectuate on the goal topic Small gap means robust selection of solution components The transfer function yields repeatable results! Even if we don t know much more than the three level cause/effect relationships Defect Density Prediction, Slide 33

22 VoC Total Cost of VoC % Cost for Learning Time Measurement Program Customer s Needs (VoC) RD VoC Ops VoC RD Total Cost of RD. Elicitation % Cost for Learning Time Origins of Learning Opportunities Total required Functional Size Acceptance Test (Ops) #LeOR Customer Total Cost of Defect Removal Origins of Defect found Total Functionality delivered Use Cases (RD) #UFP PI RD VAL RD Defect Density Prediction, Slide 34 Technical Feature (PI) #Csfu TS Total Cost of TS % Cost for Learning Time Origins of Learning Opportunities Total implemented Functional Size FT PI Integration Tests (FT) #LeOR Application Test (VAL) #LeOR VAL Total Cost of Testing % Cost for Bug Removal Origins of Bugs found Total tested Functional Size VER & PI Total Cost of Construction % Cost of Reviews and Refactoring Origins of Mistakes and Improvements Total implemented Functional Size

23 VoC based on T T (UFP) Initial Sigma = Initial VoC - Learning Effectiveness Final Sigma = Target VoC Defect Prediction Customer s Needs (VoC) RD VoC Ops VoC RD based on UFP Initial Sigma = Initial RD - Review Effectiveness - Removed VoC LeOR's Final Sigma = Target RD Acceptance Test (Ops) #LeOR Customer compared with UFP Initial Sigma = Target Customer! Use Cases (RD) #UFP TS RD VAL RD!!! TS based on Csfu Initial Sigma = Initial TS - Review Effectiveness - Removed VoC, RD LeOR's Final Sigma = Target TS Application Test (VAL) #LeOR VAL compared with UFP Initial Sigma = Initial VAL - Testing Effectiveness - Removed VoC to VER LeOR's Final Sigma = Target VAL Defect Density Prediction, Slide 35 Technical Feature (TS) #Csfu #Csfu VER TS Integration & Tests (VER & PI) #LeOR VER & PI compared with Csfu Initial Sigma = Initial VER - Unit VAL / Peer Review Effectiveness - Removed VoC, RD, TS, VER LeOR's Final Sigma = Target PI

24 You Need One Concept Measure Learning Time Not cost of finding and removing defects Let your team be proud of time spent for learnings and mistakes found Never blame anybody for his or her shortcomings Always blame the process when it didn t detect defects in time! Defect Density Prediction, Slide 36

25 And You Need Two Kind of Numbers: The Effectiveness of your defect removal techniques Reviews, Tests Verification & Validation The Defect Cost Propagation Ratio From VoC to RD From RD to TS From TS to VER/PI From VER/PI to VAL From VAL to Customer Rules of Thumb: Defect Density Prediction, Slide 37

26 Defect Propagation Limits LSL USL Five Defect Propagation Functions: VoC RD RD TS TS VER/PI VER/PI VAL VAL Customer Set Initial Sigma, Target Sigma Defines the Specification Limits = Cost budget for defect removal! Balance Time-to-Market against your Sigma goals! Defect Density Prediction, Slide 38

27 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 39

28 A Practical Example Using the prediction matrix T T, mistakes not detected in previous stages are transferred to next step Where the LeOR costs more Where missed LeOR s finally become defects, when released to customers VoC Effort spent Initial Leor Review Effectiveness Remaining Leor % 11.4% 11.0 VoC-F.1 Trendy Program % 15% 3.8% VoC-F.2 Wide selection % 11% 1.6% VoC-F.3 Easy to get tickets % 7% 1.5% VoC-Q.1Complete information % 11% 2.2% VoC-Q.2Secure reservation % 11% 2.4% RD Inherited Leor Initial Leor VAL Effectiveness Remaining Leor Transfer Factor 60% Brsu 33% 28.5% RD-1.1 Easy access to the movie theatre ticket shop 31% % 36% 2.7% RD-1.2 Let user select date and time of the performance 33% % 44% 6.7% RD-1.3 Graphically represent the theatre for seat selection 48% % 54% 8.3% RD-1.4 Select tickets and confirm purchase (w ith push-button 35% % 38% 6.4% RD-1.5 Print movie tickets at the home printer (no refund) 37% % 39% 4.5% RD-2.1 Record ticket printing and feed purchases into accoun 35% % 46% 5.2% RD-2.2 Manage several movie theatres 20% % 36% 1.8% RD-2.3 Manage performances 37% % 40% 5.2% RD-2.4 Show remaining seat status per performance 23% % 37% 2.4% RD-2.5 Create statistics and reports 24% % 38% 1.5% TS Inherited Leor Initial Leor Remaining Leor VER Effectiveness Transfer Factor 30% Csfu 44% 80.6% TS-1.1 Content Publishing 23% % 15% 3.8% TS-1.2 Content Scheduling 49% % 28% 4.8% TS-1.3 Authentification 31% % 18% 1.7% TS-1.4 Transaction Log 56% % 26% 5.4% TS-2.1 Transaction Handling 52% % 29% 7.2% TS-2.2 Accounting 28% % 21% 5.5% TS-2.3 Fraud detection 42% % 22% 7.7% TS-3.1 Graphical seat selection 49% % 28% 7.9% TS-3.2 Seat assignment 26% % 16% 2.8% TS-3.3 Seat grouping 36% % 20% 4.1% TS-3.4 Seating propositions 52% % 29% 7.0% TS-3.5 Release reservations 28% % 21% 1.8% TS-4.1 Supports all brow sers 44% % 22% 4.9% TS-4.2 Records ow nership 46% % 27% 2.9% TS-4.3 Legal validity 50% % 28% 4.9% TS-5.1 Provides traceability 57% % 27% 6.5% TS-5.2 Trend analysis 48% % 24% 1.7% TS-5.3 Strategic success 61% % 32% 4.0% Defect Density Prediction, Slide 40

29 The Starting Point Six Sigma effectively predicts defect density, i.e. the Learning Opportunities Ratio LeOR Provided you have some initial starting point Which you can get from previous projects Rule of thumb: For inexperienced teams: Initial Sigma = 1 For good, knowledgeable teams: Initial Sigma = 2 For very experienced teams only: Initial Sigma = 2.5 Target Sigma depends from industry Target Sigma = 4 is very good already! Defect Density Prediction, Slide 41

30 Agenda What is Six Sigma? Defects in Software Measuring Defects Defect Prediction based on Function Points Fully Deployed Six Sigma for Testing Example Conclusions Defect Density Prediction, Slide 42

31 Conclusion Six Sigma effectively predicts defect density Requires sophisticated measurement program And sound mathematical statistics! Don t underestimate impact of early stages Voice of the Customer Requirements Elicitation You must measure Functional Size! Defect Density Prediction, Slide 43

32 Thank You. Defect Density Prediction, Slide 44