Statistical Process Control for Software

Transcription

1 Statistical Process Control for Software Anita D. Carleton Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Sponsored by the U.S. Department of Defense 1999 by Carnegie Mellon University 1

2 Topics Introduction Statistical Thinking and Process Thinking Understanding Variation Control Charts Software Eample 1 Steps to Get Started and FAQs 2

3 Process Thinking Focus on the processes to improve quality and productivity Managers must focus on fiing processes, not blaming people Management action uses data from the process to guide decisions Recognize that variation is present in all processes and that it is an opportunity for improvement

4 Fundamental aioms Statistical Thinking all work is a series of interconnected processes all processes are variable understanding variation is the basis for management by fact and systematic improvement Source: G. Britz, How to Teach thers to Apply Statistical Thinking, Quality Progress, June 1997.

5 Interpretation Requires Analysis Data Analysis Interpretation Input Transformation utput Separation of signal from noise requires rigorous analysis procedures. This allows inferences to be drawn to guide decisions and actions. 5

6 Understanding Variation While every process displays variation, some processes display controlled variation, while others display uncontrolled variation. - Walter Shewhart Common cause or controlled variation - due to normal or inherent activities among the process components Assignable cause or uncontrolled variation - due to anomalies in the process total variation = common cause variation + assignable cause variation

7 Process Behavior Variation and Stability Variation = process noise + process anomalies Stable process = process anomalies Stable process = Controlled process 7

8 Stability Is the process that we are managing behaving predictably? Stable process = process in statistical control = sources of variability due to common causes 8

9 The Concept of Controlled Variation Frequency of Measured Values time Variation in Measured Values Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers 9

10 The Concept of Uncontrolled Variation Frequency of Measurement Value time Variation in Measurement Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers 1

11 Statistical Process Control A phenomenon will be said to be controlled when, through the use of past eperience, we can predict, at least within limits, how the phenomenon may be epected to vary in the future. Walter A. Shewhart,

12 Why SPC for Software Development? To understand the reliability of human processes To establish bounds on management epectations To understand patterns and causes of variations To validate metric analysis used to forecast and plan Source: T. Keller, Applying SPC Techniques to Software Development--A Management Approach, 1999 SEPG Conference.

13 Control Charts - 1 Upper Control Limit (UCL) Specification Limits Lower Control Limit (LCL) Control Limits Specification Limits Limits Determined by Process Performance Measurements (Voice of the process) Set by customer, engineer, etc. (Voice of the customer)

14 Control Charts - 2 Inspection Process Performance UCL Defects Per KLC Voice of the Process LCL Time

15 Why Control Charts? Control charts let you know what your processes can do, so that you can set achievable goals. Individual Values Backlog of Unresolved Problem Reports UCL=32.9 CL=2.4 LCL=7.89 They represent the voice of the process. Control charts provide the evidence of stability that justifies predicting process performance. 15

16 Detecting Instabilities and ut-of- Control Situations - 1 To test for instabilities in processes, we eamine control charts for instances and patterns that signal nonrandom behavior. Values falling outside the control limits and unusual patterns within the running record suggest that assignable causes eist. 16

17 Detecting Instabilities and ut-of- Control Situations - 2 Test 1: A single point falls outside the 3-sigma control limits. Test 2: At least two of three successive values fall on the same side of, and more than two sigma units away from, the center line. Test 3: At least four out of five successive values fall on the same side of, and more than one sigma unit away from, the center line. Test 4: At least eight successive values fall on the same side of the center line. Source: Western Electric Handbook 17

18 Detecting Instabilities and ut-of- Control Situations - 3 Test 4: 8 successive points on same side of centerline Individuals Test 3: 4 out of 5 signals in zone B Subgroup No.. Test 1:Single point outside of zone C UCL Zone C Zone B Zone B Test 2: 2 out of three beyond zone B Zone A CL Zone A Zone C LCL

19 An Eample Process in Statistical Control Frequency of Measured Values time Variation in Measured Values Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers 19

20 X-Bar and Range Charts for a Process that Is In Control UCL Average (X-bar) CL LCL t1 t2 t3 t4 t5 UCLR Range (R) CLr t1 t2 t3 t4 t5 Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers 2

21 An Eample ut-of-control Process Frequency of Measurement Value time Variation in Measurement Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers 21

22 X-Bar and Range Charts for a Process that Is ut of Control UCL Average (X-bar) CL LCL t1 t2 t3 t4 t5 UCLR Range (R) CLR Source: Adapted from Understanding Statistical Process Control, Wheeler & Chambers t1 t2 t3 t4 t5 22

23 Source: W. Florac and A. Carleton, Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley, June Eample: Summary of Defect Types Found During Component Inspections Component Totals Defects Defect Type Number of Defects per Type per Component Function Interface Timing Algorithm Checking Assignment Build/Pkg Document

24 Xm R Charts for the Total Number of Defects Found in Component Inspections Moving Range Total Defects Component Number UCL = 44.9 CL = 22.4 LCL = UCL = 27.6 CL = 8.45 Source: W. Florac and A. Carleton, Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley, June 1999.

25 Control Charts for Individual Defect Types Defect Type = Function Defect Type = Timing Defect Type = Checking Defect Type = Build/Pkg Defect Type = Interface Defect Type = Algorithm Defect Type = Assignment Defect Type = Documentation Source: W. Florac and A. Carleton, Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley, June 1999.

26 Revised Control Charts for Each of the Defect Types Defect Type = Function X X Defect Type = Timing Defect Type = Checking X Defect Type = Build/Pkg. X X Defect Type = Interface X Defect Type = Algorithm X X X Defect Type = Documentation X X Defect Type = Assignment X X

27 Improved Process- Reduction in Defect Insertion Moving Range Total Defects Component Inspection Sequence UCL=24.99 CL=15.81 LCL=6.634 UCL=11.27 CL=3.45 Source: W. Florac and A. Carleton, Measuring the Software: Statistical Process Control for Software Process Improvement, Addison-Wesley, June 1999.

28 Clarify business goals and strategy Yes Identify and prioritize issues No New goals, strategy? Evaluating Select and define measures No Yes New issues? Process Performance Measure process performance No Yes New measures? Is process stable? No Remove assignable causes Yes Is process capable? Yes No Change process Continual improvement

29 1 Steps to Get Started Get familiar with SPC techniques; refer to the SEI measurement guidebook and related books and papers on SPC. 2. btain a SPC tool. 3. Identify process issues. 4. Identify process performance attributes. 5. Select and define measures. 6. Collect data. 7. rganize the data and ensure principles underlying SPC hold (e.g., homogeniety) 29

30 1 Steps to Get Started Plot/graph data. 9. Eamine each plot/graph for process stability, process shifts, and assignable causes. (a) If process NT STABLE, THEN: - cannot determine the capability of the process - no basis to predict outcomes - action: understand why the process is not stable and determine what steps can be taken to achieve stability (b) If process is STABLE, THEN: - capability of the process can be determined and used to compare future process performance - action: predict future performance and/or eamine other process constituents 1. Run additional analysis as situation requires. 3