Defect Detection Prevention

Size: px

Start display at page:

Download "Defect Detection Prevention"

Patrick Bates
5 years ago
Views:

1 Defect Detection Prevention CS/SE 3RA3 Ryszard Janicki Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada Ryszard Janicki Defect Detection Prevention 1/11

2 Previous Lecture: Risk Analysis Risk identification Risk assessment Risk control what system-specific risks? countermeasures likely? as new reqs severe, likely consequences? Ryszard Janicki Defect Detection Prevention 2/11

3 Impact matrix Effectiveness matrix Optimal Balance DDP = Defect Detection Prevention Technique and tool developed at NASA in 2003 Quantitative support for Identify-Assess-Control cycles Three steps: Elaborate risk ikimpact matrix Elaborate countermeasure Effectiveness matrix Determine optimal balance risk reduction / countermeasure cost Ryszard Janicki Defect Detection Prevention 3/11

4 Three Steps Impact matrix Effectiveness matrix Optimal Balance 1 Build a risk-consequence table (impact matrix) with domain experts for: prioritizing risks by critical impact on all objectives highlighting the most risk-driving objectives 2 Build a risk-countermeasure table with domain experts for: estimating risk reduction by alternative countermeasures highlighting most globally effective countermeasures 3 Determine optimal balance: risk reduction vs. countermeasure cost Ryszard Janicki Defect Detection Prevention 4/11

5 Impact matrix: example for library system For each objective obj and risk r, we provide: Impact(r, obj) = estimated loss of satisfaction of obj by r 0 (no loss) 1 (total loss) Risk Criticality: RC(r) = Likelihood(r) obj Impact(r, obj)weight(obj) Objective Loss: Loss(obj) = Weight(obj) r Impact(r, obj)likelihood(r)

Effectiveness matrix: example for library system For each countermeasure cm, weighted risk r, we provide: Reduction(cm, r) = estimated reduction of r if cm

6 Effectiveness matrix: example for library system For each countermeasure cm, weighted risk r, we provide: Reduction(cm, r) = estimated reduction of r if cm applied 0 (no reduction) 1 (risk eliminated) Combines Reduction: CR(r) = 1 cm (1 Reduction(cm, r)) Overall Effect: OE(cm) = r Reduction(cm, r)criticality(r)

7 Impact matrix Effectiveness matrix Optimal Balance Determine optimal balance: risk reduction vs. countermeasure cost Cost of each countermeasure cm to be estimated with domain experts DDP can then visualize: risk balance charts: residual impact of each risk on all objectives if cm is selected optimal combinations of countermeasures for risk balance under cost constraints simulated annealing search for near-optimal solutions optimality criterion can be set by user e.g. maximize satisfaction of objectives under this cost threshold minimize cost above this satisfaction threshold Ryszard Janicki Defect Detection Prevention 7/11

8 Example ( System) Risk Consequence Table (Note, the losses are decimals between 0 and 1, where 0 is no loss and 1 is total loss): Objectives Risks > Door closes on passenger 0.2) Passengers reach destination in reasonable time frame (weight: 0.1) Passengers are safe (weight: 0.6) system correctly accounts for the elevator s status (weight: 0.2) s respond to calls in a reasonable time frame (weight: 0.1) Risk Criticality Braking system fails 0.1) Doors open in between floors 0.2) stops in between floors 0.3) usable by passengers during fire alarm 0.5) Loss obj.

9 Example ( System) Risk Countermeasure Table: Countermeasures Risks > Door closes on passenger 0.2) Have a door sensor on the inner and outer elevator doors Have software independent centrifuge brakes Have sensors to determine when an elevator has fully arrived at a floor Put the elevators to out of service mode during emergency Combined Risk Reduction Breaking system fails 0.1) Doors open in between floors 0.2) stops in between floors 0.3) usable by passengers during fire alarm 0.5) Overall Effect of Countermeasure

10 Impact matrix Effectiveness matrix Optimal Balance Example () From the tables it is easy to see that the greatest risk criticality comes from the elevator being useable during a fire alarm. The countermeasure of having the out of service mode is the most effective countermeasure, according to the chart. This countermeasure should be given priority when the new system gets implemented. The chart shows in order which risks are most in need of attention, and what countermeasures are best at fixing the risks. This table will be used to provide weighting to the countermeasures when analyzing how to spend the budget. Ryszard Janicki Defect Detection Prevention 10/11

11 IMPORTANT Impact matrix Effectiveness matrix Optimal Balance NEVER TREAT LITERARY NUMBERS OBTAINED BY ANY OF THE METHODS DISCUSSED! NEVER TREAT THE EQUATIONS AS LAWS OF NATURE! ALWAYS TREAT THE OUTCOME AS A DIRECTION, NOT AN EXACT RESULT! Ryszard Janicki Defect Detection Prevention 11/11