PERFORMANCE MEASURES

Transcription

1 PERFORMANCE MEASURES Of predictive systems

2 DATA TYPES Binary Data point Value A FALSE B TRUE C TRUE D FALSE E FALSE F TRUE G FALSE Real Value Data Point Value a 32.3 b.2 b 2. d. e 33 f.65 g 72.8

3 ACCURACY 46% 54% Category A Category B % 99%

4 MATTHEWS CORRELATION COEFFICIENT Accuracy 2/25= 84% PPV 2/23 = 87% FP A TP A Not A Experimental Assignment A Not A TN Prediction FN Not A

5 MATTHEWS CORRELATION COEFFICIENT Accuracy 2/25= 84% PPV 2/23 = 87% FP TP Sens = TP AP Spec = TN AN TP TN FN FP CC = PP AN AP PN A A Not A Experimental Assignment A Not A Prediction TN FN Not A

6 MATTHEWS CORRELATION COEFFICIENT Accuracy 2/25= 84% PPV 2/23 = 87% FP TP Sens = TP AP Spec = TN AN TP TN FN FP CC = PP AN AP PN A A =.95 =.25 =.39 Not A Experimental Assignment A Not A Prediction TN FN Not A

7 SENSITIVITY/SPECIFICITY FP Sens = TP AP Spec = TN AN TP TN FN FP CC = PP AN AP PN A TP A =.95 =.25 =.39 Not A A Not A TN Not A FN =

8 SENSITIVITY/SPECIFICITY FP Sens = TP AP Spec = TN AN TP TN FN FP CC = PP AN AP PN A TP A =.43 =.75 =.37 Not A A Not A TN Not A FN =

9 FROM REAL LIFE

10 REAL VALUE Measured affinity Predicted affinity

11 REAL VALUE Measured affinity Predicted affinity PCC = i (a i ā)(p i p) i (a i ā) 2 i (p i p) 2

12 ROC CURVES Measured affinity Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

13 ROC CURVES Measured affinity Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

14 ROC CURVES Measured affinity Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

15 ROC CURVES Measured affinity TP Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

16 ROC CURVES Measured affinity AP TP Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

17 ROC CURVES Measured affinity TN AP TP Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

18 ROC CURVES Measured affinity TN AP AN TP Predicted affinity Sens = TP AP AUC = Spec = TN AN f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

19 Sens ROC CURVES Measured affinity TN AP AN TP Predicted affinity Sens = TP AP AUC = Spec = TN AN - spec f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

20 Sens ROC CURVES Measured affinity TN AP AN TP Predicted affinity Sens = TP AP AUC = Spec = TN AN AUC=.5 - spec f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

21 Sens ROC CURVES Measured affinity TN AP AN TP Predicted affinity Sens = TP AP AUC = Spec = TN AN AUC=.8 AUC=.5 - spec f(x)dx x = - specificity (false positive rate) and f(x) is sensitivity (true positive rate)

22 CALCULATING A ROC CURVE True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

23 CALCULATING A ROC CURVE True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

24 CALCULATING A ROC CURVE 4 True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

25 CALCULATING A ROC CURVE 4 True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

26 CALCULATING A ROC CURVE 4 True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

27 CALCULATING A ROC CURVE 4 2 True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

28 CALCULATING A ROC CURVE 4 2 True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

29 CALCULATING A ROC CURVE True positive False negative False positive True negative ECCB/ISMB-29 - Immunological Bioinformatics Tutorial

30 AUC = f(x)dx Threshold TP FN TP/(TP+FN) FP TN FP/(FP+TN) >,8 4,29 2,8 >,6 8 6,57 3,23 >,4 3,79 6 7,46 >,2 3,93 9 4,69 > 4 3 True positives rate AUC =.5 AUC =. AUC = False positives rate

31 DEALING WITH SEQUENCE REDUNDANCY

32 OUTLINE

33 OUTLINE What is data redundancy?

34 OUTLINE What is data redundancy? Why is it a problem?

35 OUTLINE What is data redundancy? Why is it a problem? How can we deal with it?

36 DATABASES ARE REDUNDANT CENTER Biological reasons Some protein functions, or sequence motifs are more common than others Laboratory artifacts Some protein families have been heavily investigated, others not FOR BIOLOGICAL SEQUENCE ANALYSIS Mutagenesis studies makes large and almost identical replica

37 DATA REDUNDANCY MHC restricted peptides ALAKAAAAM ALAKAAAAN ALAKAAAAR ALAKAAAAT ALAKAAAAV GMNERPILT GILGFVFTM TLNAWVKVV KLNEPVLLL AVVPFIVSV

38 Sequence identity? ACDFG ACEFG Blast e-values Often too conservative Other 8% ID versus 24% ID What is similarity? DFLKKVPDDHLEFIPYLILGEVFPEWDERELGVGEKLLIKAVA MATGIDAKEIEESVKDTGDL-GE DVLLGADDGSLAFVP SEFSISPGEKIVFKNNAGFPHNIVFDEDSIPSGVDASKISMSEEDLLNAKGE

39 OLE LUND ET AL. %ID = 29/sqrt(alen) (PROTEIN ENGINEERING 997) Alen=; %ID=29 Alen=3: %ID=53 DSSP secondary structure identity in alignments as a function of the alignment length and the percent sequence identity

40 MHC BINDING PEPTIDES 9mer :%id = 29 9 = 97% 5mer :%id = 29 5 = 75% = 89% < 97% = 73% < 75%