Capstone Project Highway Crash Prediction

Transcription

1 Capstone Project Highway Crash Prediction Alfredo Escriba Project Presentation March 33 th 2017 Crash Risk Medium High

2 Executive Summary - The Problem Accidents are a major disruption in Highways Accidents have a significant human and economic impact Departments of Transportation ( DOTs) do not have a tool to anticipate Accidents Pre-Accident conditions have a pattern so can be partially predicted If accidents can be detected, Preventive actions and Mitigation strategies could be implemented This Project is about detecting the Risk of Accidents

3 Executive Summary Previous research & Focus Previous papers have reached up to 60% accident prediction Very controlled and favorable data sets Not reality of a Traffic Operations Center ( TOC ) Never got implemented at TOC Project has been done With real traffic data as is With a focus on usability at Traffic Operations Center ( TOC )

4 Executive Summary - The Findings & Conclusions Conclusions: Pre- Accident conditions can be anticipated with 66% accident detection and with an operational affordable balance of alarms. Remaining % is due probably to human behavior Individual Human behavior will be traceable soon with Connected Vehicle technology Connected Vehicle opens a fascinating potential for next steps in this Project

5 Executive Summary The project What A Heat Map of Risk of Accident Where 16 Miles of highway 18 to 46 different segments of study Both directions When Data available for 2012 to ,209 crashes Accident Density

6 Data MECE Diagram What is the likelihood of a crash to happen, where and when, given certain known conditions on the Highway Traffic Factors Environmental Factors Time Factors # Vehicles Vehicles per Hour Speed Occupancy Weather Conditions: Rain Snow Fog Ice Light Conditions Sunset Down Day Night Date Time of Day Day of week Holiday

7 Data Incident Data MECE Traffic Factors Sources 1 Excel file per year About Incidents per year About 300 accidents per year Available information Type of Incident Date and Time ( year, day, hour, minute, second ) Location ( latitude and longitude ) Quality Date and time are for when Incident was reported to TMC, not neccesarily for when it actually happened Location is approximate matching to segment Environmental Factors Time Factors

8 Data Traffic Data MECE Traffic Factors Sources: Detector Data Advanced Transportation Management System Probe Data Detector Data 3.3 Mill observations / 18 files Traditional traffic detector devices 2012 to minute interval x 18 segments Speed, Count, Occupancy Available per Lane Good quality when it exists Probe Data 12.8 Mill observations / 1 file GPS probed data 2012 to minute interval x 46 segments Only Speed Consistent availability from 2012 to Good quality ( it is commercialized ) Environmental Factors Time Factors

9 Data Other Data MECE Weather Data Sources: National Weather Service 1 file per month x 48 months Available in 1 hour intervals Hourly Precipitation, Rain, Fog, Snow and Ice Traffic Factors Environmental Factors Time Factors Light conditions Built based on sunset and down along the year Type of Day Built based on calendar

10 Data Working with the data Main challenges New to data management SAS intensive learning Cleansing data, merging files Cleansing data is Time consuming Work with uncomplete data sets ( missing dates, missing records ) Traffic Factors Lessons learned Structure properly data coding macros / functions / coding in general Structure properly data integrity review process MECE Environmental Factors Time Factors

11 Models Models Problem is a Rare Event problem: 0.04% of events Target is Accident = 1 if accident / 0 if no accident 73 predictors, including Interaction terms Worked using SAS Enterprise Miner Used RUS* technique 50-50, & for Train. 70 % Train set & 30 % Validation set Models used Logistic Regression ( Stepwise and Backwards ) Decision Tree ( splitting-rule based on : Entropy, Gini, Chi ) * Rare Event Under Sampling

12 Models Models Next pages show illustration of the results of different models run and what it would look like in the TOC 06 January accidents A busy day in the TOC

13 Models Decision Tree (Chi) Detector Model Decision Tree - Chi Population 1,310,730 Number of Accidents 168 False Positives 256,890 False Positives Rate 19.6% False Negatives 55 False Negatives Rate 0.0% True Positives 113 True Positive Rate 67.3% ** False positives #, means 1 alarm every 2.5 minutes 65% Detection, but too many false positives and too scattered Impractical Model

14 Models Decision Tree Entropy Probe Model Decision Tree - Enthropy Population 3,840,683 Number of Accidents 197 False Positives 323,082 False Positives Rate 8.4% False Negatives 90 False Negatives Rate 0.0% True Positives 107 True Positive Rate 54.3% ** False positives #, means 1 alarm every 2 minutes 54% Detection, but still more than $300 K false positives Impractical Model

15 Models Decision Tree ( Chi ) Detector Model Decision Tree - Chi Population 1,310,730 Number of Accidents 168 False Positives 151,383 False Positives Rate 11.5% False Negatives 69 False Negatives Rate 0.0% True Positives 99 True Positive Rate 58.9% ** False positives #, means 1 alarm every 4 minutes 59% Detection and affordable 11% of False alarms

16 Models Model Decision Tree - Chi Population 1,310,730 Number of Accidents 168 False Positives 151,383 False Positives Rate 11.5% False Negatives 69 False Negatives Rate 0.0% True Positives 99 True Positive Rate 58.9% Decision Tree ( Chi ) Detector Using High and Medium Risk (50%) Thresholds No Accident Accident Total % High Risk 424, ,108 10% Medium Risk 597, ,546 14% Low Risk 3,346, ,346,831 77% Total 4,368, ,369,485 Accidents Anticipated % Accidents Not Anticipated % 73% Detection but 24% of alarms ( 2 alarms every 5 minutes )

17 Models Decision Tree ( Entropy ) Detector Model Decision Tree - Enthropy Population 1,310,730 Number of Accidents 168 False Positives 105,307 False Positives Rate 8.0% False Negatives 81 False Negatives Rate 0.0% True Positives 87 True Positive Rate 51.8% ** False positives #, means 1 alarm every 6 minutes 52% Detection and affordable 8% of False alarms

18 Models Model Decision Tree - Enthropy Population 1,310,730 Number of Accidents 168 False Positives 105,307 False Positives Rate 8.0% False Negatives 81 False Negatives Rate 0.0% True Positives 87 True Positive Rate 51.8% Decision Tree ( Entropy ) Detector Using High and Medium Risk Thresholds No Accident Accident Total % High Risk 426, ,286 10% Medium Risk 283, ,631 6% Low Risk 3,658, ,658,568 84% Total 4,368, ,369,485 Accidents Anticipated % Accidents Not Anticipated % 66% Detection and manageable 16% of alarms ( 1 alarm / 3.6 minutes ) & 306 accidents in High Risk level. This is the proposed model

19 Models Decision Tree ( Chi ) Detector Heat Map examples on different days

20 Models Model Statistics Decision Tree ( Entropy ) Detector Main Predictors are based on Speed and the relative changes of Speed, Count and Occupancy, between the Segment and the segments Upstream and Downstream, within 10 and 15 minutes ago. Changes and differences between Upstream / Segment / Downstream create instability that favors accident conditions Most relevant Predictors Speed ^ 2 is the predictor that creates bigger separation of clusters. Limit is between below or above 60 Miles / hour. At high speeds ( above 60 Miles / hour ) If speed upstream has decreased more than 10.5 Miles/per hour since 10 minutes ago, probability of accident is 90%. Otherwise: If difference between segment and downstream in the Change of # of vehicles since 10 minutes ago is bigger that 57 Vehicles, probability of accident is above 90%. At low speeds ( below 60 Miles / hour ) If difference between segment and downstream in the Change in speed since 15 minutes ago, is bigger than 4.5 Miles / hour AND Difference in Count in vehicles on segment and downstream is significant, probability of accident is above 90%

21 Models - Conclusions Model Conclusions Detector Data produces similar or better results than Probe data with less segments Decision Tree works better than Logistic Regression Occupancy & Count provide additional accuracy than simple Speed data Heat Map levels: 90 < Prob = High Risk ( 3 ) 50 < Prob < =90 = Medium ( 2 ) Prob <= 50 Low 66% of accidents will fall into Medium and High Level areas Medium and High Level still will be around 16% occurrence Heat Map looks manageable in terms of warnings for operator, but requires validation at TOC More accuracy does not seem achievable with current existing data at TOCs

22 Conclusions How system will work Dynamic Heat Map to be used and integrated into Operators console: ATMS system receives predictor values from detectors ( speed, count, ) Algorithm calculates Risk Level Risk Level is displayed in TOC Operator console highway map: TOC implements Mitigation actions: Select CCTV camera and monitor area at risk while Level is Medium or Higher If possible modulate speed Other actions such as Service Patrol System will require regular calibration and retraining

23 Conclusions Next Steps Trial at TOC: Gather feedback on usability and effectiveness: Frequency of Medium and High levels Evaluate Disruption to Operations are alarms too frequent? Measure effect on Operations after implementation: Number of accidents detected and time to detect Response time Impact of accidents If number of alarms is affordable try more accurate model up to Operators acceptable threshold Calibrate and Train model regularly ( TBD)

24 Impact Challenges & Insights Working alone & not a Data guy & not a programmer Data preparation and cleansing Co-existence with day to day business Traffic conditions + Human behavior have a big % of Accidents cause. Real Impact at work TBD once trials have occurred at TOC. Estimated value TBD : ) Impact of the Program Tremendous!! Professional development Now I know what this is about. I have touched it! 100% of the 78 Smart City proposals included a mention to Data Analytics

25 Capstone Project Highway Crash Prediction Appendix Crash Risk Medium High

26 Models - Predictors Variable Description Segment Cnt Number of vehicles Current Occ Occupancy. Measure of % of time road is occupied. Sense of Density Current Spd Speed in Miles per hour Current UCnt Number of vehicles Upstream USpd Speed in Miles per hour Upstream DCnt Number of vehicles Downstream DOcc Occupancy. Measure of % of time road is occupied. Sense of Density Downstream DSpd Speed in Miles per hour Downstream Dvph Vehicles per hour Downstream dcnt_5 Diference of Number of vehicles for current measure versus 5 minutes before: Cnt - Cnt(lag5) Current dcnt_10 Diference of Number of vehicles for current measure versus 10 minutes before: Cnt - Cnt(lag10) Current dcnt_15 Diference of Number of vehicles for current measure versus 15 minutes before: Cnt - Cnt(lag15) Current docc_5 Diference of Occupancy for current measure versus 5 minutes before: Occ - Occ(lag5) Current docc_10 Diference of Occupancy for current measure versus 10 minutes before: Occ - Occ(lag10) Current docc_15 Diference of Occupancy for current measure versus 15 minutes before: Occ - Occ(lag15) Current dspd_5 Diference of Speed for current measure versus 5 minutes before: Spd - Spd(lag5) Current dspd_10 Diference of Speed for current measure versus 10 minutes before: Spd - Spd(lag10) Current dspd_15 Diference of Speed for current measure versus 15 minutes before: Spd - Spd(lag15) Current UdCnt_5 Diference of Number of vehicles for current measure versus 5 minutes before: Cnt - Cnt(lag5) Upstream UdCnt_10 Diference of Number of vehicles for current measure versus 10 minutes before: Cnt - Cnt(lag10) Upstream UdCnt_15 Diference of Number of vehicles for current measure versus 15 minutes before: Cnt - Cnt(lag15) Upstream UdOcc_5 Diference of Occupancy for current measure versus 5 minutes before: Occ - Occ(lag5) Upstream UdOcc_10 Diference of Occupancy for current measure versus 10 minutes before: Occ - Occ(lag10) Upstream UdOcc_15 Diference of Occupancy for current measure versus 15 minutes before: Occ - Occ(lag15) Upstream UdSpd_5 Diference of Speed for current measure versus 5 minutes before: Spd - Spd(lag5) Upstream UdSpd_10 Diference of Speed for current measure versus 10 minutes before: Spd - Spd(lag10) Upstream UdSpd_15 Diference of Speed for current measure versus 15 minutes before: Spd - Spd(lag15) Upstream DdCnt_5 Diference of Number of vehicles for current measure versus 5 minutes before: Cnt - Cnt(lag5) Downstream DdCnt_10 Diference of Number of vehicles for current measure versus 10 minutes before: Cnt - Cnt(lag10) Downstream DdCnt_15 Diference of Number of vehicles for current measure versus 15 minutes before: Cnt - Cnt(lag15) Downstream Variable Description Segment DdCnt_15 Diference of Number of vehicles for current measure versus 15 minutes before: Cnt - Cnt(lag15) Downstrea m DdOcc_5 Diference of Occupancy for current measure versus 5 minutes before: Occ - Occ(lag5) Downstrea m DdOcc_10 Diference of Occupancy for current measure versus 10 minutes before: Occ - Occ(lag10) Downstrea m DdOcc_15 Diference of Occupancy for current measure versus 15 minutes before: Occ - Occ(lag15) Downstrea m DdSpd_5 Diference of Speed for current measure versus 5 minutes before: Spd - Spd(lag5) Downstrea m DdSpd_10 Diference of Speed for current measure versus 10 minutes before: Spd - Spd(lag10) Downstrea m DdSpd_15 Diference of Speed for current measure versus 15 minutes before: Spd - Spd(lag15) Downstrea m difcntd Difference of Count between current site and downstream segment Current difoccd Difference of Occupancy between current site and downstream segment Current difspdd Difference of Speed between current site and downstream segment Current difcntu Difference of Count between current site and upstream segment Current difoccu Difference of Occupancy between current site and upstream segment Current difspdu Difference of Speed between current site and upstream segment Current difcntd_i Difference of Count between current site and downstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current difoccd_i Difference of Occupancy between current site and downstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current difspdd_i Difference of Speed between current site and downstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current difcntu_i Difference of Count between current site and upstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current difoccu_i Difference of Occupancy between current site and upstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current difspdu_i Difference of Speed between current site and upstream segment for 5, 10 and 15 minutes before (lag i. i = 5, 10 and 15) Current rain If raining = 1, else 0 All speed_sqr Interaction term = speed * speed Current speed_snow Interaction term = speed * snow Current fog if fog = 1, else 0 All snow if snow=1, else 0 All ice if ice=1, else 0 All HourlyPrecipitation hourly precipitation in mm/hour All twlight 0 if day condition, 1 if dawn, 2 if sunset, 3 if night All holiday 0 if normal day, 1 if Holiday, 2 if PreHoliday, 3 if After Holiday All day of week Day of the week from Monday to Sunday All Month Month if the year from 1 to 12 All

27 Models Predictors. Multicolinearity High Correlations between Lane data and Segment data. Lane data was dropped Almost Perfect Correlation between Vehicles per hour and Count. VPH was dropped

28 Models Interaction Terms Interaction terms were added for speed and snow and speed and fog

29 Models Quadratic terms Variance change vs Event Quadratic terms were added for Speed on segment, Upstream and Downstream

30 12:05 AM 12:30 AM 12:55 AM 1:20 AM 1:45 AM 2:10 AM 2:35 AM 3:00 AM 3:25 AM 3:50 AM 4:15 AM 4:40 AM 5:05 AM 5:30 AM 5:55 AM 6:20 AM 6:45 AM 7:10 AM 7:35 AM 8:00 AM 8:25 AM 8:50 AM 9:15 AM 9:40 AM 10:05 AM 10:30 AM 10:55 AM 11:20 AM 11:45 AM 12:10 PM 12:35 PM 1:00 PM 1:25 PM 1:50 PM 2:15 PM 2:40 PM 3:05 PM 3:30 PM 3:55 PM 4:20 PM 4:45 PM 5:10 PM 5:35 PM 6:00 PM 6:25 PM 6:50 PM 7:15 PM 7:40 PM 8:05 PM 8:30 PM 8:55 PM 9:20 PM 9:45 PM 10:10 PM 10:35 PM 11:00 PM 11:25 PM 11:50 PM Capstone Project Highway Crash Prediction Data Speed profiles 70 Speed vs Hour by Day of the Week Sunday Monday Tuesday Wednesday Thursday Friday Saturday Avg

31 12:05 AM 12:30 AM 12:55 AM 1:20 AM 1:45 AM 2:10 AM 2:35 AM 3:00 AM 3:25 AM 3:50 AM 4:15 AM 4:40 AM 5:05 AM 5:30 AM 5:55 AM 6:20 AM 6:45 AM 7:10 AM 7:35 AM 8:00 AM 8:25 AM 8:50 AM 9:15 AM 9:40 AM 10:05 AM 10:30 AM 10:55 AM 11:20 AM 11:45 AM 12:10 PM 12:35 PM 1:00 PM 1:25 PM 1:50 PM 2:15 PM 2:40 PM 3:05 PM 3:30 PM 3:55 PM 4:20 PM 4:45 PM 5:10 PM 5:35 PM 6:00 PM 6:25 PM 6:50 PM 7:15 PM 7:40 PM 8:05 PM 8:30 PM 8:55 PM 9:20 PM 9:45 PM 10:10 PM 10:35 PM 11:00 PM 11:25 PM 11:50 PM Capstone Project Highway Crash Prediction Data Speed profiles 70 Speed vs Hour by Type of Day N H PH AH Avg

32 Data Accidents vs Weather Distribution of accidents / month effect of rain Distribution of accidents / month effect of snow Distribution of accidents / month effect of fog

33 Data Accidents vs Hour, Day and Month Distribution of accidents vs hour of the day Distribution of accidents vs day of the week Distribution of accidents vs month

34 Models Probe Data. Models Summary Data Probe Data Probe RUS Proportion Validation Data RUS Proportion Validation Data Model Backwards Model Backwards Model Backwards Model Backwards Population 8,959,691 Population 3,840,683 Population 8,959,691 Population 3,840,683 Number of Accidents 449 Number of Accidents 197 Number of Accidents 449 Number of Accidents 197 False Positives 1,654,908 False Positives Rate 18.5% False Positives 709,088 False Positives Rate 18.5% False Positives 709,838 False Positives Rate 7.9% False Positives 303,639 False Positives Rate 7.9% False Negatives 184 False Negatives Rate 0.0% False Negatives 72 False Negatives Rate 0.0% False Negatives 238 False Negatives Rate 0.0% False Negatives 101 False Negatives Rate 0.0% True Positives 265 True Positive Rate 59.0% True Positives 125 True Positive Rate 63.5% True Positives 211 True Positive Rate 47.0% True Positives 96 True Positive Rate 48.7% Model Stepwise Model Stepwise Model Stepwise Model Stepwise Population 8,959,691 Population 3,840,683 Population 8,959,691 Population 3,840,683 Number of Accidents 449 Number of Accidents 197 Number of Accidents 449 Number of Accidents 197 False Positives 1,518,584 False Positives Rate 16.9% False Positives 650,146 False Positives Rate 16.9% False Positives 643,390 False Positives Rate 7.2% False Positives 275,513 False Positives Rate 7.2% False Negatives 188 False Negatives Rate 0.0% False Negatives 80 False Negatives Rate 0.0% False Negatives 251 False Negatives Rate 0.0% False Negatives 108 False Negatives Rate 0.0% True Positives 261 True Positive Rate 58.1% True Positives 117 True Positive Rate 59.4% True Positives 198 True Positive Rate 44.1% True Positives 89 True Positive Rate 45.2% Model Decision Tree - Enthropy Model Decision Tree - Enthropy Model Decision Tree - Enthropy Model Decision Tree - Enthropy Population 8,959,691 Population 3,840,683 Population 8,959,691 Population 3,840,683 Number of Accidents 449 Number of Accidents 197 Number of Accidents 449 Number of Accidents 197 False Positives 1,210,884 False Positives Rate 13.5% False Positives 519,099 False Positives Rate 13.5% False Positives 754,552 False Positives Rate 8.4% False Positives 323,082 False Positives Rate 8.4% False Negatives 206 False Negatives Rate 0.0% False Negatives 93 False Negatives Rate 0.0% False Negatives 204 False Negatives Rate 0.0% False Negatives 90 False Negatives Rate 0.0% True Positives 243 True Positive Rate 54.1% True Positives 104 True Positive Rate 52.8% True Positives 245 True Positive Rate 54.6% True Positives 107 True Positive Rate 54.3% Model Decision Tree - Gini Model Decision Tree - Gini Model Decision Tree - Gini Model Decision Tree - Gini Population 8,959,691 Population 3,840,683 Population 8,959,691 Population 3,840,683 Number of Accidents 449 Number of Accidents 197 Number of Accidents 449 Number of Accidents 197 False Positives 866,796 False Positives Rate 9.7% False Positives 371,066 False Positives Rate 9.7% False Positives 756,070 False Positives Rate 8.4% False Positives 323,450 False Positives Rate 8.4% False Negatives 197 False Negatives Rate 0.0% False Negatives 92 False Negatives Rate 0.0% False Negatives 202 False Negatives Rate 0.0% False Negatives 94 False Negatives Rate 0.0% True Positives 252 True Positive Rate 56.1% True Positives 105 True Positive Rate 53.3% True Positives 247 True Positive Rate 55.0% True Positives 103 True Positive Rate 52.3% Model Decision Tree - Chi Model Decision Tree - Chi Model Decision Tree - Chi Model Decision Tree - Chi Population 8,959,691 Population 3,840,683 Population 8,959,691 Population 3,840,683 Number of Accidents 449 Number of Accidents 197 Number of Accidents 449 Number of Accidents 197 False Positives 981,335 False Positives Rate 11.0% False Positives 420,590 False Positives Rate 11.0% False Positives 945,761 False Positives Rate 10.6% False Positives 405,103 False Positives Rate 10.5% False Negatives 195 False Negatives Rate 0.0% False Negatives 92 False Negatives Rate 0.0% False Negatives 194 False Negatives Rate 0.0% False Negatives 92 False Negatives Rate 0.0% True Positives 254 True Positive Rate 56.6% True Positives 105 True Positive Rate 53.3% True Positives 255 True Positive Rate 56.8% True Positives 105 True Positive Rate 53.3%

35 Models Detector Data. Models Summary Data Detector Data Detector RUS Proportion Validation Data RUS Proportion Validation Data Model Backwards Model Backwards Model Backwards Model Backwards Population 3,058,755 Population 1,310,730 Population 3,058,755 Population 1,310,730 Number of Accidents 379 Number of Accidents 168 Number of Accidents 379 Number of Accidents 168 False Positives 617,715 False Positives Rate 20.2% False Positives 264,976 False Positives Rate 20.2% False Positives 262,590 False Positives Rate 8.6% False Positives 112,997 False Positives Rate 8.6% False Negatives 127 False Negatives Rate 0.0% False Negatives 64 False Negatives Rate 0.0% False Negatives 176 False Negatives Rate 0.0% False Negatives 89 False Negatives Rate 0.0% True Positives 252 True Positive Rate 66.5% True Positives 104 True Positive Rate 61.9% True Positives 203 True Positive Rate 53.6% True Positives 79 True Positive Rate 47.0% Model Stepwise Model Stepwise Model Stepwise Model Stepwise Population 3,058,755 Population 1,310,730 Population 3,058,755 Population 1,310,730 Number of Accidents 379 Number of Accidents 168 Number of Accidents 379 Number of Accidents 168 False Positives 585,680 False Positives Rate 19.1% False Positives 251,269 False Positives Rate 19.2% False Positives 248,059 False Positives Rate 8.1% False Positives 106,666 False Positives Rate 8.1% False Negatives 132 False Negatives Rate 0.0% False Negatives 65 False Negatives Rate 0.0% False Negatives 176 False Negatives Rate 0.0% False Negatives 93 False Negatives Rate 0.0% True Positives 247 True Positive Rate 65.2% True Positives 103 True Positive Rate 61.3% True Positives 203 True Positive Rate 53.6% True Positives 75 True Positive Rate 44.6% Model Decision Tree - Enthropy Model Decision Tree - Enthropy Model Decision Tree - Enthropy Model Decision Tree - Enthropy Population 3,058,755 Population 1,310,730 Population 3,058,755 Population 1,310,730 Number of Accidents 379 Number of Accidents 168 Number of Accidents 379 Number of Accidents 168 False Positives 384,545 False Positives Rate 12.6% False Positives 163,876 False Positives Rate 12.5% False Positives 245,854 False Positives Rate 8.0% False Positives 105,307 False Positives Rate 8.0% False Negatives 132 False Negatives Rate 0.0% False Negatives 75 False Negatives Rate 0.0% False Negatives 159 False Negatives Rate 0.0% False Negatives 81 False Negatives Rate 0.0% True Positives 247 True Positive Rate 65.2% True Positives 93 True Positive Rate 55.4% True Positives 220 True Positive Rate 58.0% True Positives 87 True Positive Rate 51.8% Model Decision Tree - Gini Model Decision Tree - Gini Model Decision Tree - Gini Model Decision Tree - Gini Population 3,058,755 Population 1,310,730 Population 3,058,755 Population 1,310,730 Number of Accidents 379 Number of Accidents 168 Number of Accidents 379 Number of Accidents 168 False Positives 517,340 False Positives Rate 16.9% False Positives 221,816 False Positives Rate 16.9% False Positives 210,705 False Positives Rate 6.9% False Positives 90,537 False Positives Rate 6.9% False Negatives 113 False Negatives Rate 0.0% False Negatives 72 False Negatives Rate 0.0% False Negatives 156 False Negatives Rate 0.0% False Negatives 90 False Negatives Rate 0.0% True Positives 266 True Positive Rate 70.2% True Positives 96 True Positive Rate 57.1% True Positives 223 True Positive Rate 58.8% True Positives 78 True Positive Rate 46.4% Model Decision Tree - Chi Model Decision Tree - Chi Model Decision Tree - Chi Model Decision Tree - Chi Population 3,058,755 Population 1,310,730 Population 3,058,755 Population 1,310,730 Number of Accidents 379 Number of Accidents 168 Number of Accidents 379 Number of Accidents 168 False Positives 601,890 False Positives Rate 19.7% False Positives 256,890 False Positives Rate 19.6% False Positives 354,327 False Positives Rate 11.6% False Positives 151,383 False Positives Rate 11.5% False Negatives 93 False Negatives Rate 0.0% False Negatives 55 False Negatives Rate 0.0% False Negatives 135 False Negatives Rate 0.0% False Negatives 69 False Negatives Rate 0.0% True Positives 286 True Positive Rate 75.5% True Positives 113 True Positive Rate 67.3% True Positives 244 True Positive Rate 64.4% True Positives 99 True Positive Rate 58.9%

36 Models Model Statistics Logistic Regression ( Stepwise ) Detector 65-35

37 Models SAS Enterprise Miner: Model selection diagram for Probe Data with RUS proportion