AN INDEPENDENT VALIDATION OF QRISK ON THE THIN DATABASE

Transcription

1 AN INDEPENDENT VALIDATION OF QRISK ON THE THIN DATABASE Dr Gary S. Collins Professor Douglas G. Altman Centre for Statistics in Medicine University of Oxford

2 TABLE OF CONTENTS LIST OF TABLES... 3 LIST OF FIGURES SCOPE & PURPOSE OF DOCUMENT BACKGROUND PREPARING AND ANALYSING THIN SOFTWARE FOR INDEPENDENT VALIDATION REMOVING INELIGIBLE PATIENTS METHODS RESULTS FROM THE INDEPENDENT VALIDATION THIN characteristics Completeness of risk factors Baseline characteristics of eligible patients Calibration and discrimination statistics Distribution of high risk patients Observed and predicted risks Classification of patients at high risk and low risk Summary REVIEW OF THE UEA PAPER DATA FOR THE MAIN ANALYSIS Missing Data Family History Townsend scores ANALYSIS RESPONSE TO UEA CONCLUSIONS SUMMARY DOCUMENTATION & DATA SUPPLIED TO CSM REFERENCES Page 2 of 49

3 LIST OF TABLES Table 1: Summary of sequentially removing ineligible patients...6 Table 2: Summary of patients in Heart, UEA and CSM independent validation studies...11 Table 3: Completeness of recording for risk factors for men and women aged years initially free from CVD and diabetes in the Heart, UEA and CSM independent validation studies...12 Table 4: Baseline recorded clinical characteristics for men and women aged years initially free from CVD and diabetes in the Heart, UEA and CSM independent validation studies...13 Table 5: QRISK calibration and discrimination statistics for predicted 10-year risk of cardiovascular disease in the THIN cohort Table 6: Framingham calibration and discrimination statistics for predicted 10-year risk of cardiovascular disease in the THIN cohort...15 Table 7: Percentage of patients with CVD risk score 20% by sex and Townsend fifth...16 Table 8: Percentage of patients with CVD risk score 20% by sex and age...17 Table 9: Observed, QRISK and Framingham risk by sex and age group...18 Table 10: Ratio of predicted (QRISK) to observed risks of cardiovascular disease event at 10 years across tenths of estimated risk Table 11: Ratio of predicted (FRAMINGHAM) to observed risks of cardiovascular disease event at 10 years across tenths of estimated risk Table 12: patients with 20% risk according to QRISK and Framingham...21 Table 13: Reference data by age and sex based on QResearch (1995) for replacing missing values. 24 Table 14: Coefficients for the Framingham risk equation...24 Table 15: % of patients by age and sex with CVD risk score 20% from UEA paper, CSM independent validation and CSM reproduction of UEA analysis...32 Table 16: % of patients by Townsend fifth and sex with CVD risk score 20% from UEA paper, CSM independent validation and CSM reproduction of UEA analysis...33 Table 17: Agreement of models (original table from UEA paper) QRISK versus Framingham...34 Table 18: Additional diagnostic tests presented in the UEA paper (QRISK) - Women...35 Table 19: Additional diagnostic tests presented in the UEA paper (QRISK) - Men...36 Table 20: Additional diagnostic tests presented in the UEA paper (Framingham) - Women...37 Table 21: Additional diagnostic tests presented in the UEA paper (Framingham) - Men...37 Table 22: Model coefficients used in UEA analysis...45 Table 23: Mean values used in the UEA analysis...46 Page 3 of 49

4 LIST OF FIGURES Figure 1: Box-and-whisker plot of predicted risk (QRISK) by sex and age group...21 Figure 2: Predicted versus observed risk for QRISK and Framingham by QRISK tenth...22 Figure 3: Predicted versus observed risk for QRISK and Framingham by 5-year age bands...23 Figure 4: ROC curve by model, sex with cut-points...38 Figure 5: Sensitivity by sex and model Figure 6: Specificity by sex and model Figure 7: QRISK ROC curve with cutpoints by sex...41 Figure 8: Accuracy by risk model Figure 9: QRISK discrimination (Women)...43 Figure 10: QRISK discrimination (Men)...44 Page 4 of 49

5 1 SCOPE & PURPOSE OF DOCUMENT QRISK is a new cardiovascular disease risk prediction algorithm published in This document has two main objectives To report the findings of an independent validation i of the QRISK algorithm on the THIN database. To report the findings of an independent review of the unpublished Robins et al paper 1 and identify the reasons for discrepancies between that paper and the QRISK investigators own validation study described in a paper published in Heart 2. 2 BACKGROUND In July 2007, Hippisley-Cox et al published a paper in the BMJ describing the derivation and validation of QRISK, a new cardiovascular disease risk prediction algorithm 3. This was a novel risk prediction algorithm which includes traditional risk factors (age, sex, systolic blood pressure, smoking status, serum cholesterol/hdl ratio, presence of diabetes, ECG-LVH) included in the Framingham equation 4 but also includes body mass index, family history of cardiovascular disease, social deprivation and the use of blood pressure treatment. The QRISK algorithm then underwent a revision to account for statin use and changes to the multiple imputation method to account for missing data. This revised algorithm was then used in an second validation study designed to test the performance of QRISK in practices contributing to a second independent THIN dataset. This study was published in Heart 2 and the paper includes a comparison of the model performance statistics from the original QRESEARCH cohort and the THIN cohort. Subsequently, an unpublished paper by Robins et al from the University of East Anglia (UEA) was submitted to NICE in response to its consultation on the Lipid Modification guidelines, which included an independent evaluation of QRISK in the THIN database. The authors concluded that they could not replicate the results published in the Heart paper. That disagreement has highlighted the need for an independent evaluation of the QRISK algorithm in the THIN dataset to address the apparent discrepancies between these papers and to consider whether any of the issues raised in the Robins et al paper are substantial with regard to QRISK. We report here the findings of independent analyses conducted at the Centre for Statistics in Medicine (CSM), in Oxford. i From this point forward this will be referred to as the CSM independent validation. Page 5 of 49

6 3 PREPARING AND ANALYSING THIN 3.1 SOFTWARE FOR INDEPENDENT VALIDATION To enhance the credibility of the independent validation all analysis were performed in R statistical software (versions and 2.7.2), a different software package from STATA (version 10.1) which was used for the derivation of the QRISK 3, validation of the QRISK 2 and the UEA analysis REMOVING INELIGIBLE PATIENTS Due to memory constraints in R, STATA (IC 10.1) was used in the merging of the main data file (outputmrk3.csv) and the updated family history (FH_csv_records.csv). Table 1 reports the number of patients removed at each stage, in a sequential manner as per the Heart and UEA paper; that is patients with prior CVD were first removed followed by patients with invalid dates then patients aged <35 years or >74 years, then those with missing Townsend fifth data, followed by those with existing diabetes and finally those patients taking statins we removed. Sample characteristics of the final dataset once ineligible patients have been removed can be found in Section 4 (Table 2). We are able to reproduce the exact the number of patients excluded at each point in the sequential removal of patients as reported in the Heart paper. We are, however, unable to replicate the results in the UEA paper, the problem arises when removing patients with invalid dates for which we were unable to replicate (see 5.1 for details). Table 1: Summary of sequentially removing ineligible patients CSM Heart paper UEA paper independent validation All patients 1,787,169 1,787,169 1,787,169 Number of practices Prior CVD 120, , ,281 Invalid dates 2,253 2,581 2,253 Age < 35 years 284, , ,492 Age > 74 years 155, , ,248 Missing Townsend fifth 114, , ,123 Prior diabetes 28,148 28,034 28,148 Prior statins 9,824 9,819 9,824 Eligible patients 1,072,800 1,072,289 1,072, METHODS Predicted 10-year CVD risk for each patient was calculated using the QRISK algorithm and the Framingham equation using model coefficients from Table 14. Observed 10-year CVD risk for patients in the THIN cohort were obtained using 10-year Kaplan-Meier estimates. Confidence Page 6 of 49

7 intervals are presented at the 95% level. Confidence intervals for risk are calculated using the log-log transformation. Calibration measures how closely predicted 10-year CVD risk agrees with observed 10-year CVD risk. This was achieved by comparing each tenth of predicted risk for by calculating the proportion of predicted to observed CVD risk, separately for men and for women. In addition the ratio of predicted to observed 10-year CVD risk was calculated for each sex and overall, where a value of 1 is indicative of good agreement. The Brier score was also calculated which is a measure of accuracy and is the average squared deviation between predicted and observed risk, where a lower score represents higher accuracy. Discrimination is the ability of the risk prediction model to differentiate between patients who experience a CVD event during the study and those who do not. This is quantified by calculating the receiver operating characteristic (ROC) curve statistic; here a value of 1 represents perfect discrimination. In addition values of sensitivity and specificity are calculated for the cut-off predicted 10-year CVD risk of 20% and ROC curves are presented graphically. We also calculated the D statistic 5 and R 2 statistic 6 (which is derived from the D statistic) which are measures of discrimination and explained variation respectively and are specific to censored survival data. Higher values of D indicate improved discrimination where an increase of 0.1 over other risk prediction models is a good marker of improved prognostic separation. The proportion of patients who have a predicted 10-year CVD risk of 20% or more according to QRISK and Framingham were calculated by age, sex and Townsend fifth. 4 RESULTS FROM THE INDEPENDENT VALIDATION The following tables (Table 2 through to Table 8) correspond to Tables 1 to 6 published in the Heart paper with the addition of UEA results and our independent validation results presented alongside. 4.1 THIN characteristics Table 2 summarises characteristics of the THIN database after ineligible patients have been removed. We can confirm that the CSM independent validation analyses exactly reproduces the patient characteristics reported in the Heart paper. To summarise Table 2, there were 1,072,800 patents who met all eligibility criteria from 274 practices, of whom 529,813 were men (49.39%). The observed 10-year risk of a cardiovascular event was 6.55% (95% CI: 6.43 to 6.68) for women and 9.87% (95% CI: 9.71 to 10.03) for men. The UEA paper, whilst not exactly replicating the patient characteristics in the Heart paper, reported very similar but not identical characteristics. Page 7 of 49

8 4.2 Completeness of risk factors The completeness of recording for each risk factor is presented in Table 3 split by gender. We can confirm that the CSM independent validation and results presented in the Heart paper are in agreement. Complete data for all risk factors considered were available for 26.9% of women and 25.5% of men. There were markedly high levels of missing data for total serum cholesterol (59.1% women; 59.6% men) and HDL (70.6% women; 71.4% men) for both men and women. Completeness of risk factors reported in the UEA paper was slightly different. 4.3 Baseline characteristics of eligible patients Baseline characteristics of the cohort, split by gender, are presented in Table 4; we can confirm that the CSM independent validation and Heart paper are in agreement. Values reported in the UEA paper are again not dissimilar to either the Heart or the CSM independent validation. 4.4 Calibration and discrimination statistics Table 5 and Table 6 presents calibration and discrimination statistics for the Framingham and QRISK scores, and we can confirm that results from the CSM independent validation are in total agreement with those published in Heart. Results presented by the UEA team on calibration and discrimination were limited to ROC values which all differ slightly from those in the Heart paper and CSM validation. 4.5 Distribution of high risk patients Table 7 and Table 8 shows the distribution of patients at high risk (scores 20%) for QRISK and Framingham by Townsend fifth and sex and age band and sex respectively. We can confirm that we are in agreement with the corresponding tables published in the Heart validation study. We disagree with the analogous tables published in the UEA paper. Compared to the Heart paper or CSM validation the UEA study consistently showed many more patients at high risk as defined by QRISK, whereas for the Framingham score the UEA study showed more men but fewer women at high risk within deprivation categories. 4.6 Observed and predicted risks Table 9 shows the observed and predicted risks by age group for men and women separately and Figure 1 displays this information graphically. QRISK scores appear to closely match observed risk estimates when compared to the Framingham model. Predicted risks for both the QRISK and Framingham models appear better for women than men. The Framingham model appears to overestimate risk for men, whilst the QRISK underestimates risk though to a lesser extent. QRISK estimates of risk for women appear very good for all age bands when compared to Framingham, the Framingham equation overestimates risk for younger patients whilst underestimates risk for older patients (most notably those in the age bracket). Table 10 and Table 11 along with Figure 2 show the observed and predicted risks by tenths of predicted risk and Figure 3 by age group. These tables of observed and predicted risk broken down Page 8 of 49

9 tenth or age group were not presented in either the Heart paper or the UEA paper for the THIN cohort. For comparison, we present our results alongside risks that were published on the QRESEARCH cohort published in the derivation paper in the BMJ. We have included Figure 2 and Figure 3 are they clearly show the similarity between observed and expected risk for QRISK when compared to Framingham. A model with good predictions would lie along the diagonal line. When presented for men by 5-year age bands (Figure 3), Framingham clearly over-estimates risk compared to QRISK. For women, it can be seen that Framingham over-estimates risk for ages years, yet underestimates for women aged 65 years and over. Compared to Framingham, QRISK appears to provide improved estimates of 10-year CVD risk. 4.7 Classification of patients at high risk and low risk An important aspect when introducing a new risk prediction model is the classification of patients into high and low risk and the number of patients that would be reclassified to a different risk category using QRISK when compared to Framingham. Patients are classified as being at high risk if their predicted risk is 20% or more and at low risk if their predicted risk is less than 20%. The number of patients categorised as high and low risk using QRISK and Framingham (using a 20% cut-off threshold) are reported in Table 12. In total 75,557 patients (7%) would be classified as being at high risk when using the QRISK prediction model compared to 132,077 patients (12%) if using the Framingham model. Our CSM analyses show that overall 85,010 patients in the THIN cohort would be reclassified from high risk using the Framingham model to low risk using the QRISK model and vice-versa. We have 1 extra patient when compared to results presented in the Heart paper. We have identified this patient as having a Framingham score of in the CSM independent validation and in the Heart analysis. We can confidently assume this discrepancy is due to a difference in numerical precision between STATA and R and is only highlighted as this patient is on the cusp of high and low risk. In these reclassified patients 70,765 (83.24%) were reclassified as low-risk with an observed risk of 17.35% (95% CI: to 17.90), whilst the observed risk for those upgraded from low on Framingham to high on the QRISK was 23.70% (95% CI: to 25.03). Of the patients, were male (71.5%) and of these, men (95.2%) were reclassified from high risk on Framingham to low risk on QRISK with an observed risk of 17.50% (95% CI: to 18.13), whilst only 3548 (5.8%) were reclassified from low on Framingham to high on QRISK (observed risk of 25.47%, 95% CI: to 28.07). Of the 24,263 women that were reclassified, 13,566 (55.9%) were reclassified from high risk on Framingham to low risk on QRISK with an observed risk of 16.82% (95% CI: to 17.99), and conversely, the 10,697 (44.9%) patients reclassified as low risk on Framingham to high risk on QRISK had an observed risk of 23.11% (95% CI: to 24.66). Page 9 of 49

10 4.8 Summary Overall, results from our independent validation analyses have confirmed the results that were presented in the Heart paper. To summarise, we have exactly replicated the exclusion of patients and the final dataset which was used in the analysis of the Heart paper. Baseline clinical characteristics reported in our independent analysis agree exactly with those published in the Heart paper. We have independently and exactly replicated the discrimination and calibration statistics that were published in Heart. With the exception of a minor discrepancy of one patient in the exact number of patients being classified as high and low risk which we believe is due to a numerical precision difference between R and STATA, we have produced identical numbers of patients at high and low risk. A strength of our confirmation of the Heart paper is the use of a different statistical package used in conducting our analysis. We were unable to replicate the results presented in the UEA study. Page 10 of 49

11 Table 2: Summary of patients in Heart, UEA and CSM independent validation studies Heart paper (% of total) UEA paper (% of total) CSM independent validation (% of total) Total number of practices Total number of patients in cohort 1,072,800 1,072,289 1,072,800 Women 542,987 (50.61) 542,783 (50.62) 542,987 (50.61) Men 529,813 (49.39) 529,506 (49.38) 529,813 (49.39) Total person years of observation 5,357,218 5,356,566 5,357,218 Deprivation breakdown Patients in Townsend fifth 1 (most affluent) 296,929 (27.68) 296,715 (27.67) 296,929 (27.68) Patients in Townsend fifth 2 244,821 (22.82) 244,640 (22.81) 244,821 (22.82) Patients in Townsend fifth 3 220,362 (20.54) 220,238 (20.54) 220,362 (20.54) Patients in Townsend fifth 4 179,811 (16.76) 179,793 (16.77) 179,811 (16.76) Patients in Townsend fifth 5 (most deprived) 130,877 (12.20) 130,903 (12.21) 130,877 (12.20) Event Type Total patients with incident CVD events 44,152 44,375 44,152 Number with incident CHD events 30,797 (69.8) Not stated 30,797 (69.8) Number with incident stroke/tia 13,355 (30.2) Not stated 13,355 (30.2) 10-year risk of CVD events (95% CI) Women 6.55 (6.43 to 6.68) 6.56 (6.44 to 6.68) 6.55 (6.43 to 6.68) Men 9.87 (9.71 to 10.03) 9.84 (9.69 to 10.00) 9.87 (9.71 to 10.03) Page 11 of 49

12 Table 3: Completeness of recording for risk factors for men and women aged years initially free from CVD and diabetes in the Heart, UEA and CSM independent validation studies Risk factors Men (n=529,813) Heart paper UEA paper CSM independent validation Women (n=542,987) Men (n=528,734) Women (n=541,959) Men (n=529,813) Women (n=542,987) Age 529,813 (100) 542,987 (100) 528,734 (100) 541,959 (100) 529,813 (100) 542,987 (100) Sex 529,813 (100) 542,987 (100) 528,734 (100) 541,959 (100) 529,813 (100) 542,987 (100) Body mass index (kg/m 2 ) 391,703 (73.9) 451,155 (83.1) 390,922 (73.9) 450,258 (83.1) 391,703 (73.9) 451,155 (83.1) Systolic blood pressure (mm Hg) 457,217 (86.3) 511,423 (94.2) 456,034 (86.3) 510,274 (94.2) 457,217 (86.3) 511,423 (94.2) Total serum cholesterol (mmol/l) 213,793 (40.4) 221,869 (40.9) 213,189 (40.3) 221,139 (40.8) 213,793 (40.4) 221,869 (40.9) HDL (mmol/l) 151,782 (28.7) 159,690 (29.4) 151,387 (28.6) 159,135 (29.4) 151,782 (28.6) 159,690 (29.4) Smoking status 460,616 (86.9) 508,680 (93.7) 459,667 (86.9) 507,676 (93.7) 460,616 (86.9) 508,680 (93.7) Complete data for all risk factors 135,114 (25.5) 145,994 (26.9) 135,164 (25.6) 145,779 (26.9) 135,114 (25.5) 145,994 (26.9) Page 12 of 49

13 Table 4: Baseline recorded clinical characteristics for men and women aged years initially free from CVD and diabetes in the Heart, UEA and CSM independent validation studies Characteristics Heart paper UEA paper CSM independent validation Men Women Men Women Men Women Age at baseline, median (interquartile range) 48 (40 to 57) 49 (41 to 59) 48 (Not stated) 49 (Not stated) 48 (40 to 57) 49 (41 to 59) Body mass index (kg/m 2 ), mean (SD) 26.6 (4.0) 26.1 (4.9) 26.6 (4.0) 26.1 (4.9) 26.6 (4.0) 26.1 (4.9) Systolic blood pressure (mm Hg), mean (SD) (19.4) (21.0) (19.4) (21.0) (19.4) (21.0) Total serum cholesterol (mmol/l), mean (SD) 5.7 (1.1) 5.8 (1.2) 5.7 (1.1) 5.8 (1.2) 5.7 (1. 1) 5.8 (1.2) HDL (mmol/l), mean (SD) 1.3 (0.4) 1.6 (0.4) 1.3 (0.4) 1.6 (0.4) 1.3 (0.4) 1.6 (0.4) Total serum cholesterol/hdl ratio, mean (SD) 4.5 (1.3) 3.9 (1.2) 4.5 (1.3) 3.9 (1.2) 4.5 (1.3) 3.9 (1.2) Current smoker 141,113 (26.6) 124,094 (22.9) 141,026 (26.6) 124,047 (22.9) 141,113 (26.6) 124,094 (22.9) Family history of CHD in first-degree relative < 60 years 18,638 (3.5) 22,922 (4.2) 17,793 (3.4) 21,958 (4.0) 18,638 (3.5) 22,922 (4.2) Receiving antihypertensive treatment 35,066 (6.6) (10.5) 35,291 (6.7) 55,125 (10.2) 35,066 (6.6) 56,886 (10.5) Receiving ACE inhibitors 11,718 (2.2) 12,901 (2.4) 11,628 (2.2) 12,808 (2.4) 11,718 (2.2) 12,901 (2.4) Receiving β-blockers 16,700 (3.2) 27,554 (5.1) 16,549 (3.1) 27,283 (5.0) 16,700 (3.2) 27,554 (5.1) Receiving calcium channel blockers 9,847 (1.9) 11,147 (2.1) 9,761 (1.8) 11,035 (2.0) 9,847 (1.9) 11,147 (2.1) Receiving thiazides 10,630 (2.0) 23,391 (4.3) 10,532 (2.0) 5,586 (1.0) 10,630 (2.0) 23,391 (4.3) Page 13 of 49

14 Table 5: QRISK calibration and discrimination statistics for predicted 10-year risk of cardiovascular disease in the THIN cohort Statistic ROC statistic (95% CI) D statistic (95% CI) R 2 statistic (95% CI) Heart paper UEA paper CSM independent validation Men Women Men Women Men Women ( to ) (1.374 to 1.414) (31.09 to 32.31) ( to ) (1.532 to 1.580) (35.94 to 37.34) 0.75 (Not stated) Not stated Not stated 0.78 (Not stated) Not stated Not stated (1.375 to 1.414) (31.09 to 32.31) (1.533 to 1.580) (35.94 to 37.34) Brier score Not stated Not stated Predicted / observed Not stated Not stated Predicted / observed (overall) 0.88 Not stated 0.88 Page 14 of 49

15 Table 6: Framingham calibration and discrimination statistics for predicted 10-year risk of cardiovascular disease in the THIN cohort Statistic ROC statistic (95% CI) D statistic (95% CI) R 2 statistic (95% CI) Heart paper UEA paper CSM independent validation Men Women Men Women Men Women ( to ) (1.235 to 1.275) (26.69 to 27.93) ( to ) (1.354 to 1.402) (30.45 to 31.91) 0.75 (Not stated) Not stated Not stated 0.77 (Not stated) Not stated Not stated (1.235 to 1.274) (26.69 to 27.93) (1.354 to 1.401) (30.45 to 31.91) Brier score Not stated Not stated Predicted / observed Not stated Not stated Predicted / observed (overall) 1.23 Not stated 1.23 Page 15 of 49

16 Table 7: Percentage of patients with CVD risk score 20% by sex and Townsend fifth Townsend fifth Heart Paper UEA paper CSM Independent validation Framingham QRISK CVD Framingham QRISK CVD QRISK CVD Framingham (CHD + Stroke) Women First Second Third Fourth Fifth Men First Second Third Fourth Fifth Total Page 16 of 49

17 Table 8: Percentage of patients with CVD risk score 20% by sex and age Age band (years) Heart Paper UEA paper CSM Independent validation Framingham QRISK CVD Framingham QRISK CVD QRISK CVD Framingham (CHD + Stroke) Women All women Men All women All patients Page 17 of 49

18 Table 9: Observed, QRISK and Framingham risk by sex and age group Observed CVD risk at 10 years Mean predicted risk at 10 years (QRISK) Mean predicted risk at 10 years (Framingham) Women Men Page 18 of 49

19 Table 10: Ratio of predicted (QRISK) to observed risks of cardiovascular disease event at 10 years across tenths of estimated risk QRISK (QRESEARCH COHORT) CSM independent validation (THIN COHORT) Tenth Predicted risk Observed risk Ratio Predicted risk Observed risk Ratio Women First Second Third Fourth Fifth Sixth Seventh Eighth Ninth Tenth Overall Men First Second Third Fourth Fifth Sixth Seventh Eighth Ninth Tenth Overall All patients Page 19 of 49

20 Table 11: Ratio of predicted (FRAMINGHAM) to observed risks of cardiovascular disease event at 10 years across tenths of estimated risk QRISK (QRESEARCH COHORT) CSM independent validation (THIN COHORT) Tenth Predicted risk Observed risk Ratio Predicted risk Observed risk Ratio Women First Second Third Fourth Fifth Sixth Seventh Eighth Ninth Tenth Overall Men First Second Third Fourth Fifth Sixth Seventh Eighth Ninth Tenth Overall All patients Page 20 of 49

21 Figure 1: Box-and-whisker plot of predicted risk (QRISK) by sex and age group years years years years years years years years 60 predicted risk women men women men women men women men women men women men women men women men Table 12: patients with 20% risk according to QRISK and Framingham QRISK SCORE FRAMINGHAM SCORE Women Men Overall <20 20 Totals <20 20 Totals <20 20 Totals <20 506,478 13, , ,000 57, , ,478 70, , ,697 12,246 22,943 3,548 49,066 52,614 14,245 61,312 75,557 Totals 517,175 25, , , , , , ,077 1,072,800 Page 21 of 49

22 Figure 2: Predicted versus observed risk for QRISK and Framingham by QRISK tenth predicted (QRISK) QRISK Framingham male predicted (QRISK) QRISK Framingham female observed (kaplan-meier) observed (kaplan-meier) Page 22 of 49

23 Figure 3: Predicted versus observed risk for QRISK and Framingham by 5-year age bands predicted (QRISK) QRISK Framington male predicted (QRISK) QRISK Framington female observed (kaplan-meier) observed (kaplan-meier) Page 23 of 49

24 Table 13: Reference data by age and sex based on QResearch (1995) for replacing missing values Sex Age band Cholesterol / HDL ratio Systolic blood pressure Body mass index Female years Female years Female years Female years female years Female years Female years Female years Male years Male years Male years Male years Male years Male years Male years Male years Table 14: Coefficients for the Framingham risk equation Coefficients CHD Stroke (including TIA) θ θ β Female log(age) log(age) log(age) female log(age) 2 female log(sbp) Cigarettes (Y/N) log(tsc/hd) Diabetes Diabetes female ECG-LVH ECG-LVJ male - - Page 24 of 49

25 5 REVIEW OF THE UEA PAPER This section details findings from our independent review of the UEA paper. As much as possible, any discrepancies between the Heart analyses and UEA analyses will be highlighted and examined. 5.1 DATA FOR THE MAIN ANALYSIS The inclusion / exclusion criteria set out in the Heart paper were specified as 1. Include patients registered between 1 January 1995 and 31 March 2006 (inclusive) 2. Exclude patients with prior diagnosis of CVD 3. Exclude patients with invalid dates 4. Exclude patients under the age of 35 years 5. Exclude patients 75 years and over 6. Exclude patients with missing Townsend scores 7. Exclude patients with pre-existing diabetes 8. Exclude patients who were taking statins Exclusion point (3) invalid dates was not explicitly defined in either the original BMJ paper or the Heart paper. The unpublished QRISK STATA code supplied from the University of Nottingham defined invalid dates as those dates where the censoring outcome was later than the study entry date. The UEA paper defined invalid as clinical dates ± 15 years from the entry date. We are unable to reproduce the numbers quoted in the UEA paper as the definition of clinical dates is ambiguous. If clinical dates/values are related to dates when blood pressure, BMI, HDL and cholesterol ratio are measured then the number of invalid dates defined above are Exclusion point (8) was not explicitly defined in the Heart paper in terms of a time window. The QRISK user guide (page 7) defined this as follows: Patients were considered to be on a drug at baseline (i.e. entry to the study) if they had two or more prescriptions in total AND their first recorded date of prescription preceded the entry date AND the last recorded date was at least after 28 days after the entry date Missing Data As defined in the QRISK user manual: Prescription data for statins and hypertensive medication which had invalid dates were treated as missing values. That is, a prescription date occurring before the patient s date of birth or after the end of the study would be discounted. Page 25 of 49

26 It is unclear how this was addressed in the UEA paper, as the description of dealing with missing data is unclear. The UEA authors state they have followed the imputation model suggested by the QRISK authors but we are unable to confirm this. In the Heart paper, missing data were replaced using age-sex matched reference data which was listed in the QRISK user manual 7. The UEA group had no access to these values and replaced missing values with mean values from the THIN dataset. In addition, the Heart paper replaced missing cholesterol/hdl ratios using the age-sex reference data whilst the UEA paper replaced cholesterol and HDL values separately and then calculated the cholesterol/hdl ratio Family History In the Heart paper and this subsequent CSM independent validation, an additional family history dataset was used to check and improve the completeness of the family history data. This additional data was supplied to the QRESEARCH team by EPIC on 07 September 2007 and was included in the pack of material made available to CSM for our independent validation. It is unclear and probably unlikely from the UEA report that this dataset was indeed incorporated into their analysis. As noted in the QRISK user guide and confirmed in the CSM independent validation, this information increased the total number of patients with a family history of premature CHD by 2,572 from 63,103 to 65, Townsend scores The THIN database has Townsend scores coded categorically 1 to 5. The Heart validation substituted these categories by median values from fifths based on a national postcode table mapped to Townsend scores at output area. The UEA paper does include Townsend scores in their analysis but they do not state what was done with the Townsend categories and it is unclear what they actually did. The CSM independent validation used values supplied by the QRISK researchers (as per the Heart paper). 5.2 ANALYSIS To date the underlying QRISK algorithm has yet to be published in the public domain; it does not appear in either the derivation or validation papers 2 3. Published hazard ratios have been published in the BMJ paper relating to version 1.0 of the algorithm and updated hazard ratios for QRISK version 1.1 have been published online as a rapid response 8. In addition, mean values which are used in the algorithm were not included in either the BMJ or Heart paper. Values used in the UEA paper appear to have been incorrectly extracted from the baseline clinical characteristics of the derivation cohort presented in Table 2 of the BMJ paper (Table 23 of this document). Tables 8 and 9 of the UEA report show marked differences between their implementation of the QRISK algorithm and that published in Heart. These two tables report the percentage of patients that have predicted 10-year CVD risk 20% or more (high risk) by deprivation index and age group. The Page 26 of 49

27 UEA paper reported proportions of patients at high risk that are significantly higher than those published in Heart and our own independent analyses of magnitude roughly 2-5 times higher when presented by Townsend fifth and report suspiciously large differences when presented by age group. The UEA team did not have the relevant baseline survivor estimates at their disposal when implementing the QRISK algorithm as used in the Heart paper. Incorrectly specifying baseline survival rates can potentially lead to significant differences in predicted 10-year CVD risk estimates. However, we believe we have successfully replicated the UEA analysis to identify the reasons for the discrepancy (with a small degree of error). It is our opinion that the UEA authors have at the outset identified and implemented an incorrect model. We believe the UEA authors have assumed the QRISK model has the form k 10 year CVD RISK = S 0 exp β i (xi mi ) i =1 where S 0 is the baseline survivor function, β i are the model coefficients, x i are the patient s recorded value for parameter i and m i are the population mean scores for parameter i. Whilst both the UEA assumed model and the actual QRISK model are of Cox proportional hazards regression form, the QRISK model is not exactly as specified above, as fractional polynomial methods 9 were used in the model building process to examine non-linear risk relations as described in the BMJ paper 3. We are under strict agreement not to reproduce the QRISK algorithm in any form in this current document. The misspecification of the QRISK model at the outset will have had a significant impact on subsequent analyses presented in the UEA paper, which we shall highlight and discuss. Regarding the Framingham model, the UEA authors remark on the similarity of Framingham scores between their implementation and that of the Heart paper. We feel this should not be surprising as the full Framingham model with coefficient values has been published and so is publicly available. We believe the slight differences between the UEA and Heart Framingham results are attributable to the subtle differences in interpretation of patient eligibility and hence the resulting slightly different patient datasets used in the Heart and UEA analyses. Table 15 shows the proportion of patients at high risk by age band separately by sex. Our replication of the UEA analysis is labelled CSM (UEA approach) which we believe is close enough to the UEA results for us to assume we have correctly specified the model the UEA team have assumed. We have included the CSM (UEA approach) purely to illustrate where we believe the UEA authors have made their error early in their analyses. The final column in the table, which is the column of interest, shows the results from the CSM independent validation, using the correct QRISK model which, as we have shown in Section 3.3 are in agreement with the results presented in the Heart paper. The results of the UEA analysis and our subsequent replication of their analysis clearly Page 27 of 49

28 highlight the overestimation of the number of patients at high risk as a result of incorrectly specifying the risk model. A similar pattern of results can be in Table 16, which shows the proportion of patients at high risk by Townsend fifth and sex. The UEA analysis clearly overestimates the risk of a CVD event, which is pronounced in women. Table 17 shows the agreement of classification of high and low risk patients by the QRISK and Framingham models. These results were not presented in the Heart paper. However, so we can address issues raised in the UEA paper, we have conducted analyses using the risk model we assumed the UEA team have used and the correct version of QRISK and our results are presented alongside the results presented in the UEA paper and complement Table 12. As an incorrect model was used in the UEA analyses, we shall not discuss their results nor our implementation of their approach, but we will summarise results from implementing the correct QRISK model, so as to avoid confusing the reader. Rows labelled High-High represent those patients who have predicted risk of 20% or more on both QRISK and Framingham. Similarly, rows labelled Low-Low represent those patients who have predicted risk less than 20% on QRISK and Framingham. Rows labelled High- Low are those patients who have a predicted risk of 20% or more on QRISK but less than 20% on Framingham. Rows labelled Low-High are those who have a predicted risk less than 20% on QRISK but a risk of 20% of more on Framingham. Overall, both QRISK and Framingham agree in the classification of 518,724 women (95.54%). This breaks down as QRISK and Framingham agreeing on the classification of 506,478 women (93.28%) being at low risk and 12,246 women (2.26%) being at high risk 2.26%. However, QRISK and Framingham disagree on the classification of 24,263 women (4.47%) with 10,697 women being classified at high risk using QRISK (observed risk of 23.11%, 95% CI: to 24.61) and low risk with Framingham, and 13,566 women classified as high risk with Framingham and low risk with QRISK (observed risk of 16.82%, 95% CI: to 17.95). For men, both models agree on the classification of 469,066 men (88.53%) as being at high or low risk. This breaks down as both models agreeing on the classification of 420,000 men (79.27%) as being at low risk, and 49,066 men (9.26%) being at high risk. The Framingham model classified a significantly larger number (n=57,199) of men as being at high risk (10.80%) with observed risk of 17.50% (95% CI: to 18.12). The QRISK model classified 3,548 men as being at high risk whilst the Framingham model classified these patients as low risk (observed risk of 25.47%, 95% CI: to 27.93). Due to the UEA team incorrectly specifying the model, the agreement of models tables (described above for the correct analysis) are incorrect in the UEA report. In addition, the UEA authors have incorrectly interpreted some of the results from their equivalent tables of agreement of models. For example, the authors incorrectly concluded that the models agree on only just over 4% of all high risk Page 28 of 49

29 predictions, whereas, the correct interpretation is that the models agree in classifying just over 4% of the sample as being at high risk. Table 18 and Table 19 correspond to Tables 11 and 14 of the UEA report for QRISK and cover additional aspects of discrimination not reported in the Heart paper but deemed by the UEA team to be of interest. The incorrect specification of the QRISK model by the UEA authors means that the original tables in the UEA paper are meaningless. We have replicated their analysis using the model we believe they specified for consistency and in addition we have repeated the analysis they indeed wanted to show using the correct QRISK model for comparison and to rectify their conclusions. Table 20 and Table 21 are the corresponding tables for the Framingham model. We shall not discuss the analogous results presented in the UEA paper as they are incorrect. However, so their additional analyses are addressed, we shall provide a brief summary which refers to our Table 18 through to Table 21. For high risk patients as defined by the QRISK model, the sensitivities for women and men are 0.18 and 0.30 respectively compared to 0.17 and 0.48 for the Framingham model. QRISK values of specificity are 0.96 and 0.91 for women and men respectively, compared to 0.96 and 0.81 for the Framingham model. Figure 4 through to Figure 8 are supplementary plots showing ROC curves by model, sensitivity, specificity, QRISK ROC curves by sex and model accuracy. In addition, Figure 9 and Figure 10 show the distribution of QRISK scores for those patients experiencing a CVD event and those who do not. We can observe from Figure 9 and Figure 10 that as expected those experiencing a CVD event generally score higher than those who do not. For both men and women, of those not experiencing a CVD event the majority, as expected have a low 10-year CVD risk and those who experience a CVD event have higher predicted CVD risk compared to those who do not. 5.3 RESPONSE TO UEA CONCLUSIONS The UEA authors have listed a number of conclusions from their analysis (shown below in italic text) which we will address in turn. The dataset used was a good match for the dataset used in the validation study in Heart. Most of the small discrepancies can be due to vague definition in selection criteria. CSM response: We agree with the authors comment. Such a small discrepancy will not have an important effect on any subsequent analysis especially given the size of the dataset. Page 29 of 49

30 The dataset has some data quality problems, notably missing data, untimely clinical values and limited time at risk. CSM response: We agree there are considerable amounts of missing data for completeness of recording total serum cholesterol and HDL. Regarding limited time at risk, median is ~5 years, however the sample size involvement is large, 36,483 patients have over 10 years of data and 191, 784 (18% have 8 years or more of data) and we feel this is unlikely to cause any major problems. The clinical values in the data have a bias towards the unhealthy range to imputation based on existing clinical values as proposed by the QRISK team for validation of models may be inadequate. CSM response: We are unclear about their argument here. Our QRISK scores do not always agree with those published in the validation study. CSM response: This is due to the UEA authors specifying an incorrect model for QRISK. According to the calibration assessment for the imputed dataset, all models over-estimate, and QRISK shows the worst fit particularly for women. All models estimate worse for the higher age/higher incidence groups. CSM response: This statement is incorrect because it is based on results from an analysis which has implemented an incorrect QRISK model. Section 3.1 examines this is issue in detail and confirms the Heart analysis. The QRISK scores obtained for the high risk groups (predicted risk over 20% in a 10-year period) do not agree with the scores we obtain, and the disagreements are sometimes considerable. CSM response: This statement is incorrect because it is based on results from an analysis which has implemented an incorrect QRISK model. Section 3.1 examines this is issue in detail and confirms the Heart analysis. For our discrimination test results, ASSIGN appears to give better results. CSM response: The ASSIGN model uses Scottish Index of Multiple Deprivation which is not recorded in THIN. In addition, ASSIGN uses number of cigarettes smoked per day, which again is not recorded Page 30 of 49

31 in the THIN database, only whether the patient is a smoker or not. Thus we are sceptical to as whether UEA team were indeed able to accurately calculate the ASSIGN score. Furthermore, as previously mentioned, the UEA version of the QRISK model is also incorrect, leading this conclusion to be incorrect. Looking at the agreement between models, ASSIGN and QRISK agree more often and Framingham (CHD+S). When QRISK and ASSIGN disagree the K-M incidence suggest that ASSIGN is providing better classification. CSM response: This statement is incorrect because it is based on results from an analysis which has implemented an incorrect QRISK model. Section 3.1 examines this is issue in detail and confirms the Heart analysis. Page 31 of 49

32 Table 15: % of patients by age and sex with CVD risk score 20% from UEA paper, CSM independent validation and CSM reproduction of UEA analysis Age band (years) QRISK UEA paper ii Framingham (CHD + Stroke) CSM iii (UEA approach) CSM Independent validation iv QRISK QRISK Framingham Women All women Men All men All patients ii As presented in the UEA paper iii CSM analysis implementing the model assumed by the UEA team iv CSM analysis using the correct QRISK model Page 32 of 49

33 Table 16: % of patients by Townsend fifth and sex with CVD risk score 20% from UEA paper, CSM independent validation and CSM reproduction of UEA analysis Townsend fifth QRISK UEA paper Framingham (CHD + Stroke) CSM (UEA approach) CSM Independent validation QRISK QRISK Framingham Women First Second Third Fourth Fifth Men First Second Third Fourth Fifth Total Page 33 of 49

34 Table 17: Agreement of models (original table from UEA paper) QRISK versus Framingham % Records UEA paper CSM (reproduction of UEA analysis) CSM independent validation Crude Incidence % K-M Incidence % % Records Crude Incidence % K-M Incidence % % Records Crude Incidence % K-M Incidence % Women Low-Low High-High High-Low Low-High Men Low-Low High-High High-Low Low-High Page 34 of 49

35 Table 18: Additional diagnostic tests presented in the UEA paper (QRISK) - Women CSM UEA paper Women (reproduction of UEA analysis) CSM independent validation Event No event Total Event No event Total Event No event Total QRISK 20% QRISK < 20% Total Sensitivity (95%CI) 0.46 (0.45 to 0.47) 0.47 (0.46 to 0.48) 0.18 (0.18 to 0.19) Specificity (95%CI) 0.86 (0.85 to 0.86) 0.85 (0.85 to 0.85) 0.96 (0.96 to 0.96) Positive likelihood ratio v (95%CI) 3.20 (3.14 to 3.25) 3.10 (3.05 to 3.15) 4.92 (4.76 to 5.09) Negative likelihood ratio vi (95%CI) 0.63 (0.62 to 0.64) 0.62 (0.61 to 0.63) 0.85 (0.84 to 0.85) Diagnostic odds ratio (95%CI) 5.08 (4.93 to 5.23) 4.98 (4.83 to 5.14) 5.81 (5.58 to 6.05) Overall misclassification rate (%) vii v Positive likelihood ratio provides an estimate of the odds of favouring CHD being present given QRISK score 20% [=sensitivity/(1-specificity)] vi Negative likelihood ratio provides an estimate of the odds of favouring CHD being absent given QRISK score < 20% [=(1-sensitivity)/specificity] vii Not presented in the UEA paper Page 35 of 49