Cross-validated AUC in Stata: CVAUROC
|
|
|
- Dorcas Shaw
- 5 years ago
- Views:
Transcription
1 Cross-validated AUC in Stata: CVAUROC Miguel Angel Luque Fernandez Biomedical Research Institute of Granada Noncommunicable Disease and Cancer Epidemiology Spanish Stata Conference 24 October 2018 MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
2 Contents 1 Cross-validation 2 Cross-validation justification 3 Cross-validation methods 4 cvauroc 5 References MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
3 Cross-validation Definition Cross-validation is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice (note: performance = model assessment). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
4 Cross-validation Definition Cross-validation is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice (note: performance = model assessment). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
5 Cross-validation Applications However, cross-validation can be used to compare the performance of different modeling specifications (i.e. models with and without interactions, inclusion of exclusion of polynomial terms, number of knots with restricted cubic splines, etc). Furthermore, cross-validation can be used in variable selection and select the suitable level of flexibility in the model (note: flexibility = model selection). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
6 Cross-validation Applications However, cross-validation can be used to compare the performance of different modeling specifications (i.e. models with and without interactions, inclusion of exclusion of polynomial terms, number of knots with restricted cubic splines, etc). Furthermore, cross-validation can be used in variable selection and select the suitable level of flexibility in the model (note: flexibility = model selection). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
7 Cross-validation Applications MODEL ASSESSMENT: To compare the performance of different modeling specifications. MODEL SELECTION: To select the suitable level of flexibility in the model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
8 Cross-validation Applications MODEL ASSESSMENT: To compare the performance of different modeling specifications. MODEL SELECTION: To select the suitable level of flexibility in the model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
9 MSE Regression Model f (x) = f (x 1 + x 2 + x 3 ) Y = βx 1 + βx 2 + βx 3 + ɛ MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
10 MSE Regression Model f (x) = f (x 1 + x 2 + x 3 ) Y = βx 1 + βx 2 + βx 3 + ɛ Y = f (x) + ɛ MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
11 MSE Regression Model f (x) = f (x 1 + x 2 + x 3 ) Y = βx 1 + βx 2 + βx 3 + ɛ Y = f (x) + ɛ MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
12 MSE Expectation E(Y X 1 = x 1, X 2 = x 2, X 3 = x 3 ) MSE E[(Y ˆf (X)) 2 X = x] MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
13 MSE Expectation E(Y X 1 = x 1, X 2 = x 2, X 3 = x 3 ) MSE E[(Y ˆf (X)) 2 X = x] MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
14 Bias-Variance Trade-off Error descomposition MSE = E[(Y ˆf (X)) 2 X = x] = Var(ˆf (x 0 )) + [Bias(ˆf (x 0 ))] 2 + Var(ɛ) Trade-off As flexibility of ˆf increases, its variance increases, and its bias decreases. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
15 BIAS-VARIANCE-TRADE-OFF Bias-variance trade-off Choosing the model flexibility based on average test error Average Test Error E[(Y ˆf (X)) 2 X = x] And thus, this amounts to a bias-variance trade-off. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
16 BIAS-VARIANCE-TRADE-OFF Bias-variance trade-off Choosing the model flexibility based on average test error Average Test Error E[(Y ˆf (X)) 2 X = x] And thus, this amounts to a bias-variance trade-off. Rule More flexibility increases variance but decreases bias. Less flexibility decreases variance but increases error. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
17 BIAS-VARIANCE-TRADE-OFF Bias-variance trade-off Choosing the model flexibility based on average test error Average Test Error E[(Y ˆf (X)) 2 X = x] And thus, this amounts to a bias-variance trade-off. Rule More flexibility increases variance but decreases bias. Less flexibility decreases variance but increases error. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
18 Bias-Variance trade-off Regression Function MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
19 Overparameterization George E.P.Box,( ) Quote, 1976 All models are wrong but some are useful Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration (...). Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
20 Overparameterization George E.P.Box,( ) Quote, 1976 All models are wrong but some are useful Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration (...). Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
21 Justification AIC and BIC AIC and BIC are both maximum likelihood estimate driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior. AIC = -2*ln(likelihood) + 2*k, k = model degrees of freedom MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
22 Justification AIC and BIC AIC and BIC are both maximum likelihood estimate driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior. AIC = -2*ln(likelihood) + 2*k, k = model degrees of freedom BIC = -2*ln(likelihood) + ln(n)*k, k = model degrees of freedom and N = number of observations. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
23 Justification AIC and BIC AIC and BIC are both maximum likelihood estimate driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior. AIC = -2*ln(likelihood) + 2*k, k = model degrees of freedom BIC = -2*ln(likelihood) + ln(n)*k, k = model degrees of freedom and N = number of observations. There is some disagreement over the use of AIC and BIC with non-nested models. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
24 Justification AIC and BIC AIC and BIC are both maximum likelihood estimate driven and penalize free parameters in an effort to combat overfitting, they do so in ways that result in significantly different behavior. AIC = -2*ln(likelihood) + 2*k, k = model degrees of freedom BIC = -2*ln(likelihood) + ln(n)*k, k = model degrees of freedom and N = number of observations. There is some disagreement over the use of AIC and BIC with non-nested models. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
25 Justification Fewer assumptions Cross-validation compared with AIC, BIC and adjusted R 2 provides a direct estimate of the ERROR. Cross-validation makes fewer assumptions about the true underlying model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
26 Justification Fewer assumptions Cross-validation compared with AIC, BIC and adjusted R 2 provides a direct estimate of the ERROR. Cross-validation makes fewer assumptions about the true underlying model. Cross-validation can be used in a wider range of model selections tasks, even in cases where it is hard to pinpoint the number of predictors in the model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
27 Justification Fewer assumptions Cross-validation compared with AIC, BIC and adjusted R 2 provides a direct estimate of the ERROR. Cross-validation makes fewer assumptions about the true underlying model. Cross-validation can be used in a wider range of model selections tasks, even in cases where it is hard to pinpoint the number of predictors in the model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
28 Cross-validation strategies Cross-validation options Leave-one-out cross-validation (LOOCV). k-fold cross validation. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
29 Cross-validation strategies Cross-validation options Leave-one-out cross-validation (LOOCV). k-fold cross validation. Bootstraping. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
30 Cross-validation strategies Cross-validation options Leave-one-out cross-validation (LOOCV). k-fold cross validation. Bootstraping. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
31 K-fold Cross-validation K-fold Technique widely used for estimating the test error. Estimates can be used to select the best model, and to give an idea of the test error of the final chosen model. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
32 K-fold Cross-validation K-fold Technique widely used for estimating the test error. Estimates can be used to select the best model, and to give an idea of the test error of the final chosen model. The idea is to randmoly divide the data into k equal-sized parts. We leave out part k, fit the model to the other k-1 parts (combined), and then obtain predictions for the left-out kth part. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
33 K-fold Cross-validation K-fold Technique widely used for estimating the test error. Estimates can be used to select the best model, and to give an idea of the test error of the final chosen model. The idea is to randmoly divide the data into k equal-sized parts. We leave out part k, fit the model to the other k-1 parts (combined), and then obtain predictions for the left-out kth part. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
34 K-fold Cross-validation K-fold MSE k CV = k k 1 = n k n MSE k i C k (y i (ŷ i ))/n k Seeting K = n yields n-fold or leave-one-out cross-validation (LOOCV) MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
35 K-fold Cross-validation K-fold MSE k CV = k k 1 = n k n MSE k i C k (y i (ŷ i ))/n k Seeting K = n yields n-fold or leave-one-out cross-validation (LOOCV) MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
36 Model performance: Internal Validation (AUC) AUC The AUC is a global summary measure of a diagnostic test accuracy and discrimination. The greater the AUC, the more able is the test to capture the trade-off between Se and Sp over a continuous range. An important aspect of predictive modeling is the ability of a model to generalize to new cases. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
37 Model performance: Internal Validation (AUC) AUC The AUC is a global summary measure of a diagnostic test accuracy and discrimination. The greater the AUC, the more able is the test to capture the trade-off between Se and Sp over a continuous range. An important aspect of predictive modeling is the ability of a model to generalize to new cases. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
38 Model performance: Internal Validation (AUC) AUC The AUC is a global summary measure of a diagnostic test accuracy and discrimination. The greater the AUC, the more able is the test to capture the trade-off between Se and Sp over a continuous range. An important aspect of predictive modeling is the ability of a model to generalize to new cases. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
39 Predictive performance: internal validation AUC Evaluating the predictive performance (AUC) of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. K-fold cross-validation can be used to generate a more realistic estimate of predictive performance when the number of observations is not very large (Ledell, 2015). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
40 Predictive performance: internal validation AUC Evaluating the predictive performance (AUC) of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. K-fold cross-validation can be used to generate a more realistic estimate of predictive performance when the number of observations is not very large (Ledell, 2015). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
41 Predictive performance: internal validation AUC Evaluating the predictive performance (AUC) of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. K-fold cross-validation can be used to generate a more realistic estimate of predictive performance when the number of observations is not very large (Ledell, 2015). MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
42 CVAUROC cvauroc cvauroc implements k-fold cross-validation for the AUC for a binary outcome after fitting a logistic regression model and provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named fit. GitHub cvauroc development version MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
43 CVAUROC cvauroc cvauroc implements k-fold cross-validation for the AUC for a binary outcome after fitting a logistic regression model and provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named fit. GitHub cvauroc development version Stata ssc ssc install cvauroc MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
44 CVAUROC cvauroc cvauroc implements k-fold cross-validation for the AUC for a binary outcome after fitting a logistic regression model and provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named fit. GitHub cvauroc development version Stata ssc ssc install cvauroc MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
45 CVAUROC cvauroc cvauroc implements k-fold cross-validation for the AUC for a binary outcome after fitting a logistic regression model and provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named fit. GitHub cvauroc development version Stata ssc ssc install cvauroc MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
46 cvauroc Stata command syntax cvauroc Syntax cvauroc depvar varlist [if] [pw] [Kfold] [Seed] [, Cluster(varname) Detail Graph] MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
47 Classical AUC estimation. use gen lbw = cond(bweight<2500,1,0.). logistic lbw mage medu mmarried prenatal fedu mbsmoke mrace order Logistic regression Number of obs = 4,642 lbw Odds Ratio Std. Err. z P> z [95% Conf. Interval] -+ mage medu mmarried prenatal fedu mbsmoke mrace order predict fitted, pr. roctab lbw fitted ROC -Asymptotic Normal Obs Area Std. Err. [95% Conf. Interval] 4, MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
48 Crossvalidated AUC using cvauroc. cvauroc lbw mage medu mmarried prenatal fedu mbsmoke mrace order, kfold(10) seed(12) 1-fold... 2-fold... 3-fold... 4-fold... 5-fold... 6-fold... 7-fold... 8-fold... 9-fold fold... Random seed: 12 ROC -Asymptotic Normal Obs Area Std. Err. [95% Conf. Interval] 4, MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
49 cvauroc detail and graph options // Using detail option to show the table of cutoff values and their respective Se, S // and likelihood ratio values.. cvauroc lbw mage medu mmarried prenatal1 fedu mbsmoke mrace fbaby, kfold(10) seed(3489) detail Detailed report of sensitivity and specificity Correctly Cutpoint Sensitivity Specificity Classified LR+ LR- ( >=.019 ) % 0.00% 6.01% ( >=.025 ) 99.64% 0.18% 6.16% ( >=.026 ) 99.64% 0.39% 6.36% (...) Omitted results ( >=.272 ) 1.08% 99.93% 93.99% ( >=.273 ) 0.72% 99.93% 93.97% ( >=.300 ) 0.36% 99.95% 93.97% // Using the "graph" option to display the ROC curve. cvauroc lbw mage medu mmarried prenatal1 fedu mbsmoke mrace fbaby, kfold(10) seed(3489) graph. graph export "your_path/figure1.eps", as(eps) preview(off) MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
50 cvauroc: Cross-validated AUC Sensitivity Specificity MA Luque Fernandez Area (ibs.granada) under ROC curve Cross-validated = AUC in Stata: CVAUROC 24 October / 28
51 Conclusion cvauroc Evaluating the predictive performance of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. However, cvauroc is user-friendly and helpful k-fold internal cross-validation technique that might be considered when reporting the AUC in observational studies. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
52 Conclusion cvauroc Evaluating the predictive performance of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. However, cvauroc is user-friendly and helpful k-fold internal cross-validation technique that might be considered when reporting the AUC in observational studies. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
53 Conclusion cvauroc Evaluating the predictive performance of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. However, cvauroc is user-friendly and helpful k-fold internal cross-validation technique that might be considered when reporting the AUC in observational studies. MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
54 References MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
55 References MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
56 Thank you THANK YOU FOR YOUR TIME MA Luque Fernandez (ibs.granada) Cross-validated AUC in Stata: CVAUROC 24 October / 28
Cross-validation. Miguel Angel Luque Fernandez Faculty of Epidemiology and Population Health Department of Non-communicable Diseases.
Cross-validation Miguel Angel Luque Fernandez Faculty of Epidemiology and Population Health Department of Non-communicable Diseases. October 29, 2015 Cancer Survival Group (LSH&TM) Cross-validation October
Cross-validation. Miguel Angel Luque Fernandez Faculty of Epidemiology and Population Health Department of Non-communicable Disease.
Cross-validation Miguel Angel Luque Fernandez Faculty of Epidemiology and Population Health Department of Non-communicable Disease. August 25, 2015 Cancer Survival Group (LSH&TM) Cross-validation August
Notes for laboratory session 2
Notes for laboratory session 2 Preliminaries Consider the ordinary least-squares (OLS) regression of alcohol (alcohol) and plasma retinol (retplasm). We do this with STATA as follows:. reg retplasm alcohol
Selection and Combination of Markers for Prediction
Selection and Combination of Markers for Prediction NACC Data and Methods Meeting September, 2010 Baojiang Chen, PhD Sarah Monsell, MS Xiao-Hua Andrew Zhou, PhD Overview 1. Research motivation 2. Describe
Sociology Exam 3 Answer Key [Draft] May 9, 201 3
![Sociology Exam 3 Answer Key [Draft] May 9, 201 3](/thumbs/96/127470588.jpg "Sociology Exam 3 Answer Key [Draft] May 9, 201 3")
Sociology 63993 Exam 3 Answer Key [Draft] May 9, 201 3 I. True-False. (20 points) Indicate whether the following statements are true or false. If false, briefly explain why. 1. Bivariate regressions are
Biostatistics II
Biostatistics II 514-5509 Course Description: Modern multivariable statistical analysis based on the concept of generalized linear models. Includes linear, logistic, and Poisson regression, survival analysis,
Syntax Menu Description Options Remarks and examples Stored results Methods and formulas References Also see
Title stata.com teffects ra Regression adjustment Syntax Syntax Menu Description Options Remarks and examples Stored results Methods and formulas References Also see teffects ra (ovar omvarlist [, omodel
Multivariate dose-response meta-analysis: an update on glst
Multivariate dose-response meta-analysis: an update on glst Nicola Orsini Unit of Biostatistics Unit of Nutritional Epidemiology Institute of Environmental Medicine Karolinska Institutet http://www.imm.ki.se/biostatistics/
1 Introduction. st0020. The Stata Journal (2002) 2, Number 3, pp
The Stata Journal (22) 2, Number 3, pp. 28 289 Comparative assessment of three common algorithms for estimating the variance of the area under the nonparametric receiver operating characteristic curve
UNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Midterm, 2016
UNIVERSITY of PENNSYLVANIA CIS 520: Machine Learning Midterm, 2016 Exam policy: This exam allows one one-page, two-sided cheat sheet; No other materials. Time: 80 minutes. Be sure to write your name and
Applying Machine Learning Methods in Medical Research Studies
Applying Machine Learning Methods in Medical Research Studies Daniel Stahl Department of Biostatistics and Health Informatics Psychiatry, Psychology & Neuroscience (IoPPN), King s College London daniel.r.stahl@kcl.ac.uk
Outline. Introduction GitHub and installation Worked example Stata wishes Discussion. mrrobust: a Stata package for MR-Egger regression type analyses
mrrobust: a Stata package for MR-Egger regression type analyses London Stata User Group Meeting 2017 8 th September 2017 Tom Palmer Wesley Spiller Neil Davies Outline Introduction GitHub and installation
m 11 m.1 > m 12 m.2 risk for smokers risk for nonsmokers
SOCY5061 RELATIVE RISKS, RELATIVE ODDS, LOGISTIC REGRESSION RELATIVE RISKS: Suppose we are interested in the association between lung cancer and smoking. Consider the following table for the whole population:
ROC Curves. I wrote, from SAS, the relevant data to a plain text file which I imported to SPSS. The ROC analysis was conducted this way:
ROC Curves We developed a method to make diagnoses of anxiety using criteria provided by Phillip. Would it also be possible to make such diagnoses based on a much more simple scheme, a simple cutoff point
Age (continuous) Gender (0=Male, 1=Female) SES (1=Low, 2=Medium, 3=High) Prior Victimization (0= Not Victimized, 1=Victimized)
Criminal Justice Doctoral Comprehensive Exam Statistics August 2016 There are two questions on this exam. Be sure to answer both questions in the 3 and half hours to complete this exam. Read the instructions
Least likely observations in regression models for categorical outcomes
The Stata Journal (2002) 2, Number 3, pp. 296 300 Least likely observations in regression models for categorical outcomes Jeremy Freese University of Wisconsin Madison Abstract. This article presents a
Assessing Studies Based on Multiple Regression. Chapter 7. Michael Ash CPPA
Assessing Studies Based on Multiple Regression Chapter 7 Michael Ash CPPA Assessing Regression Studies p.1/20 Course notes Last time External Validity Internal Validity Omitted Variable Bias Misspecified
Sensitivity, Specificity and Predictive Value [adapted from Altman and Bland BMJ.com]
![Sensitivity, Specificity and Predictive Value [adapted from Altman and Bland BMJ.com]](/thumbs/86/93672915.jpg "Sensitivity, Specificity and Predictive Value [adapted from Altman and Bland BMJ.com]")
Sensitivity, Specificity and Predictive Value [adapted from Altman and Bland BMJ.com] The simplest diagnostic test is one where the results of an investigation, such as an x ray examination or biopsy,
RISK PREDICTION MODEL: PENALIZED REGRESSIONS
RISK PREDICTION MODEL: PENALIZED REGRESSIONS Inspired from: How to develop a more accurate risk prediction model when there are few events Menelaos Pavlou, Gareth Ambler, Shaun R Seaman, Oliver Guttmann,
Up-to-date survival estimates from prognostic models using temporal recalibration
Up-to-date survival estimates from prognostic models using temporal recalibration Sarah Booth 1 Mark J. Rutherford 1 Paul C. Lambert 1,2 1 Biostatistics Research Group, Department of Health Sciences, University
Daniel Boduszek University of Huddersfield
Daniel Boduszek University of Huddersfield d.boduszek@hud.ac.uk Introduction to Logistic Regression SPSS procedure of LR Interpretation of SPSS output Presenting results from LR Logistic regression is
Multiple Linear Regression Analysis
Revised July 2018 Multiple Linear Regression Analysis This set of notes shows how to use Stata in multiple regression analysis. It assumes that you have set Stata up on your computer (see the Getting Started
MIDAS RETOUCH REGARDING DIAGNOSTIC ACCURACY META-ANALYSIS
MIDAS RETOUCH REGARDING DIAGNOSTIC ACCURACY META-ANALYSIS Ben A. Dwamena, MD University of Michigan Radiology and VA Nuclear Medicine, Ann Arbor 2014 Stata Conference, Boston, MA - July 31, 2014 Dwamena
Knowledge Discovery and Data Mining. Testing. Performance Measures. Notes. Lecture 15 - ROC, AUC & Lift. Tom Kelsey. Notes
Knowledge Discovery and Data Mining Lecture 15 - ROC, AUC & Lift Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-17-AUC
3. Model evaluation & selection
Foundations of Machine Learning CentraleSupélec Fall 2016 3. Model evaluation & selection Chloé-Agathe Azencot Centre for Computational Biology, Mines ParisTech chloe-agathe.azencott@mines-paristech.fr
Statistical reports Regression, 2010
Statistical reports Regression, 2010 Niels Richard Hansen June 10, 2010 This document gives some guidelines on how to write a report on a statistical analysis. The document is organized into sections that
1. Objective: analyzing CD4 counts data using GEE marginal model and random effects model. Demonstrate the analysis using SAS and STATA.
LDA lab Feb, 6 th, 2002 1 1. Objective: analyzing CD4 counts data using GEE marginal model and random effects model. Demonstrate the analysis using SAS and STATA. 2. Scientific question: estimate the average
Daniel Boduszek University of Huddersfield
Daniel Boduszek University of Huddersfield d.boduszek@hud.ac.uk Introduction to Multinominal Logistic Regression SPSS procedure of MLR Example based on prison data Interpretation of SPSS output Presenting
Overview. All-cause mortality for males with colon cancer and Finnish population. Relative survival
An overview and some recent advances in statistical methods for population-based cancer survival analysis: relative survival, cure models, and flexible parametric models Paul W Dickman 1 Paul C Lambert
Chapter 17 Sensitivity Analysis and Model Validation
Chapter 17 Sensitivity Analysis and Model Validation Justin D. Salciccioli, Yves Crutain, Matthieu Komorowski and Dominic C. Marshall Learning Objectives Appreciate that all models possess inherent limitations
Estimating average treatment effects from observational data using teffects
Estimating average treatment effects from observational data using teffects David M. Drukker Director of Econometrics Stata 2013 Nordic and Baltic Stata Users Group meeting Karolinska Institutet September
BMI 541/699 Lecture 16
BMI 541/699 Lecture 16 Where we are: 1. Introduction and Experimental Design 2. Exploratory Data Analysis 3. Probability 4. T-based methods for continous variables 5. Proportions & contingency tables -
MS&E 226: Small Data
MS&E 226: Small Data Lecture 10: Introduction to inference (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 17 What is inference? 2 / 17 Where did our data come from? Recall our sample is: Y, the vector
ROC Curve. Brawijaya Professional Statistical Analysis BPSA MALANG Jl. Kertoasri 66 Malang (0341)
ROC Curve Brawijaya Professional Statistical Analysis BPSA MALANG Jl. Kertoasri 66 Malang (0341) 580342 ROC Curve The ROC Curve procedure provides a useful way to evaluate the performance of classification
Classification. Methods Course: Gene Expression Data Analysis -Day Five. Rainer Spang
Classification Methods Course: Gene Expression Data Analysis -Day Five Rainer Spang Ms. Smith DNA Chip of Ms. Smith Expression profile of Ms. Smith Ms. Smith 30.000 properties of Ms. Smith The expression
Mantel-Haenszel Procedures for Detecting Differential Item Functioning
A Comparison of Logistic Regression and Mantel-Haenszel Procedures for Detecting Differential Item Functioning H. Jane Rogers, Teachers College, Columbia University Hariharan Swaminathan, University of
STATISTICAL METHODS FOR DIAGNOSTIC TESTING: AN ILLUSTRATION USING A NEW METHOD FOR CANCER DETECTION XIN SUN. PhD, Kansas State University, 2012
STATISTICAL METHODS FOR DIAGNOSTIC TESTING: AN ILLUSTRATION USING A NEW METHOD FOR CANCER DETECTION by XIN SUN PhD, Kansas State University, 2012 A THESIS Submitted in partial fulfillment of the requirements
< N=248 N=296
Supplemental Digital Content, Table 1. Occurrence intraoperative hypotension (IOH) using four different thresholds of the mean arterial pressure (MAP) to define IOH, stratified for different categories
validscale: A Stata module to validate subjective measurement scales using Classical Test Theory
: A Stata module to validate subjective measurement scales using Classical Test Theory Bastien Perrot, Emmanuelle Bataille, Jean-Benoit Hardouin UMR INSERM U1246 - SPHERE "methods in Patient-centered outcomes
Pattern of Comorbidities among Colorectal Cancer Patients in Spain
Pattern of Comorbidities among Colorectal Cancer Patients in Spain Miguel Angel Luque-Fernandez, Daniel Redondo-Sánchez, Miguel Rodriguez-Barranco, M Carmen Carmona-Garcia, Rafael Marcos-Gragera, Maria
Estimating and Modelling the Proportion Cured of Disease in Population Based Cancer Studies
Estimating and Modelling the Proportion Cured of Disease in Population Based Cancer Studies Paul C Lambert Centre for Biostatistics and Genetic Epidemiology, University of Leicester, UK 12th UK Stata Users
Immortal Time Bias and Issues in Critical Care. Dave Glidden May 24, 2007
Immortal Time Bias and Issues in Critical Care Dave Glidden May 24, 2007 Critical Care Acute Respiratory Distress Syndrome Patients on ventilator Patients may recover or die Outcome: Ventilation/Death
Statistics, Probability and Diagnostic Medicine
Statistics, Probability and Diagnostic Medicine Jennifer Le-Rademacher, PhD Sponsored by the Clinical and Translational Science Institute (CTSI) and the Department of Population Health / Division of Biostatistics
What is Regularization? Example by Sean Owen
What is Regularization? Example by Sean Owen What is Regularization? Name3 Species Size Threat Bo snake small friendly Miley dog small friendly Fifi cat small enemy Muffy cat small friendly Rufus dog large
EPI 200C Final, June 4 th, 2009 This exam includes 24 questions.
Greenland/Arah, Epi 200C Sp 2000 1 of 6 EPI 200C Final, June 4 th, 2009 This exam includes 24 questions. INSTRUCTIONS: Write all answers on the answer sheets supplied; PRINT YOUR NAME and STUDENT ID NUMBER
Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism
Levy et al. Molecular Autism (2017) 8:65 DOI 10.1186/s13229-017-0180-6 RESEARCH Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism Sebastien
Analysis of TB prevalence surveys
Workshop and training course on TB prevalence surveys with a focus on field operations Analysis of TB prevalence surveys Day 8 Thursday, 4 August 2011 Phnom Penh Babis Sismanidis with acknowledgements
Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers
Tutorial in Biostatistics Received 21 November 2012, Accepted 17 July 2013 Published online 23 August 2013 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.5941 Graphical assessment of
Math 215, Lab 7: 5/23/2007
Math 215, Lab 7: 5/23/2007 (1) Parametric versus Nonparamteric Bootstrap. Parametric Bootstrap: (Davison and Hinkley, 1997) The data below are 12 times between failures of airconditioning equipment in
Assessment of a disease screener by hierarchical all-subset selection using area under the receiver operating characteristic curves
Research Article Received 8 June 2010, Accepted 15 February 2011 Published online 15 April 2011 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.4246 Assessment of a disease screener by
4. Model evaluation & selection
Foundations of Machine Learning CentraleSupélec Fall 2017 4. Model evaluation & selection Chloé-Agathe Azencot Centre for Computational Biology, Mines ParisTech chloe-agathe.azencott@mines-paristech.fr
DISEASE-SPECIFIC REFERENCE EQUATIONS FOR LUNG FUNCTION IN PATIENTS WITH CYSTIC FIBROSIS
DISEASE-SPECIFIC REFERENCE EQUATIONS FOR LUNG FUNCTION IN PATIENTS WITH CYSTIC FIBROSIS Michal Kulich, PhD, Margaret Rosenfeld, MD, MPH, Jonathan Campbell, MS, Richard Kronmal, PhD, Ron L. Gibson, MD,
Sensitivity, specicity, ROC
Sensitivity, specicity, ROC Thomas Alexander Gerds Department of Biostatistics, University of Copenhagen 1 / 53 Epilog: disease prevalence The prevalence is the proportion of cases in the population today.
Generalized Mixed Linear Models Practical 2
Generalized Mixed Linear Models Practical 2 Dankmar Böhning December 3, 2014 Prevalence of upper respiratory tract infection The data below are taken from a survey on the prevalence of upper respiratory
Regression Discontinuity Analysis
Regression Discontinuity Analysis A researcher wants to determine whether tutoring underachieving middle school students improves their math grades. Another wonders whether providing financial aid to low-income
Quantifying the added value of new biomarkers: how and how not
Cook Diagnostic and Prognostic Research (2018) 2:14 https://doi.org/10.1186/s41512-018-0037-2 Diagnostic and Prognostic Research COMMENTARY Quantifying the added value of new biomarkers: how and how not
Sarwar Islam Mozumder 1, Mark J Rutherford 1 & Paul C Lambert 1, Stata London Users Group Meeting
2016 Stata London Users Group Meeting stpm2cr: A Stata module for direct likelihood inference on the cause-specific cumulative incidence function within the flexible parametric modelling framework Sarwar
Name: emergency please discuss this with the exam proctor. 6. Vanderbilt s academic honor code applies.
Name: Biostatistics 1 st year Comprehensive Examination: Applied in-class exam May 28 th, 2015: 9am to 1pm Instructions: 1. There are seven questions and 12 pages. 2. Read each question carefully. Answer
Modeling unobserved heterogeneity in Stata
Modeling unobserved heterogeneity in Stata Rafal Raciborski StataCorp LLC November 27, 2017 Rafal Raciborski (StataCorp) Modeling unobserved heterogeneity November 27, 2017 1 / 59 Plan of the talk Concepts
Evidence-Based Assessment: From simple clinical judgments to statistical learning
Evidence-Based Assessment: From simple clinical judgments to statistical learning Eric A. Youngstrom, PhD University of North Carolina at Chapel Hill Looking at clinical decision-making from many angles
Final Exam. Thursday the 23rd of March, Question Total Points Score Number Possible Received Total 100
General Comments: Final Exam Thursday the 23rd of March, 2017 Name: This exam is closed book. However, you may use four pages, front and back, of notes and formulas. Write your answers on the exam sheets.
HOW TO BE A BAYESIAN IN SAS: MODEL SELECTION UNCERTAINTY IN PROC LOGISTIC AND PROC GENMOD
HOW TO BE A BAYESIAN IN SAS: MODEL SELECTION UNCERTAINTY IN PROC LOGISTIC AND PROC GENMOD Ernest S. Shtatland, Sara Moore, Inna Dashevsky, Irina Miroshnik, Emily Cain, Mary B. Barton Harvard Medical School,
4 Diagnostic Tests and Measures of Agreement
4 Diagnostic Tests and Measures of Agreement Diagnostic tests may be used for diagnosis of disease or for screening purposes. Some tests are more effective than others, so we need to be able to measure
Before we get started:
Before we get started: http://arievaluation.org/projects-3/ AEA 2018 R-Commander 1 Antonio Olmos Kai Schramm Priyalathta Govindasamy Antonio.Olmos@du.edu AntonioOlmos@aumhc.org AEA 2018 R-Commander 2 Plan
Today. HW 1: due February 4, pm. Matched case-control studies
Today HW 1: due February 4, 11.59 pm. Matched case-control studies In the News: High water mark: the rise in sea levels may be accelerating Economist, Jan 17 Big Data for Health Policy 3:30-4:30 222 College
Introduction to regression
Introduction to regression Regression describes how one variable (response) depends on another variable (explanatory variable). Response variable: variable of interest, measures the outcome of a study
Small Sample Bayesian Factor Analysis. PhUSE 2014 Paper SP03 Dirk Heerwegh
Small Sample Bayesian Factor Analysis PhUSE 2014 Paper SP03 Dirk Heerwegh Overview Factor analysis Maximum likelihood Bayes Simulation Studies Design Results Conclusions Factor Analysis (FA) Explain correlation
Final Exam - section 2. Thursday, December hours, 30 minutes
Econometrics, ECON312 San Francisco State University Michael Bar Fall 2011 Final Exam - section 2 Thursday, December 15 2 hours, 30 minutes Name: Instructions 1. This is closed book, closed notes exam.
Logistic Regression and Bayesian Approaches in Modeling Acceptance of Male Circumcision in Pune, India
20th International Congress on Modelling and Simulation, Adelaide, Australia, 1 6 December 2013 www.mssanz.org.au/modsim2013 Logistic Regression and Bayesian Approaches in Modeling Acceptance of Male Circumcision
NOVEL BIOMARKERS PERSPECTIVES ON DISCOVERY AND DEVELOPMENT. Winton Gibbons Consulting
NOVEL BIOMARKERS PERSPECTIVES ON DISCOVERY AND DEVELOPMENT Winton Gibbons Consulting www.wintongibbons.com NOVEL BIOMARKERS PERSPECTIVES ON DISCOVERY AND DEVELOPMENT This whitepaper is decidedly focused
Implementing double-robust estimators of causal effects
The Stata Journal (2008) 8, Number 3, pp. 334 353 Implementing double-robust estimators of causal effects Richard Emsley Biostatistics, Health Methodology Research Group The University of Manchester, UK
Learning from data when all models are wrong
Learning from data when all models are wrong Peter Grünwald CWI / Leiden Menu Two Pictures 1. Introduction 2. Learning when Models are Seriously Wrong Joint work with John Langford, Tim van Erven, Steven
Assessment of performance and decision curve analysis
Assessment of performance and decision curve analysis Ewout Steyerberg, Andrew Vickers Dept of Public Health, Erasmus MC, Rotterdam, the Netherlands Dept of Epidemiology and Biostatistics, Memorial Sloan-Kettering
Cover Page. The handle holds various files of this Leiden University dissertation
Cover Page The handle http://hdl.handle.net/1887/28958 holds various files of this Leiden University dissertation Author: Keurentjes, Johan Christiaan Title: Predictors of clinical outcome in total hip
Diagnostic Test of Fat Location Indices and BMI for Detecting Markers of Metabolic Syndrome in Children
Diagnostic Test of Fat Location Indices and BMI for Detecting Markers of Metabolic Syndrome in Children Adegboye ARA; Andersen LB; Froberg K; Heitmann BL Postdoctoral researcher, Copenhagen, Denmark Research
! Mainly going to ignore issues of correlation among tests
2x2 and Stratum Specific Likelihood Ratio Approaches to Interpreting Diagnostic Tests. How Different Are They? Henry Glick and Seema Sonnad University of Pennsylvania Society for Medical Decision Making
ethnicity recording in primary care
ethnicity recording in primary care Multiple imputation of missing data in ethnicity recording using The Health Improvement Network database Tra Pham 1 PhD Supervisors: Dr Irene Petersen 1, Prof James
MODEL SELECTION STRATEGIES. Tony Panzarella
MODEL SELECTION STRATEGIES Tony Panzarella Lab Course March 20, 2014 2 Preamble Although focus will be on time-to-event data the same principles apply to other outcome data Lab Course March 20, 2014 3
Item Analysis: Classical and Beyond
Item Analysis: Classical and Beyond SCROLLA Symposium Measurement Theory and Item Analysis Modified for EPE/EDP 711 by Kelly Bradley on January 8, 2013 Why is item analysis relevant? Item analysis provides
HIV risk associated with injection drug use in Houston, Texas 2009: A Latent Class Analysis
HIV risk associated with injection drug use in Houston, Texas 2009: A Latent Class Analysis Syed WB Noor Michael Ross Dejian Lai Jan Risser UTSPH, Houston Texas Presenter Disclosures Syed WB Noor No relationships
Predicting Breast Cancer Survival Using Treatment and Patient Factors
Predicting Breast Cancer Survival Using Treatment and Patient Factors William Chen wchen808@stanford.edu Henry Wang hwang9@stanford.edu 1. Introduction Breast cancer is the leading type of cancer in women
METHODS FOR DETECTING CERVICAL CANCER
Chapter III METHODS FOR DETECTING CERVICAL CANCER 3.1 INTRODUCTION The successful detection of cervical cancer in a variety of tissues has been reported by many researchers and baseline figures for the
The Pretest! Pretest! Pretest! Assignment (Example 2)
The Pretest! Pretest! Pretest! Assignment (Example 2) May 19, 2003 1 Statement of Purpose and Description of Pretest Procedure When one designs a Math 10 exam one hopes to measure whether a student s ability
Index. Springer International Publishing Switzerland 2017 T.J. Cleophas, A.H. Zwinderman, Modern Meta-Analysis, DOI /
Index A Adjusted Heterogeneity without Overdispersion, 63 Agenda-driven bias, 40 Agenda-Driven Meta-Analyses, 306 307 Alternative Methods for diagnostic meta-analyses, 133 Antihypertensive effect of potassium,
This tutorial presentation is prepared by. Mohammad Ehsanul Karim
STATA: The Red tutorial STATA: The Red tutorial This tutorial presentation is prepared by Mohammad Ehsanul Karim ehsan.karim@gmail.com STATA: The Red tutorial This tutorial presentation is prepared by
Strategies for handling missing data in randomised trials
Strategies for handling missing data in randomised trials NIHR statistical meeting London, 13th February 2012 Ian White MRC Biostatistics Unit, Cambridge, UK Plan 1. Why do missing data matter? 2. Popular
CSE 258 Lecture 2. Web Mining and Recommender Systems. Supervised learning Regression
CSE 258 Lecture 2 Web Mining and Recommender Systems Supervised learning Regression Supervised versus unsupervised learning Learning approaches attempt to model data in order to solve a problem Unsupervised
An application of the new irt command in Stata
An application of the new irt command in Stata Giola Santoni 2015 Nordic and Baltic Stata Users Group meeting, Stockholm September 4, 2015 Aging Research Center (ARC), Department of Neurobiology, Health
Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Chapter 10 Analysing and Presenting Results Petra Macaskill, Constantine Gatsonis, Jonathan Deeks, Roger Harbord, Yemisi Takwoingi.
Benchmark Dose Modeling Cancer Models. Allen Davis, MSPH Jeff Gift, Ph.D. Jay Zhao, Ph.D. National Center for Environmental Assessment, U.S.
Benchmark Dose Modeling Cancer Models Allen Davis, MSPH Jeff Gift, Ph.D. Jay Zhao, Ph.D. National Center for Environmental Assessment, U.S. EPA Disclaimer The views expressed in this presentation are those
Mark J. Anderson, Patrick J. Whitcomb Stat-Ease, Inc., Minneapolis, MN USA
Journal of Statistical Science and Application (014) 85-9 D DAV I D PUBLISHING Practical Aspects for Designing Statistically Optimal Experiments Mark J. Anderson, Patrick J. Whitcomb Stat-Ease, Inc., Minneapolis,
Model evaluation using grouped or individual data
Psychonomic Bulletin & Review 8, (4), 692-712 doi:.378/pbr..692 Model evaluation using grouped or individual data ANDREW L. COHEN University of Massachusetts, Amherst, Massachusetts AND ADAM N. SANBORN
Regression Output: Table 5 (Random Effects OLS) Random-effects GLS regression Number of obs = 1806 Group variable (i): subject Number of groups = 70
Regression Output: Table 5 (Random Effects OLS) Random-effects GLS regression Number of obs = 1806 R-sq: within = 0.1498 Obs per group: min = 18 between = 0.0205 avg = 25.8 overall = 0.0935 max = 28 Random
Multi-state survival analysis in Stata
Multi-state survival analysis in Stata Michael J. Crowther Biostatistics Research Group Department of Health Sciences University of Leicester, UK michael.crowther@le.ac.uk Italian Stata Users Group Meeting
Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models, 2nd Ed.
Eric Vittinghoff, David V. Glidden, Stephen C. Shiboski, and Charles E. McCulloch Division of Biostatistics Department of Epidemiology and Biostatistics University of California, San Francisco Regression
Discrimination and Reclassification in Statistics and Study Design AACC/ASN 30 th Beckman Conference
Discrimination and Reclassification in Statistics and Study Design AACC/ASN 30 th Beckman Conference Michael J. Pencina, PhD Duke Clinical Research Institute Duke University Department of Biostatistics
Classical Psychophysical Methods (cont.)
Classical Psychophysical Methods (cont.) 1 Outline Method of Adjustment Method of Limits Method of Constant Stimuli Probit Analysis 2 Method of Constant Stimuli A set of equally spaced levels of the stimulus
Quantifying cancer patient survival; extensions and applications of cure models and life expectancy estimation
From the Department of Medical Epidemiology and Biostatistics Karolinska Institutet, Stockholm, Sweden Quantifying cancer patient survival; extensions and applications of cure models and life expectancy