MODEL SELECTION STRATEGIES. Tony Panzarella

Transcription

1 MODEL SELECTION STRATEGIES Tony Panzarella

2 Lab Course March 20, Preamble Although focus will be on time-to-event data the same principles apply to other outcome data

3 Lab Course March 20, Developing a multivariable prediction model Select clinically relevant predictors for possible inclusion in the model Evaluate the quality of the data and how to handle missing data Data handling decisions Choosing a strategy for selecting the important variables in the final model Deciding how to model continuous variables Selecting measures of model performance or predictive accuracy

4 AUTOMATIC SELECTION ROUTINES

5 Lab Course March 20, Forward Selection Variables are added to the model one at a time At each stage the variable added is the one which gives the largest decrease in the value of -2LogL on its inclusion The process ends when each of the remaining variables fails to reduce -2LogL by a pre-specified amount (typically couched as a significance level)

6 Lab Course March 20, Backward elimination Full model is fit first Variables are excluded one at a time At each stage the variable omitted is the one that increases -2LogL by the smallest amount by its exclusion The process ends when the next candidate for deletion increases the value of -2LogL by more than a pre-specified amount.

7 Lab Course March 20, Stepwise Operates similarly to forward selection However, a variable that is included can be considered for exclusion at a later stage Thus after adding a variable, the procedure then checks whether any previously included variable can be deleted

8 Lab Course March 20, Best Subsets Provides a computational efficient way to screen all possible models The procedure requires a criterion to judge a model Given the criterion the software screens all models containing q covariates and reports the covariates in the best, say n, models for q=1,2,3,,p, where p denotes the number of covariates SAS uses the score test proc phreg data=myeloma; model Time*VStatus(0)=LogBUN HGB Platelet Age LogWBC Frac LogPBM Protein SCalc / selection=score best=3; run;

9 Lab Course March 20, The PHREG Procedure Regression Models Selected by Score Criterion Number of Score Variables Chi-Square Variables Included in Model LogBUN HGB Platelet LogBUN HGB LogBUN Platelet LogBUN SCalc LogBUN HGB SCalc LogBUN HGB Age LogBUN HGB Frac LogBUN HGB Age SCalc LogBUN HGB Frac SCalc LogBUN HGB LogPBM SCalc LogBUN HGB Age Frac SCalc LogBUN HGB Age LogPBM SCalc LogBUN HGB Age LogWBC SCalc LogBUN HGB Age Frac LogPBM SCalc LogBUN HGB Age LogWBC Frac SCalc LogBUN HGB Platelet Age Frac SCalc LogBUN HGB Platelet Age Frac LogPBM SCalc LogBUN HGB Age LogWBC Frac LogPBM SCalc LogBUN HGB Platelet Age LogWBC Frac SCalc LogBUN HGB Platelet Age LogWBC Frac LogPBM SCalc LogBUN HGB Platelet Age Frac LogPBM Protein SCalc LogBUN HGB Platelet Age LogWBC Frac Protein SCalc LogBUN HGB Platelet Age LogWBC Frac LogPBM Protein SCalc

10 Lab Course March 20, Disadvantages of automatic routines They typically lead to one particular subset of variables, rather than a set of equally good ones The subsets found might be different for different selection routines They generally tend not to account for the hierarchic principle Dependent on the stopping rule It does not foster critical thinking about the problem

11 Lab Course March 20, Collett The model selection strategy depends to some extent on the purpose of the study

12 Lab Course March 20, Collett Chow et al. (2002) Main goal: Investigate what explanatory variables, in a palliative care setting, are associated with overall survival

13 Lab Course March 20, Collett Fosker et al. (2013) The Importance of Poor Performance Status in Personalizing Palliative Radiotherapy Towards the End of Life

14 Lab Course March 20, Collett Step 0: Identify a set of explanatory variables that have the potential for being included in a model This approach assumes that all variables are considered to be on an equal footing, and there is no a priori reason to include any specific variables (like treatment). Steps 1-4: Determine the combination of variables to be included In practice, there will not be a unique combination of variables; there are likely to be a number of equally good models

15 Lab Course March 20, Collett If the number of potential explanatory variables (including interactions, non-linear terms etc.) is not too large, it might be feasible to consider all combinations of terms Pay due regard to the hierarchic principle and use the statistic -2Log(Likelihood) Use AIC to compare possible models

16 Lab Course March 20, Collett When the number of variables is relatively large, the number of possible models that need to be fitted can be computationally expensive Automatic selection routines might seem to be an attractive option Forward selection Backward elimination Stepwise

17 Lab Course March 20, Collett Step 1: Fit a univariate model for each covariate, and identify the predictors significant at some level p1, say 0.20.

18 Lab Course March 20, Collett Step 2: Fit a multivariate model with all significant univariate predictors, and use backward selection to eliminate nonsignificant variables at some level p2, say 0.10.

19 Lab Course March 20, Collett Step 3: Starting with final step (2) model, consider each of the non-significant variables from step (1) using forward selection, with significance level p3, say 0.10.

20 Lab Course March 20, Collett Step 4: Do final pruning of main-effects model (omit variables that are non-significant, add any that are significant), using stepwise regression with significance level p4.

21 Lab Course March 20, Collett At this stage, you may also consider adding interactions between any of the main effects currently in the model, under the hierarchical principle.

22 Lab Course March 20, Collett Collett recommends using a likelihood ratio test for all variable inclusion/exclusion decisions.

23 Lab Course March 20, Collett Statistical criteria alone should not guide the model selection strategy It may not be appropriate to include particular combinations of variables It might be unwise to omit some non statistically significant variables

24 Lab Course March 20, Hosmer, Lemeshow and May Purposeful selection Step 1: Fit a multivariable model containing all variables significant in the univariable analysis at the 0.20 to 0.25 significance level, and any other variables not selected using this criterion but judged to be of clinical importance

25 Lab Course March 20, Hosmer, Lemeshow and May Note: If there are many covariates that show a statistically significant association with survival you can rank order the covariates based on p-values using only the most highly significant variables. Include one covariate per ten events.

26 Lab Course March 20, Hosmer, Lemeshow and May Step 2: Use Wald test p-values of the individual coefficients to identify covariates that might be deleted Cautioned not to delete too many seemingly non-significant variables at one time Confirm above by using partial likelihood test

27 Lab Course March 20, Hosmer, Lemeshow and May Step 3: Assess whether removal of the covariate has produced an important change in the coefficients of the variables remaining in the model. A value of 20% is used as an indicator of important change. If the variable excluded is an important confounder reintroduce it into the model. This process continues until no variables can be deleted.

28 Lab Course March 20, Hosmer, Lemeshow and May Step 4: Add to the model, one at a time, all variables excluded from the initial multivariable model to confirm that they are neither statistically significant nor a confounder Result referred to as the preliminary main effects model

29 Lab Course March 20, Hosmer, Lemeshow and May Step 5: Test linearity of the continuous covariates This is referred to as the main effects model

30 Lab Course March 20, Hosmer, Lemeshow and May Step 6: Are interactions needed? Use 0.05 significance level. Use Wald p-value and partial likelihood ratio test as described earlier

31 Lab Course March 20, Hosmer, Lemeshow and May Step 7: Final Model Check model assumptions, goodness-of-fit

32 Lab Course March 20, Machin, Cheung, Parmar Explanatory variables are categorized 1. Fundamental to research design (D) 2. Those that influence outcome or are confounders (K) 3. Uncertain influence (Q)

33 Lab Course March 20, Strategies Forced-entry Significance tests Change in estimates of hazard ratios

34 Lab Course March 20, Forced-entry Include variables in the model according to research design or prior opinion. This could include a non-statistically significant variable. E.g. treatment variable in a RCT Include variables known to be influential in their ability to confound the primary association of interest The resulting model (with statistically non-significant effects) could have a reduced efficiency

35 Lab Course March 20, Significance testing Step-up or step-down procedures where selection is manual, not automated

36 Lab Course March 20, Change in estimates If our purpose is to obtain a suitable estimate of the HR for a key variable the significance-testing strategy may not be successful in selecting confounders Compare HR Crude with the adjusted estimate HR Adjusted for a clinically important difference. A 10% change is suggested.

37 Lab Course March 20, Practical considerations Due to the effects of bias if more than 20% of the data points are missing for a variable exclude it from the modeling process. If missing data comprise < 5% then the bias introduced will likely be small. Check to see how any automatic selection routines handle missing data In practice one can start with missing data excluded at the early stages of the selection process but bring them back into the process as it becomes more clear which variables are likely to be in the final model

38 Lab Course March 20, Practical considerations Significance level to use? Err on the side of caution. Use 0.10 generally and 0.2 for the change-in-estimates method

39 Lab Course March 20, Practical considerations Univariable analysis per se is not recommended Rationale for univariable screening if an explanatory variable is associated with an outcome variable this association may be the result of confounding However, if an explanatory variable is not associated with an outcome variable in a univariable analysis, there is no gain in further examining it in a multivariable analysis This argument is flawed; it overlooks the possibility of confounding which may suppress a genuine relation; so-called negative confounding

40 Lab Course March 20, Positive vs. Negative Confounding Positive confounding An association is found between an exposure variable and outcome but in reality there is no association. The spurious association is caused by the confounder OR the association is stronger than it appears because of the confounder Negative confounding - An association is not found between an exposure variable and outcome but in reality there is an association. The true association is suppressed by the confounder OR the association is weaker than it appears in reality because of a confounder

41 Lab Course March 20, Higher education in women Outcome: Higher breast cancer incidence Nulliparous True Magnitude Higher education in women Apparent Magnitude Lower breast cancer incidence Outcome: Lower breast cancer incidence

42 Lab Course March 20, Steyerberg The problem of overfitting already starts with considering too many candidate predictors in a data set. The problem is difficult to solve with standard statistical techniques which are used by default in medical research. The uncertainty of model selection is an important source of overfitting.

43 Lab Course March 20, Steyerberg Improvements can be sought by limiting the necessity for selection by using subject matter knowledge, especially in relatively smaller data sets (also advocated by Harrell) Use better algorithms to discover patterns in the data (e.g. LASSO) LASSO is a penalized estimation technique where the estimated regression coefficients are constrained such that the sum of their scaled absolute values falls below some constant k chosen by cross-validation This type of constraint forces some regression coefficients towards zero (which helps with overfitting problem) and some to exactly zero (helping with variable selection)

44 Lab Course March 20, Royston et al. No consensus exits on the best method for selecting variables Two main strategies: Full model approach all candidate variables are included. This model is claimed to avoid overfitting and selection bias and provide correct standard errors and P values. However, the full model is not always easy to define Backward elimination approach the choice of significance level has a major effect on the number of variables selected. Selection of predictors by significance testing is known to produce selection bias (regression coefficients overestimated) and optimism as a result of overfitting. Overfitting leads to worse prediction in independent data

45 Lab Course March 20, Example 1 - Chow et al. (Collett approach)

46 Lab Course March 20,

47 Lab Course March 20,

48 Lab Course March 20,

49 Lab Course March 20,

50 Lab Course March 20,

51 Lab Course March 20,

52 Lab Course March 20, Example 2 Fosker et al. (Harrell approach) The Importance of Poor Performance Status in Personalising Palliative Radiotherapy Towards the End of Life The goal of our project is to define a clinically relevant ECOG PS based algorithm that would enable accurate prediction of patients with shorter life expectancies (< 3-4 months).

53 Lab Course March 20,

54 Lab Course March 20, Multivariate Analysis Cox Proportional Hazards model results NOTE: ECOG=0 as reference category for variable ECOG Parameter P-value Hazard 95% CI Ra o Lower Upper Age Age 75+ < Brain mets Yes < ECOG 1 < ECOG 2 < ECOG 3 < ECOG 4 < Gender Male < Primary Lung <

55 Lab Course March 20, Conclusions One size doesn t fit all hard to conclude there is a best approach. Cutting to the chase is not appropriate to describe multivariable modeling building A good model is one chosen by using a careful, well thought out covariate selection process that gives thought consideration to issues of adjustment and interactions and thoroughly evaluates the model for assumptions, influential observations, and tests for goodness-of-fit (Hosmer and Lemeshow 2008)

56 Lab Course March 20, References Collett D. Modelling Survival Data in Medical Research. Chapman and Hall Hosmer DW, Lemeshow S, May S. Applied Survival Analysis Regression Modeling of Time-to-event Data 2 nd edition Wiley Machin D, Cheung YB, Parmar MKB. Survival Analysis A Practical Approach. Wiley Steyerberg EW. Clinical Prediction Models. Springer Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model BMJ June 2009 Volume 338 pp Harrell FE. Regression Modeling Strategies. Springer New York.