THE EFFECT OF NUMBER OF IMPUTATIONS ON PARAMETER ESTIMATES IN MULTIPLE IMPUTATION WITH SMALL SAMPLE SIZES MAURICE KAVANAGH DISSERTATION

Transcription

1 THE EFFECT OF NUMBER OF IMPUTATIONS ON PARAMETER ESTIMATES IN MULTIPLE IMPUTATION WITH SMALL SAMPLE SIZES by MAURICE KAVANAGH DISSERTATION Submitted to the Graduate School of Wayne State University, Detroit, Michigan In partial fulfillment of the requirements for the degree of DOCTOR OF EDUCATION 2018 MAJOR: EVALUATION AND RESEARCH Approved By: Advisor Date ii

2 ACKNOWLEDGEMENTS I would like to acknowledge my advisor, Dr. Shlomo Sawilowsky for his guidance, encouragement, and sense of humor throughout the process. I would also like to acknowledge and thank my committee members, Dr. Irwin Jopps for his kindness and gentle guidance, and Dr. Monte Piliawsky for his sparkling conversations, wit, and encouragement, especially with our shared passion and sanity preserving addiction, running. I would like to acknowledge the support and friendship of Tiana Bosley, the yin to my yang throughout the many courses we did together. I would like to acknowledge Janet Leslie for her encouragement and support as I began the process, and Tinysha McCord for her energy, encouragement and support as I completed it. I especially want to acknowledge the gentle, unwavering support and mentoring of Nelia Afonso, one of the warmest, kindest human beings I have ever had the privilege of encountering. Thank you. iii

3 TABLE OF CONTENTS Acknowledgements ii List of Tables vii List of Figures xiv CHAPTER 1 Introduction 1 Impact of Missing Data 2 Remediating Missing Data 3 Statement of the Problem 4 Limitations 5 Assumptions 6 Definitions 6 CHAPTER 2 Literature Review 9 Classifications of Missing Data 9 Proportion of Missing Data 10 Ignorable Versus Non-ignorable Missingness 10 Classic Data Handling Methods 12 Deletion 12 iv

4 Weighting 14 Data Replacement 15 Explicit Methods of Replacement 15 Implicit Methods 17 Contemporary Methods of Replacement: Multiple Imputation and Maximum Likelihood Estimation 19 Multiple Imputation 19 Likelihood Estimation 24 Numbers of Imputations 26 CHAPTER 3 Methodology 29 Sampling 29 Procedure 30 CHAPTER 4 Results 35 Data Set Adjustments 35 Tables of Organization 35 Regression Analyses, Intact Samples 35 Imputation Procedures 36 v

5 Regression Analysis, Imputed Data Sets 37 Changes to Alpha Values, Imputed Data Sets 37 Out of s Beta, Post Imputation 38 Confidence Interval Changes, Imputed Data Sets 40 CHAPTER 5 Discussion 43 Narrowed Confidence Intervals, Imputed Data Sets 43 Changes to Alpha Values, Imputed Data Sets 45 Manipulating Number Of Imputations, Sample Size, And Percent Missing_ 45 Out of s Beta, Post Imputation 47 Parent Data Set 48 Software Issues 49 Conclusion 50 Future Directions 50 APPENDIX A: Computer Coding and Syntax 52 APPENDIX B: Regression Coefficient Tables 56 APPENDIX C: Tables of Confidence Interval Widths 116 References 128 vi

6 Abstract 134 Autobiographical Statement 135 vii

7 LIST OF TABLES Table 1: Table 2: Table 3: Total Number of Observations, Eight Variables With Intact Data Set 29 Number of Missing Observations, Eight Variables With Data Missing on Six 30 Example Intact 20 Record Sample Chosen From Original Database 31 Table 4: Field Name Descriptions for Variables Used in Study 32 Table 5: Example Regression Analysis Results 33 Table 6: Table 7: Data Set N=20 After 20% MAR Deletion On Age, Education, Family, Race, Sex, And Income 34 Regression Analysis, Confidence Interval Width Predicted By Sample Size, Percent Missing, And Number Imputations 41 Table B1: Intact data, sample size 20, 10% deletion rate to be applied 56 Table B2: Sample size 20, 10% deletion rate, 20 imputations 56 Table B3: Sample size 20, 10% deletion rate, 40 imputations 57 Table B4: Sample size 20, 10% deletion rate, 60 imputations 57 Table B5: Sample size 20, 10% deletion rate, 100 imputations 58 Table B6: Intact data, sample size 20, 20% deletion rate to be applied 58 Table B7: Sample size 20, 20% deletion rate, 20 imputations 59 Table B8: Sample size 20, 20% deletion rate, 40 imputations 59 Table B9: Sample size 20, 20% deletion rate, 60 imputations 60 Table B10: Sample size 20, 20% deletion rate, 100 imputations 60 Table B11: Intact data, sample size 20, 30% deletion rate to be applied 61 viii

8 Table B12: Sample size 20, 30% deletion rate, 20 imputations 61 Table B13: Sample size 20, 30% deletion rate, 40 imputations 62 Table B14: Sample size 20, 30% deletion rate, 60 imputations 62 Table B15: Sample size 20, 30% deletion rate, 100 imputations 63 Table B16: Intact data, sample size 20, 40% deletion rate to be applied 63 Table B17: Sample size 20, 40% deletion rate, 20 imputations 64 Table B18: Sample size 20, 40% deletion rate, 40 imputations 64 Table B19: Sample size 20, 40% deletion rate, 60 imputations 65 Table B20: Sample size 20, 40% deletion rate, 100 imputations 65 Table B21: Intact data, sample size 40, 10% deletion rate to be applied 66 Table B22: Sample size 40, 10% deletion rate, 20 imputations 66 Table B23: Sample size 40, 10% deletion rate, 40 imputations 67 Table B24: Sample size 40, 10% deletion rate, 60 imputations 67 Table B25: Sample size 40, 10% deletion rate, 100 imputations 68 Table B26: Intact data, sample size 40, 20% deletion rate to be applied 68 Table B27: Sample size 40, 20% deletion rate, 20 imputations 69 Table B28: Sample size 40, 20% deletion rate, 40 imputations 69 Table B29: Sample size 40, 20% deletion rate, 60 imputations 70 Table B30: Sample size 40, 20% deletion rate, 100 imputations 70 Table B31: Intact data, sample size 40, 30% deletion rate to be applied 71 Table B32: Sample size 40, 30% deletion rate, 20 imputations 71 Table B33: Sample size 40, 30% deletion rate, 40 imputations 72 ix

9 Table B34: Sample size 40, 30% deletion rate, 60 imputations 72 Table B35: Sample size 40, 30% deletion rate, 100 imputations 73 Table B36: Intact data, sample size 40, 40% deletion rate to be applied 73 Table B37: Sample size 40, 40% deletion rate, 20 imputations 74 Table B38: Sample size 40, 40% deletion rate, 40 imputations 74 Table B39: Sample size 40, 40% deletion rate, 60 imputations 75 Table B40: Sample size 40, 40% deletion rate, 100 imputations 75 Table B41: Intact data, sample size 50, 10% deletion rate to be applied 76 Table B42: Sample size 50, 10% deletion rate, 20 imputations 76 Table B43: Sample size 50, 10% deletion rate, 40 imputations 77 Table B44: Sample size 50, 10% deletion rate, 60 imputations 77 Table B45: Sample size 50, 10% deletion rate, 100 imputations 78 Table B46: Intact data, sample size 50, 20% deletion rate to be applied 78 Table B47: Sample size 50, 20% deletion rate, 20 imputations 79 Table B48: Sample size 50, 20% deletion rate, 40 imputations 79 Table B49: Sample size 50, 20% deletion rate, 60 imputations 80 Table B50: Sample size 50, 20% deletion rate, 100 imputations 80 Table B51: Intact data, sample size 50, 30% deletion rate to be applied 81 Table B52: Sample size 50, 30% deletion rate, 20 imputations 81 Table B53: Sample size 50, 30% deletion rate, 40 imputations 82 Table B54: Sample size 50, 30% deletion rate, 60 imputations 82 Table B55: Sample size 50, 30% deletion rate, 100 imputations 83 x

10 Table B56: Intact data, sample size 50, 40% deletion rate to be applied 83 Table B57: Sample size 50, 40% deletion rate, 20 imputations 84 Table B58: Sample size 50, 40% deletion rate, 40 imputations 84 Table B59: Sample size 50, 40% deletion rate, 60 imputations 85 Table B60: Sample size 50, 40% deletion rate, 100 imputations 85 Table B61: Intact data, sample size 100, 10% deletion rate to be applied 86 Table B62: Sample size 100, 10% deletion rate, 20 imputations 86 Table B63: Sample size 100, 10% deletion rate, 40 imputations 87 Table B64: Sample size 100, 10% deletion rate, 60 imputations 87 Table B65: Sample size 100, 10% deletion rate, 100 imputations 88 Table B66: Intact data, sample size 100, 20% deletion rate to be applied 88 Table B67: Sample size 100, 20% deletion rate, 20 imputations 89 Table B68: Sample size 100, 20% deletion rate, 40 imputations 89 Table B69: Sample size 100, 20% deletion rate, 60 imputations 90 Table B70: Sample size 100, 20% deletion rate, 100 imputations 90 Table B71: Intact data, sample size 100, 30% deletion rate to be applied 91 Table B72: Sample size 100, 30% deletion rate, 20 imputations 91 Table B73: Sample size 100, 30% deletion rate, 40 imputations 92 Table B74: Sample size 100, 30% deletion rate, 60 imputations 92 Table B75: Sample size 100, 30% deletion rate, 100 imputations 93 Table B76: Intact data, sample size 100, 40% deletion rate to be applied 93 Table B77: Sample size 100, 40% deletion rate, 20 imputations 94 Table B78: Sample size 100, 40% deletion rate, 40 imputations 94 xi

11 Table B79: Sample size 100, 40% deletion rate, 60 imputations 95 Table B80: Sample size 100, 40% deletion rate, 100 imputations 95 Table B81: Intact data, sample size 200, 10% deletion rate to be applied 96 Table B82: Sample size 200, 10% deletion rate, 20 imputations 96 Table B83: Sample size 200, 10% deletion rate, 40 imputations 97 Table B84: Sample size 200, 10% deletion rate, 60 imputations 97 Table B85: Sample size 200, 10% deletion rate, 100 imputations 98 Table B86: Intact data, sample size 200, 20% deletion rate to be applied 98 Table B87: Sample size 200, 20% deletion rate, 20 imputations 99 Table B88: Sample size 200, 20% deletion rate, 40 imputations 99 Table B89: Sample size 200, 20% deletion rate, 60 imputations 100 Table B90: Sample size 200, 20% deletion rate, 100 imputations 100 Table B91: Intact data, sample size 200, 30% deletion rate to be applied 101 Table B92: Sample size 200, 30% deletion rate, 20 imputations 101 Table B93: Sample size 200, 30% deletion rate, 40 imputations 102 Table B94: Sample size 200, 30% deletion rate, 60 imputations 102 Table B95: Sample size 200, 30% deletion rate, 100 imputations 103 Table B96: Intact data, sample size 200, 40% deletion rate to be applied 103 Table B97: Sample size 200, 40% deletion rate, 20 imputations 104 Table B98: Sample size 200, 40% deletion rate, 40 imputations 104 Table B99: Sample size 200, 40% deletion rate, 60 imputations 105 Table B100: Sample size 200, 40% deletion rate, 100 imputations 105 Table B101: Intact data, sample size 500, 10% deletion rate to be applied 106 xii

12 Table B102: Sample size 500, 10% deletion rate, 20 imputations 106 Table B103: Sample size 500, 10% deletion rate, 40 imputations 107 Table B104: Sample size 500, 10% deletion rate, 60 imputations 107 Table B105: Sample size 500, 10% deletion rate, 100 imputations 108 Table B106: Intact data, sample size 500, 20% deletion rate to be applied 108 Table B107: Sample size 500, 20% deletion rate, 20 imputations 109 Table B108: Sample size 500, 20% deletion rate, 40 imputations 109 Table B109: Sample size 500, 20% deletion rate, 60 imputations 110 Table B110: Sample size 500, 20% deletion rate, 100 imputations 110 Table B111: Intact data, sample size 500, 30% deletion rate to be applied 111 Table B112: Sample size 500, 30% deletion rate, 20 imputations 111 Table B113: Sample size 500, 30% deletion rate, 40 imputations 112 Table B114: Sample size 500, 30% deletion rate, 60 imputations 112 Table B115: Sample size 500, 30% deletion rate, 100 imputations 113 Table B116: Intact data, sample size 500, 40% deletion rate to be applied 113 Table B117: Sample size 500, 40% deletion rate, 20 imputations 114 Table B118: Sample size 500, 40% deletion rate, 40 imputations 114 Table B119: Sample size 500, 40% deletion rate, 60 imputations 115 Table B120: Sample size 500, 40% deletion rate, 100 imputations 115 Table C1: Confidence Interval Widths, Data Set N=20, Missing 10% 116 Table C2: Confidence Interval Widths, Data Set N=20, Missing 20% 116 Table C3: Confidence Interval Widths, Data Set N=20, Missing 30% 117 xiii

13 Table C4: Confidence Interval Widths, Data Set N=20, Missing 40% 117 Table C5: Confidence Interval Widths, Data Set N=40, Missing 10% 118 Table C6: Confidence Interval Widths, Data Set N=40, Missing 20% 118 Table C7: Confidence Interval Widths, Data Set N=40, Missing 30% 119 Table C8: Confidence Interval Widths, Data Set N=40, Missing 40% 119 Table C9: Confidence Interval Widths, Data Set N=50, Missing 10% 120 Table C10: Confidence Interval Widths, Data Set N=50, Missing 20% 120 Table C11: Confidence Interval Widths, Data Set N=50, Missing 30% 121 Table C12: Confidence Interval Widths, Data Set N=50, Missing 40% 121 Table C13: Confidence Interval Widths, Data Set N=100, Missing 10% 122 Table C14: Confidence Interval Widths, Data Set N=100, Missing 20% 122 Table C15: Confidence Interval Widths, Data Set N=100, Missing 30% 123 Table C16: Confidence Interval Widths, Data Set N=100, Missing 40% 123 Table C17: Confidence Interval Widths, Data Set N=200, Missing 10% 124 Table C18: Confidence Interval Widths, Data Set N=200, Missing 20% 124 Table C19: Confidence Interval Widths, Data Set N=200, Missing 30% 125 Table C20: Confidence Interval Widths, Data Set N=200, Missing 40% 125 Table C21: Confidence Interval Widths, Data Set N=500, Missing 10% 126 Table C22: Confidence Interval Widths, Data Set N=500, Missing 20% 126 Table C23: Confidence Interval Widths, Data Set N=500, Missing 30% 127 Table C24: Confidence Interval Widths, Data Set N=500, Missing 40% 127 xiv

14 LIST OF FIGURES Figure 1: Regression Analysis, Confidence Interval Vs Sample Size 42 xv

15 1 CHAPTER 1 INTRODUCTION Surveys with missing observations are an expected occurrence in social science research (Rubin, 1976), as noted by Widaman (2006) who opined, "The presence of non-trivial levels of attrition is the norm in longitudinal studies (p. 43) Newman (2014) suggested research reports should be viewed with suspicion if data sets were claimed to be complete. Contrary to Enders (2004), who asserted that missing data in social science constitute a statistical nuisance (p. 431) to be eliminated, others suggested missingness in a dataset may constitute part of the total story. For example, timed performance tests. Comparing sequential blank responses to numbers of completed items may be part of a performance measure (Glas & Pimentel, 2008). Rubin (1987) suggested that items which preclude all but specific responses, for example job title, can be expected to return a portion of blank responses. Designs where subsets of available data are collected from larger samples (Baraldi & Enders, 2010) are also examples where useful information is available when the response cell is empty. Rutkowski (2011), however, counseled that biases conferred by missing data may be present even when missingness is due to study design. Rubin (1987) posited missing data are observable, versus inferred, and are therefore finite. Therefore, when referring to missing data, it should be thought of as data which exist but not recorded. Lee and Cai (2012) supported this counsel, equating missing data to latent or unobserved variables.

16 2 Impact of Missing Data Analyses where all potentially available data are not considered when creating estimates risks return of misleading or faulty inferences. Although the value of a missing response may be biased on speculation, ignorance of the true value invokes additional levels of uncertainty. One consequence of missing data pertains to statistical power, as it is in part a function of sample size. Hence, missing data immediately begets a loss of power. Data missing from comparison groups in an asymmetrical fashion may represent differences between groups, and symmetrical patterns of missingness may include common, potentially confounding, unexplored issues. Either pattern may suggest biasing influences which may affect study outcomes. Even when built into the design, Rutkowski (2011) suggested that instruments with intentionally missing data are subject to unintended biases. Not all missing data are equally problematic, however. Occasional random missing observations from a study may likely be of little consequence, while patterns of missingness may constitute a minor nuisance, major concern, or even a focus for study. Remediating missing data without considering the background and causes of the missingness may appear to address the problem at hand, but may actually mask substantial issues (Hair, Black, Babin, Anderson, & Tatham, 2006). Knowledge and understanding of the study population is an important consideration for remediating missing data as indicated by Little and Rubin (2002), who suggested that data replacement by the data collector as opposed to the data analyst is preferable, as not

17 3 only is the collector likely more familiar with the data source, but they may also be the best or only person with access to the original data pool. In support for Little and Rubin (2002), Widamin (2006) opined that the optimal way of handling missing data depends on the rate of missingness. Similarly, Rubin (2003) suggested that even though groups of missing and collected data may have the same mean, they likely have different distributions. This lent credence to the contention that missing observations are a subset of the larger data pool or may even form unique subset of the population. The common message conferred by the above is that familiarity with the data and its background will serve the researcher well when assessing both the impact and remediation of missing data. Remediating Missing Data The three most common approaches for handling missing data include 1) deletion, 2) weighting, and 3) replacement (Little & Rubin, 1989). Deletion methods remove or ignore partial or entire cases where observations are missing. Weighting adds emphasis to remaining observations to compensate for loss. Replacement methods attempt to repair the data set by imputing surrogate values in place of blank observations. Data replacement may be accomplished using either readily available values of convenience or purposively generated ones. Similar to deletion and weighting, classic, ad hoc data replacement techniques are relatively straightforward to use; however, all three methods tend to distort parameter estimates, especially as sample sizes decrease, rates of missingness increase, or as patterns of missingness become more complex.

18 4 Contemporary purposive data replacement techniques, including multiple imputation (MI), and maximum likelihood estimation (ML) address concerns of parameter distortion by using existing data to estimate replacement values. Unlike ad hoc methods, purposive methods track the data (Rubin, 1996, p. 478), potentially resulting in more accurate estimates. Contemporary purposive data handling methods can ameliorate problems inherent in older methods; however, they too may have issues. Multiple imputation procedures can be somewhat unwieldy to implement, return a slightly different result each time, and run the risk of increased parameter estimation errors as sample size decreases and assumption violations increase. Likelihood estimation procedures require intact covariates to be successfully applied. However, covariates can restrict the generalizability of the model, and may introduce unwanted interaction effects or unpredictable biases. As MI has become increasingly accessible through commercial software packages such as SPSS or SAS, guidelines may be beneficial in helping to gauge the optimal number of imputations which balance under and overly precise estimates. Although estimate biases tend to diminish as sample size increases, the problem remains or exacerbates as sample size decreases. Smaller samples will require more effort to generate greater numbers of imputations in an attempt to neutralize bias. Some of these issues appear to have been addressed in the multiple imputation option available in more recent versions of SPSS. While many users will likely address missing data using SPSS defaults of five imputations and ten iterations, users are able to

19 5 specify much higher numbers of each, which risk creation of overly precise and potentially inaccurate estimates. Statement of the Problem Although the number of imputations required to obtain confidence intervals within 10% bias of the original complete data set will increase as the rate of missingness increases and sample size decreases, there is a point where further increasing the number of imputations may spawn overly precise and potentially misleading estimates. It is therefore hypothesized that there exists an optimal range of imputations which balance over and under specified parameter estimates. It is further hypothesized that the number of imputations will vary with 1) sample size decreases and 2) increased rates of missingness. The purpose of the current study, therefore, is to explore the effect of numbers of imputations on parameter estimate biases in smaller sample sizes with varying levels of missingness. The effect of varying the sample size and level of missingness with the number of imputations needed to return parameter estimates with biases within 10% of the original complete data will be explicated. Limitations For the current study, a small sample will be drawn from a larger data set. The randomly drawn sample may not be a true representative of the population. Data deletion and replacement will be simulated using a computer and, as such, may not accurately reflect reality. By its nature, multiple imputation is expected to return a different set of estimates each time it is conducted; therefore, the exact imputation results and resulting parameter estimations may not be readily reproducible.

20 6 Assumptions The following assumptions are made. The chosen sample and parent data set are approximately normally distributed. Variables chosen for study are continuous and approximately normally distributed. Missing data may be classified as missing at random. Definitions Bayesian: A typology of inference where prior information contributes to the likelihood of subsequent outcomes. Bias: Systematic variation of a parameter estimator from one created using full information. Comparison group: A population subset where participants are assigned to the experimental condition. Confidence interval (C.I.): A range of probabilities within a specified interval from the parameter of interest. Convergence: The process of two variables approaching or converging upon the same value. Covariate: An ancillary or auxiliary variable used to predict the value or outcome of a variable of interest. Distribution: The range and frequency of observed and expected frequency of unobserved values. Efficiency: A measure of the accuracy or precision of a parameter estimator. Estimate: (noun) The projected value of a parameter. Factor analysis: A methodology whereby groups of variables are assessed for patterns

21 7 of correlation. Heteroscedasticity: An inconsistent and or unpredictable pattern of variance in a distribution. Imputation: A procedure or procedures whereby values are added to the data set. Inference, statistical: A conclusion based upon the assessment of likelihood. Interaction: Influence or confounding effect of variables upon one another. Latent variable: A variable which is unobserved but assumed to exist. Log likelihood: Use of natural logarithm to estimate probability or likelihood. Misspecification: An incorrect or mismatched variable used for creating parameter estimates. Monotone: A pattern of missingness in longitudinal studies where data is consistently missing after a specific point in time. Monte Carlo: A methodology which uses random sampling for generating arrays of simulated values. Observation: A datum or single data point. Parameter: A value which defines or describes a characteristic of a distribution, e.g. the mean. Population: An entire group of interest. Posterior distribution: A distribution which uses prior information as conditions or components of parameter predictors. Power, statistical: The ability of a statistical test to detect an effect if an effect exists. Predictor line: A trend line used in regression analysis to predict or estimate a value. Regression: A method of estimating or predicting a value by comparing its fit with known

22 8 patterns, usually through graphical means and use of predictor or regression lines. Sample: A selection or subset of a population. Specification: The choice of variables used to generate parameter estimates. Standard error (SE): The average magnitude of difference between a random observation and the true or hypothesized value. Stochastic: A random variable. Type I error: Incorrectly inferring an effect exists. Type II error: Incorrectly inferring an effect does not exist. Univariate: Single variable, usually in reference to descriptive statistics.

23 9 CHAPTER 2 LITERATURE REVIEW Classifications of Missing Data Data may be classified as missing as primarily at the item, scale, or subject levels. Item level missing data refer to single empty data cells where no between-cell connection for the missingness is evident. Blank entries may be a result of participants omitting items with intention to return but not doing so, choosing not to answer, or simply skipping items unintentionally. Single item missingness may also be due to factors outside of individual participant control, such as coding, or recording anomalies among myriad other possible causes. Scale or construct level missingness refers to subsets of items within individual questionnaires (Newman, 2014). Subsets may include construct or thematically linked groupings of questions that participants feel unable or unwilling to answer. Scale level missingness may also refer to patterns where blank responses are returned on all items beyond a certain point within the survey. Such patterns of missingness may be due to participant fatigue or insufficient time for completion (Glas & Pimentel, 2008), the latter being a common feature of timed surveys or tests where gross count of responses forms part of the final measure. Subject level missingness includes surveys returned devoid of responses, often a result of deliberate or inadvertent participant non-response. Potential participants may choose not to respond, they may n=be unable to respond, or their responses were not recorded. Data loss for technical reasons, e.g. loss of groups of records, may also be a root cause of subject level missingness.

24 10 Proportion of Missing Data As the proportion of missing data increases, the challenges for deriving accurate parameter estimates and, in turn, inferences increase. Determining the approximate level of missingness where inferences are considered unacceptably compromised can prove challenging when no clear distinction between low, moderate and high levels of missingness is evident in the literature. Widamin (2006) described missingness in the 1-2% range as relatively minor, and Schafer (1997) categorized as minor any levels of missingness below 5%. Rubin (2003) described levels of missingness below 30% as modest, and Widamin (2006) considered levels greater than 25% relatively high. As the rate of missingness of a variable approaches extreme levels, it may be beneficial to consider the contention by Rubin (2003) that data sets with levels of missingness in excess of 50% may no longer be considered a problem of missing data. In essence, if less than half of the data of interest is observed, then that data which has been gathered essentially becomes a subsample of the overall sample space. Ignorable Versus Non-Ignorable Missingness Data loss at any of item, scale, or subject levels may or may not be random occurrences. Data may also be missing in non-random patterns if interactions within and between levels exist. For example, a subset of potential respondents may refuse to respond to an entire survey if they find parts of it objectionable. Understanding patterns of missingness and the reasons behind nonresponse can aid the researcher considering remediation of missing data. Rubin (1976) suggested the answer to the question of when is it appropriate to ignore missing data lies in the distinction of random versus nonrandom patterns of missingness. Rubin (1976)

25 11 postulated tenets for classification of missing data into ignorable and non-ignorable categories. He suggested classification as ignorable connotes any fluctuations in the unobserved versus observed may be attributable to random variation. Rubin (1976) cautioned, however, that non-responses are rarely ignorable. Data described as missing completely at random (MCAR) are those where there is no discernible pattern to the missingness, and missing at random (MAR) include responses which exhibit a pattern not in conjunction with the dependent variable (Little & Rubin, 2002). Missing not at random (MNAR) data is an evident relationship between the dependent variable and the pattern of missingness. Widamin (2006) suggested both MAR and MCAR be grouped under ignorable, and MNAR be considered non-ignorable. Newman (2014) suggested mechanisms of missingness, e.g. MCAR, MAR, MNAR may be more useful if viewed on a continuum, and argued rarely if ever are data entirely MCAR or MNAR. Along similar lines, Little and Zanganeh (2013) advocated for relaxing the rules surrounding ignorability of missing data, and suggested it may be acceptable to follow guidelines for the MAR condition for data sets whose patterns of missingness are MNAR if the variables or parameters under consideration meet criteria for MAR. Even when no evident relationship exists between the patterns of missingness, it may be prudent to not dismiss the existence of unaccounted for interactions. It may also be advisable to consider groupings of participants whose surveys are missing data as part of a de facto subgroup. It then becomes incumbent upon the researcher to uncover any patterns of missingness and to make a determination if a pattern is ignorable or non-ignorable. For example, Glas and Pimentel (2008) suggested classifying missing

26 12 data as non-ignorable in surveys where the missingness is related to survey design, e.g. surveys which track counts of completed items as a measure of proficiency. They argue that ignoring missing data in this context constitutes a bias. Ultimately, when considering the ignorability of missing data, it is incumbent upon the researcher determine if patterns of missingness represent a meaningful threat to potential statistical inferences. Classic Data Handling Methods Deletion Deletion methods or complete case methods only use records where complete data are available. Methods include listwise deletion and pairwise deletion. With listwise deletion, only complete cases are included and any with missing values are ignored. For pairwise deletion, only cases where both variables are available for individual comparisons are included in the analysis. Listwise deletion discards entirely any cases missing any variable information. Although such a method appears to be an expedient way to circumvent the complication of incomplete records, Roth, Switzer, and Switzer (1999) argued listwise deletion ensures only part of the available data are used. This aligned with Enders (2001) who made the sobering observation that given a 5% rate of missingness in a three-predictor regression model, employing listwise deletion can expect to lose an estimated 18.6% of their subject data (p. 716). Additionally, the smaller sample size resulting from such actions may not only decrease statistical power; it may also consolidate remaining cases into tighter clusters with less variance, thus increasing the likelihood of Type I errors (Baraldi & Enders, 2010).

27 13 According to Enders (2001), Peugh and Enders (2004), Baraldi and Enders (2010), and Thoemmes and Mohan (2015), successful use of listwise deletion requires the strict assumption of MCAR, and according to Cheema (2014), would likely only function at lower levels of missingness. Newman (2014) stated that records converted from item to record level missingness subsequently become more difficult to restore due to a lack of information available from which to make estimates. Widamin (2006) furthered this argument, and suggested additional biases may be introduced by use of listwise deletion, especially if the deleted cases have something in common. This agreed with Rubin (2003), who indicated that subjects with missing information should be considered a subset of the population. In a study on predictive risk factors in the intensive care unit, (ICU), Chevret, Seaman, and Resche-Rigon (2015) implied listwise deletion can mask the possibility of excluded patients being MNAR, this introducing potential biases which can erode the precision of the parameter estimator model. In limited support of listwise deletion, Schafer (1997) opined that use of listwise deletion at levels of missingness below 5% may be acceptable but is only practical for univariate missing data. Pairwise deletion removes cases from individual cell pair comparisons where a subject s data are missing from one of two variables. Although this method may appear more conserving of data, it introduces concerns. Roth et al. (1999), Baraldi and Enders (2010), and Newman (2014) indicated that each variable will contribute differing numbers of unmatched comparisons; such mismatches making total scale comparisons difficult while yielding inaccurate standard errors. As with listwise deletion, the discounting of blank cells by

28 14 pairwise deletion may bias parameter estimates if the emptiness of the cell itself forms part of the overall data profile. Although either method will necessarily erode statistical power simply by virtue of data loss, Widamin (2006), Newman (2014), and Thoemmes and Mohan (2015) agreed the losses are markedly greater with listwise deletion. This aligned with Little and Rubin (2002) who were dismissive of either method of complete case analysis except in very limited circumstances. Jones (1996) described deletion as wasteful of data, while Newman (2014) went further, decrying deliberate deletion of data as disrespectful of the human being who gave their time and effort to provide the information. Weighting According to Little and Rubin (1989), weighting of cases to replace missing data involves discarding records with any missing components and assigning a weighting value to the remaining intact cases to compensate for those lost to deletion. Rubin (1987) opined that weighting of cases in order to replace missing or discarded ones assumes all other values in the weighted case are similar to the replaced one. Rubin (1987) further stated the assumptions become invalid when multivariate statistics are involved. Similarly, Little and Rubin (1989) noted that case weighting can be useful in limited instances and with monotone data patterns; however, using weightings will complicate computation of standard error estimates. Cheema (2014) argued more directly against weighting, and stated missing values due to item non-response cannot be fixed by using weights (p. 489).

29 15 Data Replacement Replacement of missing values may be accomplished in myriad ways, using procedures of widely varying levels of complexity. Little and Rubin (2002) referred to explicit methods where the missing value is replaced with an existing one, and implicit methods where a value is inserted with the assumption of its compatibility. Explicit Methods of Replacement The most straightforward method for replacing missing values involves substituting or imputing the arithmetic mean. Although an easy procedure to implement, mean substitution creates several fundamental problems. Widamin (2006) stated that substituting the mean for an actual value missing from the sample pool will contribute a zero measure of variance to parameter estimates. As occurrences of mean substitution increase in a sample, the alleged sample distribution will increasingly consolidate around the mean, resulting in underestimates of variance and standard errors and increases in the likelihood of Type I errors (Newman, 2014; Rubin, 1987). Widamin (2006) suggested that narrowing of estimated distributions may also play havoc with estimates of correlation which rely on overlaps between variables. Another issue with imputing the mean in place of missing observations is voiced by Rubin (1987), who suggested that each occurrence of mean imputation not only artificially reduces estimates of standard error, but the additional variable counts yield erroneous degrees of freedom measures, thus further biasing parameter estimates. Little and Rubin (2002) described mean imputation as a special case of regression imputation, essentially where the mean functions as a placeholder for missing observations. Not unlike mean substitution, regression imputation uses existing

30 16 values tom predict the value of missing observations. Although this may seem a convenient way to plug an empty spot with what appears to be an ideal placeholder, Widamin (2006) suggested that regression imputation is a double-edged sword. Predictions made using regression lines produce too good (p. 52) to be true values which fall directly on the prediction line. This aligns with Azur, Stuart, Frangakis, and Leaf (2011) who cautioned on misleading overly precise results (p. 41) resulting from a single imputations such as mean substation. Again, the narrowed range of estimated standard error in such distributions will increase the chance of Type I error, or the mirage (Newman, 2014, p. 378) of an effect when none is present. For either of the above types of single imputation where a surrogate missing value is fabricated from existing observations, both Widamin (2006) and Cheema (2014) raised doubts about the validity. Widamin (2006) expressed concerns that real unobserved values may actually be different from imputed ones. This aligns with concerns expressed by Cheema (2014), who suggested that replacing missing values with what amounts to a guess at the mean raises the question whether the estimated value is actually the mean at all. Rubin (2003) furthered the argument of representativeness, proposing that even if the collection of missing records has the same mean as those which have been observed, they may have a different distribution pattern. Rubin (1978) succinctly summarized the problem with single value imputation stating, imputing one value for a missing datum cannot be correct in general, because we don t know what value to impute with certainty (if we did, it wouldn t be missing) (p. 21).

31 17 A third method of explicit replacement modeling, stochastic regression imputation, was described by Little and Rubin (2002) as a regression imputation plus a residual, drawn to reflect uncertainty in the predicted value (p. 60). Baraldi and Enders (2010) agreed that although the stochastic procedure does add an error term, it does it only once, yielding an artificially small confidence interval and commensurate uptick in the likelihood of Type I error. The opposite was claimed by Widamin (2006), who expressed concerns that addition of a random component into the absolute value of imputation may yield an entry out of realistic range or is otherwise implausible. Although their admonition to prevent this happening the researcher must be familiar with the characteristics of the original data appears to have been addressed by Little and Rubin s (2002) description of the component as a residual of the predicted value, their caution to remain mindful of potential unforeseen interactions does provide food for thought on overall context when choosing to replace missing data. Implicit Methods Hot deck imputation involves replacing one missing value with another copied from a different respondent with a similar response pattern from the same population. In a Monte Carlo simulation, Roth et al. (1999) counseled not to immediately discard the hot deck procedure, as it does actually introduce some level of variability. Schenker and Welsh (1988) also saw hot deck imputation as potentially useful when used with large sample sizes, relatively few covariates and small numbers of possible values. Daniel and Kenward (2012) considered hot deck helpful under MAR conditions; however, they suggested that it may actually become a hindrance under MNAR, especially if other participant variates are actually different.

32 18 Although the schemata behind this fairly popular procedure can be rather complex (Little & Rubin, 2002), in the end, the substitute value is at best only a static estimate of the true value (Rubin, 1987). Rubin (1987) suggested that straight draws from the hot deck may not be sufficient to capture the dynamic nature of sampling: instead, it was suggested to follow a Bayesian paradigm (p. 126) which has variability between draws built into the model. Cold deck imputation is similar to hot deck except that the replacement is drawn from a comparable but preexisting data set (Little & Rubin, 2002). The preexisting and static nature of the data source sampled using cold deck imputation may be even more removed from the population from which the replacement is sought. Other replacement methods include straight substitution of one missing participant with another from the population; the zero imputation method of replacing missing values with zero, dummy variable adjustment, and last observation carried forward procedures among others (Little & Rubin, 2002; Cheema, 2014; Barnes, Lindborg, & Seaman Jr., 2006). These methods are straightforward to implement, but can yield parameter estimates biased towards Type I errors. A serious problem according to Rubin (1987) with replacement techniques is that although the substituted data are only estimates of the true values, they are almost always treated by researchers as real responses. The flaw is in unwittingly making inferences and decisions with potentially inaccurate information. Peugh and Enders (2004) suggested all such ad hoc replacement techniques are fundamentally flawed and should be abandoned.

33 19 Contemporary Methods of Replacement: Multiple Imputation and Maximum Likelihood Estimation Rubin (1978) and Rubin (1987) described a method, multiple imputation (MI) where the researcher creates a series of copies using the original data, replacing individual missing observations with values randomly drawn from the existing data pool. The result is a series of similar but unique data sets, each representing plausible versions of the intact data set. Analysis procedures envisioned for the complete data are applied separately to each copy and the results pooled. The pooled estimates are then averaged to create final parameter estimates. Maximum likelihood (ML) methods such as estimation maximization (EM) algorithms (Little & Rubin, 2002) and full information maximum likelihood (FIML) use existing information to essentially triangulate on unobserved values (Peugh & Enders, 2004). The resulting complete data set contains a mix of observed values and empty cells occupied by a best guess at the real value. Multiple Imputation Rubin (1987), Little and Rubin (2002), and Carlin (2014) referred to multiple imputation as a Bayesian grounded approach where information from prior sampling informs subsequent draws. Plausible imputation values and error terms are based upon knowledge of existing data, with each sampling from the data set consisting of a random draw which, when recorded, is added back to the pool for use in subsequent draws. Unlike static imputation methods such as mean and regression imputation or hot and cold deck imputations, in MI, the sample space updates with each imputation (Baraldi & Enders, 2010). Over subsequent sampling draws, the adaptive behavior the MI process

34 20 yields a more distinct sample space, while preserving the element of randomness within the sampling pool. The random element allows, according to Rubin (1987), the imputed collection values to represent the uncertainty (p. vii) about which of the group values corresponds to the actual missing item. Subsequent imputations may or may not more closely approximate the actual value of the missing item; however, van Buuren (2007) noted that the pooling of parameter estimates tends to offset the effects of any extreme values which may appear. MI resembles stochastic regression imputation (Baraldi & Enders, 2010; Enders, 2010). However, the potential for implausible entries (Widamin, 2006), and increase in likelihood of Type I error (Baraldi & Enders, 2010) inherent in stochastic imputation are tempered by the multiple and likely divergent values returned by MI procedures. The aggregated effect of multiple imputations is in agreement with Widamin (2006) who, like van Buuren (2007), noted that the convergent behavior of MI confers the benefits of stochastic regression imputation, while mitigating the effects of extreme values. Although multiple imputation by design adds a measure of uncertainty to parameter estimates, there is support for the scaffolding influence provided by the inclusion of covariates and or auxiliary variable in the imputation model. Rubin (1978), Rubin (1996), and Crawford, Tennstedt, and McKinlay (1995) counseled using auxiliary variables as a form of frameworking. Van Buuren (2007) suggested that including auxiliary variables may help support the ignorabilty assumption of missing data. This aligned with Schenker and Welsh (1988), who suggested the ignorability assumption becomes more plausible with the addition of covariates to help explain or contextualize

35 21 nonresponses, and Enders and Gottschall (2011), who advocated for carefully selected auxiliary variables to satisfy the MAR assumption, (p. 37). Although Rubin (2003) indicated most MI models function under the assumption of ignorabilty, in reality, as noted earlier, nonresponse is rarely truly ignorable (Rubin, 1976; 1987). In an example of nonignorable missingness, Engels and Diehr (2003) proposed using participants prior data to predict those subsequently lost to attrition in longitudinal studies. They eschewed use of MI, suggesting simulations which prove that the utility of MI methodology are conducted using unrealistic patterns of missingness. Although their allegations may appear superficially logical in context of their focus, the conclusion was based on the extreme MNAR nature of their target participants. Comparing MI to data augmentation (DA), Merkle (2011) suggested that DA, essentially MI draws alternating with simulated, i.e. Monte Carlo values, may in some circumstances function better than MI. They found DA returned less biased estimates with smaller sample sizes and higher levels of missingness. Of note is their discussion of the differences in the narrower context of factor analysis, with this procedure functioning for them in both imputation and analysis roles. They acknowledged, however, this advantage is modest, and most evident with the better specified DA imputation model. In essence, the conceded the less precise nature of MI methodology may actually make it more tolerant of misspecified models. Essentially, more tolerant of the inherent messiness of real world data. In comparing MI to doubly robust estimators, Carpenter et al. (2006) appeared to contradict Merkle (2011). Carpenter et al. (2006) suggested MI works better with correctly specified models. Their model, based on inverse probabilities, essentially

36 22 weightings, more strongly supported the choice of parameter estimators, and disallowed some apparent flexibility inherent in the uncertainty aspect of MI. This ran contrary to Rubin (1987), who suggested an advantage of MI was that the data manager no longer needed to risk the bias of misleadingly constrained values potentially conferred by the use of weightings to shore up holes in the data. Although limited support for use of weightings may be found in Little and Rubin (2002), Carpenter et al. (2006) undercut the utility of their version of waiting by offering caveats on the restrictive circumstances under which it may be valid. Hughes, Sterne, and Tilling (2014), not unlike Carpenter et al. (2006), suggested mismatch between assumptions around the data set and the imputation model used can yield misleading results. They expressed concerns regarding overly precise estimates in the presence of heteroscedasticity, especially as sample size decreases. This aligned with Schafer (1997), who cautioned that an overly focused imputation model may restrict the utility of a data set. Hughes et al. (2014) discussed an estimator proposed by Robins and Wang (2000) which appeared more robust in the presence of specification and compatibility mismatches. Both Robins and Wang (2000) and Hughes et al. (2014) conceded the Robins and Wang (2000) estimator appeared to function best in the presence of mismatches. Robins and Wang (2000) further admitted their procedure may not function with smaller data sets. Steele, Wang, and Rafferty (2010) also proposed a method of enhancing the imputations with samples taken from a simulated normal posterior distribution. Although

37 23 this appeared to narrow their confidence intervals, they indicated their method for improving upon Rubin (1987) had limited utility. Kim (2004) noted the Rubin (1987) estimators carried large biases with smaller sample sizes in linear regression applications. The expressed concern in instances of stratified data, suggesting that such further subdivision would exacerbate biases. They proposed use of an estimator, which not unlike a logarithmic coefficient, seemed to better attenuate variance fluctuations with smaller sample sizes. In line with Kim (2004), Barnard and Rubin (1999) suggested an adjustment to reduce the biases inherent in smaller sample sizes, as did Graham and Schafer (1999), who suggested transforming variables back to their original scale afterwards (p. 9). Although Kim (2004) showed their estimator increasing efficiency and shrinking biases in a simulation study, on closer inspection, the most notable differences begin to emerge with the 40% response rate condition. This raised the question of practicality of the Kim (2004) variant when Rubin (2003) suggested that MI can, but was not intended, to work at levels of missingness above 50% due to concerns of loss of efficiency and validity. In comparing MI to ML in small univariate normal samples, von Hippel (2013) argued against MI on the grounds of inefficiency and bias. This aligns with Yuan, Yang- Wallentin, and Bentler (2012) who stated that MI will tend to return biased, less efficient estimates than ML in the presence of violations of distribution conditions. Their suggestion that MI may fare better than ML with smaller sample sizes when performed under conditions of a properly specified posterior distribution agreed with Carpenter et al. (2006) and their belief that MI functions better with more precisely specified models.