THE IMPACT OF UNDERREPORTING ON MMPI-2-RF SUBSTANTIVE SCALE SCORES. A thesis submitted. To Kent State University in partial

Transcription

1 THE IMPACT OF UNDERREPORTING ON MMPI-2-RF SUBSTANTIVE SCALE SCORES A thesis submitted To Kent State University in partial Fulfillment of the requirements for the Degree of Masters of Arts by Adam Hoge Crighton April 2017 Copyright All rights reserved Except for previously published materials

2 Thesis written by Adam Hoge Crighton B.S., Eastern Kentucky University, 2014 M.A., Kent State University, 2017 Approved by Yossef S. Ben-Porath, Ph.D., Advisor Maria S. Zaragoza, Ph.D., Chair, Department of Psychological Sciences James L. Blank, Ph.D., Dean, College of Arts and Sciences

3 TABLE OF CONTENTS iii LIST OF TABLES... iv ACKNOWLEDGEMENTS... v INTRODUCTION... 1 RESEARCH DESIGNS USED TO INVESTIGATE RESPONSE STYLES... 3 MMPI-2-RF VALIDITY SCALES... 4 VALIDITY SCALES AS MODERATORS AND SUPPRESSORS OF ASSOCIATIONS BETWEEN SUBSTANTIVE SCALES AND CRITERION MEASURES... 8 THE CURRENT STUDY... 9 METHODOLOGY PARTICIPANTS MEASURES PROCEDURES RESULTS MMPI-2-RF SCALES CRITERION VALIDITY CORRELATIONAL ANALYSES ABSOLUTE MEDIAN CORRELATIONS INCREMENTAL PREDICTIVE UTILITY OF UNDERREPORTING VALIDITY SCALES MODERATION ANALYSES DISCUSSION TABLES APPENDICES REFERENCES iii

4 LIST OF TABLES Table 1. Minnesota Multiphasic Personality Inventory-2-Restructured Form Standard Instructions (SI), Very Virtuous (VV) and Well-Adjusted (WA) Groups Mean Scores (N = 275) Table 2. Means and Standard Deviations of Criterion Measures for Groups who Completed the MMPI-2-RF under Standard Instructions (SI), Instructions to Appear Very Virtuous (VV), and Instructions to Appear Well-Adjusted (WA) (N = 275) Table 3. Correlations between Criterion Measures and MMPI-2-RF Scales for (1) Those who Completed the MMPI-2-RF under Standard Instructions, (2) Those asked to appear Very Virtuous, and (3) Those asked to appear Very Well-Adjusted (N s range from 273 to 275) Table 4. Hierarchical Logistic Regression Analysis Results for Predicting Standard Instructions Group vs. Very Virtuous Group (n = 196) Table 5. Hierarchical Logistic Regression Analysis Results for Predicting Standard Instructions Group vs. Well Adjusted Group (n = 194) Table 6. Hierarchical Logistic Regression Analysis Results for Predicting Well-Adjusted Group vs. Very Virtuous Group (n = 160) Table 7. Bivariate Correlations Between MMPI-2-RF Restructured Clinical Scales and Conceptually-Relevant External Criteria (N s Range From 273 to 275) Table 8. Significance for College Sample Moderated Linear Regression Equations, Slope Differences, and Associated R 2 Values (N s Range From 273 to 275) iv

5 Acknowledgements I would like to thank my advisor, Dr. Yossef Ben-Porath for his continued support and vital contributions to my professional development. Additionally, I would like to thank Dr. John Gunstad, Dr. Beth Spitznagel, and Dr. Manfred van Dulmen for their feedback and support as part of my thesis committee. I would also like to thank my colleagues from the MMPI-2-RF research group: Bill Menton, Adam Hicks, Ryan Marek, Anthony Tarescavage, Jacob Finn, Katy Martin-Fernandez, and Jessica Tylicki for their feedback throughout this project. v

6 Introduction Self-report inventories are commonly administered during psychological evaluations in a variety of contexts. These inventories can be susceptible to various forms of response bias, and other forms of invalid responding, which can lead to inaccurate results and interpretations (Forbey, Lee, Ben-Porath, Arbisi, & Garland, 2013). The term validity, in reference to psychological tests, is typically used to describe the psychometric properties of test scores, which include construct, content, and criterion validity, all of which together comprise the term instrument validity. Another important type of validity associated with psychological tests is protocol validity, which refers to whether an individual test taker s approach to answering items on the test impacts the interpretability of the resulting test scores (Ben-Porath, 2012). Ben- Porath (2012) described two general forms of response bias that threaten the protocol validity of self-report inventories: non-content based- (NCIR) and content based invalid responding (CBIR). NCIR occurs in protocols where the test-taker either fails to respond, or engages in a pattern of random or fixed responding, whereas CBIR involves exaggeration or fabrication of symptoms or problems (overreporting), or minimizing or denying them (underreporting). Although both overreporting and underreporting have been investigated in various contexts, there is a dearth of research examining the direct impact underreporting has on psychological test scores. Historically, there have been various conceptualizations and terms used to describe minimization or denial of symptoms when responding to self-report inventories. Paulhus and Reid (1991) differentiatied between unintentional and intentional underreporting as occurring through either self-deceptive enhancement (SDE) or impression management (IM), terms 1

7 initially proposed by Block (1965) and Paulhus (1984), respectively. When an individual engages in SDE, he or she believes that the responses are accurate, whereas IM occurs when an individual knowingly represents himself or herself in an overly positive manner. Although the motivation behind these response styles is different, these two forms of underreporting can produce similar results of attenuated test scores, regardless of whether the individual intentionally or unintentionally underreports, thus making it inherently difficult to differentiate between these two forms of underreporting. Some of the terms used to describe this form of response bias include, but are not limited to underreporting, faking good, and socially desirable responding. Throughout this study, the term underreporting will be used for consistency and ease, and the present study focuses specifically on intentional underreporting (Impression Management). Underreporting is inherently more common in contexts in which evaluees have an incentive to present themselves in an especially positive light, including pre-surgical screening (Ambwani et al., 2013; Block, Ben-Porath, & Marek, 2013; Marek et al., 2015; Tarescavage, Wygant, Boutacoff, & Ben-Porath, 2013; Walfish, 2007), pre-employment evaluations (Lowmaster & Morey, 2012; Tarescavage, Corey, & Ben-Porath, 2014), and parental fitness and child custody evaluations (Archer, Hagan, Mason, Handel, & Archer, 2012; Sellbom & Bagby, 2008). Underreporting has been shown to attenuate scores on self-report measures of psychological dysfunction (Anastasi & Urbina, 1997; Forbey et al., 2013). Implications of attenuated scores in these settings include less than adequately-informed decisions when hiring new employees in the context of preemployment psychological evaluations, as well as misclassifying individuals as being at lower risk for poor surgical outcomes during presurgical evaluations. In light of these inherent consequences of underreporting in clinical and nonclinical 2

8 settings, it is recommended to administer measures that incorporate validity scales that assess for content based invalid responding (Ambwani et al., 2013; Forbey, Lee, Ben-Porath, Arbisi, & Gartland, 2013). Research Designs used to Investigate Response Styles A number of methods are used commonly to investigate various response styles. Rogers (2008) lists among these methods the known-groups comparison, differential prevalence design, bootstrapping comparisons, and simulation designs. Although the current study solely utilizes a simulation design, the differences between these various methods will be discussed, as will some of the strengths and limitations of each. In a known-groups design, groups classified in real-world conditions (e.g., groups that are known to differ on measures [e.g., response bias measures]) are compared. Known-groups comparisons have very strong external validity, because they are tested in contexts where they have real world incentives to underreport. However, this comes at the expense of internal validity due to the lack of control over assignment to the groups in an experimental manner, as well as often the absence of standardized methods available in controlled simulations. Differential prevalence designs utilize groups of individuals who are expected to have a specific response style based on the inherent nature of the evaluation, who are compared with groups of individuals who are not expected to have that specific response style (e.g., individuals undergoing child custody evaluations are expected to underreport to a greater extent than undergraduate college students). Similar to known-groups comparison, differential prevalence design has weak internal validity due to the aforementioned lack of control. However, this design has moderate external validity because individuals in these natural settings have potential 3

9 consequences for adoption of specific response styles (e.g., a child custody litigant may have a higher likelihood of achieving custody if he or she successfully underreports). Bootstrapping comparisons (Rogers & Bender, 2003) use multiple detection strategies associated with a specific response style to classify groups (i.e., stringent cut scores on several measures), which increases sensitivity to reduce false positives, while maintaining acceptable specificity. Bootstrapping comparisons have moderately strong external validity due to the realworld nature of the settings, participants, and incentives involved in this design. Again, internal validity is weaker because there is no researcher control over the assignment of groups, and the absence of standardized procedures. Simulation designs randomly assign individuals to various conditions where they will receive instructions to engage in a specific response style (i.e., underreporting, overreporting, random responding, standard instructions), and these groups are either compared with each other (e.g., underreporting vs. standard instructions) or compared with relevant clinical groups, depending on the nature of the study. Simulation designs have the strongest internal validity because the procedures are standardized, and they can include manipulation checks. Because these are simulations rather than real-word assessments, external validity is weaker; however incentives can, to some extent, be incorporated in these designs. MMPI-2-RF Validity Scales One family of broadband personality and psychopathology instruments that includes validity scales to assess for the various forms of invalid responding previously described is the Minnesota Multiphasic Personality Inventory family of instruments, which includes the MMPI (Hathaway & McKinley, 1943), MMPI-2 (Butcher et al., 2001), and the MMPI-2-Restructured Form (MMPI-2-RF; Ben-Porath & Tellegen, 2008/2011). Specifically, the MMPI instruments 4

10 contain measures of non content-based invalid responding (i.e., nonresponding, random responding, or fixed responding) and content-based invalid responding (i.e., underreporting, and overreporting) that assess for protocol validity. The MMPI-2 Restructured Form (MMPI-2-RF; Ben-Porath & Tellegen, 2008/2011) is a theoretically and empirically derived revision of the MMPI-2 that contains revised versions of the nine MMPI-2 validity scales developed to assess the protocol validity of the responses provided by the test taker to determine whether the substantive scale scores should be considered valid, invalid, or should be interpreted with caution (Tellegen & Ben-Porath, 2008/2011). Specifically, the MMPI-2-RF includes two Validity Scales designed to detect non-content based invalid responding: Variable Response Inconsistency Scale (VRIN-r) and True Response Inconsistency Scale (TRIN-r). These two scales use item pairs to determine whether an individual is engaging in random responding (VRIN-r) based on inconsistent responses to item pairs, or fixed responding (TRIN-r) based on acquiescent or counter-acquiescent responding. Additionally, the MMPI-2-RF contains five over-reporting Validity Scales: Infrequent Responses (F-r) scale, Infrequent Psychopathology Responses (Fp-r) scale, Infrequent Somatic Responses (Fs) scale (Wygant, Ben-Porath, & Arbisi, 2004), Symptom Validity (FBS-r) scale, and Response Bias Scale (RBS; Gervais et al., 2008). F-r is a measure of infrequent responding, and uses items that are rarely endorsed by the normative sample. Fp-r is a measure of infrequent responding, which uses items that are not frequently endorsed by psychiatric inpatients, and serves as a measure that is more robust to confounds of legitimate psychopathology. Fs was developed by Wygant, Ben-Porath, and Arbisi (2004) to detect somatic overreporting using items that were infrequently endorsed in medical samples, with the idea that individuals who engage in somatic overreporting may report multiple uncommon symptoms. FBS-r is a measure designed 5

11 to detect noncredible somatic and cognitive complaints among personal injury litigants thought to be malingering. RBS was developed by Gervais and colleagues (2008) as a measure to differentiate individuals who passed and individuals who failed performance validity tests. Relevant to the current investigation, the MMPI-2-RF underreporting validity scales were developed using factor analyses of items from three MMPI-2 underreporting scales (L, K, and S), in addition to items from the Positive Malingering Scale (Mp) and the Wiggins (1959) Social Desirability Scale (Sd). These additional items from Mp and Sd were included after Baer and Miller s (2002) meta-analyses, in which they found that Mp and Sd added incrementally to L, K, and S in the detection of underreporting. These factor analyses were replicated in a variety of samples, resulting consistently in a two-factor solution. The resulting two, non-overlapping scales were the Uncommon Virtues (L-r) scale and the Adjustment Validity (K-r-) scale. L-r contains 14 items that assess the extent to which individuals present themselves in an uncommonly favorable light, denying shortcomings and minor faults acknowledged by most people. K-r comprises 14 items that assess the extent to which individuals present themselves as being well-adjusted, with higher scores indicating remarkable levels of adjustment. However, it is noted in the interpretive manual that high scores on K-r may also reflect individuals who have legitimately higher levels of adjustment (Ben-Porath & Tellegen, 2008/2011). Relatively few studies have examined the utility of L-r and K-r in the detection of underreporting (Detrick & Chibnall, 2014; Sellbom & Bagby, 2008;). Using a differential prevalence design, Detrick and Chibnall (2014) examined a group of applicants seeking employment as police officers, who were administered the MMPI-2-RF once during the preemployment psychological evaluation (high demand condition) and once after completing the police academy (low demand condition), with the notion that the individuals would be more 6

12 likely to underreport under the high demand condition because of the potential consequences of answering honestly. They found that individuals scored significantly higher on the underreporting Validity Scales (i.e., L-r and K-r) during the high demand condition (i.e., during pre-employment evaluations) than they did during the low demand condition (i.e., after graduating from the academy). Additionally, they found that underreporting most notably impacted MMPI-2-RF scores on the EID (Emotional/Internalizing Dysfunction) and BXD (Behavioral/Externalizing Dysfunction) scales, as well as the lower-order scales associated within these domains. Finally, they calculated change scores (i.e., high demand scale scores minus low demand scale scores) for the underreporting Validity Scales and the substantive scales, using the Validity Scale change scores in regressions as predictors of substantive scale change scores. The results indicated that L-r predicted change in several behavior-focused scales and K-r predicted change in several emotional-focused scales, which taken together indicates that underreporting has a significant impact on substantive scale scores. Using a combination of a simulation and differential prevalence design, Sellbom and Bagby (2008) found that L-r and K-r were able to differentiate both college students and psychiatric inpatients with schizophrenia who were instructed to underreport on the MMPI-2-RF from college students given standard instructions for the measure, with higher L-r and K-r scores for the underreporting groups compared to the standard instruction group. In several studies examining police officer applicants, individuals undergoing employment pre-screening evaluations in a police department exhibited higher scores on K-r and L-r and lower mean scores on most substantive scales when compared with the general population (Tarescavage, Brewster, Corey, & Ben-Porath, 2014; Tarescavage, Corey, & Ben- 7

13 Porath, 2014; Tarescavage, Corey, Gupton, & Ben-Porath, 2015). These findings across studies are consistent with the expectations for this type of evaluation. Validity Scales as Moderators and Suppressors of Associations between Substantive Scales and Criterion Measures Another method for determining the utility of validity scales is to assess whether scores on these scales moderate associations between substantive scale scores and relevant external criterion measures, with the notion that at higher values of the validity scales, correlations would be lower between the substantive scales and external criterion. Because of null findings in specific validity scale moderation studies, and generally inconsistent findings across validity scale studies, it has been argued that validity scales do not moderate these associations, and therefore use of these scales is not justifiable (Li & Bagger, 2006; McGrath, Mitchell, Kim, & Hough, 2010). McGrath and colleagues (2010) conducted a literature review of 41 studies that used response bias indicators as either moderators or suppressors, finding weak support for validity scale scores moderating associations between psychopathology indicators and related criterion measures. Specifically, the authors concluded that because there are inconsistent findings in studies examining the degree to which response bias indicators (i.e., validity scales) enhance the predictive validity of a substantive measure, through moderation or suppression, these indicators may not be sufficient to use in applied settings. Li and Bagger (2006) conducted a meta-analysis of studies to determine whether intentional or unintentional underreporting (i.e., impression management and self-deception, respectively) created spurious effects on the associations between scores on personality measures and external criterion (e.g., work performance). Similar to McGrath and colleagues (2010), the 8

14 results of this meta-analysis did not support social desirability (neither impression management nor self-deception) as suppressors for the associations between measures of personality and performance. However, there were significant limitations to studies included in each of these reviews, including the use of outdated measures (e.g., California Psychological Inventory [CPI]; Gough, 1957), the use of the big five model of normal personality as a predictor (rather than a model of pathological personality), range restriction within the samples, and the review included samples where individuals had little or no motivation/incentive to underreport. In response to McGrath and colleagues (2010) claims, Rohling and colleagues (2011) identified major problems with the review, including the narrow focus of articles included in the review, misinterpretation of results, and the authors disregard for evidence that confirmed validity scale moderation effects. Morey (2012) agreed with Rohling and collegues response to the article, and argued that the limited power of the studies included in the review increased the probability of Type II error, thus providing an explanation for the lack of significant findings in these studies. Due to the inconsistent findings across studies, and because these reviews had noteworthy limitations, it is necessary to further examine the utility of validity scales as moderators in research designs and samples where underreporting is prevalent (i.e., simulation studies and real-world data sets). The Current Study The current study is intended to examine the impact of underreporting on MMPI-2-RF substantive scale score validity, investigate the discriminant validity of the underreporting scales in differentiating various forms of simulated underreporting, and examine the moderating effects 9

15 of the MMPI-2-RF underreporting Validity Scale scores on substantive scale score criterion validity. Six hypotheses were tested in the current investigation: First, we hypothesized that the group given instructions to appear very virtuous will have significantly higher scores on L-r than both the group given instructions to appear well-adjusted psychologically and the group given standard instructions. Similarly, the group given instructions to appear well-adjusted psychologically will have significantly higher scores on K-r than both the group given instructions to appear very virtuous and the group given standard instructions. Second, we hypothesized that the two simulation groups (underreporters) will have significantly lower mean scores across the majority of MMPI-2-RF substantive scales (with the exception of the Suicidal/Death Ideation (SUI) Scale, given the range restriction for this scale expected in the current sample), as well as the Interest Scales [Aesthetic-Literary Interests (AES) and Mechanical-Physical Interests (MEC)], given the content of these scales and the unlikelihood of motivation to underreport on these items) than will members of a control group (those given standard instructions). Third, we hypothesized that instructions to appear either very virtuous or well-adjusted psychologically will attenuate correlations between MMPI-2-RF substantive scale scores and criterion measures associated with those scales. That is, correlations between scores on the MMPI-2-RF substantive scales and scores on associated criterion measures will be significantly lower for underreporters than for those given standard instructions for the MMPI-2-RF. 10

16 Fourth, we hypothesized that L-r will add incrementally to K-r in predicting whether individuals belong to the group given instructions to appear very virtuous or the control group. Fifth, we hypothesized that K-r will add incrementally to L-r in predicting whether individuals belong to the group given instructions to appear well-adjusted psychologically or the control group. Finally, we hypothesized that the scores on the MMPI-2-RF underreporting Validity Scales (L-r and K-r) will moderate the associations between MMPI-2-RF substantive scales and relevant criterion measures. 11

17 Methodology Participants Participants were 432 male and female college students enrolled in general psychology courses at a large, Midwestern university in the United States. Individuals who produced noncontent-based invalid MMPI-2-RF protocols (Cannot Say 18 and/or VRIN-r T 80 and/or TRIN-r T 80 and/or who were unable to identify what instructions they were given for completing the MMPI-2-RF and/or reported they did not follow their instructions for completing the MMPI-2-RF (based on answers to the three manipulation check questions found in Appendix A) were excluded from the study. The final sample included 275 students (210 female, 65 male) with an average age of 18.7 years (SD = 2.0) and 12.4 years of education (SD = 0.7). The sample was predominantly Caucasian (84%), as well as 14% African-American, and 2% other. Participants who were excluded did not differ on age, sex, race, or years of education from those who were included. Participants were randomly assigned to one of three groups. The three groups did not differ on sex, age, or race, but the Well-Adjusted (see below) group was slightly less educated (M = 12.3, SD =.56) than the Very Virtuous group (M = 12.5, SD =.87), Hedge s g =.27. Measures Minnesota Multiphasic Personality Inventory-2 Restructured Form (MMPI-2-RF; Ben-Porath & Tellegen, 2008/2011). The MMPI-2-RF is a 338 item restructured version of the MMPI-2. The MMPI-2-RF contains 51 scales organized in a higher order fashion congruent with emerging models of personality and psychopathology (Kotov et al., 2011; Markon, Krueger, 12

18 & Watson, 2005; Sellbom, Ben-Porath, & Bagby, 2008; Watson, 2005). These scales include nine Validity Scales, three Higher-Order scales, nine Restructured Clinical scales, five Somatic/Cognitive Specific Problem scales, nine Internalizing Specific Problem scales, four Externalizing Specific Problem scales, five Interpersonal Specific Problem scales, two Interest scales, and five Personality Psychopathology Five (PSY-5) scales. Psychometric data, including normative scores, reliability estimates, and validity coefficients, are available in the MMPI-2-RF Technical Manual (Tellegen & Ben-Porath, 2008/2011). Antisocial Behavior Questionnaire (ABQ; Sellbom & Verona, 2004). The ABQ is a 16-item self-report assessment of delinquent behavior. Sellbom and Verona (2004) modified the measure from a previous self-report delinquency questionnaire (Hirschi, Hindelang, & Weis, 1980; Lynam, Whiteside, & Jones, 1999). The ABQ is highly correlated with the total score on the Psychopathic Personality Inventory (PPI; Lilienfeld & Andrews, 1996) and the subscale Impulsive Antisociality (Sellbom & Verona, 2004). Internal consistency of a =.81 was found by Sellbom (2007) and in the current study we found internal consistency of a =.72. Alcohol Use Disorders Identification Test (AUDIT; Babor, Higgins-Biddle, Saunders, & Monteiro, 2001). The AUDIT is a 10-item screening measure that assesses excessive drinking and has been validated among samples of university students, drug users, primary care patients, and elderly hospital patients (Babor et al., 2001). It is correlated with other measures of alcohol abuse: the Michigan Alcohol Screening Test, r =.88 (Bohn, Babor, & Kranzler, 1995) and CAGE Questionnaire, r =.78 (Clements, 1998). The AUDIT exhibited good internal consistency in the current study (a =.82). Cognitive Difficulties Scale (CDS; McNair & Kahn, 1984). The CDS is a 39-item selfreport measure of memory problems. Items encompass specific memory impairments using a 5-13

19 point Likert-type scale (with 0 = not at all and 4 = very often). The CDS exhibited good internal consistency in the current study (a =.91). Derouesné and colleagues (1993) found that the CDS was positively correlated with neuropsychological measures. Drug Abuse Screening Test 20-Item Version (DAST-20; Skinner, 1982). The DAST is a self-report screening tool developed to assess for problematic drug use. Responses to items are in a Yes / No format. Yudko et al. (2007) found that the DAST-20 correlated with other measures of problematic drug use, including the Clinician Rating Scale for Drug Use (r =.40), and the Addiction Severity Index Drug Composite Score (r =.42) and Alcohol Composite Score (r =.33). Cocco and Carey (1998) reported internal consistency for DAST-20 of α =.78, and the DAST-20 exhibited good internal consistency in the present study (a =.77). Emotionality, Activity, Sociability, Impulsivity Temperament Survey II (EASI-II; Buss & Plomin, 1975). The EASI-II was designed to measure the four personality characteristics considered to be temperaments. The EASI-II consists of 20 items rated on a 5-point Likerttype scale, with 1 = a little and 5 = a lot. Emotionality assesses the intensity of individuals reactions to stimuli. Activity assesses the energy level reported by individuals. Sociability assesses the amount of affiliativeness an individual reports. Impulsivity assesses tendencies to react rapidly to environmental cues. The EASI-II exhibited questionable to good internal consistency for each factor in the current study (Emotionality, a =.80; Activity, a =.65; Sociability, a =.64; Impulsivity, a =.75). General Self-Efficacy Scale (GSE; Schwarzer & Jerusalem, 1995). The GSE is a scale consisting of 10 self-report items and was developed to assess self-efficacy. The scale uses a 4- point Likert-type scale ranging from 1 = not at all true to 4 = exactly true. Corrected item-total correlations were high for the test within the United States (Chronbach s α =.87) and for other 14

20 countries (Chronbach s α ranged from.75 to.91) in a study that examined the universal nature of the construct of self-efficacy (Scholz, Doña, Sud, & Schwartzer, 2002). The GSE exhibited good internal consistency in the present study (a =.82). Rosenberg Self-Esteem Scale (SES; Rosenberg, 1989). The SES is a 10-item self-report measure assessing to what extent people believe they are good enough. Items are rated on a 4- point Likert-type scale, which includes strongly agree, agree, disagree, and strongly disagree. The SES exhibited good internal consistency in the present study (a =.90). Symptom Checklist 90 Revised (SCL-90-R; Derogatis, 1994). The SCL-90-R includes 90-items designed to assess a variety of psychopathological symptoms across nine primary domains of mental health problems, as well as three global indices of psychopathology. Items are rated on a 5-point Likert-type scale with low scores indicating the individual is not at all distressed by a symptom and high scores indicating the individual is extremely distressed by the symptom. The nine dimensions of this measure are highly correlated with MMPI Clinical Scales that are conceptually similar, and reliability and validity information is available in the technical manual (Derogatis, 1994). Post-Study Questionnaire (See Appendix A). The post-study questionnaire used in the current study was a three-item questionnaire that was administered after the participants completed the study. The questionnaire included the following items: What instructions did you receive for completing the MMPI-2-RF?, I followed the instructions, I got tired and bored and answered some of the questions without reading them carefully. The criteria for exclusion based on the answers to these questions is described in the following procedures. Procedures 15

21 Participants were tested in small groups led by trained research assistants. First, participants were administered all the criterion measures under standard instructions. Once participants completed all criterion measures, each was randomly assigned to one of three groups: (1) completing the MMPI-2-RF under standard instructions (Standard Instructions Group; original n = 146; final n = 115), (2) completing the MMPI-2-RF under instructions to appear very virtuous to get a highly desirable job (Very Virtuous Group; original n = 149; final n = 81), or (3) completing the MMPI-2-RF under instructions to appear very well-adjusted psychologically to get a highly desirable job (Well-Adjusted Group; original n = 137; final n = 79). Appendix B includes the scripts that were displayed on the screen (and played over the headphones) to participants. After completing the MMPI-2-RF, participants were administered the post-study questionnaire, which included the three manipulation check items used in the current study: What instructions did you receive for completing the MMPI-2-RF?, I followed the instructions, I got tired and bored and answered some of the questions without reading them carefully. If participants answered the first item correctly based on the instructions they received, answered the second item with most of the time or all of the time, and answered the third item with less than half of the time or not at all, they were considered compliant and were included in the analyses. Participants who did not meet any single one of these criteria were excluded from further analyses. These exclusions were based on recommendations from previous research in which significant differences were found between underreporting simulators who passed the manipulation check and those who failed in a study examining the utility of incorporating a manipulation check in simulated underreporting studies (Crighton, Marek, Dragon, & Ben-Porath, 2016). 16

22 Results MMPI-2-RF Scales Analyses of Variance (ANOVAs) were conducted to examine group differences (Standard Instructions, Very Virtuous, and Well-Adjusted) on the MMPI-2-RF scales. Hedge s g effect size estimates were calculated to correct for unequal sample sizes for comparisons between the Standard Instructions group and the Very Virtuous group, between the Standard Instructions group and the Well-Adjusted group, as well as between the Very Virtuous group and the Well-Adjusted group, where values of.20,.50, and.80 reflect small, medium, and large effect sizes, respectively (Hedges & Olkin, 1985). Table 1 includes mean profiles and standard deviations for the three groups, information about which group mean scores were statistically different, and associated effect sizes. For the Validity Scales, univariate F tests indicated significant group differences for Variable Response Inconsistency (VRIN-r), Infrequent Responses (F-r), Infrequent Psychopathology Responses (Fp-r), Infrequent Somatic Responses (Fs), Symptom Validity (FBS-r), Response Bias Scale (RBS), Uncommon Virtues (L-r), and Adjustment Validity (K-r), with the Standard Instruction group scoring lower on the overreporting Validity Scales, and higher on the underreporting Validity Scales than the two underreporting groups, which can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on L-r (g = -1.20) and K-r (g = -1.11), and the Standard Instructions group versus the Well-Adjusted group on L-r (g = -1.40) and K-r (g = -1.22). 17

23 For the Higher Order Scales, univariate F tests indicated significant group differences for two of the scales (Emotional/Internalizing Dysfunction [EID] and Behavioral/Externalizing Dysfunction [BXD]), but not for Thought Dysfunction (THD), with all differences in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups, which can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on EID (g = 1.11), and the Standard Instructions group versus the Well-Adjusted group on EID (g = 1.40) and BXD (g =.96). For the RC Scales, univariate F tests were calculated, indicating statistically significant group differences for all nine RC Scales (Demoralization [RCd], Somatic Complaints [RC1], Low Positive Emotions [RC2], Cynicism [RC3], Antisocial Behavior [RC4], Ideas of Persecution [RC6], Dysfunctional Negative Emotions [RC7], Aberrant Experiences [RC8], and Hypomanic Activation [RC9]), all in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups, which can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on RCd (g = 1.06) and RC7 (g =.96), and the Standard Instructions group versus the Well-Adjusted group on RC4 (g = 1.18), RCd (g = 1.17), RC7 (g = 1.10), RC2 (g =.93), RC1 (g =.85), and RC6 (g =.83). For the Somatic/Cognitive SP Scales, univariate F tests indicated statistically significant group differences for all five scales (Malaise [MLS], Head Pain Complaints [HPC], Neurological Complaints [NUC], and Cognitive Complaints [COG]), all in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups. These results can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on MLS (g =.89), and the Standard 18

24 Instructions group versus the Well-Adjusted group on HPC (g = 1.03), MLS (g =.89), and NUC (g =.81). For the nine Internalizing SP Scales, univariate F tests indicated statistically significant group differences for seven of the scales(suicidal/death Ideation [SUI], Helplessness/Hopelessness [HLP], Self-Doubt [SFD], Inefficacy [NFC], Stress/Worry [STW], Anxiety [AXY], and Anger Proneness [ANP]), with no differences on Behavior Restricting Fears (BRF) and Multiple Specific Fears (MSF), and all differences in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups. These results can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on NFC (g = 1.02), STW (g =.93), and SFD (g =.90), and the Standard Instructions group versus the Well-Adjusted group on STW (g = 1.03), SFD (g = 1.17), NFC (g =.92), ANP (g =.87), and HLP (g =.83). For the Externalizing SP Scales, univariate F tests were calculated, indicating statistically significant group differences for three of the scales (Juvenile Conduct Problems [JCP], Substance Abuse [SUB], and Aggression [AGG]), with no differences on Activation (ACT), and all differences in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups. These results can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on AGG (g =.64), and the Standard Instructions group versus the Well-Adjusted group on AGG (g =.68). For the five Interpersonal SP Scales, univariate F tests were calculated, indicating statistically significant group differences on three of the scales (Family Problems [FML], Social Avoidance [SAV], and Shyness [SHY]), with no differences on Interpersonal Passivity (IPP) or 19

25 Disaffiliativeness (DSF), all in the expected direction of the Standard Instructions group scoring lower than the two underreporting groups. These results can be found in Table 1. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on FML (g =.77), and the Standard Instructions group versus the Well-Adjusted group on FML (g = 1.11). For the two Interest Scales, univariate F tests indicated statistically significant group differences on Mechanical-Physical Interests (MEC), with the underreporting groups scoring lower than Standard Instructions group, but there were no differences on Aesthetic-Literary Interests (AES). Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on MEC (g = -.66), and the Standard Instructions group versus the Well-Adjusted group on MEC (g = -.41). For the five PSY-5 Scales, univariate F tests indicated statistically significant group differences on four of the scales (Psychoticism [PSYC-r], Disconstraint [DISC-r], Negative Emotionality/Neuroticism [NEGE-r], and Introversion/Low Positive Emotionality [INTR-r]), with no differences on Aggressiveness (AGGR-r), with all differences in the expected direction of the underreporting groups scoring lower than the Standard Instructions group. Effect size estimates indicated the largest effects for the Standard Instructions group versus the Very Virtuous group on NEGE-r (g = 1.02), and the Standard Instructions group versus the Well- Adjusted group on NEGE-r (g = 1.04). In addition to these comparisons, descriptive statistics are presented for each of the three groups on the criterion measures, all of which can be found in Table 2. Criterion Validity Correlational Analyses 20

26 To examine the effect of underreporting on criterion validity estimates, correlations were calculated between MMPI-2-RF scales and criterion measures for the three groups, which can be found in Table 3. Criterion measures that were hypothesized to be conceptually related to each scale of the MMPI-2-RF were denoted using a or a (-). Included in the table are the bivariate correlations for all three groups. Cohen s q (Cohen, 1977), which is an effect size for comparing correlations, was computed to compare correlations between MMPI-2-RF scales and conceptually-relevant criterion measures for (a) the Standard Instructions group versus the Very Virtuous group, and (b) the Standard Instructions group versus the Well-Adjusted group. Cohen (1977) suggests interpretive guideline for Cohen s q as follows: values between.10 and.29 indicate the difference between two correlations has a small effect; values between.30 and.49 indicate a medium effect; and values at or above.50 indicate a large effect. Although Table 3 includes results for all possible bivariate correlations, only associations that were hypothesized to be different between the MMPI-2-RF and external criterion measures will be interpreted (indicated by /(-)). It was hypothesized that correlations between the MMPI-2-RF substantive scales and relevant criterion measures would be significantly stronger in the Standard Instruction group than in either of the underreporting groups. Differences were found for the majority of the hypothesized associations, with some of the most notable examples being between RCd (Demoralization) and the SCL-90-R Depression scale. RCd was highly associated with the Depression scale for the Standard Instructions group (r =.77), which indicates that these two scales measure similar constructs (i.e., demoralization/distress). However, these same scales were associated at r =.36 for the Very Virtuous group and at r = -.01 for the Well-Adjusted group, with large effect size estimates associated with each comparison. Another notable 21

27 example was the correlation between RC1 (Somatic Complaints) and the SCL-90-R Somatization scale. As expected, RC1 was highly associated with the Somatization scale for the Standard Instructions group (r =.61), indicating that these two scales measure similar constructs. However, these same scales were associated at r =.16 for the Very Virtuous group and at r = -.03 for the Well-Adjusted group, with large effect size estimates associated with each comparison. Whereas other correlations were not generally as clearly different as these examples, the results indicated that overall correlations were attenuated substantially by underreporting. Absolute Median Correlations The last two rows in Table 3 include the absolute median correlations for (a) expected associations between each of the 41 substantive MMPI-2-RF scales and criteria and (b) associations between MMPI-2-RF scales and criteria that were significantly correlated.30. The latter were calculated because of uncertainty about the accuracy of the hypothesized associations. These absolute median correlations provide a summary of differences in correlations between the Standard Instructions group and each of the underreporting groups for each substantive MMPI-2-RF scale with (a) conceptually-relevant and (b) statistically-related criterion measures. These analyses were conducted to determine the proportion of the 42 substantive MMPI-2-RF scales that had median correlations that were significantly different between the Standard Instructions group and the underreporting groups. For the median correlations of expected relationships, there were 8 (23.81%) scales with medium and zero scales with large effects for median correlation differences between the Standard Instructions group and the Very Virtuous group. When comparing the Standard Instructions group and the Well-Adjusted group, there were 13 (30.95%) scales that had medium effects and an additional 5 (11.90%) scales had large effects. When examining median 22

28 correlations for those correlations that were significant at or above.30, There were 6 (14.29%) scales with medium and zero scales with large effects when comparing the Standard Instructions group and the Very Virtuous group. When comparing the Standard Instructions group and the Well-Adjusted group, there were 19 (45.24%) scales that had medium effects and an additional 3 (7.14%) scales had large effects. Incremental Predictive Utility of Underreporting Validity Scales Hierarchical logistic regression analyses were conducted to determine the incremental utility of the MMPI-2-RF underreporting Validity Scales (L-r and K-r) in predicting whether individuals were given standard instructions or instructions to underreport (analyses were conducted for each underreporting group), as well as in predicting whether individuals were given instructions to appear very virtuous or instructions to appear well-adjusted. Log-likelihood ratio tests were used to compare model fit. Omega prime (ω ) effect size estimates were calculated based on the Chi-square statistic to determine the variance accounted for in the models, where.10 is a small effect,.30 is a medium effect, and.50 is a large effect (Cohen, 1988). The results of these analyses can be found in Tables 4-6. Regarding the prediction of whether individuals were in the Standard Instructions group or the Very Virtuous group, these analyses indicated a large effect for both L-r (ω =.54) and K-r (ω =.50), and a small effect for K-r adding incrementally to L-r (Δω =.16), and a small to medium effect for L-r adding incrementally to K-r (Δω =.26). Regarding the prediction of whether individuals were in the Standard Instructions group or the Well-Adjusted group, these analyses indicated a large effect for both L-r (ω =.61) and K- 23

29 r (ω =.54), and a small effect for K-r adding incrementally to L-r (Δω =.17), and a medium effect for L-r adding incrementally to K-r (Δω =.33). Finally, regarding the prediction of whether individuals were in the Very Virtuous group or the Well-Adjusted group, these analyses indicated virtually no effect for both L-r (ω =.01) and K-r (ω =.03), and K-r did not add incrementally to L-r (Δω =.05), and L-r did not add incrementally to K-r (Δω =.04). Moderation Analyses To examine moderation effects, hierarchical moderated multiple regression analyses were conducted using PROCESS for SPSS (Hayes, 2012), in which the MMPI-2-RF substantive scale, the MMPI-2-RF Underreporting validity scale (i.e., L-r or K-r), and an interaction between these scales were entered into the model. Incremental improvements in model fit indicated whether overall moderation was present and 95% Confidence Intervals were examined to determine whether there was any overlap in the slopes at high levels of the moderator versus lower levels of the moderation. After determining whether there were slope differences, the plotted moderation graphs were visually examined to determine whether the slope differences were in the hypothesized direction (i.e., higher validity scale scores were associated with lower criterion validity). To determine which moderation analyses would be conducted, all of the external criteria included in the study were examined first to decide which of these variables would be expected to be associated with any of the MMPI-2-RF RC Scales. Next, bivariate correlations between RC Scales and conceptually-associated criteria were examined, which can be found in Table 7. These correlations were examined in order to determine whether the criteria expected to be conceptually associated to the RC scales were statistically correlated. Because bivariate 24

30 associations are necessary for meaningful moderations to be examined, no moderated regression analyses on any RC Scale/Criterion Measure pairs with correlations weaker than.20 were conducted, because this is magnitude of correlation is commonly used as a benchmark for practically significant associations in the MMPI literature. After pairings of RC Scale/Criterion Measure with associations weaker than.20 were removed, moderation analyses were conducted to examine whether L-r and K-r moderated associations between the remaining RC Scale/Criterion Measure pairings. Next, these analyses were examined to determine whether there was an overall moderation effect, as well as whether there were slope differences (and if these differences were in the expected direction of weaker associations between the RF Scale and Criterion Measure pairs for higher scores [1 SD above the mean] on L-r and K-r compared to lower scores [1 SD below the mean] on these underreporting Validity Scales), and the change in R 2, all of which can be found in Table 8. Overall, 78 moderated regressions were examined, with 15 (19.23%) found to be statistically significant at a conservative Bonferroni-corrected alpha of.05/39 (39 moderations conducted for each of the two MMPI-2-RF underreporting Validity Scales) = L-r significantly moderated two of 39 (5.13%) regressions, whereas K-r significantly moderated 13 of 39 (33.33%) regressions, with all of these overall moderation effects including significant slope differences in the expected direction, with lower scores on L-r and K-r exhibiting stronger slopes than higher scores on L-r and K-r. Across all of these moderated regressions, the average change in variance explained (ΔR 2 ) was.0196, indicating that an average of 1.96% of incremental variability was accounted for when the moderator was added. When only statistically significant moderations were examined, the average change in variance explained increased to 4.8%. 25

31 In addition to examining the average change in all moderations, average change in variance explained (ΔR 2 ) was examined for L-r and K-r separately. For L-r, the average change in variance explained was 1.49%, and increased to 5.00% when only significant moderations were examined. For K-r, the average change in variance explained was 2.5%, and increased to 4.69% when only significant moderations were examined. 26

32 Discussion The purpose of this study was to examine the impact of underreporting on MMPI-2-RF substantive scale scores, investigate the discriminant validity of the underreporting scales in differentiating various forms of simulated underreporting, and examine the moderating effects of the MMPI-2-RF underreporting Validity Scale scores on substantive scale score criterion validity. An experimental simulation design was utilized in which participants were administered several criterion measures under standard instructions and were then randomly assigned to one of three sets of instructions for completing the MMPI-2-RF: underreporting instructions to appear very virtuous, underreporting instructions to appear well-adjusted psychologically, or standard instructions. The first hypothesis that the group given instructions to appear very virtuous would have significantly higher scores on L-r than both the group given instructions to appear well-adjusted psychologically and the group given standard instructions was partially supported. Specifically, the Very Virtuous group had significantly higher scores on L-r than the Standard Instructions group (Hedge s g = -1.20), but there was no difference between the Very Virtuous group and the Well-Adjusted group (Hedge s g =.02). Similarly, the second part of the first hypothesis that the group given instructions to appear well-adjusted psychologically would have significantly higher scores on K-r than both the group given instructions to appear very virtuous and the group given standard instructions yielded similar results. The Well-Adjusted group had significantly higher scores on K-r than the Standard Instructions group (Hedge s g = -1.22), but there was no difference between the Very Virtuous group and the Well-Adjusted group (Hedge s g = -.05). 27