Improving Accuracy in the. through the use of Technology

Size: px

Start display at page:

Download "Improving Accuracy in the. through the use of Technology"

Darrell Evans
6 years ago
Views:

1 Improving Accuracy in the Assessment of Malingering through the use of Technology Lisa Drago Piechowski, PhD, ABPP American School of Professional Psychology, Washington DC Key Points Assessment of malingering is complex Objective assessment methods are generally more accurate than decisions based on clinical judgment. Computer administration of testing saves time, allows the collection of additional data, and reduces error. Computer scoring of testing saves time, significantly reduces error, and allows for more complex data analyses. Computer based narrative interpretation has not been found to improve accuracy or save time. 1

2 Understanding Malingering Reported functioning Actual functioning 2

3 Dimensions of dissimulation Unintentional Intentional Positive Presentation Negative Presentation Lack of insight, narcissism, naiveté, unconscious denial, excessive optimism Pessimism, poor self image, hopelessness Symptom denial, exaggeration of positive qualities Symptom exaggeration, malingering, attempt to prove case, factitious disorder Assessment of Malingering Psychopathology: create a believable set of symptoms, a likely onset and course of the disorder, and plausible effects on their day to day functioning Cognitive Dysfunction: effortful failure : while appearing to try hard, incorrect responses given, the pattern of which are consistent with known patterns of cognitive dysfunction. 3

4 Caveat: Malingering can co exist with valid symptoms Individuals can have one disorder and malinger in the way of faking or exaggerating symptoms of other disorders. Individuals can have a condition and exaggerate or fake the symptoms of that condition, the persistence of symptoms of that t condition, or the impact of those symptoms on their functioning. Dimensions of Assessment 4

5 Detecting Malingering Method tool Norms basis of comparison Decision rule how the tool is used Norms Ideal method (Known Groups): Give the test tto a group of honest people Give the test to a group of malingerers Compare the two groups More common method (Simulators): Give the test to a group of college students Give the test to another group of college students and ask them to act like malingers Compare the two groups 5

6 Sensitivity & Specificity Example: Airport Security Metal detector goes off Metal detector does not go off Person has weapon True positive False negative Person has change in pocket False positive True negative Sensitivity If sensitivity is high, metal detector goes off for all metal objects. That way, no weapons are missed, but a lot of people who don t have weapons get stopped. Person has weapon Person has change in pocket Metal detector True positive False positive goes off Metal detector does not go off. False negative True negative 6

7 Specificity If specificity is high, the metal detector goes off only for large heavy metal objects. That way, no law abiding citizens get stopped, but some weapons will not be detected. Person has weapon Person has change in pocket Metal detector True positive False positive goes off Metal detector does not go off. False negative True negative And what does this mean? Tests of malingering are only as good as the decision rules that areused used. If the cut score is too low (high sensitivity), honest responders will be classified as malingerers. If the cut score is too high (high specificity), many malingerers will be missed. Decision rules are only as good as the method by which they were derived. 7

8 Questions to ask How sensitive is the test? How likely l is it that t an honest responder will be incorrectly identified as malingering? How specific is the test? How likely is it that a malingerer will be missed? What is the classification rate? What percentage of the individuals tested are correctly identified as malingering or not malingering? Methods of detecting malingering Clinical (examples) Embedded Scales (examples) Direct observation MMPI 2/MMPI 2 RF VIP Stand Alone Tests (examples) Cross validation with external sources PAI VSVT Red flags MCMI III TOMM TSI WMT SIRS 8

9 Clinical Methods STRENGTHS Allows incorporation of data from variety of sources. Focuses on individual differences/circumstances WEAKNESSES Not standardized Requires subjective interpretation No decision rules Susceptible to confirmation bias Less accurate than objective, empirically based methods. Embedded Scales STRENGTHS Does not require administration of additional tests. Different scales can assess different aspects of dissimulation. WEAKNESSES Scales vary greatly in terms of accuracy of detection. Not all scales designed for psycho legal contexts. Individual scales may contain relatively few items. Standardized May not have firmly established decision rules. 9

10 Stand Alone Tests STRENGTHS Designed specifically for detection of malingering in psycho legal contexts. Standardized. Many have sophisticated decision rules based on item difficulty and pattern of responses. WEAKNESSES Requires administration of additional tests. Each test only measures one aspect of dissimulation. Standardization may be based on simulators rather than known groups. One method or several methods? Research supported aggregating results of multiple measures for the detection of malingering. Failing 2 SVT s = strong evidence of probable malingering. Failing 3 SVT s = very strong evidence of probable bbl or dfiit definite malingering i From Larrabee (2008), Clin Neuropsychologist, 22 (4),

11 Similar methods or different methods? High correlations among the raw scores of multiple measures of malingering, but also significant discrepancies. Concluded certain measures may more accurately detect certain types of distorted performance in a given set of circumstances or with certain types ofexaminees. From Farkas et al. (2006), Behav Sci & Law, 24 (5), Improving Accuracy 11

12 What s needed to improve accuracy? Use of objective, empirically based methods. Reduction of human error. Time to collect more and varied data. Interpretation of data based on empiricallyderived decision rules. Use of Computer Based Administration & Scoring Research has demonstrated that computeradministered tests essentially equivalent to paper and pencil tests. Administration is standardized reducing possible errors. Requires less time for administration, scoring, and interpretation, allowing time for collection of additional data. Examinees less likely to make errors in recording responses. Improves accuracy of scoring. Allows seamless inclusion of stand alone measures with a test battery. 12

13 Hand scoring can increase scoring errors Clerical errors relatively common in hand scoring MMPI 2 profiles. (Iverson & Barton, 1999, J of Clin Psych, 55 (1), MMPI hand scoring and keyboarding errors as high as 34% in 1 setting. (Allard & Faust, 2000, Assessment, 7 (2), Hand scoring of psychological ltests t resulted in significant error rates. (Simons et al., 2002, Assessment, 9 (3), Hand scoring limits both amount & accuracy of data Hand scored MMPI 2 Profile Computer scored MMPI 2 Profile 13

Statistical rules based on empirical data? Conclusions based on developer s clinical experience?

14 Computer scoring allows for more sophisticated data analysis Example: Valid VIP Example: Invalid VIP Computer Generated Test Interpretation (CBTI) Suggest hypotheses, don t provide conclusions. What is basis for hypotheses? Statistical rules based on empirical data? Conclusions based on developer s clinical experience? Clinical judgment required to select relevant hypotheses and weigh evidence. Does not necessarily saveexaminer examiner time. Interpretations may not be appropriate for psycho legal contexts. 14

15 Questions to ask What method was used to generate each statement? tt t? Why were some statements incorporated into examiner s report and not others? How did the examiner weigh each hypothesis? Are the words used in the report those of the examiner or the author of the CBTI? In conclusion: What we know about improving accuracy Malingering can take many forms. Different forms require different detection strategies. All detection strategies are not created equal. Decision rules are based on classification rates how likely is it that the decision is incorrect? The use of multiple and varied detection strategies is likely to yield more accurate results. Computer based administration and scoring improves accuracy, maximizes time, and facilitates the collection of more data. Computer based narrative interpretation has not been demonstrated to improve accuracy. 15

The Albany Consistency Index for the Test of Memory Malingering

Archives of Clinical Neuropsychology 27 (2012) 1 9 The Albany Consistency Index for the Test of Memory Malingering Jessica H. Gunner 1, *, Andrea S. Miele 1, Julie K. Lynch 2, Robert J. McCaffrey 1,2 1