A Psychometric Evaluation of the Rorschach Comprehensive System s Perceptual Thinking Index

Transcription

1 PERCEPTUAL DAO AND THINKING PREVATTINDEX JOURNAL OF PERSONALITY ASSESSMENT, 86(2), Copyright 2006, Lawrence Erlbaum Associates, Inc. A Psychometric Evaluation of the Rorschach Comprehensive System s Perceptual Thinking Index Tam K. Dao and Frances Prevatt Department of Educational Psychology and Learning Systems Florida State University In this study, we investigated evidence for reliability and validity of the Perceptual Thinking Index (PTI; Exner, 2000a, 2000b) among an adult inpatient population. We conducted reliability and validity analyses on 107 patients who met the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision; American Psychiatric Association, 2000) criteria for a schizophrenia-spectrum disorder (SSD) or mood disorder with no psychotic features (MD). Results provided support for interrater reliability as well as internal consistency of the PTI. Furthermore, the PTI was an effective index in differentiating SSD patients from patients diagnosed with an MD. Finally, the PTI demonstrated adequate diagnostic statistics that can be useful in the classification of patients diagnosed with SSD and MD. We discuss methodological issues, implications for assessment practice, and directions for future research. Psychosis applies to a state of being (i.e., psychotic state) as well as distinct diagnostic entities (Courvoisie, Labellarte, & Riddle, 2001). In the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision [DSM IV TR]; American Psychiatric Association, 2000), a number of psychotic disorders are presented primarily associated with schizophrenia. Schizophrenia is both the most classic and most common psychotic disorder (American Psychiatric Association, 2004). According to an annual report by the Veterans Health Administration (2001), the population of patients with psychosis continues to rise, up from 1.9% in 1999 to 2.7% in 2000, with schizophrenia affecting approximately 1% (Abreu & Filips, 2001) to 1.5% (American Psychiatric Association, 2004) of the entire world population. Based on the DSM IV TR (American Psychiatric Association, 2000), psychosis can be defined as delusions or prominent hallucinations without insight into their pathological nature (p. 297). A criterion-based definition of psychosis would consist of other positive symptoms of schizophrenia such as disorganized speech and behavior. In addition, a conceptual definition of psychosis centers on the loss of ego boundaries or a gross impairment in reality testing (American Psychiatric Association, 2000, p. 297). Based on these definitions, it is not surprising that there are a number of disorders that fall under the general title of psychotic disorders (Kaplan & Sadock, 1998). The evaluation of psychosis utilizes a wide range of psychological tests (Kaplan & Sadock, 1998). The Rorschach Comprehensive System (RCS; Exner, 2003) has demonstrated some utility in discriminating between criterionbased diagnoses. For instance, a number of studies have found positive results regarding the ability of the RCS to differentiate individuals with psychotic versus nonpsychotic disorders (Hilsenroth, Fowler, & Padawer, 1998; Ilonen et al., 1999; Jorgensen, Anderson, & Dam, 2000; Meyer, 1993; Netter & Viglione, 1994). These results have led some investigators to conclude that the ability of the RCS to detect the bizarre and illogical processes often seen in schizophrenia is probably one of its best-validated features (Lilienfeld, Wood, & Garb, 2000; Vincent & Harman, 1991; Wood, Nezworski, & Garb, 2003). Previously, the most widely used criteria on the RCS for the evaluation of psychosis was the Schizophrenia Index (SCZI; Exner, 1993; Hilsenroth et al., 1998). The SCZI was developed to assist in the evaluation of schizophrenia and related disorders (Viglione, 1999), particularly in the four basic areas of inaccurate perception, disordered thinking, inadequate controls, and interpersonal ineptness (Hilsenroth et al., 1998). Although a number of past studies have indicated that the SCZI may be helpful in differentiating psychotic patients from other clinical groups (e.g., Ganellen, 1996a, 1996b; Hilsenroth et al., 1998; Ilonen et al., 1999), Exner (2000a, 2000b) has made further revisions with a new index, the Perceptual Thinking Index (PTI). According to Exner (2000a), when applying the cutoff value of 4, the SCZI routinely identifies between 65% and 80% of persons diagnosed as schizophrenic. However, the SCZI has also been shown to falsely identify approximately 10% to 20% of per-

2 PERCEPTUAL THINKING INDEX 181 sons with other relatively serious problems such as those suffering from a major affective disturbance. Exner (2000a) also noted that the false positive rate was substantial among preadolescent and adolescent populations. According to Exner (2000a), these factors prompted a series of new studies concerned with improving the validity of the SCZI, particularly its ability to identify persons who have cognitive disturbances. To date, the PTI has replaced the SCZI as the preferred index for assessing cognition prior to interpreting other variables that may be related to thought disturbance (Exner, 2000a). Although the SCZI has been shown to have a.94 correlation with the PTI (Smith, Baity, Knowles, & Hilsenroth, 2001), suggesting that previous findings on the SCZI will generalize to the PTI, additional data are still warranted on the reliability and validity of the PTI. The PTI is comprised of eight Rorschach variables that are arranged based on a combination of different values on five empirical criteria. It measures both perceptual oddities and cognitive slippage (Smith et al., 2001). Furthermore, the PTI contains two variables new to the RCS (Exner, 2003): XA% and WDA%. The new variable XA% is defined as the sum of all Form Quality plus (+), ordinary (o), and unusual (u) responses divided by R. The variable WDA% is calculated by dividing the sum of +, o, and u responses given to the W and D areas by the sum of all responses given to the W and D areas. Possible scores on the PTI range from 0 to 5. Exner (2000) did not report PTI cutoff scores for adult populations because he promoted a dimensional approach to interpreting the PTI. Current research on the PTI is limited. Preliminary research on patients with schizophrenia has indicated that the distribution of PTI scores does not appear to be markedly different from the distribution of SCZI scores. For instance, Exner (2000a) reported that in a group of 110 individuals having a DSM diagnosis of schizophrenia, 84 individuals had SCZI values of 4 or greater and 62 of those 84 had values of 5 or 6. In comparison, the distribution of PTI scores for the 110 individuals revealed that 61 had values of 4 or 5 and 22 had values of 3. A review of the literature produced two studies that have examined the validity of the PTI among children and adolescents and non-u.s. adult populations. Smith et al. (2001) investigated the relationship of Rorschach variables (PTI, SCZI, M, and X %) to thought disorder indexes of a behavior rating scale and a self-report measure among children and adolescents. Using a cutoff score of 3 (representing approximately 1 SD above the mean for the sample), Smith et al. found that the PTI differentiated between those patients with and without elevated thought disorder scores on the rating scale and self-report. In addition, the PTI, unlike the SCZI, significantly differentiated between patients with clinically significant symptoms on the parent rating scale. According to Smith et al., the differences in performance between the PTI and SCZI suggest that the PTI might be a more valid measure of thought disorder in children and adolescents than the SCZI. More recently, Ritsher (2004) investigated the relationships between the Rorschach and Minnesota Multiphasic Personality Inventory (MMPI; Hathaway & McKinley, 1943) and schizophrenia spectrum diagnoses in a Russian sample of 180 adult psychiatric patients. Ritsher found modest support for both the SCZI and PTI, but not the MMPI indicators (Sc, Sc 3,Sc 6, and BIZ), in detecting psychosis. Despite consistent findings regarding the ability of the SCZI to identify psychotic patients, researchers have cautioned against the use of some Rorschach variables due to issues of reliability and validity. Wood, Nezworkski, and Stejskal (1996) offered a number of methodological recommendations for research employing Rorschach variables and indexes (Hilsenroth et al., 1998): (a) consideration of interrater reliability of the various Comprehensive System scores in both ideal and field conditions, (b) using welldefined and rigorous diagnostic criteria (e.g., from semistructured interviews based on the DSM (4th ed. [DSM IV]; American Psychiatric Association, 1994) when examining the validity of RCS variables and indexes, (c) employing procedures that ensure that diagnosticians are blind to Rorschach results, (d) investigating ecological validity of the Rorschach scores, and (e) reporting measures of diagnostic performance. Based on recommendations by Wood et al. (1996), in this study, we investigated the psychometric properties of the Rorschach PTI in a sample of adult inpatients with DSM IV diagnoses of schizophrenia-spectrum disorders (SSD). A comparison group included adult inpatients with diagnoses of mood disorders without psychotic features (MD). In the study, we examined (a) the internal consistency of the PTI, (b) the ability of the PTI to differentiate patients diagnosed with SSD from MD, (c) the predictive validity of the PTI, and (d) the diagnostic performance of the PTI. Based on the established variables and criteria for the PTI (Exner, 2000a), XA% and WDA% were anticipated to be higher in the MD group because these two variables indicate conventional responses. Alternately, X %, Sum Level 2 Special Score, FABCOM2, WSUM 6, M %, PTI Criteria 1, PTI Criteria 2, PTI Criteria 3, PTI Criteria 4, PTI Criteria 5, and PTI Criteria 6 were anticipated to be higher in the SSD group because these variables indicate distorted perceptions or problematic thinking. Participants METHOD The sample consisted of 107 patients drawn from an archival search of files at a 60-bed inpatient psychiatric facility in an urban area in the southeastern United States. The aggregate sample ranged in age from 18 to 74 years (M = 32), and 41% of all participants were male (n = 44). As for ethnic composi-

3 182 DAO AND PREVATT tion, 67% were White (n = 72), 24% were African American (n = 26), 7% were Hispanic (n = 7), and 2% were Asian (n = 2). The SSD group ranged in age from 18 to 74 years (M = 34), and 48% were male (n = 24). The ethnic composition of the SSD group consisted of 70% White (n = 32), 20% African American (n = 9), 9% Hispanic (n = 4), and 2% Asian (n = 1). The MD group ranged in age from 18 to 72 years (M = 31), and 36% were male (n = 22). The ethnic composition of the SSD group consisted of 66% White (n = 40), 28% African American (n = 17), 5% Hispanic (n = 3), and 2% Asian (n = 1). To assess for potential confounding group demographic variables between the SSD (n = 46) and the MD (n = 61) groups, we performed chi-square tests on gender and ethnicity. For gender, the chi square test indicated no significant difference, χ 2 (1, N = 107) = 1.49, p >.05, two-tailed, with φ=.01. A chi-square test comparing White to others (i.e., African American, Hispanic, and Asian American) indicated no significant difference, χ 2 (1, N = 107) = 1.90, p >.66 (twotailed), with φ =.01. In regards to age, unpaired t tests indicated no significant difference, t(105) = 1.04, p >.05, twotailed, with d =.19. SSD and MD groups were formed based on primary admission diagnosis. Diagnoses of the SSD group consisted of schizophrenia (n = 32), schizophreniform (n = 1), schizoaffective disorder (n = 8), and psychotic disorder not otherwise specified (n = 5). Diagnoses of the MD group consisted of major depressive disorder without psychotic features (n = 45), dysthymia (n = 8), depressive disorder not otherwise specified without psychotic features (n = 5), Bipolar I disorder (n = 2), and Bipolar II disorder (n = 1). The SSD group had 38 patients with comorbid diagnoses compared to 52 patients from the MD group who had comorbid diagnoses. The types of comorbid diagnoses in the SSD group consisted of Axis I and Axis II disorders. Axis I disorders for the SSD group included posttraumatic stress disorder, alcohol dependence, polysubstance dependence, generalized anxiety disorder, and panic disorder. Axis II disorders for the SSD group consisted of schizoid personality disorder and obsessive compulsive personality disorder. Similar to the SSD group, the types of comorbid diagnoses in the MD group consisted of Axis I and Axis II disorders. A number of Axis I disorders were found that included generalized anxiety disorder, alcohol dependence, polysubstance dependence, posttraumatic disorder, bulimia nervosa, and attention deficit/hyperactive disorder. Axis II disorders for the MD consisted borderline personality disorder and antisocial personality disorder. Procedure We obtained institutional Review Board approval for all procedures from the hospital facility as well as from our University Human Subject s Committee. Administration and scoring of the Rorschach protocols followed Exner s (1993) Comprehensive System guidelines with the structural summary produced through the Rorschach Interpretation Assistance Program (Version 5; Exner, Weiner, & PAR Staff, 2001). We performed a search of medical records of adult inpatients admitted to the hospital within the previous 4 years ( ). The initial sample consisted of 600 adult psychiatric inpatients that had received a DSM IV admission diagnosis by a psychiatrist in consultation with a licensed social worker. We excluded cases if they did not meet the following five criteria: 1. We excluded participants if they were not diagnosed with one of two of the following categories of psychiatric disorders: (a) SSDs consisting of schizophrenia, schizophreniform, and schizoaffective disorder; or (b) nonpsychotic affective disorders consisting of major depressive disorder without psychotic features, dysthymia, and depressive disorder not otherwise specified without psychotic features. Based on this criterion, we excluded a total of 432 cases, leaving 168 cases having diagnoses of SSD or MD. 2. We excluded participants if the Rorschach was not administered and scored according to the Comprehensive System s guidelines (Exner, 1993). Based on this criterion, we excluded a total of 5 participants, leaving a sample size of 163 participants. 3. We excluded participants if Rorschach protocols were illegible or incomplete (i.e., structural summary, location sheets, etc.). Based on this criterion, we excluded a total of 2 participants, leaving a sample size of 161 participants. 4. We excluded participants if Rorschach protocols consisted of less than the minimum of 14 responses. A number of studies have cautioned against interpreting Rorschach protocols containing fewer than 14 responses (Exner, 1993; Exner & Weiner, 1995). Based on this criterion, we excluded a total of 2 participants, decreasing the sample size to 159 participants. 5. We excluded participants if Rorschach protocols contained proportions of Lambda responses greater than 1.0. This criterion was based on Weiner (1996) who suggested defensiveness on the part of the patient with Lambda > 1.0. Based on this criterion, we excluded a total of 52 participants, leaving a final sample size of 107 participants. The process of admission and diagnosis at the hospital entailed a licensed psychiatrist and clinical social worker performing separate comprehensive intake evaluations of symptoms and history via a semistructured interview and in accordance with the DSM IV TR. Intake diagnoses were established through consensus by the psychiatrist and clinical social worker within 48 hr of admission. For both the SSD and MD groups, the Structured Clinical Interview for

4 PERCEPTUAL THINKING INDEX 183 DSM IV Axis I Disorders: Clinical Versions (SCID CV; Michael, Spitzer, Gibbon, & Williams, 1996) and the Structured Clinical Interview for DSM IV personality disorders (SCID II; Michael, Spitzer, Gibbon, & Williams, 1997) were used to determine Axis I and Axis II disorders. In all cases, the intake diagnosis was established before the Rorschach was administered. A psychology intern who was blind to the diagnosis administered the Rorschach within the first 7 days following a patient s admission to the hospital. Working under the direct supervision of the director of psychological services who was also blind to the diagnosis, the final Rorschach scores were reached via consensus. Psychology interns consisted of upper level graduate psychology students who had completed at least 2 years in a psychology-oriented doctoral program. Furthermore, these interns received formal instruction within an academic setting as well as specialized Rorschach training at various psychology associations. Between 2000 and 2003, a number of graduate psychology interns administered Rorschach protocols. However, the supervising clinical psychologist remained the same, thus reducing the potential problems due to multiple raters. The director of psychological services, also a licensed clinical psychologist, had over 11 years of extensive training in providing Rorschach administration, scoring, interpretation, and supervision. The Rorschach was not given to all patients as part of the admission procedure. However, the Rorschach was routinely administered as part of an assessment battery with patients who exhibited cognitive disturbances and/or depressive symptoms. Instrument The Rorschach Inkblot Test consists of 10 inkblots (5 black and white and 5 containing colors). One of the most commonly taught (Hilsenroth & Handler, 1995) systems for administering, scoring, and interpreting Rorschach responses is the RCS (Exner, 2003). These studies have examined the following overall psychometric properties of the RCS: test retest reliability (Gronnerod, 2003), interrater reliability (Archer & Krishnamurthy, 1997; Exner, 1993; McDowell & Acklin, 1996; Meyer et al., 2002), convergent validity (Archer & Krishnamurthy, 1993, 1997; Greenwald, 1997), and discriminant validity (Ball, Archer, Gordon, & French, 1991). In this study, we used the following RCS variables: the total PTI, the five PTI criteria, X + %, WDA%, X %, Sum Level 2 Special Score, FABCOM2, R, WSUM6, and M. To estimate interrater reliability, 20 Rorschach protocols were chosen at random using a numbers table and rescored independently by a licensed psychologist who was blind to the original Rorschach scores as well as to patients diagnoses. Using Meyer s (1999) formulas for estimating kappa for RCS score segments, we calculated interrater agreement for the segments that include PTI variables. We conducted intraclass correlation (ICC) reliability analyses for RCS variables and PTI criteria. Table 1 contains the percentages of observed agreement, Kappa coefficients for the PTI segments, and interpretive guidelines based on Cicchetti (1994). Table 2 contains the ICC coefficients for the PTI variables, PTI criteria, and interpretive guidelines based on Cicchetti. The range of percentage of observed agreements, Kappa coefficients, and ICC coefficients were good to excellent, which indicates that the PTI variables, criteria, and score segments can be reliably scored. These results were consistent with interrater reliability results reported by Hilsenroth et al. (1998), Meyer et al. (2002), and Smith et al. (2001). Criteria and Segments TABLE 1 Response Level Interrater Reliability for Scoring PTI-Relevant Response Segments in 20 Rorschach Protocols % Agreement Kappa Coefficient Interpretation a Developmental Good Quality Determinants Excellent Form Quality Good Cognitive Special Scores b Good Other Special Scores c Excellent Note. PTI = Perceptual Thinking Index. a Interpretations based on Cicchetti s (1994) guidelines. b Cognitive Special Scores = deviant verbalization, incongruence combination, deviant response, fabulized combinations, inappropriate logic, and contamination. c Other Cognitive Special Scores = abstract content, aggressive movement, cooperative movement, color projection, morbid content, good or poor human representation, personal, and perseveration. TABLE 2 Summary Score Interrater Reliability for PTI Variables Across 20 Rorschach Protocols Variables and Criteria Intraclass Correlation Coefficient a Interpretation b XA%.84 Excellent WDA%.74 Good X %.82 Excellent Sum Level 2 Special Score.81 Excellent FABCOM2.79 Excellent WSUM6.71 Good M %.87 Excellent PTI Criteria 1.88 Excellent PTI Criteria 2.88 Excellent PTI Criteria 3.81 Excellent PTI Criteria 4.80 Excellent PTI Criteria 5.84 Excellent Note. PTI = Perceptual Thinking Index. a Intraclass correlation coefficients were computed using a one-way random effects model. b Interpretations based on Cicchetti s (1994) guidelines.

5 184 DAO AND PREVATT RESULTS The data analyses proceeded in four sequential steps. We conducted two separate analyses to assess internal consistency of the PTI. The PTI consists of five separate criteria, and these criteria are constructed from eight different individual variables. First, we calculated the Kuder Richardson formula 20 (KR 20) on the PTI to assess internal consistency of the five separate criteria. Second, we computed corrected item-to-total correlations (two-tailed) between scores on each of the five criteria and the PTI total score as well as for each of the nine variables that make up the criteria and the PTI total score. In doing so, the corrected item-to-total correlations provided an estimate of how well the different variables converge on a common construct. Table 3 contains the results of these two analyses. Results of the KR 20 and the corrected item-to-total correlations suggest adequate internal consistency as well as convergence of the different variables and criteria on a common construct. Nunnally and Bernstein (1994) suggested a standard of.70 or greater on the KR 20 as a reliable statistic to evaluate internal consistency and.30 or greater on corrected item-to-scale correlations as possessing adequate convergence on an index defining a construct (Hilsenroth et al., 1998; Smith et al., 2001). Next, we conducted analyses of validity. Table 4 contains the descriptive statistics, whereas Table 5 contains the t-test statistics for the PTI variables, criteria, and total score for the full sample as well as the group-specific samples. We computed the t-test statistics to examine differences between means of the SSD and MD groups across PTI variables, criteria, and total score. Because there are multiple t tests, we made a Bonferroni adjustment to protect against inflated Type I error (α =.05/14 =.00357). The inclusion of R in the analyses was to determine whether discrepancies between the two Variable TABLE 3 Internal Consistency Statistics for the PTI KR 20 Corrected Item-to- Total Correlations PTI Criteria 1 through PTI Criteria XA%.487 WDA%.460 X %.470 Sum Level 2 Special Score.506 FABCOM2.582 R.519 WSUM6.524 M %.459 PTI Criteria PTI Criteria PTI Criteria PTI Criteria PTI Criteria Note. N = 107 protocols. KR 20 was calculated for the five criteria on the PTI. PTI = Perceptual Thinking Index; KR 20 = Kuder Richardson formula 20. groups could be attributed to differences in number of RCS responses. There were no differences between the two groups on total number of responses; therefore, this can be ruled out as a potential confound to the results of the study. Examination of individual variables comprising the PTI revealed that the MD group evidenced significantly higher scores than the SSD group on the variables XA% and WDA%. Alternately, the MD group evidenced significantly lower scores than the SSD group on the variables X %, Sum Level 2 Special Score, FABCOM2, and M %. Examination of the PTI criterion scores revealed that the SSD group scored significantly higher than the MD group on all five criterion scores as well as the total PTI score. To assess for practical importance, we computed Cohen s (1988) d effect sizes using pooled variance, and we adjusted them for unequal sample sizes. The results indicate that there were moderate to large effect sizes for all hypothesized PTI variables and indexes. There is some concern due to the large number of cases excluded due to a Lambda > 1.0 (n = 52). Although other investigations of the Rorschach have excluded cases with Lambda > 1.0 (e.g., Blais, Hilsenroth, Castlebury, Fowler, & Baity, 2001; Hilsenroth et al., 1998; Smith, Baity, Knowles, & Hilsenroth, 2001), it would be useful to know the impact of this decision. To avoid biasing the sample, we decided to analyze a post hoc subset of cases, all with Lambda > 1.0. Of the 52 excluded cases, 22 were patients with SSD. We compared these 22 cases to 22 randomly selected (of the remaining 30) patients with MD. Table 6 shows the results of this analysis. Although this analysis had less statistical power than the initial analysis, the pattern of results was similar to the initial data set. There were moderate to large effect sizes for all PTI variables and indexes. Given that Exner (2000a) advocated that the PTI be used as a continuous scale with higher values more indicative of a thought disturbance, we computed a Spearman (Rho) correlation coefficient to examine the relation between the PTI total score with the dichotomously coded MD and SSD groups. As would be expected based on the results in Table 5, the correlation was substantial: ρ =.62, p <.05, two-tailed. To further assess the ability of the PTI to differentiate among diagnostic groups, we conducted a logistic regression analysis with the PTI total score regressed on the dichotomous group diagnosis. Preliminary analyses consisted of identifying potential outliers and observations of excessive influence. We identified one outlier (2.99) that had a larger standardized Pearson residual than the threshold of 2.5, which is considered an adequate case index for the identification of outliers for univariate and multivariate studies. Pearson residual is defined to be the standardized difference between the observed frequency and the predicted frequency. It measures the relative deviations between the observed and fitted values (Tate, 1998). However, an inspection of β indexes showing the change in model coefficients if individual observations were to be dropped did not suggest excessive influence of any individual observations;

6 PERCEPTUAL THINKING INDEX 185 TABLE 4 Descriptive Statistics for PTI Variables and Criteria for the Aggregate Sample and the SSD and MD Groups Descriptive Statistics Total Sample a SSD Group b MD Group c Variable M SD Min Max Med Skew Kurtosis M SD Min Max Med Skew Kurtosis M SD Min Max Med Skew Kurtosis R XA% WDA% X % Sum Level 2 Special Score d d FABCOM d d WSUM M % PTI PTI PTI d d PTI PTI Total PTI Note. PTI = Perceptual Thinking Index; SSD = Schizophrenia-Spectrum disorder; MD = mood disorder with no psychotic features; min = minimum; max = maximum; med = median. a N = 107. b N = 46. c N = 61. d Skewness and kurtosis were not available because all patients had scores of zero. TABLE 5 t-test Statistics and Effect Sizes Comparing SSD a to MD b Groups in the Primary Analysis When All Protocols Have Lambda 1.0 TABLE 6 t-test Statistics Comparing SSD a to MD b Groups When All Protocols Have Lambda > 1.0 Variable t Statistics (df = 105) p Value Cohen s d Variable t Statistics (df = 105) p Value Cohen s d R XA% WDA% X % 6.1 < Sum Level 2 Special Score 4.0 < FABCOM2 3.9 < WSUM6 4.1 < M-% 3.9 < PTI < PTI < PTI < PTI < PTI < Total PTI 8.2 < Note. The Bonferroni adjusted p value is Cohen s d statistics were calculated using pooled variance and adjusted for unequal sample sizes. SSD = Schizophrenia-spectrum disorder; MD = mood disorder with no psychotic features; PTI = Perceptual Thinking Index. a n = 46. b n = 61. R XA% 6.1 < WDA% 5.9 < X % 5.6 < Sum Level 2 Special Score FABCOM WSUM M % 3.9 < PTI < PTI < PTI < PTI < PTI < Total PTI 7.2 < Note. The Bonferroni adjusted p value is SSD = schizophreniaspectrum disorder; MD = mood disorder with no psychotic features; PTI = Perceptual Thinking Index. a n = 22. b n = 22.

7 186 DAO AND PREVATT therefore, we did not drop any observations from the analysis. Table 7 shows the PTI total score reliably distinguished SSD and MD patients. For the PTI total score, the odds of being in the SSD group were estimated to increase by a factor of 2.82 when the PTI score increased by 1. A test of the model goodness of fit, the Hosmer Lemeshow test (Tabachnick & Fidell, 1996), produced a fail to reject decision χ 2 (1, N = 107) = 11.7, p >.05, a result consistent with the assumption that the specified logistic model was correct. There was no reason to believe that there was any violation of the independence assumption. Moreover, the measure of the independent variable was reliable, and we assumed that the effect of any technical violation of the assumption that all independent variables are measured exactly was not problematic. According to Ganellen (1996a), test scores are most useful if they maximize accurate identification of individuals as having or not having a particular diagnosis as well as minimizing the possibility of misclassification. The following diagnostic efficiency statistic descriptions and computations are based on Streiner (2003), and we present them as they relate to this study: Variable 1. Sensitivity is defined as the proportion of people diagnosed with SSD who are detected by the PTI. 2. Specificity is the proportion of people who do not meet diagnostic criteria for SSD and are correctly identified as not SSD by the PTI. 3. False positive rate refers to the proportion of individuals tested who do not meet the diagnostic criteria for SSD but have been identified as having that disorder. 4. False negative rate refers to the proportion of individuals tested who meet the diagnostic criteria for SSD but have not been identified as having that disorder. 5. Positive predictive power is the percent of individuals classified by the PTI as having SSD who truly have the particular disorder. 6. Negative predictive power is the percent of individuals classified by the PTI as not having SSD who truly do not have the particular disorder. 7. Overall correct classification rate refers to the proportion of individuals correctly identified by the PTI as having SSD or MD. TABLE 7 Logistic Regression for PTI Total Scores Predicting SSD Versus MD Classification β Wald Test Odds Ratio 95% Confidence Interval for Odds Ratio PTI total score 1.04* a 1.96 to 4.07 Constant Note. PTI = Perceptual Thinking Index; SSD = schizophrenia-spectrum disorder; MD = mood disorder with no psychotic features. a Odds ratio associated with a PTI score increase of one. *p < Kappa represents the level of agreement between the PTI and the diagnostic criteria beyond that accounted for by chance alone. Table 8 provides diagnostic efficiency statistics for three different PTI cutoff scores for the differentiation of SSD patients from MD patients. An examination of diagnostic efficiency statistics revealed five major findings: 1. Sensitivity, negative predictive power, false positive rate, and overall correct classification rates were lower when higher PTI cutoff scores were used. 2. Specificity, positive predictive power, and false negative rate were higher when higher PTI cutoff scores were used. 3. Participants from the SSD group produced PTI scores as high as Participants in the MD group produced PTI scores of no greater than The overall correct classification rate and kappa coefficient indicate a cutoff of 3 is optimal in this sample. DISCUSSION In this study, we examined the psychometric properties of the PTI in a U.S. sample of adult inpatients diagnosed with DSM IV TR diagnoses of SSDs. Overall, reliability analyses of the PTI suggest that it can be reliably scored and possesses adequate internal consistency. Interrater reliability statistics for this study were similar to robust interrater reliability statistics reported on other Rorschach variables (Meyer et al., 2002; Smith et al., 2001; Viglione & Taylor, 2003). Internal consistency analyses met the commonly accepted alpha level of.70 on the KR 20 and.30 on the corrected item-to-scale correlations (Nunnally & Bernstein, 1994), which demonstrates adequate convergence among PTI variables and criteria. The internal consistency results revealed levels of reliability comparable to those reported by Hilsenroth et al. (1998) on the SCZI. The findings on interrater reliability as well as internal consistency on the PTI are noteworthy given the continuing debate regarding the reliability of variables and indexes within the RCS. Validity analyses demonstrated that the PTI could be used effectively to differentiate psychotic patients from patients with mood disorders without psychotic features. There were significant differences between the means of the SSD and MD groups across PTI variables, criteria, and PTI total score, all in the expected direction. SSD patients scored significantly higher on variables indicative of poor Form Quality, implausible relationships, illogical or circumstantial thinking, and dissociated or distorted thinking. Alternately, MD patients scored higher on the two variables new to the RCS: XA% and WDA%. These variables essentially measure the absence of poor Form Quality responses. Across variables,

8 PERCEPTUAL THINKING INDEX 187 TABLE 8 Diagnostic Efficiency Statistics for Different PTI Cutoff Scores Cutoff Score Sensitivity Specificity Positive Predictive Power Negative Predictive Power False Positive Rate False Negative Rate Overall Correct Classification Cohen s Kappa PTI PTI PTI = Note. PTI = Perceptual Thinking Index. Cohen s d values ranged from 0.74 and above, which suggests moderate to large effect sizes. Previous investigations have excluded protocols with Lambda > 1.0 (e.g., Blais et al., 2001; Hilsenroth et al., 1998; Smith et al., 1999). There is legitimate reason to do this; Exner (1993) suggested that high Lambda represents an organizational style that tends to supercede other stylistic features. As such, those features central to psychotic thinking might be masked if high Lambda protocols are included in the analyses. However, if a large proportion of psychotic patients have high Lambda protocols, the PTI becomes less useful without knowing whether it can be utilized on all participants. Therefore, we analyzed a separate sample of SSD and MD participants, all with high Lambda protocols. Even with the small sample size, the pattern of scores was similar to the larger sample, and effect sizes remained in the moderate to large range for all PTI variables. These combined findings lend even stronger support for the ability of the PTI to differentiate those characteristics of psychotic disorders most associated with cognitive rather than emotional functioning. Paralleling the mean differences, logistic regression indicated that PTI total scores could distinguish between SSD and MD patients, and the odds of being in the SSD group were estimated to increase by a factor of 2.82 when the PTI score increased by 1. The diagnostic efficiency statistics also showed that the PTI performed well in the classification of SSD and MD patients. Cutoff scores always involve decisions regarding the types of errors most tolerable given the nature of the assessment. Cutoff scores of 4 or = 5 resulted in a zero false positive rate; that is, no MD patients were misclassified as SSD. However, these scores led to what might be considered unacceptably high false negative rates: that is, individuals diagnosed with SSD who were not identified as having SSD by the PTI. In addition, the overall classification rates and kappa indicated that the cutoff score of 3 provided the best overall classification, with a reasonable trade-off between false positives and false negatives. Clearly, the use of a particular cutoff score will be contingent on the nature and consequences of the decision being made. Based on the diagnostic efficiency statistics descriptions, we determined the coefficients for sensitivity and specificity independent of the base rate, or prevalence, of being diagnosed with SSD (Streiner, 2003). The coefficients for positive and negative predictive power, on the other hand, take into account the prevalence rate of being diagnosed with SSD in this particular sample. This distinction is important given this sample consisted of 43% of patients diagnosed with SSD and 57% of patients diagnosed with MD, percentages that may not be comparable to the full range of diagnoses encountered in an inpatient setting. As a result, values for positive and negative predictive power would need to be adjusted to reflect local base rates in a particular practice setting. It would be valuable for future studies to examine the psychometric properties of the PTI among various groups of patients (e.g., personality disorder Cluster A, B, and C; see Hilsenroth et al., 1998) in different treatment settings (e.g., outpatient setting). The population that a particular program serves (e.g., inpatient hospital, outpatient) will influence the number of individuals who exhibit the signs and symptoms characteristic of those diagnosed in the SSD group, thus impacting the prevalence rate for SSD in that setting. For instance, an impatient psychiatric hospital might have a greater incidence of patients diagnosed with SSD among its population when compared to an outpatient treatment center. This is important to consider given that prediction of psychiatric disorder with low prevalence rates often results in a high proportion of misclassification (Derogatis & DellaPietra, 1994). Consequently, future studies should examine the psychometric properties of the PTI in various populations and settings to account for the different base rates of SSD. The findings of this study have implications for assessment practice; however, there are several limitations to consider, the first of which involves the assigning of patients to either the SSD or the MD groups. As suggested by others, when examining psychometric properties of tests using classical test theory techniques such as internal consistency measures, it is important to consider that the interpretations drawn from these analyses cannot be separated from the population being studied. Consequently, the somewhat restricted range of participants utilized for this study might have reduced the internal consistency statistics. It is plausible that if a greater range of participants (i.e., patients and nonpatients) are utilized, improvements on internal consistency statistics would be observed. Second, although in this study, we used the SCID CV and SCID II in diagnosing patients, we failed in this study to assess reliability of diagnoses using multiple raters. Third, the majority of patients in both the SSD and MD groups had comorbid diagnoses; thus, some caution is warranted in formulating interpretations pertaining to the true effects of thought

9 188 DAO AND PREVATT disturbance of the PTI. Despite the limitation inherent when employing patients with comorbid diagnoses to examine one particular diagnostic group, this limitation can also be viewed as addressing the ecological validity of the study given that a large number of patients in the clinical settings are diagnosed with co-occurring disorders (Kaplan & Sadock, 1998). REFERENCES Abreu, A. C., & Filips, J. K. (2001). Psychiatry. In M. A. Graber & M. Lanternier (Eds.), University of Iowa family practice handbook (pp ). St. Louis, MO: Mosby. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: Author. American Psychological Association. (2004). Schizophrenia. Retrieved July 2, 2004, from Archer, R. P., & Krishnamurthy, R. (1993). A review of MMPI and Rorschach interrelationships in adult samples. Journal of Personality Assessment, 61, Archer, R. P., & Krishnamurthy, R. (1997). MMPI A and Rorschach indices related to depression and conduct disorder: An evaluation of the incremental validity hypothesis. Journal of Personality Assessment, 69, Ball, J. D., Archer, R. P., Gordon, R. A., & French, J. (1991). Rorschach Depression indices with children and adolescents: Concurrent validity findings. Journal of Personality Assessment, 57, Blais, M. A., Hilsenroth, M. J., Castlebury, F., Fowler, C. J., & Baity, M. R. (2001). Predicting DSM IV cluster B personality disorder criteria from MMPI 2 and Rorschach data: A test of incremental validity. Journal of Personality Assessment, 76, Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized instruments in psychology. Psychological Assessment, 6, Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Courvoisie, H., Labellarte, M. J., & Riddle, M. A. (2001). Psychosis in children: Diagnosis and treatment. Dialogues in Clinical Neuroscience, 3, Derogatis, L. R., & DellaPietra, L. (1994). Psychological tests in screening for psychiatric disorders. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (pp ). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Exner, J. E., Jr. (1993). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (3rd ed.). New York: Wiley. Exner, J. E., Jr. (2000a). A primer for Rorschach interpretation. Asheville, NC: Rorschach Workshops. Exner, J. E., Jr. (2000b) alumni newsletter. Asheville, NC: Rorschach Workshops. Exner, J. E., Jr. (2003). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (4th ed.). New York: Wiley. Exner, J. E., Jr., & Weiner, I. B. (1995). The Rorschach: A comprehensive system: Vol. 3. Assessment of children and adolescents (2nd ed.). Oxford, England: Wiley. Exner, J. E., Jr., Weiner, I. B., & PAR Staff. (2001). Rorschach interpretation assistance program: Version 5 for Windows [Computer software]. Odessa, FL: Psychological Assessment Resources. Ganellen, R. J. (1996a). Comparing the diagnostic efficiency of the MMPI, MCMI II, and Rorschach: A review. Journal of Personality Assessment, 67, Ganellen, R. J. (1996b). Exploring the MMPI-Rorschach relationships. Journal of Personality Assessment, 67, Greenwald, D. F. (1997). Comparisons between the Rorschach Depression Index and depression-related measures in a nonpatient sample. Psychological Reports, 80, Gronnerod, C. (2003). Temporal stability in the Rorschach method: A metaanalytic review. Journal of Personality Assessment, 80, Hathaway, S. R., & McKinley, J. C. (1943). The Minnesota Multiphasic Personality Inventory. Minneapolis: University of Minnesota Press. Hilsenroth, M. J., & Handler, L. (1995). A survey of graduate students experiences, interests, and attitudes about learning the Rorschach. Journal of Personality Assessment, 64, Hilsenroth, M. J., Fowler, J. C., & Padawer, J. R. (1998). The Rorschach schizophrenia index (SCZI): An examination of reliability, validity, and diagnostic efficiency. Journal of Personality Assessment, 70, Ilonen, T., Tiaminen, T., Karlsson, H., Lauerma, H., Leinonen, K. M., Wallenius, E., et al. (1999). Diagnostic efficiency of the Rorschach schizophrenia and depression indices in identifying first-episode schizophrenia and severe depression. Psychiatry Research, 87, Jorgensen, K., Anderson, T., & Dam, H. (2000). The diagnostic efficiency of the Rorschach depression index and the schizophrenia Index: A review. Assessment, 7, Kaplan, H. I., & Sadock, B. J. (1998). Synopsis of psychiatry (8th ed.). Baltimore: Williams & Wilkins. Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000). The scientific status of projective techniques. Psychological Science in the Public Interests, 1, McDowell, C., & Acklin, M. W. (1996). Standardizing procedures for calculating Rorschach interrater reliability: Conceptual and empirical foundations. Journal of Personality Assessment, 66, Meyer, G. J. (1993). The impact of response frequency on the Rorschach constellation indices and on their validity with diagnostic and MMPI 2 criteria. Journal of Personality Assessment, 60, Meyer, G. J. (1999). Simple procedures to estimate chance agreement and kappa for the interrater reliability of response segments using the Rorschach Comprehensive System. Journal of Personality Assessment, 72, Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J.E., Jr., Fowler, J. C., Piers, C. C., et al. (2002). An examination of interrater reliability for scoring the Rorschach comprehensive system in eight data sets. Journal of Personality Assessment, 78, Michael, B., Spitzer, R. L., Gibbon, M., & Williams, J. B. (1996). Structured Clinical Interview for DSM IV Axis I Disorders: Clinician Version (SCID CV). Washington, DC: American Psychiatric Press. Michael, B., Spitzer, R. L., Gibbon, M., & Williams, J. B. (1997). Structured Clinical Interview for DSM IV Personality Disorders (SCID II). Washington, DC: American Psychiatric Press. Netter, B. E. C., & Viglione, D. J. (1994). An empirical study of malingering schizophrenia on the Rorschach. Journal of Personality Assessment, 62, Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Ritsher, J. B. (2004). Association of Rorschach and MMPI psychosis indicators and schizophrenia spectrum diagnoses in a Russian clinical sample. Journal of Personality Assessment, 83, Smith, S. R., Baity, M. R., Knowles, E. S., & Hilsenroth, M. J. (2001). Assessment of disordered thinking in children and adolescents: The Rorschach perceptual-thinking index. Journal of Personality Assessment, 77, Streiner, D. L. (2003). Diagnosing tests: Using and misusing diagnostic and screening tests. Journal of Personality Assessment, 81, Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: HarperCollins. Tate, R. (1998). An introduction to modeling outcomes in the behavioral and social sciences. Edina, MN: Burgess International Group.

10 PERCEPTUAL THINKING INDEX 189 Veterans Health Administration. (2001). Care for veterans with psychosis in the VHA, FY01: Third Annual National Psychosis Registry report. Ann Arbor, MI: Author. Viglione, D. (1999). A review of recent research addressing the utility of the Rorschach. Psychological Assessment, 11, Viglione, D. J., & Taylor, N. (2003). Empirical support for interrater reliability of Rorschach comprehensive system coding. Journal of Clinical Psychology, 59, Vincent, K. R., & Harman, M. J. (1991). The Exner Rorschach: An analysis of its clinical validity. Journal of Clinical Psychology, 47, Weiner, I. B. (1996). Some observations on the validity of the Rorschach Inkblot Method. Psychological Assessment, 8, Wood, J. M., Nezworski, M. T., & Garb, H. N. (2003). What s right with the Rorschach? Scientific Review of Mental Health Practice, 2, Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The comprehensive system for the Rorschach: A critical examination. Psychological Science, 7, Frances Prevatt Department of Educational Psychology & Learning Systems Stone 307 Florida State University Tallahassee, FL fprevatt@coe.fsu.edu Received March 18, 2005 Revised July 18, 2005