Rescaling the Post-Traumatic Stress Disorder Checklist for Use in Primary Care

Transcription

1 MILITARY MEDICINE, 181, 9:1002, 2016 Rescaling the Post-Traumatic Stress Disorder Checklist for Use in Primary Care Phoebe K. McCutchan, MPH*; Michael C. Freed, PhD, EMT-B* ; Elizabeth C. Low, MA*; Bradley E. Belsher, PhD* ; Charles C. Engel, MD, MPH* ABSTRACT The Posttraumatic Stress Disorder (PTSD) Checklist (PCL) is a 17-item self-report measure of PTSD symptom severity that has demonstrated excellent psychometric properties across a variety of settings, purposes, and populations. The PCL is widely used in busy Department of Defense primary care settings as part of routine PTSD screening, requiring that it is easy for patients to complete and providers to score. The clinical utility of the PCL may be improved through use of a zero-anchored Likert-type response scale by providing intuitive anchors for respondents and fewer calculations for clinic staff; however, changes to the response scale may invalidate the known psychometric properties of the measure. The purpose of this study is to evaluate the equivalence of a zero-anchored PCL to the traditional one-anchored PCL. Differences in total scores were examined using inferential confidence intervals. Substantial overlap of the inferential confidence intervals and small R g (maximum probable difference) value of 0.68 indicated that the zeroanchored PCL is equivalent to the one-anchored PCL on the basis of our specified delta (amount of difference considered inconsequential). These findings support the use of a zero-anchored PCL in clinical practice, and more broadly, the use of zero-anchored measures in the larger field of psychological assessment. INTRODUCTION The Posttraumatic Stress Disorder (PTSD) Checklist (PCL) 1 is among the most widely used instruments to assess the symptoms of PTSD. 2 The 17-item self-report measure demonstrates excellent psychometric properties, 3 favorable diagnostic utility compared to other brief PTSD screening tools, 4,5 and has been studied in a variety of populations and settings, 6 to include active duty service members, 7 veterans, 8 and civilian and military primary care. 9,10 To date, the PCL remains the evidence-based instrument of choice for annual and new patient screenings of service members for PTSD required under Department of Defense policy. 11 Given the frequency of its administration and routine use in busy clinical settings, employing a zero-anchored response scale on the PCL (i.e., a 0 4 Likert-type scale, versus the traditional 1 5 scale) may enhance its utility by reducing both respondent and administrator burden. Research on Likert-type scales indicates that respondents use the listed numeric values to interpret scale labels. 12,13 For scales that are intended to assess the intensity of a single attribute (e.g., symptoms of a disorder), a zero-to-positive-values format emphasizes that the question pertains to the absence or presence of the specific *Deployment Health Clinical Center, Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury, 1335 East West Highway, Silver Spring, MD Department of Psychiatry, Center for the Study of Traumatic Stress, Uniformed Services University of the Health Sciences, 6700A Rockledge Drive, Bethesda, MD The views expressed in this article are those of the authors and do not necessarily represent those of the Department of Defense, Deployment Health Clinical Center, Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury, Uniformed Services University of the Health Sciences, National Institutes of Health, or any other organization public or private. doi: /MILMED-D attribute, and thus provides more intuitive anchors for symptom ratings. 13 From a health care provider perspective, response formats that use a zero-anchored scale allow a measure to be more easily and accurately scored, as items that are rated 0 can be excluded from the summation thus requiring fewer calculations. Increased ease of use in clinical settings will facilitate measurement-based care and minimize user error (i.e., miscalculation of score). Several instruments measuring related anxiety and depression symptom severity use a zero-anchored Likert-type response scale, and the field of psychological assessment appears to be moving in a similar direction. In the recent revision to the Diagnostic and Statistical Manual of Mental Disorders (DSM), 14 the DSM-5 Working Group endorsed the use of ratings on a 5-point scale, with 0 indicating the absence of the problem as an essential characteristic of newer assessments. 15 Accordingly, the Working Group used zero-anchored scales in the DMS-5 field trials to confirm the reliability and validity of PTSD symptoms to be included in the DSM-5 diagnosis of PTSD, 16,17 and new DSM-5-based PTSD measures have been designed consistent with this approach. 18,19 The revision of clinical measures that utilize zero-anchored Likert-type response scales, although presumed to reduce response biases, may also lead to differential response patterns and could potentially invalidate the known psychometric properties of the measure. 13,20 Without a demonstrated understanding of the psychometric attributes of an instrument, there is a risk that responses to the instrument will be misinterpreted or misused. 21 Thus, the equivalence of a modified measure must be empirically validated to ensure that it retains its effectiveness as an assessment tool. This study seeks to examine the equivalence of a modified version of the PCL that utilizes a zero-anchored scale compared to the traditionally scaled PCL in a military primary 1002

2 care setting. The initial impetus behind this study was to validate use of the zero-anchored PCL version long used in U.S. Army primary care practice. 22,23 Under this primary care protocol, all primary care visits at participating military treatment facilities are screened for PTSD using a twostaged approach with the PCL serving as the secondary screen. Given the high frequency of administration (over 160,000 PCLs were administered in as part of routine screening alone), ease of use for both respondents and clinic staff is essential. If the PCL with 0- to 4-item scoring is found to be equivalent to the 1 to 5 version, it would suggest that the zero-anchored scale may not compromise the psychometric properties of the measure and is acceptable for use in primary care practice to ease administration and scoring. More broadly, the findings may also have implications for the PCL-5, the DSM-5 PTSD severity assessment that also adopts a zero-anchored Likert-type scale, by demonstrating that such rescaling does not significantly impact the measurement of the underlying PTSD construct. METHODS Participants Participants were consecutively approached individuals presenting to the General Internal Medicine Clinic at Walter Reed National Military Medical Center. Individuals were eligible to participate if they were 18 years of age or older, medically stable at the time of recruitment, and attending a medical appointment. Individuals who self-reported a history of serious cognitive disturbance, had a visibly apparent severe medical illness, or were unable to read, write, or understand English at a sixthgrade level were excluded. Measures Demographics Questionnaire The demographics questionnaire asked participants to report their age, race/ethnicity, gender/sex, service affiliation, branch of service, and rank. PTSD Checklist The PCL is a 17-item screening tool that asks respondents to rate how bothered they have been in the past month (1 = not at all; 5 = extremely) by each of the 17 DSM-IV symptoms of PTSD. 1 It has demonstrated high internal consistency (between 0.91 and 0.94), strong convergent validity with other measures of PTSD, 3,24 and excellent test retest reliability (between 0.92 and 0.96). 1,25 Of the three validated versions of the PCL, this study used of the civilian version because it is already in wide military primary care use. Hereafter, we refer to this traditionally scored version of the PCL as the PCL-1. Zero-Anchored PTSD Checklist (PCL-0) The PCL-0 is identical to the PCL-1 described above, with the exception of a revised Likert-type response scale ranging from 0 to 4 (versus 1 5). Response option labels (i.e., not at all to extremely ) remained the same. Appointment question: Did anything occur during your appointment today that affected your responses on this questionnaire? (Yes/No). Given that psychological concerns are often addressed in primary care settings, it is possible that the content of the appointment may influence participants responses on the second version of the PCL administered in the study. 26 Therefore, this broad, internally developed contextual question was asked to identify instances in which events occurring between the administrations of the two questionnaires may have accounted for potential differences in participants responses. Procedures Individuals arriving at the clinic waiting area were consecutively approached by a trained study staff member and verbally consented to participate in a study to test a new primary care screening questionnaire. The intent to test the equivalence of an alternate response scale was not disclosed before participation in order to prevent priming or other response biases. Participants completed the demographics questionnaire and one version of the PCL prior to their appointment, and the second version of the PCL and the appointment question after their appointment. The order of PCL form administration was counterbalanced to control for potential order effects. The pretest post-test design around the medical appointment (approximately 30 minutes on average) was intended to reduce practice effects on the completion of the second form. Participants received a debriefing form detailing the full purpose of the study and a list of mental health resources on completion of the study. All study procedures were approved by the Walter Reed National Military Medical Center Institutional Review Board. Statistical Analysis The goal of equivalence testing is to determine whether the two PCL forms are functionally equal (i.e., produce equal responses, allowing for some variability within an inconsequential range). 27 In simple terms, this process entails comparing mean scores between the forms and deciding whether the differences are acceptable on the basis of practicality. Specifically, we utilized Tryon s method of inferential confidence intervals (ICIs) for dependent means to test for statistical equivalence. 28,29 Traditional descriptive confidence intervals are problematic in equivalence testing in that 95% confidence intervals can overlap substantially despite the 2 means being significantly different at the 5% level. 28,30 ICIs correct for this by adding a reduction term to narrow the descriptive confidence intervals such that nonoverlapping ICIs are statistically equivalent to classical null hypothesis significance testing at the specified significance level. To establish equivalence in the current study, it was necessary to determine first that the forms are not statistically 1003

3 different, and second that the forms are statistically equivalent. To assess for statistical difference, ICIs were constructed around each sample (i.e., PCL-1 and PCL-0 scores) mean. In calculating sample means, total scores on the PCL-0 were adjusted to match the traditional PCL-1 total scores by adding 17 to each score, such that all scores ranged from 17 to 85. A reduction factor for dependent means (which takes into account the correlation between data sets) was applied to the calculation of the ICIs. The ICIs were then examined to determine if they overlapped; ICI overlap would indicate that the alternate forms were not statistically different. Determining statistical equivalence requires an extension of the procedures described above. First, a delta (Δ) value, the amount of difference between means that is considered inconsequential on the basis of practicality, was specified. For the purposes of this study, Δ = 5, signifying a maximum acceptable difference of 5 points in PCL total score; this value is consistent with military primary care protocols in which 5 points is considered clinically meaningful change. Second, R g, the difference between the upper bound of the greater mean and the lower bound of the lesser mean (i.e., the range of the two overlapping ICIs) was calculated to determine the maximum probable difference between sample means. Lastly, R g was compared to Δ; ifr g is less than the specified Δ value (i.e., <5), then the difference is inconsequential and the means of the two forms would be considered statistically equivalent (i.e., the two PCL forms are equivalent). In order to assess the equivalence of the PCL forms across the spectrum of PTSD symptoms found in primary care patients, this study was powered on the basis of our previously published data 10 to examine two subgroups: individuals with low/minimal PTSD symptom levels (PCL-1 score < 30) and individuals with elevated PTSD symptom levels (PCL-1 score 30, using point Likert-type item scoring). Additionally, Pearson product-moment correlations were computed to examine the relationship between individual items and total scores on the PCL-1 and PCL-0; a p value of 0.05 was considered statistically significant. TABLE I. Total Score Means, 95% ICIs, and R g Values for PCL-1 and PCL-0 Forms Across Samples Full Sample (N = 120) Minimal PTSD (n = 98) Elevated PTSD (n = 22) PCL-1 M (SD) (8.61) (3.29) (9.70) ICI Upper Bound ICI Lower Bound PCL-0 M (SD) (8.84) (3.61) (10.69) ICI Upper Bound ICI Lower Bound R g M, mean; SD, standard deviation. distribution was 45% Navy, 28% Army, 7% Air Force, 20% Coast Guard, and 15% Public Health Service. Table I presents sample means, standard deviations, 95% ICIs, and R g (maximum probable difference) values for PCL-1 and PCL-0 forms. As seen in Figure 1, ICIs for the full sample demonstrated substantial overlap, indicating that means on the PCL forms are not statistically different. The R g value for the full sample was 0.68, indicating that the maximum mean difference in total scores between the two PCL versions can be expected to be less than 1 point (95% of the time). This R g value is considerably smaller than the predetermined Δ value of 5, and thus the maximum probable difference between PCL forms is inconsequential and the alternate forms are equivalent. Subgroup analyses on the basis of PTSD RESULTS Of the approximately 255 individuals approached to participate in the study, 225 agreed to participate and completed at least partial questionnaires (i.e., 30 individuals refused). Of the sample of participants, 107 individuals did not complete both PCL forms (68% were called for appointment before completing the preappointment measures, 7% provided incomplete data, and 25% were lost to follow-up) and were thus excluded from analyses. The final sample for analyses consisted of 120 participants composed equally of males (n = 60) and females (n = 60) with a mean age of (standard deviation = 19.1, range = 18 88). Racial distribution was 64% white, 25% black, 3% Hispanic, and 8% other. Most participants were retired military (44%); remaining participants were 32% active duty, 1% reserve, and 23% beneficiaries. Of those who were active duty (n = 40), the FIGURE 1. Sample means and 95% ICIs for PCL-1 and PCL-0 total scores. 1004

4 symptom levels evidenced similar ICI overlap, and R g values of 0.68 and 1.84 (minimal PTSD symptom and elevated PTSD symptom levels, respectively) confirmed statistical equivalence across the spectrum of PTSD symptomatology. The correlation coefficient between total PCL-1 and PCL-0 scores in the full sample was 0.96 ( p < 0.01). Itemlevel correlations were all highly significant, ranging from 0.66 to 0.94 ( p < 0.01). For PTSD symptom level subgroups, the total score correlation coefficients were also strongly significant (0.82 for minimal PTSD symptoms and 0.92 for elevated PTSD symptoms; p < 0.01). Item-level correlation coefficients for the subgroups ranged from 0.39 to 0.88 (minimal PTSD symptoms; p < 0.01) and 0.62 to 0.97 (elevated PTSD symptoms; p <0.01). There were six positive responses to the appointment question (two in the minimal PTSD symptom group and four in the elevated PTSD symptom group), indicating that 5% of participants perceived their responses to be affected by the interim primary care appointment. However, removing these participants from analyses had a minimal effect on R g values (full sample = 0.69, minimal symptom level subgroup = 0.56, and elevated symptom level subgroup = 2.71) and did not change the overall results of the equivalence test. DISCUSSION The results from this study indicate that the zero-anchored PCL form was equivalent to the traditional PCL that uses the 1-to-5 self-report response scales. The correlations between the two forms were consistent with the test retest reliability reported in the literature. 6 These findings suggest that the two forms yield essentially the same or highly comparable results, with analyses finding conservatively that the maximum probable difference between the two total scores is less than 1 point. In short, the forms can be considered functionally equal, and can be used interchangeably in practice with confidence. A zero-anchored PCL provides respondents with intuitive anchors to rate the presence (or absence) of symptoms, which may facilitate comprehension of the measure and reduce item nonresponse, resulting in more complete and accurate symptom assessments. 31 Additionally, a zero-anchored PCL improves the measure s clinical utility as items rated 0 can be entirely excluded from the summation to ease scoring (and reduce errors) by nurses and doctors in busy clinical settings. In the context of U.S. military primary care practice, use of the zero-anchored PCL is consistent with other clinical measures routinely used (which all utilize zero-to-positive response scales), so clinic staff does not have to remember differential scoring procedures. Beyond screening, a zeroanchored PCL may facilitate routine symptom monitoring as part of measurement-based PTSD treatment, thus improving the quality of care. Its advantages will also be relevant for researchers who use the PCL as a brief tool to determine PTSD caseness or clinical disposition, particularly since a potential reduction in item nonresponse means fewer missing data points. Findings from the current study lend support that use of the zero-anchored scale does not compromise the validity of the construct being evaluated on the basis of differential Likert-type scaling. Of note, a new version of the PCL (i.e., the PCL-5) 18 was recently published to be consistent with revisions to the diagnostic criteria for PTSD in the DSM-5 and, accordingly, includes a zero-anchored scale. Although field trials are underway to demonstrate the psychometric properties of the PCL-5, it is unlikely that these trials will specifically test the effect of the revised response scale. Thus, our findings uniquely support the PCL-5 decision to scale symptoms on a zero-anchored scale. Notably, given the cost of transitioning health information systems to the DSM-5 version of the PCL, its longer format (20 items versus the 17 items of the DSM-IV-based PCL), and the nascent body of research establishing its measurement properties, the practice in many clinical settings (e.g., military primary care settings) may involve continued use of the DSM-IV version of the PCL. New DSM-5 measures will require extensive psychometric testing and validation before they are widely implemented with confidence. Additionally, the transition from DSM-IV to DSM-5 will require significant organizational practice change, necessitating revised forms, information systems, diagnostic and treatment procedures, and staff training. Thus, DSM-IV measures will likely remain relevant for the foreseeable future. This study was limited by the size and composition of the sample. Specifically, our consecutive sample from one military primary care clinic is not demographically representative of patients seen in all primary care clinics, particularly in civilian settings. Although further validation of the revised measure in a larger and more representative sample will strengthen our conclusion of equivalence, we saw no indication of differential responding in our demographically diverse sample, even across levels of symptom severity. Additionally, in the ICI method applied in this study, the chosen Δ value (the amount of difference between means that is considered acceptable) is specific to military primary care practice recommendations, and may differ in other settings. However, given the small (less than 1 point) observed maximum probable difference between total score means, it is likely that the conclusion of equivalence will be widely applicable. Finally, the current study only examined the civilian version of the PCL. There are no published reports on the interchangeability of PCL versions, and thus future research will be necessary to confirm that the findings from this study are generalizable to the military and specific versions. 32 CONCLUSION In sum, the revised version of the PCL with a zero-anchored response scale demonstrated psychometric equivalence to the original one-anchored scale. These findings validate the use of a zero-anchored PCL in Department of Defense primary care screening protocols, and lend support to wider efforts to 1005

5 use zero-anchored response scales for measurement-based clinical practice. Use of the zero-anchored item response scale has the potential to improve the accuracy and clinical utility of the measure. ACKNOWLEDGMENT This study was internally funded by the Deployment Health Clinical Center; no external funds were received. REFERENCES 1. Weathers FW, Litz BT, Herman D, Huska JA, Keane TM: The PTSD Checklist: Reliability, Validity, and Diagnostic Utility. Annual Convention of the International Society for Traumatic Stress Studies, San Antonio, TX, Elhai JD, Gray MJ, Kashdan TB, Franklin CL: Which instruments are most commonly used to assess traumatic event exposure and posttraumatic effects? A survey of traumatic stress professionals. J Trauma Stress 2005; 18(5): Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA: Psychometric properties of the PTSD checklist (PCL). Behav Res Ther 1996; 34(8): Yeager DE, Magruder KM, Knapp RG, Nicholas JS, Frueh BC: Performance characteristics of the posttraumatic stress disorder checklist and SPAN in Veterans Affairs primary care settings. Gen Hosp Psychiatry 2007; 29(4): Prins A, Ouimette P, Kimerling R, et al: The primary care PTSD screen (PC-PTSD): development and operating characteristics. Primary Care Psychiatry 2003; 9: Wilkins KC, Lang AJ, Norman SB: Synthesis of the psychometric properties of the PTSD checklist (PCL) military, civilian, and specific versions. Depress Anxiety 2011; 28(7): Mansfield AJ, Williams J, Hourani LL, Babeu L: Measurement invariance of posttraumatic stress disorder symptoms among U.S. military personnel. J Trauma Stress 2010; 23(1): Keen SM, Kutter CJ, Niles BL, Krinsley KE: Psychometric properties of PTSD checklist in sample of male veterans. J Rehabil Res Dev 2008; 45(3): Freedy JR, Steenkamp MM, Magruder KM, et al: Post-traumatic stress disorder screening test performance in civilian primary care. Fam Pract 2010; 27(6): Gore KL, McCutchan PK, Prins A, et al: Operating characteristics of the PTSD checklist in a military primary care setting. Psychol Assess 2013; 25(3): Under Secretary of Defense for Personnel and Readiness: Integration of behavioral health personnel (BHP) services into patient-centered medical home (PCMH) primary care and other primary care service settings (DoD Instruction ). Washington, DC, Department of Defense, Available at accessed March 1, O Muircheartaigh CA, Gaskell GD, Wright DB: Evaluating numeric and verbal labels for response scales. 48th Annual Conference of the American Association for Public Opinion Research, St. Charles, IL, Available at pdf; accessed November 11, Schwarz N, Knauper B, Hippler HJ, Noelle-Neumann E, Clark L: Rating scales numeric values may change the meaning of scale labels. Public Opin Q 1991; 55(4): American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, Ed. 5. Washington, DC, American Psychiatric Association, Burke J, Kraemer H, Shaffer D: DSM-5 task force considers dimensional, severity measures. Psychiatric News, Available at psychnews.psychiatryonline.org/doi/full/ /pn psychnews_ 45_17_013; accessed April 4, Clarke DE, Narrow WE, Regier DA, et al: DSM-5 fieldtrialsinthe United States and Canada, part I: study design, sampling strategy, implementation, and analytic approaches. Am J Psychiatry 2013; 170(1): Regier DA, Narrow WE, Clarke DE, et al: DSM-5 field trials in the United States and Canada, part II: test-retest reliability of selected categorical diagnoses. Am J Psychiatry 2013; 170(1): Weathers F, Litz B, Keane T, Palmieri P, Marx B, Schnurr P: The PTSD checklist for DSM-5 (PCL-5), Available from the National Center for PTSD at adult-sr/ptsd-checklist.asp; accessed November 11, LeBeau R, Mischel E, Resnick H, Kilpatrick D, Friedman M, Craske M: Dimensional assessment of posttraumatic stress disorder in DSM-5. Psychiatry Res 2014; 218(1 2): Schwarz N: Self-reports: how the questions shape the answers. Am Psychol 1999; 54(2): Furr RM, Bacharach VR: Psychometrics: An Introduction. Thousand Oaks, CA, Sage Publications, Engel CC, Oxman T, Yamamoto C, et al: RESPECT-Mil: feasibility of a systems-level collaborative care approach to depression and posttraumatic stress disorder in military primary care. Mil Med 2008; 173(10): Wong E, Jaycox L, Ayer L, et al: Evaluating the Implementation of the re-engineering systems of primary care treatment in the military (RESPECT-Mil). Santa Monica, CA, RAND Corporation, Available at RR588/RAND_RR588.pdf; accessed November 11, Adkins JW, Weathers FW, McDevitt-Murphy M, Daniels JB: Psychometric properties of seven self-report measures of posttraumatic stress disorder in college students with mixed civilian trauma exposure. J Anxiety Disord 2008; 22(8): Ruggiero KJ, Del Ben K, Scotti JR, Rabalais AE: Psychometric properties of the PTSD Checklist-civilian version. J Trauma Stress 2003; 16(5): Podsakoff PM, Podsakoff NP, MacKenzie SB, Lee J-Y: Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 2003; 88(5): Stegner B, Bostrom A, Greenfield T: Equivalence testing for use in psychosocial and services research: an introduction with examples. Evaluation and Program Planning 1996; 19(3): Tryon WW: Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: an integrated alternative method of conducting null hypothesis statistical tests. Psychol Methods 2001; 6(4): Tryon WW, Lewis C: An inferential confidence interval method of establishing statistical equivalence that corrects Tryon s (2001) reduction factor. Psychol Methods 2008; 13(3): Goldstein H, Healy M: The graphical presentation of a collection of means. J Roy Statist Soc 1995; 158A(Part 1): Courser M, Lavrakas P: Item nonresponse and the 10-point response scale in telephone surveys. Survey Practice 2012; 5(4). 32. McDonald SD, Calhoun PS: The diagnostic accuracy of the PTSD checklist: a critical review. Clin Psychol Rev 2010; 30(8):