Reliability of the Deployment Resiliency Assessment

BRIEF REPORTS MILITARY MEDICINE, 181, 7:638, 2016 Reliability of the Deployment Resiliency Assessment Samuel E. Simon, PhD*; Kate Stewart, PhD*; Michelle Kloc, PhD ; Thomas V. Williams, PhD ; MG Margaret C. Wilmoth, USAR ABSTRACT This article describes the reliability of the instruments embedded in a mental health screening instrument designed to detect risky drinking, depression, and post-traumatic stress disorder among members of the Armed Forces. The instruments were generally reliable, however, the risky drinking screen (Alcohol Use Disorders Identification Test- Consumption) had unacceptable reliability (α = 0.58). This was the first attempt to assess psychometric properties of a screening and assessment instrument widely used for members of the Armed Forces. INTRODUCTION Military deployment and exposure to combat have been demonstrated to result in higher risk for post-traumatic stress disorder (PTSD), major depression, and substance abuse. 1 4 Beginning in 2003, the Department of Defense (DoD) mandated that all service members be required to complete a brief (Pre) Postdeployment Health Assessment (PDHA) 60 days before deployment, and postdeployment at 30 days, 90 to 180 days, and annually thereafter. 5 The PDHA is self-administered and contains questions about PTSD symptoms, depression, suicidal ideation, aggression, and mental health concerns. Administration of the PDHA is followed immediately by an interview with a credentialed health care professional for potential further additional evaluation. Recognizing the need to formalize a staged screening process for mental health conditions, the National Defense Authorization Act, signed into law in December 2011, took the PDHA requirements further by directing the DoD to administer a staged mental health assessment process for deployed members of the Armed Forces (Section 708). The Deployment Resiliency Assessment (DRA), the Air Force version of the staged mental health screening instrument, is integrated into the PDHA and is designed to identify mental health conditions including PTSD, depression, risky *Mathematica Policy Research, 955 Massachusetts Avenue, Suite 801, Cambridge, MA 02139. Altarum Institute, 2000 M Street, NW, Suite 400, Washington, DC, 20036. Defense Health ncy, 7700 Arlington Boulevard, Suite 5101, Falls Church, VA 22042-5101. Byrdine F. Lewis School of Nursing and Health Professions, Georgia State University, P. O. Box 3995, Atlanta, GA 30302-3995. This article was presented at the Academy Health s Annual Research Meeting, Baltimore, MD, June 24, 2013. This work was done under contract to the Department of Health and Human Services, Program Support Center, the U.S. Department of Defense, Short Term Policy Assessment and Studies, contract HHSP-23320095642WC. doi: 10.7205/MILMED-D-15-00229 alcohol use, and other behavioral conditions that require referral for additional care and treatment. By law, administration of the DRA is required within 2 months before the estimated date of deployment, 3 to 6 months after return from deployment, 7 to 12 months after return from deployment, and again 16 to 24 months after return from deployment. The DRA is conducted in three stages. Each service member completes the first stage, which consists of screening instruments to identify major life stressors, mental health history and mental health symptoms, medication use, alcohol use, and mental health concerns identified by the deployee. Table I shows the screening instruments contained in Stage 1 and Stage 2 of the DRA. If the Stage 1 screens for PTSD or depression are positive, the deployee completes additional items that assess symptom severity of PTSD using the PTSD Checklist Civilian version (PCL-C) and/or depression (Patient Health Questionnaire [PHQ-8]) in Stage 2 of the assessment. For individuals who screened positive at the second stage, the third stage consists of a person-to-person dialogue between the deployee and a health care provider for referrals and education if necessary. Deployees who do not screen positive at Stage 1 also interact directly with a provider to clarify responses, identify areas of concern, and refer to additional care if necessary. The screening instruments (AUDIT-C, PHQ-2, and Primary Care-PTSD [PC-PTSD]) and symptom severity instruments (PHQ-8 and PCL-C) form the basis of the DRA and were chosen for inclusion in the DRA based on prior evidence indicating they were highly reliable and valid instruments. 6 8 Despite the availability of a military version of the PCL instrument (PCL-M), the PCL-C was selected for inclusion in the DRA based on its focus on a wide range of experiences, not just those related to military experiences. A potential weakness of the DRA is that reliability and validity of these instruments were established primarily among the civilian, general medical 638

TABLE I. DRA Screening and Assessment Instruments Stage 1 (Screening) Stage 2 (Assessment) a Domain Instrument Title Scoring Details Instrument Title Scoring Details Risky Drinking Alcohol Use Three-item screen designed N/A N/A Disorders Identification Test-Consumption (AUDIT-C) to identify risky drinking behavior. AUDIT-C scores range from 0 to 12. Females with a score of 4 and higher and males with a score of 5 and higher are considered positive. Depression PTSD Patient Health Questionnaire (PHQ-2) Primary Care PTSD Screen (PC-PTSD) Two-item screen for depression; scores range from 0 to 6. A positive depression screen occurs when a respondent endorses either PHQ-2 statement as occurring more than half the days or nearly every day. Four-item PTSD screen with scores that range from 0 to 4. Endorsement of any two items results in a positive screen. a Stage 2 completed only if stage 1 screen for PTSD or depression is positive. population. However, the DoD must ensure that mental health assessments of deployed individuals are based on scientifically sound instruments that yield reliable information about the mental health of deployed forces. The goal of this study was to evaluate the reliability of the instruments embedded within the DRA administered to deployed Air Force personnel. In addition to examining reliability for the entire sample, we repeated our analyses across a range of demographic subgroups (age, sex, marital status, race, and pay grade) to ensure the applicability of the assessments to these subpopulations. METHODS This study involved secondary data analysis of demographic data and linked DRA data collected by the Air Force from January to August 2011. The Public/Private Ventures Institutional Review Board approved the study as exempt. Sample The Air Force provided DRA assessments for 99,769 service members. We selected the first DRA assessment available for each service member. Service members DRA assessment records were matched with demographic variables from the Defense Enrollment Eligibility Reporting System (DEERS) based on Social Security number matches. We matched all 99,769 service members DRA records with a corresponding DEERS record. The Stage 1 sample sizes were as follows: 99,577 for the AUDIT-C, 98,540 for the PHQ-2, and 98,580 for the PC-PTSD. A small subgroup completed Stage 2 assessments (n = 2,074). Patient Health Questionnaire (PHQ-8) PTSD Checklist- Civilian version (PCL-C) The PHQ-8 assesses the frequency of eight depression symptoms, omitting the suicide screening question from the PHQ-9. Suicide screening and assessment occurs in the person-to-person interview (stage 3) completed by all individuals. Seventeen-item validated assessment of Diagnostic and Statistical Manual of Mental Disorders-IV PTSD symptom criteria and symptom severity covering the past month. Five-levels of responses are possible for each symptom, ranging from Not at all to Extremely. Table II describes the demographic characteristics of the sample. Air Force deployees were overwhelming male and nearly half were ages 25 to 34. About 20% of sample members were officers. Most of the sample were classified as active duty (85%), whereas 10%were members of the Air Force Guard, and another 5% were listed as being in the Air Force Reserves (data not shown in Table II). Measures Screening and severity measures were coded according to DoD training documentation. We used frequency distributions to examine the relative sizes of subgroups within the analytic sample and to describe the prevalence and severity of the conditions assessed by the DRA. We used chi-square tests to assess the significance of differences in positive screens within subgroups. We inferred the reliability of each instrument by computing Cronbach s reliability (α) coefficient for each screening and severity instrument embedded within the DRA. Cronbach s α detects the internal consistency of a set of items to show how closely related the items are as a group. As a rule, values of 0.7 and higher are considered acceptable. 9 All analyses were conducted using the SAS system software, version 9.0 (SAS, Cary, North Carolina). RESULTS Prevalence and Severity of Screened Conditions Overall, screening rates of risky drinking, depression, and PTSD were generally low in this sample of Air Force deployees. 639

TABLE II. Demographic Characteristics of the Sample Characteristic N (%) Total 99,769 (100.0) Male 84,630 (84.8) Female 15,138 (15.2) Missing 1 (0.0) 18 24 28,935 (29.0) 25 34 46,236 (46.3) 35 44 19,529 (19.6) 45 64 5,050 (5.1) Missing 19 (0.0) Single, Never Married 33,327 (33.4) Married 58,542 (58.7) Divorced 7,750 (7.8) Other 134 (0.1) Missing 16 (0.0) White, Not Hispanic 72,939 (73.1) Black, Not Hispanic 13,147 (13.2) Hispanic 5,075 (5.1) Asian/Pacific Islander 4,696 (4.7) Other 3,179 (3.2) Unknown 512 (0.5) Missing 221 (0.2) Officer 18,765 (18.8) Nonofficer 80,963 (81.2) Missing 41 (0.0) Source: Analysis of merged DRA-DEERS data. TABLE III. AUDIT-C (N = 99,577) Risky Drinking One in 25 Air Force deployees met the threshold for risky drinking based on the AUDIT-C summary score (4%, see Table III). Rates of positive AUDIT-C screens were highest for females, those between the ages of 18 and 24, and those who were divorced or never married (Table III). Note that the AUDIT-C screen has different screening thresholds for men (score of 5 or higher) and women (score of 4 or higher). Depression Table III shows that overall, 1% of the sample screened positive for depression on the PHQ-2, with women and black respondents having the highest rates of depression in this sample. According to the DRA protocol, individuals who screen positive on the PHQ-2 are instructed to complete the PHQ-8. The overall size of the sample population that screened positive on the PHQ-2 was small (1%); however, about one-fourth of those service members who screened positive had moderate or severe depressive symptoms as measured by the PHQ-8 (23.2%). Post-Traumatic Stress Disorder Overall, 1.4% of the sample had a positive PTSD screen. Females, service members ages 35 to 44, and divorced individuals had the highest rates of positive screens on the PC-PTSD (Table III). According to the DRA protocol, individuals who screen positive on the PC-PTSD are instructed to complete the PCL-C. Among the 1,364 service members who screened positive on the PC-PTSD, about a third had moderate-to-severe PTSD symptoms (34.9%) as measured by the PTSD-C instrument. DRA Positive Screen Rates, Overall and by Subgroup PHQ-2 (N = 98,540) PC-PTSD (N = 98,580) Overall 4.0 1.0 1.4 Male 3.8 1.0 1.2 Female 5.7 ** 1.6 ** 2.3 ** 18 24 5.4 1.0 1.0 25 34 3.6 0.9 1.3 35 44 3.1 1.4 2.2 1 45 64 4.0 ** 1.2 ** 1.4 ** Divorced 5.1 1.4 2.2 Married 3.1 1.1 1.5 Never Married 5.4 ** 1.0 * 0.9 ** Asian/Pacific Islander 4.0 1.1 1.5 Black, Not Hispanic 3.6 1.5 1.6 White, Not Hispanic 4.1 1.0 1.3 Hispanic 3.7 * 1.2 ** 1.7 * Nonofficer 4.2 1.2 1.5 Officer 3.4 ** 0.6 ** 1.0 ** Source: Analysis of merged DRA-DEERS data. Chi-square significance of difference between subgroup categories: **p < 0.01; *p < 0.05. 640

Reliability of Screening and Severity Measures Table IV displays the reliability of the screening and severity measures, overall and within subgroups. Cronbach s α reliability estimates ranged from 0.58 (AUDIT-C) to 0.93 (PCL-C). Using 0.70 as the threshold for acceptable reliability, this finding indicates the measures retain acceptable to good reliability when measured among deployed members of the Air Force, with the exception of the AUDIT-C. The reliability coefficient for AUDIT-C was below the conventional threshold of 0.70, indicating poor reliability. Subgroup analysis revealed that the AUDIT-C had acceptable reliability estimates among younger service members. Reliability appears to be inversely related to age, as the oldest service members had the least reliable AUDIT-C scores. DISCUSSION The instruments embedded in the DRA were originally developed to screen and assess mental health conditions among clinical populations. Given the need to make sound policy decisions related to the mental health of deployed military service members, we undertook this study to confirm the psychometric properties of the instruments in the DRA. 10 This is the first study that reports the psychometric properties of the DRA measures among deployed military service members. Overall, we found preliminary evidence to support the validity of DRA in a population of deployed Air Force members. The evidence for reliability of the DRA instruments was mixed. Cronbach s reliability estimates for the TABLE IV. Cronbach s α Reliability Coefficient Across DRA Instruments, Overall and by Subgroup AUDIT-C PHQ-2 PHQ-8 PC-PTSD PCL-C Overall 0.58 0.82 0.79 0.78 0.93 Male 0.59 0.80 0.79 0.78 0.92 Female 0.48 0.85 0.77 0.78 0.94 18 24 0.70 0.79 0.80 0.74 0.92 25 34 0.57 0.81 0.77 0.78 0.92 35 44 0.49 0.84 0.79 0.78 0.94 45 64 0.45 0.84 0.70 0.79 0.93 Divorced 0.59 0.82 0.73 0.76 0.90 Married 0.53 0.82 0.79 0.78 0.93 Never Married 0.66 0.81 0.80 0.75 0.92 Asian/Pacific 0.62 0.83 0.82 0.82 0.94 Islander Black, 0.59 0.79 0.79 0.75 0.93 Not Hispanic White, 0.58 0.82 0.79 0.77 0.93 Not Hispanic Hispanic 0.60 0.82 0.77 0.78 0.94 Nonofficer 0.63 0.82 0.79 0.78 0.93 Officer 0.42 0.81 0.76 0.76 0.93 Source: Analysis of merged DRA-DEERS data. depression and PTSD instruments in the DRA ranged from acceptable (PC-PTSD and PHQ-8) to excellent (PCL-C). In contrast, the AUDIT-C instrument was the least reliable instrument in the DRA and had a reliability estimate below the conventional level of acceptable reliability. Therefore, DRA assessment of risky drinking behavior was found to be insufficiently reliable with the exception of service members between 18 and 24 (Cronbach s α = 0.70). The principal aim of deployment mental health assessments is to identify mental health conditions that require referral for additional care and treatment in order to ensure that the armed services are achieving individual and unit readiness, or the capabilities for battlefield success. Risky drinking behavior is a readiness concern for DoD as rates of heavy alcohol use among active duty military personnel increased significantly between 1998 and 2008. 11 This trend underscores DoD s need to monitor risky drinking, making the finding of lower reliability for the AUDIT-C notable. Some literature suggests that the reliability of the AUDIT-C might be lower in populationbased or nonclinical settings. In clinical populations involving patients treated in emergency rooms or alcohol treatment centers, the reliability of the AUDIT-C has been shown to be quite high (α coefficient = 0.94), whereas other studies in the general population found lower reliability (α coefficient = 0.56). 12,13 Another study of a large population-based military cohort found low reliability for questions from the PHQ items ascertaining alcohol abuse as well as for another alcohol abuse screening instrument, the CAGE questionnaire (α coefficient = 0.58, 0.64, respectively). 14 The CAGE is a four-item questionnaire designed to identify problem drinking through the following questions: (1) Have you ever felt you should Cut down on your drinking? (2) Have people Annoyed you by criticizing your drinking? (3) Have you ever felt bad or Guilty about your drinking? (4) Have you ever had adrinkfirst thing in the morning to steady your nerves or to get rid of a hangover (Eye opener)? 15 Each positive response receives a score of 1 and a total score of 2 or higher is considered clinically significant. We found the reliability estimate for the AUDIT-C to be highest among younger service members (α coefficient = 0.70). Younger service members also had the highest rates of positive AUDIT-C screens. The screening rates for risky drinking, depression, and PTSD in our sample were low. A recent article by Larson et al 16 found rates of risky drinking among active duty Army service members returning from Iraq or Afghanistan deployments was 28.8% using the AUDIT-C, a rate more than seven times higher compared to the rate in our study. Studies using the PHQ-2 as a screen for depression have found rates of 1.9 to 4.5% among deployed service members, substantially higher than the rates we found. 5,17 Rates of PTSD are also substantially higher in other studies of service members, ranging from 3.3 to 9.8%, although these rates were drawn from Army and Marine service members returning from Iraq or Afghanistan. 5,17 Combined, these findings suggest that the rates from our study are not representative of all military service members and 641

therefore underscore the need for additional examination of the reliability of these instruments among deployed service members from different branches of the military. A consequence of the low-screening rates in our sample is a notably skewed distribution for the depression and PTSD screening instruments. The PHQ-2 and PC-PTSD instruments had skewness values of 5.2 and 8.0, respectively. Because these instruments are not normally distributed, our reliability findings for these instruments should be interpreted with caution. The AUDIT-C, PCL, and PHQ-8 were much closer to a normal distribution, with skewness values <1.0. We recommend further testing of the AUDIT-C in other service branches to understand the extent to which the instrument lacks reliability, because it is possible the AUDIT-C screen is more reliable in other branches, particularly if rates of problem drinking exceed those found in this sample of Air Force service members. We note the findings of Shen, Arkes, and Williams who found differential rates of substance use disorders and depression among deployed service members across branches of the military. 18 We also recommend the nonabbreviated version of the AUDIT-C (that is, the 10-item AUDIT instrument) should be considered for testing and use in this population, as others have found the longer version of the instrument to be reliable in a nonclinical population, with an alpha of 0.75. 13 Another alternative brief instrument to assess risky drinking is the four-item CAGE questionnaire, although it too demonstrated poor reliability in a military population. 14 The external validity of this study is limited by the representativeness of our sample. Because we used data from one branch of the Armed Forces, the findings are unlikely to be generalizable to the other branches of the Armed Forces. Respondents in this sample also had relatively low-screening rates of depression and PTSD, which could affect the reliability findings reported here. Replication of our findings using data from other branches of the Armed Forces is recommended to ensure the instruments are reliable and valid for all service members. Additional work should also examine criterion validity for the first stage of mental health screens in the DRA assessing problem drinking, depression, and PTSD. Refinement of the instruments would improve the usefulness of the assessment process, individual and unit readiness, and the value of clinical information shared between health providers in the Military Health System and those from the Department of Veterans Affairs who may be called on for continued service member care. Study limitations include the fact that the DRA was recently implemented during the period when data were collected for this study. Administration of the instrument may have been refined during this period, which could have affected the findings reported here. Furthermore, as noted above, criterion validity would provide stronger evidence of the validity of the DRA assessment process and should be examined to bolster our findings based on secondary data analysis. This study was the first attempt to assess psychometric properties of a widely used screening and assessment instrument. We examined the reliability of the instruments contained within the DRA as doing so is a necessary precursor to establishing the validity of the instruments. The mental health assessment instruments embedded in the DRA were found to be generally reliable for use with this nonclinical population; the reliability of drinking behavior should be explored further among nonclinical populations. REFERENCES 1. Jordan BK, Schlenger WE, Hough R, et al: Lifetime and current prevalence of specific psychiatric disorders among Vietnam veterans and controls. Arch Gen Psychiatry 1991; 48: 207 15. 2. Kessler RC, Sonnega A, Bromet E, Hughes M, Neson CB: Posttraumatic stress disorder in the National Comorbidity Study. Arch Gen Psychiatry 1995; 52: 1048 60. 3. Prigerson HG, Maciejewski PK, Rosenheck RA: Population attributable fractions of psychiatric disorders and behavioral outcomes associated with combat exposure among US men. Am J Public Health 2002; 92: 59 63. 4. Self-reported illness and health status among Gulf War veterans: a population-based study. The Iowa Persian Gulf Study Group. JAMA 1997; 277: 238 45. 5. Hoge CW, Auchterlonie JL, Milliken CS: Mental health problems, use of mental health services, and attrition from military service after returning from deployment to Iraq or Afghanistan. JAMA 2006; 295: 102332. 6. Arroll B, Goodyear-Smith F, Crengle S, et al: Validation of PHQ-2 and PHQ-9 to screen for major depression in the primary care population. Ann Fam Med 2010; 8: 348 53. 7. Bliese PD, Wright KM, Adler AB, Cabrera O, Castro CA, Hoge CW: Validating the primary care posttraumatic stress disorder screen and the posttraumatic stress disorder checklist with soldiers returning from combat. J Consult Clin Psychol 2008; 76: 272 81. 8. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Arch Intern Med 1998; 158: 1789 95. 9. Nunnally JC: Psychometric Theory, Ed 2. New York, McGraw-Hill, 1978. 10. Milliken CS, Auchterlonie JL, Hoge CW: Longitudinal assessment of mental health problems among active and reserve component soldiers returning from the Iraq war. JAMA 2007; 298: 2141 8. 11. Bray RM, Pemberton MR, Hourani LL, et al: 2008 Department of Defense survey of health related behaviors among active duty military personnel. Research Triangle Park, RTI International, NC, 2009. Available at https:// www.documentcloud.org/documents/694941-2008-department-of-defensesurvey-of-health.html; accessed October 3, 2014. 12. Meneses-Gaya C, Zuardi AW, Loureiro SR, et al: Is the full version of the AUDIT really necessary? Study of the validity and internal construct of its abbreviated versions. Alcohol Clin Exp Res 2010; 34: 1417 24. 13. Rumpf HJ, Hapke U, Meyer C, John U: Screening for alcohol use disorders and at-risk drinking in the general population: psychometric performance of three questionnaires. Alcohol Alcohol 2002; 37: 261 8. 14. Smith TC, Smith B, Jacobson IG, Corbeil TE, Ryan MA, Millennium Cohort Study Team: Reliability of standard health assessment instruments in a large population-based cohort study. Ann Epidemiol 2007; 17: 525 32. 15. Ewing J: Detecting alcoholism. The CAGE questionnaire. JAMA 1984; 252: 1905 7. 16. Larson MJ, Mohr BA, Adams RS, Wooten NR, Williams TV: Missed opportunity for alcohol problem prevention among army active duty services members postdeployment. Am J Public Health 2014; 104: 1402 12. 17. Warner CW, Appenzeller GN, Grieger T, et al: Importance of anonymity to encourage honest reporting in mental health screening after combat deployment. Arch Gen Psychiatry 2011; 68: 1065 71. 18. Shen YC, Arkes J, Williams TV: Effects of Iraq/Afghanistan deployments on major depression and substance use disorder: analysis of active duty personnel in the US military. Am J Public Health 2012; 102(Suppl 1): S80 7. 642