2016 Workplace and Gender Relations Survey of Active Duty Members. Nonresponse Bias Analysis Report

Transcription

1 2016 Workplace and Gender Relations Survey of Active Duty Members Nonresponse Bias Analysis Report

2 Additional copies of this report may be obtained from: Defense Technical Information Center ATTN: DTIC-BRR 8725 John J. Kingman Rd., Suite #0944 Ft. Belvoir, VA Or from:

3 OPA Report No May WORKPLACE AND GENDER RELATIONS SURVEY OF ACTIVE DUTY MEMBERS: NONRESPONSE BIAS ANALYSIS REPORT Office of People Analytics (OPA) Defense Research, Surveys, and Statistics Center 4800 Mark Center Drive, Suite 06E22, Alexandria, VA

4 Contributors Many members of the Office of People Analytics (OPA) contributed to the analyses and writing of this report assessing the level and direction of potential nonresponse bias in estimates from the 2016 Workplace and Gender Relations Survey of Active Duty (WGRA). Sections of the report were written by the following authors: Eric Falk wrote Section 1: Compare Known Population Values with Weighted Survey Estimates. Jeff Schneider wrote Section 2: Analysis of OPA s Survey of Nonrespondents based on the 2016 Workplace and Gender Relations Survey of Active Duty Nonresponse Bias Survey. Jeff Schneider wrote Section 3: Evaluate the Sensitivity of Different Post-Survey Adjustments (Weighting Methods) on Survey Estimates. Reviews and comments were provided by David McGrath and Lisa Davis. Eric Falk and David McGrath guided the studies and served as primary editors. ii

5 Table of Contents Page Summary of Findings...1 Section 1: Comparison of Known Population Values With Weighted Survey Estimates...3 Summary...6 Section 2: Analysis of OPA s Survey of Nonrespondents...7 Weighting the 2016 WGRA-N...10 Summary...13 Section 3: Evaluate the Sensitivity of Different Post-Survey Adjustments (Weighting Methods) on Survey Estimates...15 OPA Weighting Methodology...15 Comparison of Adjustment Stages and Final Weights...16 Comparison of Key Estimates...17 Summary...18 References...21 List of Tables WGRA Reporting Questions Summary of Sexual Assault Reports in DSAID, by Type of Report and Service WGRA Estimates vs. Actual Number of Reported Sexual Assaults WGRA-N Comparison Questions: Control Questions WGRA-N Comparison Questions: MEO Questions Sample Disposition Codes for 2016 WGRA-N Comparison of 2016 WGRA Sample With Nonresponse Sample 2016 WGRA-N WGRA-N Final Weight Moments Comparison of WGRA Survey With Nonresponse Study Control Questions Comparison of WGRA Survey With Nonresponse Study MEO Questions Comparison Between Adjustment Factor: Standard OPA vs. 2-Stage Boosted Method for Eligibility, Completion and Poststratification Adjustments Comparison Between Standard OPA and 2-Stage Boosted Final Weights Comparison of OPA and Westat Key Survey Estimates (Males) Comparison of OPA and Westat Key Survey Estimates (Females)...18 iii

6

7 2016 WORKPLACE AND GENDER RELATIONS SURVEY OF ACTIVE DUTY MEMBERS: NONRESPONSE BIAS ANALYSIS REPORT Survey nonresponse has the potential to introduce bias in the estimates of key outcomes. To the extent that nonrespondents and respondents differ on observed characteristics, the Office of People Analytics (OPA) can use weights to adjust the sample so the weighted respondents match the full population on the most critical characteristics. This eliminates the portion of nonresponse bias (NRB) associated with those observed variables if these variables are strongly associated with the behaviors. When all NRB can be eliminated in this manner, the missingness is called ignorable or missing at random (Little & Rubin, 2002). The more observable demographic variables that were incorporated into the weights, the more plausible it is to assume that the weights eliminate any NRB. The objective of this research was to assess the extent of NRB for the estimated percentage of members who indicated experiencing a sexual assault in the prior 12 months (henceforth referred to as the sexual assault rate) in the active duty military. The level of nonresponse bias can vary for every question on the survey, but OPA focused on the sexual assault rate as this is the most important survey topic. Nonresponse bias occurs when survey respondents are systematically different from nonrespondents. Statistically, the bias in a respondent mean (e.g., sexual assault rate) is a function of the response rate and the relationship (covariance) between response propensities and the estimated statistics (i.e., sexual assault rate), and takes the following form: propensity,, where = covariance between and response NRB can occur with high or low survey response rates, but the decrease in overall survey response rates within the Department as well as in civilian studies in the past decade have resulted in a greater focus on potential NRB. OPA investigated the presence of NRB using many different methods, and are summarized in this report as follows: 1. Comparison of known population values with weighted survey estimates, 2. Analysis of OPA s survey of nonrespondents, and 3. Evaluating the sensitivity of different post-survey adjustments (weighting methods) on survey estimates. Summary of Findings NRB is difficult to assess and therefore most authors recommend averaging across several different studies to measure NRB (Montaquila & Olson, 2012). OPA has taken this approach here and conducted three studies to assess NRB in sexual assault estimates. Based on these three studies, OPA did not find evidence of significant NRB in sexual assault estimates from the 2016 WGRA. 1

8 The results from each study are summarized below: 1. Comparison of known population values with weighted survey estimates OPA compared weighted estimates of reported sexual assaults from the survey with actual reported sexual assaults to SAPRO. From this analysis, OPA concludes there is no evidence of NRB in sexual assault estimates from the 2016 WGRA. 2. Analysis of OPA s survey of nonrespondents Results from the 2016 WGRA Non- Responders Study (2016 WGRA-N) were mixed. OPA determined five of the eight estimates from overlapping questions were significantly different between the 2016 WGRA and 2016 WGRA-N which may be an indication of NRB. However, the estimates from one particular survey were not consistently more positive or negative in comparison to the other survey. Additionally, given the low response rate to the nonresponse study (under 5%, an extremely low number for a standard OPA survey) and the fact that the NRB study was a non-probability sample, the results should be used cautiously. From this analysis, OPA concludes there is little evidence of NRB in sexual assault estimates from the 2016 WGRA. 3. Evaluation of the sensitivity of different post-survey adjustments (weighting methods) on survey estimates Analysis of estimates using four different weighting methods show both the weights and key survey estimates are robust to the choice of weighting methods. From this analysis, OPA concludes that is little evidence of NRB in sexual assault estimates from the 2016 WGRA. 2

9 Section 1: Comparison of Known Population Values With Weighted Survey Estimates To assess total survey error, one common method is to compare a known parameter to a weighted estimate from the survey. 1 If OPA s sampling, measurement, weighting, and analysis methods performed well, confidence intervals of estimates should frequently contain the true parameters. In this investigation, OPA examined the number of reported sexual assaults in the active duty. A similar type of analysis was performed by the RAND Corporation for the 2014 RAND Military Workplace Study (2014 RMWS, Morral, 2015) and by OPA for the 2015 Workplace and Gender Relations Survey of Reserve Component Members (2015 WGRR, DMDC, 2016a). It is important to point out that OPA does not know the true number of sexual assaults in the active duty military. However, OPA compared the number of sexual assault reports filed by active duty members to weighted estimates from survey respondents to assess NRB (and overall total survey error). The Sexual Assault Prevention and Response Office (SAPRO) provided OPA with summary information of the number of reported sexual assaults (unrestricted and restricted), which was collected via the Defense Sexual Assault Incident Database (DSAID). For a record to be entered into DSAID, the survivor needed to complete a Victim Reporting Preference Statement (DDForm 2910) indicating whether the survivor would like to make either a restricted or unrestricted report. To match all possible 12-month reference periods for indicating a sexual assault on the 2016 WGRA, OPA requested the sexual assaults reported from July 1, 2015 through October 31, 2016 (16 months of data). For example, if a survey respondent completed the survey on July 22 and indicated they reported a sexual assault within the last year, DSAID would contain the incidence in the July 23, 2015 through July 22, 2016 timeframe. Rather than create all time frames that mirror the survey period, OPA decided to create five yearly timeframes as a basis for comparison to the estimates (see list below). In total there were 4,079 cases reported during the 16-month period provided by SAPRO and the five time periods that mirror the survey fielding period of July 22 to October 14 are: July 1, 2015 through June 30, 2016, August 1, 2015 through July 31, 2016, September 1, 2015 through August 31, 2016, October 1, 2015 through September 30, 2016, and November 1, 2015 through October 31, The summary files provided by SAPRO contained the number of sexual assaults by Service and whether the report type was restricted or unrestricted. On the 2016 WGRA survey, sexual assault survivors were asked follow-up questions to determine 1) if they filed a formal report of sexual assault, 2) type of report filed, and 3) to verify the sexual assault occurred within 1 For more information regarding the sampling and weighting of the 2016 WGRA please refer to 2016 Workplace and gender relations survey of active duty members: Statistical Methodology Report (Report No ) 3

10 the last 12 months. The 2016 WGRA survey questions regarding these behaviors are displayed in Table 1. Table WGRA Reporting Questions SAMILREPT 2016 WGRA Reporting Questions SA12MOS SAREPTYPE To ensure comparability between the DSAID file and the survey results, OPA selected cases with the following characteristics: Survivor service affiliation was either Army, Navy, Marine Corps, or Air Force, Survivor duty status was active, Survivor was in military at time of incident, and Date of incident occurred in the July 1, 2015 through October 31, 2016 timeframe. 4

11 Based on the definitions, the DSAID file provided by SAPRO contained a 12-month average of 3,097 restricted and unrestricted reports of sexual assault within the DoD Services. Table 2 summarizes sexual assault reports, by Service and type of report. Table 2. Summary of Sexual Assault Reports in DSAID, by Type of Report and Service Service/Component Restricted Unrestricted Total Army 181 1,025 1,206 Navy Marine Corps Air Force Total 608 2,489 3,097 Note: DSAID Reports were 12-month averages from July 1, 2015 to October 31, 2016 OPA used the following criteria from the survey to compare the information provided by respondents on the 2016 WGRA to actual reports in DSAID. Respondent indicated Yes to reported the sexual assault (Q129), Respondent indicated the report was restricted or unrestricted (Q131), 2 and Respondent confirmed the sexual assault occurred in the past 12 months (Q167). There were 448 DoD complete eligible respondents from the survey who indicated in Q129 they had reported a sexual assault (SAMILREPT=Yes). Of the 448 respondents, 389 indicated the sexual assault occurred in the last 12 months (Q167) and that they filed either a restricted or unrestricted report. 3 The weighted estimate based on these 389 respondents is 3,145, compared with the 3,097 average cases from DSAID during a 12 month period. The confidence interval from the survey estimate ranged from a lower bound of 2,847 to an upper bound of 3,443 and contains the true average number of 3,097 cases reported in DSAID. The estimate and the average actual are very close and only differ by 2%. Table 3 shows the 2016 WGRA number of reported sexual assaults, weighted estimates including 95 percent confidence intervals, and DSAID reported cases. For each Service, the 2016 WGRA confidence intervals contain the number of DSAID cases. For example, OPA estimates 1,259 Army reported cases of sexual assault with a 95 percent confidence interval ranging from 1,056 to 1,462, whereas the true number in DSAID for Army is 1, For purposes of this analysis those who indicated they were unsure were grouped with the unrestricted. 3 This included not only those who indicated the event definitely occurred within the last year (SA12MOS = Definitely occurred after X date) but those who are not sure if it was within the last year (SA12MOS = Not sure if it occurred before or after X date). In addition, this includes not only those who indicated that they filed a restricted or unrestricted report (SAREPTYPE = A restricted report or an unrestricted report) but also those who were unsure of the type of report filed (SAREPTYPE = Unsure what type of report was made). 5

12 RAND performed a similar analysis for the active duty in the 2014 RMWS and found that the survey estimates were systematically lower than the true DSAID number. This led RAND to conclude that sexual assault survivors that report are less likely to respond to the survey, and consequently survey estimates may be too low. It is unclear why RAND and OPA may find different results comparing weighted survey estimates to DSAID administrative data. Table WGRA Estimates vs. Actual Number of Reported Sexual Assaults Service Number of Respondents that filed a Report Weighted Total Estimate from Survey WGRA Reports 95% Confidence Interval Lower Bound of Estimate 95% Confidence Interval Upper Bound of Estimate Average Number of Reports in DSAID Army 127 1,259 1,056 1,462 1,206 Navy , Marine Corps Air Force Total 389 3,145 2,847 3,443 3,097 Note. The number of reports in DSAID is the average of the five 12-month totals within DSAID that correspond with the 2016 WGRA reference period for reporting sexual assaults, as described previously. Summary This NRB section assessed whether the 2016 WGRA survey estimates were similar to administrative data (cases in SAPRO s DSAID database). Differences between survey estimates and administrative data could be an indication of survey error, perhaps caused by nonresponse bias. 4 However, the survey estimates and the actual number of sexual assault survivors that made an unrestricted or restricted report in DSAID were very similar for each Service, and the confidence intervals contained the DSAID number in all cases. From this analysis OPA concludes there is no evidence of NRB in sexual assault estimates. 4 It is important to note that while likely accurate, errors do exist in administrative data. 6

13 Section 2: Analysis of OPA s Survey of Nonrespondents If survey respondents and nonrespondents have different sexual assault propensities that cannot be accounted for during survey weighting, it would result in biased survey estimates of sexual assault. OPA conducted a nonresponse study (2016 WGRA-N) based on a sample of nonrespondents from the original survey as another method to assess WGRA NRB. The 2016 WGRA-N was a much shorter web survey with only notifications (the 2016 WGRA had one postal notification, 10 reminders, and a paper form). The 2016 WGRA-N served two purposes. First, the 2016 WGRA-N assessed the 2016 WGRA and generated constructive feedback for future iterations of the survey. Second, the survey evaluated NRB by comparing the responses from the follow-up survey to the original survey. In particular, there were eight questions that were asked on both the 2016 WGRA and 2016 WGRA-N to assess NRB. These eight questions can be split into two groups: control questions and Military Equal Opportunity (MEO) questions. The control questions cover topics such as retention, health, and assessment of DoD policy toward gender related topics. Table 4 shows the four control questions. Table WGRA-N Comparison Questions: Control Questions 2016 WGRA-N Control Questions The four MEO questions are displayed in Table 5. 7

14 Table WGRA-N Comparison Questions: MEO Questions 2016 WGRA-N Comparison Questions If estimates from either set (control or MEO) of these matching questions were significantly different, that could be indicative of the presence of NRB in these questions (and potentially other correlated questions such as the sexual assault rate). For the 2016 WGRA, OPA selected a sample of 696,329 from the 1,291,357 DoD active duty members. There were 378, WGRA DoD nonrespondents (SAMP_DC=11), and OPA selected a sample of 100,811 for the 2016 WGRA-N. The sample was selected using the same sampling strata as the 2016 WGRA. Similar to the 2016 WGRA survey, OPA created sample disposition codes using the criteria shown in Table 6. The table shows that 2,864 of the 2016 WGRA-N sample were considered to be complete eligible respondents based on standard OPA criteria. 8

15 Table 6. Sample Disposition Codes for 2016 WGRA-N Sample Disposition Code (SAMP_DC) Conditions Sample Size Record Ineligbles: OPA identified members that 8,936 (8.9%) separated from active duty using the April Defense Enrollment Eligibility Reporting System (DEERS) Medical Point-in-Time Extract (PITE) The sampled member or a proxy reported that 2 (0.002%) 2 member was ineligible due to such reasons as "Separated," Retired, etc. The sampled member was determined to be 1 (0.001%) 3 ineligible based on their response to Q1 of the survey questionnaire Were you on active duty on [OPEN DATE]? 4 Complete Eligibles: If respondent was eligible 2,864 (2.8%) and completed 50% or more of the questions. Incomplete Eligibles: If respondent was eligible 265 (0.263%) 5 but failed to complete 50% or more of the questions Survey is refused due reasons such as too 238 (0.236%) 8 long, inappropriate/intrusive, Refused additional s,, etc. 10 Postal non-deliverable or original address is 5,215 (5.2%) non-locatable. 11 Nonrespondents: All others 83,290 (82.6%) Total 100,811 (100%) Table 7 shows population, sample size, respondents, and response rates by key domains (e.g., gender) for both the 2016 WGRA and 2016 WGRA-N. The weighted response rates for both surveys have similar patterns, although rates are much lower for the 2016 WGRA-N. For example, Air Force and female members were the highest responders for both the 2016 WGRA (34.9% and 28.4%) and 2016 WGRA-N (6.9% and 4.9%). It is important to highlight the difficulty in getting initial nonresponders to respond, and the lower response rates in the WGRA- N were expected. However, while the low response rates were expected, OPA cautions against drawing strong conclusions from a survey with a 4% response rate. 9

16 Table 7. Comparison of 2016 WGRA Sample With Nonresponse Sample 2016 WGRA-N 2016 WGRA 2016 WGRA-N Domain Domain Population Variable Sample Eligible Response Sample Eligible Response Size Responses Rates Size Responses Rates Sample Full Sample 1,291, , , % 100,811 2, % Service Army 474, ,584 44, % 45,856 1, % Navy 324, ,326 28, % 23, % Marine Corps 183, ,936 14, % 17, % Air Force 308, ,483 44, % 13, % Gender Male 1,089, ,197 93, % 81,217 2, % Female 202, ,132 39, % 19, % Paygrade E1 E4 565, ,995 38, % 67,703 1, % Group E5 E9 498, ,162 60, % 23,870 1, % W1 O3 145,830 60,069 19, % 6, % O4 O6 81,122 27,103 12, % 2, % Note: Both sets of response rates are weighted by sampling weights. Weighting the 2016 WGRA-N Because 2016 WGRA respondents were not eligible for the 2016 WGRA-N, it is not a probability sample of all active duty DoD members and there is no method to create base weights that represent the full active duty population. However, weights can be constructed that approximately represent the active duty population by raking. 5 OPA weighted the 2016 WGRA- N using the following process: OPA raked complete eligibles to population totals using the following seven dimensions: Gender (male and female), Service (Army, Navy, Marine Corps, Air Force), Three different paygrade groupings including a 2 level group (enlisted and officer), a 5 level group including (E1-E4, E5-E9, W1-W5, O1-O3 and O4-O6), and a 7 level group (E1-E3, E4, E5-E6, E7-E9, W1-W5, O1-O3 and O4-O6), Deployed in the last 12 months (Yes or No), and Family status (single with children, single without children, married with children, married without children). 5 Raking, otherwise known as iterative proportional fitting, is a method for adjusting weights to match known population characteristics. 10

17 Population totals for these raking totals were determined based on the total 2016 WGRA sample population. OPA computed final weights based on the raking adjustment. Table 8 shows standard moments for the final rake adjusted weights. Table 8. WGRA-N Final Weight Moments Moment Final Weight Mean Standard Deviation Max 1, % % % % % % % % % Min 99.6 Table 9 shows the estimates and corresponding margins of error for the four control questions that overlapped between the two surveys. The margins of error for the 2016 WGRA are small, whereas they are larger for the WGRA-N because there are fewer respondents. The 2016 WGRA estimated that 62.8% of active duty members indicated they were likely or very likely to stay on active duty compared with 64.4% on the 2016 WGRA-N. OPA judged these estimates to be similar, but did not compute statistical significance because the WGRA-N has a very low response rate and is also subject to NRB (e.g., WGRA-N respondents may not represent a random sample of 2016 WGRA nonrespondents). Estimates for the health question and sexual harassment question show more differences. WGRA-N respondents (58.3%) were less likely to report their health was good/excellent than WGRA respondents (67.6%) (Q23); however, for Q24 and Q25 WGRA-N respondents were less likely to indicate that the DoD climate was more of a problem for sexual harassment (WGRA-N, 6.6%; WGRA, 8.1%) and assault (WGRA-N, 7.5%; WGRA, 8.3%). 11

18 Table 9. Comparison of WGRA Survey With Nonresponse Study Control Questions NRB WGRA Q22 Q23 Q24 Q25 Q194 Q195 Q209 Q210 Question RETENTION: Assuming you could stay [in the Active Duty], how likely is it you would choose to do so? (% Saying Likely or Very Likely) HEALTH: In general, would you say your health is? (% Saying Good or Excellent) HOW ARE WE DOING: In your opinion, has sexual harassment in the military become more or less of a problem over the last 2 years? (% Saying More of a Problem) HOW ARE WE DOING: In your opinion, has sexual assault in the military become more or less of a problem over the last 2 years? (% Saying More of a Problem) Weighted Estimates 2016 WGRA 2016 WGRA-N 62.8% ± % ± % ± % ± % ± % ± % ± % ± 1.2 Table 10 shows that the estimates for the MEO questions on the WGRA-N differed from the 2016 WGRA. For example, WGRA-N estimates of hostile work environment (Q29) and gender discrimination (Q31) are both several percentage points higher than 2016 WGRA. For example, 5.7% of 2016 WGRA respondents indicated they heard someone make a gender discrimination comment, compared with 10.6% on the WGRA-N. In contrast, Q30 and Q32 estimates are similar across the two studies. Table 10. Comparison of WGRA Survey With Nonresponse Study MEO Questions NRB WGRA Question Q29 Q30 Q31 Q32 Q23 Q46 Q24 Q47 Since [X Date], did you hear someone from work say that [men] [women] are not as good as [women] [men] at your particular job, or that [men] [women] should be prevented from having your job? Do you think their beliefs about [men] [women] ever harmed or limited your career? For example, did they hurt your evaluation/fitness report, affect your chances of promotion or your next assignment? Since [X Date], do you think someone from work mistreated, ignored, excluded, or insulted you because you are a [man] [woman]? Do you think this treatment ever harmed or limited your career? For example, did it hurt your evaluation/fitness report, affect your chances of promotion or your next assignment? Weighted Estimates 2016 WGRA 2016 WGRA-N 5.7% ± % ± % ± % ± % ± % ± % ± % ±

19 Summary The purpose of this NRB analysis was to compare estimates from eight questions (four control questions and four MEO questions) asked identically on the 2016 WGRA-N and 2016 WGRA to assess NRB. If estimates were substantively and statistically different, this could be evidence of NRB for the 2016 WGRA estimates. Comparing the WGRA-N and 2016 WGRA is difficult because there is no clear pattern that 2016 WGRA respondents are more positive or negative. To summarize, three questions show WGRA-N respondents as more negative (Q23, Q29, Q31), three questions are similar (Q22, Q30, Q32), and for two questions WGRA-N respondents are more positive (Q24 and Q25). It appears WGRA-N respondents report more gender issues in Q29 and Q31, but at the same time have a more positive view that the military is improving on these issues. Thus, these results could indicate a potential sign of NRB, but it is difficult to determine the direction (e.g., whether 2016 WGRA may under- or overestimate correlates of the sexual harassment and sexual assault rates). While OPA would have hoped that all differences in estimates were within margins of error, this did not occur. NRB can vary question by question and the combination of MEO and control questions might not effectively represent NRB on sexual assault questions. Finally, the WGRA-N survey was a non-probability sample and had a very low response rate (under 5%) and OPA recommends cautious interpretation of this study. 13

20

21 Section 3: Evaluate the Sensitivity of Different Post-Survey Adjustments (Weighting Methods) on Survey Estimates Production weights for the 2016 WGRA were produced by first developing models that account for each member s propensity of experiencing unwanted sexual behaviors, and then using those estimated propensities throughout the weighting process. This method is consistent with RAND s approach for the 2014 RMWS and the OPA-Westat approach for the 2015 WGRR (hereafter called 2-Stage Boosted ) but represents a difference from how OPA conducts survey weighting across non-gender related surveys. For this study, OPA independently developed a set of weights using standard OPA methods to assess the effects of different weighting approaches on survey estimates. This section uses the OPA weights as a validity check to determine if large differences in the weights exist, and if these potential differences lead to more or less NRB in survey estimates. OPA Weighting Methodology OPA s standard weighting procedures have many similarities to the methods recently used by RAND in 2014 and OPA-Westat in Both methods estimate response propensities and make weighting adjustments based on the inverse of those propensities. However, there are two key differences; first, RAND and OPA-Westat used gradient-boosted decision trees (GBM and xgboost) to estimate the propensities, while standard OPA weighting uses single Chi-squared Automatic Interaction Detection classification trees (CHAID). Second, RAND and OPA-Westat first estimate propensities for several sexual assault characteristics, and then use those estimated propensities to predict survey response. Standard OPA weighting skips this step and directly models survey eligibility and response propensities. In addition to the production weights described in 2016 Department of Defense Workplace and Gender Relations Survey of Active Duty Members: Statistical Methods Report, OPA created a second set of weights for the 2016 WGRA using the following three steps: Step 1: Adjust weights for nonresponse based on eligibility as follows: Transfer the weight of the 568,191 nonrespondents (SAMP_DC = 8, 9, 10, 11) to the 157,891 cases with known eligibility (SAMP_DC = 2, 3, 4, 5). A decision tree technique based on Chi-square tests, was used to determine the probability of eligibility for the survey (known eligibility vs. unknown eligibility). Weighting adjustment factors for eligibility were computed as the inverse of the logistic model-predicted probabilities. The model was weighted using the sampling weight (base weight). Predictors in the eligibility model were the same variables used as in the 2 Stage production weights with the exception of the modelpredicted probabilities for unwanted sexual behaviors. Step 2: Adjust weights for survey completion as follows: Transfer the eligibility weight (created in Step 1) of the 5,603 incomplete survey responses (SAMP_DC = 5) to the 151,010 complete-eligible respondents (SAMP_DC = 4). Weighting adjustments for completion use the same methodology as Step 1 (CHAID and logistic model). 15

22 Step 3: Create final weights The weights were raked to match population totals and to reduce variance and bias unaccounted for by the previous weighting adjustments. OPA calculated the final weight as the product of adjustment factors in Steps 1, 2 and 3. The raking process followed the exact same steps and used the same program as the 2016 WGRA production weights. Comparison of Adjustment Stages and Final Weights Table 11 compares the standard OPA methodology and 2-Stage Boosted Methods (2014 RAND, 2015 OPA-Westat) for each of the weight adjustments discussed in Steps 1 through 3: eligibility, completion, and raked. The comparison shows the univariate distribution of each weighting adjustment factor. The results indicate that in aggregate both methods have very similar univariate distributions across adjustment Steps 1 through 3. Although the OPA method carries slightly more variance, all adjustments outside of the tails seem to convey the same adjustments in both instances. Table 11. Comparison Between Adjustment Factor: Standard OPA vs. 2-Stage Boosted Method for Eligibility, Completion and Poststratification Adjustments Standard OPA (Direct Method) 2-stage Boosted Method (Final Weights) Statistic Eligibility Completion Raking Eligibility Completion Raking Mean Standard Deviation % Max % % % % Q % Median % Q % % % % Min Table 12 extends the comparison of OPA standard and 2-stage boosted methods by showing the distribution of final weights. The final weight takes into account all of the previous weighting adjustments. OPA sees somewhat erratic behavior at the tails for the maximum 16

23 weight value, but are very close in most of the other quantiles. OPA concludes that overall both methods produce similar distributions of survey weights. Table 12. Comparison Between Standard OPA and 2-Stage Boosted Final Weights Moments Standard OPA 2-Stage Boosted Mean Standard Deviation % Max % % % % Q % Median % Q % % % % Min Comparison of Key Estimates Finally, differences in weighted survey estimates for sexual assault based on the 2-Stage Boosted and OPA standard weighting methods are compared in Table 13 and Table 14. Each table shows seven estimates associated with sexual assault and sexual harassment for males and females. For example, the estimates of any sexual assault occurring with females using OPA standard methods was 5.3% and the estimate was 5.4% using the 2-Stage Boosted weighting approach (Table 14). All comparisons are nearly identical for both weighting approaches. 17

24 Table 13. Comparison of OPA and Westat Key Survey Estimates (Males) Question Variable Standard OPA 2-Stage Boosted Sexual Quid Pro Quo QPQ 0.3% ± % ± 0.04 Sexual Assault-Penetrative SA_PEN 0.3% ± % ± 0.04 Sexual Assault-Any Type SA_RATE 0.8% ± % ± 0.1 Sexual Assault-Attempted Touch SA_TOUCH 0.5% ± % ± 0.1 Gender Discrimination SDISC 2.0% ± % ± 0.1 Sexual Harassment SEXHAR 5.7% ± % ± 0.2 Sexual Assault Rate Adjusted for telescoping SA_R_ADJ 0.8% ± % ± 0.1 Overall all of the other estimates are nearly identical. In addition, all of the confidence intervals for either weighting method overlap. Table 14. Comparison of OPA and Westat Key Survey Estimates (Females) Question Variable Standard OPA 2-Stage Boosted Sexual Quid Pro Quo QPQ 2.0% ± % ± 0.2 Sexual Assault-Penetrative SA_PEN 2.2% ± % ± 0.2 Sexual Assault-Any Type SA_RATE 5.3% ± % ± 0.3 Sexual Assault-Attempted Touch SA_TOUCH 3.0% ± % ± 0.2 Gender Discrimination SDISC 13.6% ± % ± 0.4 Sexual Harassment SEXHAR 20.8% ± % ± 0.5 Sexual Assault Rate Adjusted for telescoping SA_R_ADJ 4.9% ± % ± 0.3 Finally, while not displayed here, OPA also replaced the decision tree methodology in the direct estimates from CHAID to use two different alternative methods 1) Recursive Partioning Trees (RPART) in R and 2) xgboost in R. Both of these additional methodologies provided similar results compared to the 2-Stage Boosted and the Direct Estimate CHAID. Summary The direct OPA and 2-Stage Boosted weighting methods were conducted independently using different software (OPA standard weights used SAS and SPSS; 2-Stage Boosted used R and SAS) and methodology, but the overall results are strikingly similar across all of the intermediate weighting steps, the final weights, and a comparison of key estimates. In addition, all of the confidence intervals for either weighting method overlap, and none of the estimates are 18

25 statistically significant. This insight is particularly powerful given the differences in the methods. As mentioned earlier, the 2-Stage Boosted Method was designed to reduce variance in the estimates, and Table 13 and Table 14 show that this goal is achieved. In conclusion, 2016 WGRA estimates of sexual assault and sexual harassment are very robust to the choice of weighting methods. It should be noted that OPA conducted these comparisons for the 2015 WGRR and found similar results (DMDC, 2016a). Note that this study does not assess the level of NRB in 2016 WGRA estimates, but instead assesses whether different weighting methods differentially alter the level of NRB. OPA concludes that the choice of weighting methods does not substantially alter the level of NRB in 2016 WGRA estimates. 19

26

27 References DMDC. (2016) Workplace and gender relations survey of reserve component members: Statistical Methodology Report (Report No ). Alexandria, VA: Author. Little, R.J., & Rubin, D.B. (2002). Statistical analysis with missing data (2 nd ed.). New York: John Wiley & Sons, Inc. doi: / Montaquila, J.M. and Olson, K. M. (2012). Practical Tools for Nonresponse Bias Studies. Retrieved from Morral, A.R., Gore, K.L., & Schell, T.L. (Eds.). (2015). Sexual assault and sexual harassment in the U.S. military: Volume 4. Investigations of potential bias in estimates from the 2014 RAND military workplace study (No. RR-870/2-OSD). Santa Monica, CA: RAND Corporation. OPA. (2016a) Workplace and gender relations survey of active duty members: Statistical Methodology Report (Report No ). Alexandria, VA: Author. 21

28

29 This page is reserved for insertion of Standard Form 298, page 1 -- this is best accomplished by replacing this page after the document has been converted to PDF

30