An Empirical Study to Evaluate the Performance of Synthetic Estimates of Substance Use in the National Survey of Drug Use and Health

Transcription

1 An Empirical Study to Evaluate the Performance of Synthetic Estimates of Substance Use in the National Survey of Drug Use and Health Akhil K. Vaish 1, Ralph E. Folsom 1, Kathy Spagnola 1, Neeraja Sathe 1, Art Hughes 2 Joint Statistical Meetings, 2013, Montréal, Canada 1 RTI International. 2 Center for Behavioral Health Statistics and Quality (CBHSQ), Substance Abuse and Mental Health Services Administration (SAMHSA). 1 RTI International is a trade name of Research Triangle Institute.

2 Background Information The National Survey on Drug Use and Health (NSDUH) provides national, State, and substate data on substance use and mental health in the civilian, noninstitutionalized population aged 12 or older. Sponsored and managed by the Substance Abuse and Mental Health Services Administration since Data collection and processing conducted under contract with RTI International since

3 NSDUH Design NSDUH employs a State-based design with an independent, multistage area probability sample within each State and the District of Columbia. The data is collected via face-to-face interview at the respondent's place of residence. Oversample persons aged 12 to 25 years, with 1/3 of sample in 12 to 17, 18 to 25, and 26 or Older age groups. Target 67,500 respondents/year. 3

4 NSDUH Annual Sample Characteristics States No. of SSRs No. of Segments per SSR No. of Resp. per Segment Total Annual Sample CA, FL, IL, MI, NY, OH, PA, TX ,600 Remaining 42 States & D.C All States ,500 State sampling regions (SSRs) are contiguous geographic areas designed to yield approximately the same number of interviews. Within each SSR, 48 census tracts were selected with probability proportional to population size. Within sampled census tracts, adjacent census blocks were combined to form the second-stage sampling units or area segments. One area segment was selected within each sampled census tract with probability proportional to population size. 4

5 Past Validation Studies NSDUH sample was expanded in 1999 to produce estimates in every state. Small Area Estimation (SAE) methodology is used on two years of pooled NSDUH data to improve precision of state estimates and produce 2-year moving average state change estimates. Validation study on pseudo states with n=300 per age group demonstrated that state-level SAEs were more precise than their design-based counterparts while exhibiting small levels of bias (Wright, 2002 and 2003) 5

6 Past Validation Studies (cont.) SAE methodology is also being used to produce substate estimates on 3 years of pooled NSDUH data. The substate regions (about 360) are usually defined as groups of contiguous counties within a state. An overall sample size of 200 was first used as minimum for defining the substate regions. To satisfy increasing interest in smaller areas and estimates by age group, currently, some substate areas are based on n s as small as 50 in a particular age group. Hughes et al. (2012) demonstrated the validity of NSDUH substate estimates. 6

7 Motivation 7

8 Objective The objective of our paper is to evaluate the quality of synthetic estimates using NSDUH data. The quality of synthetic estimates would depend on several factors such as model fit, left over variation between counties after adjusting for fixed effects, and the similarity between the characteristics of missing and non-missing counties. 8

9 Method Create reliable benchmark estimates for comparing the synthetic estimates with these reasonably true values for a variety of outcome measures (low, medium, and high prevalence rates). We chose the following outcome measures Past month binge alcohol use (~24%), Past month use of cigarettes (~22%), Past year use of cocaine (~2.3%), and Past month use of marijuana (~9.4%). 9

10 Method (cont.) Each of the 8 large states was partitioned into two parts with n=~3600 resulting in 8x2=16 benchmark areas and their design based estimates were treated as true value. It is like each of the large states has 2 substate areas/counties. Part 1 of each of the 8 large states was formed to have lower average prevalence rates than part 2 across the 4 outcome measures. This was done to simulate real life situations where counties within states have varying prevalence rates i.e. some are higher and some are lower than the state prevalence rates, and some of the county level rates are similar to the state prevalence rates. 10

11 Method (cont.) 11 2 Parts of Each the 8 Large States Part 1 of CA Part 2 of CA Pseudo Sub-state Areas CA1_1 CA1_2... CA1_15 CA1_16 CA2_1 CA2_2... CA2_15 CA2_16 FL1_1 FL1_2. Part 1 of FL.. FL1_15 FL1_ Created 16 pseudo sub-state areas with n=225 (approximately) in each of the 16 parts i.e. randomly partitioned each part s sample of area segments into 16 replicated subsamples. From the previous two validation studies, n=225 appears to be sufficient to produce reliable SAEs as compared to the corresponding design based estimates.

12 Method (cont.) 12

13 Method (cont.) ( ε ) 13

14 Method (cont.) The synthetic estimates for pseudo sub-state area-i (i=1...16) within the benchmark area-d (d=1 16) are constructed as universe level averages of the person-k level predicted values: p dik ( syn) exp( X ˆ dkβ + Z dkε) = 1+ exp( X ˆ β + Z ε) dk dk where ε ~ MVN(0, Wˆ ). The corresponding SAEs are constructed from predicted values: p dik ( SAE) exp( X ˆ ˆ dkβ + Z dkηdi ) = 1+ exp( X ˆ β + Z ˆ η ) dk dk di where are the estimated random effects for pseudo sub-state area-i within the benchmark area-d based on the associated survey data. ηˆdi 14

15 Results rabias( ˆ θ ) d = 16 1 ˆ θdi i= 1 abs[ 16 ( θd )] θ d rrmse( ˆ θ ) d = 16 1 ˆ θdi i= 1 16 ( θ ) θ d d 2 15

16 Table 1. Outcome Binge Cigarette Cocaine Marijuana Results (cont.) Age Group RABIAS (%) RRMSE(%) DB SAE SYN SAE SYN DB ,2, ,2, ,2, ,2, Age Group=0:12 or older, 1:12 to 17, 2:18 to 25, 3: 26 or Older The statistics shown above are overall averages. 16

17 Table 2. Outcome Binge Cigarette Cocaine Marijuana Results (cont.) Age Group Ratio of Widths of 95% Confidence Intervals DB/SAE Synthetic/SAE ,2, ,2, ,2, ,2, Age Group=0:12 or older, 1:12 to 17, 2:18 to 25, 3: 26 or Older The statistics shown above are overall averages. 17

18 Discussion and Future Research Bias of DB Bias of SAE Bias of Synthetic RRMSE of SAE RRMSE of Synthetic RRMSE DB (except in one case) These results are not unexpected. For Cocaine, performance of all three estimators deteriorates with SAEs showing less deterioration than others. RABIAS and RRMSE of synthetic estimates is very close to each other which is also an expected result. This is due to the fact that synthetic estimates are formed using the fixed parameter estimates which exhibit very little variance and therefore do not change across the 16 pseudo sub-state areas within a benchmark area. 18

19 Discussion and Future Research (cont.) Synthetic estimates performed better than expected especially for higher prevalence outcome measures (Binge, Cigarettes). This may be due to the fact that after adjusting for age, race, gender, and a few other covariates there was not enough variability remaining among our pseudo sub-state areas to benefit from the sample data based sub-state area level random effects. This needs to be investigated further. The prediction intervals for synthetic estimates are much wider than expected. That is, the simulation mean square errors are much smaller than the model based mean square errors for the synthetic estimates. This also needs to be investigated further. 19

20 Discussion and Future Research (cont.) This is a limited simulation study and does not fully account for real life situations where sub-state areas within states generally exhibit wide ranging characteristics. Intuitively, if the missing areas lie close to the fixed regression line then their synthetic estimates will be ok. Otherwise, they will be highly biased. A more comprehensive simulation study is needed to further evaluate the quality of synthetic estimates. 20

21 Acknowledgments The NSDUH was funded by the Substance Abuse and Mental Health Services Administration, Center for Behavioral Health Statistics and Quality, under Contract No. HHSS C and conducted under RTI Project No The views expressed in this presentation do not necessarily reflect the official policies of the U.S. Department of Health and Human Services, nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government. Contact Information: Akhil K. Vaish 21