Running Head: EXAMINING THE FACTOR STRUCTURE OF THE ACS 1

Transcription

1 Running Head: EXAMINING THE FACTOR STRUCTURE OF THE ACS 1 Tables: 4 Addressing Psychometric Limitations of the Attentional Control Scale Via Bifactor Modeling and Item Modification Kate Clauss & Joseph R. Bardeen Department of Psychology, Auburn University, United States NOTICE: this is the author s version of a work that was accepted for publication in Journal of Personality Assessment. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Journal of Personality Assessment. Author Note Correspondence concerning this article should be addressed to Joseph R. Bardeen, Department of Psychology, Auburn University, Auburn, AL Voice: ; E- mail: jbardeen@auburn.edu.

2 FACTOR STRUCTURE OF THE ACS 2 Abstract The purpose of this three-part study was to identify and correct psychometric limitations of the Attentional Control Scale (ACS: Derryberry & Reed, 2002) via bifactor modeling and item modification. In Study 1 (N = 956), results from exploratory and confirmatory factor analyses (EFA and CFA) suggested that the multidimensionality of the ACS might be a function of a method effect (i.e., reverse coding). In Study 2 (N = 478), reverse-coded items were recoded in a straightforward manner and submitted to EFA. Results supported retention of 15 items and two factors. In Study 3 (N = 410), CFA was used to test the model identified in Study 2 and compare it to competing models (i.e., one-factor, bifactor). The bifactor model exhibited the best fit to the data. However, results from bifactor analysis, suggested that the structure of the ACS is more consistent with a unidimensional rather than multidimensional model. Additionally, the second domain-specific factor appears to be redundant with the general factor and both domain-specific factors are poorly defined and may be of little practical value. Taken together, results caution the use of the ACS subscales independent of the total score. Moreover, they support coding ACS items in a straightforward manner. Keywords: attentional control scale, attention, bifactor, exploratory factor analysis, confirmatory factor analysis

3 FACTOR STRUCTURE OF THE ACS 3 Addressing Psychometric Limitations of the Attentional Control Scale Via Bifactor Modeling and Item Modification Attentional control the effortful allocation of attention toward goal relevant information in the face of conflicting prepotent attentional demands (Sarapas, Weinberg, Langenecker, & Shankman, 2017) has been implicated in a wide variety of negative psychological outcomes, including depression (Fergus, Bardeen, & Orcutt, 2012; Kanske & Kotz, 2012), anxiety (Derryberry & Reed, 2002), and posttraumatic stress (Bardeen & Read, 2010). A growing body of evidence suggests that attentional control may protect those who are vulnerable to maladaptive psychological outcomes from experiencing such outcomes. For example, higher levels of self-reported attentional control decrease the likelihood that individuals (a) who perceive themselves as having emotion regulation deficits will abandon goal-directed behavior when experiencing distress (Bardeen, Tull, Dixon-Gordon, Stevens & Gratz, 2015), (b) who use worry and thought suppression to regulate emotional distress will experience higher levels of anxiety (Fergus et al. 2012), (c) who have public-speaking anxiety will exhibit deficits in speech performance (Jones, Fazio & Vasey, 2012), and (d) who have higher trait anxiety will respond with panic symptoms to a CO2 challenge (Richey, Keough, & Schmidt, 2012). Additionally, prospective research supports the distress buffering effects of attentional control. For example, individuals with relatively better self-reported attentional control, measured prior to a traumatic event, are less likely to exhibit posttraumatic stress in the acute aftermath of a traumatic event in comparison to individuals with relatively worse self-reported attentional control (Bardeen, Fergus, & Orcutt, 2015). Importantly, attentional control abilities can be improved through clinical intervention (Jha, Krompinger, & Baime, 2007). Thus, attentional control appears to

4 FACTOR STRUCTURE OF THE ACS 4 have important transdiagnostic and treatment implications that highlight the importance of psychometrically sound self-report measures of this construct. The Attentional Control Scale (ACS) was developed to assess a general capacity for attentional control, with correlated subfactors, (Derryberry & Reed, 2002: p. 226). Specifically, items from scales developed to measure voluntary attentional focusing and shifting (Derryberry & Rothbart, 1988) were submitted to exploratory factor analysis (EFA). Although Derryberry and Reed (2002) reported that results of an unpublished manuscript resulted in a multidimensional self-report measure consisting of an overarching attentional construct and the following three correlated lower-order domain-specific factors (i.e., focusing, shifting between tasks, and flexibly controlling thought), the psychometric properties of these domain-specific factors were not reported in the 2002 paper. Additionally, subsequent psychometric investigations of the ACS identified two domain-specific factors (Judah, Grant, Mills, & Lechner, 2014; Ólafsson et al., 2011). Regarding the ACS total score, evidence suggests that the measure is internally consistent and has criterion-related validity (Bardeen, & Daniel, 2017; Derryberry & Reed, 2002; Judah et al., 2014; Quigley, Wright, Dobson, & Sears, 2017). For example, the ACS is negatively associated with measures of trait and state anxiety, depressive symptoms (r = -.55; Derryberry & Reed, 2002; Reinholdt-Dunne, Mogg, & Bradley, 2009), negative affect (Healy, 2010), and neuroticism (Verwoerd, de Jong, & Wessel, 2008). Additionally, the ACS is positively associated with measures of positive emotionality (Derryberry & Reed, 2002) and higher scores are predictive of affect regulation in response to a stressor (Bardeen & Read, 2010) and increased activation in brain areas associated with topdown emotion regulation (i.e., prefrontal cortex, anterior cingulate cortex; Matthews, Yiend, & Lawrence, 2004). To date, evidence of convergent validity of the ACS with performance-based

5 FACTOR STRUCTURE OF THE ACS 5 measures that are purported to assess attentional control is limited. While there is some evidence of medium-sized associations between the ACS total score and performance-based measures of working memory updating and response inhibition (e.g., Judah et al., 2014), some studies have failed to replicate these findings (e.g., Quigley et al., 2017). Ólafsson and colleagues (2011) translated the ACS into Icelandic and a large sample of university students (n = 361) completed the measure. Following the removal of item 9 (due to low inter-item correlation), principal components analysis (PCA) with oblique rotation was used to examine the factor structure of this version of the ACS. It should be noted that PCA is primarily a data reduction technique, whereas EFA is used to identify the underlying factor structure of a measure (see Abdi & Williams, 2010). Ólafsson et al. (2011) also conducted parallel analysis, which has been put forth as a more accurate method of factor extraction (O Connor, 2000). Five factors exhibited eigenvalues greater than one, but only the first two factors had eigenvalues that exceeded those randomly generated in parallel analysis. In the proposed two factor solution, nine items loaded onto Factor I (i.e., items: 1, 2, 3, 4, 5, 6, 7, 8, 12), and 10 items loaded onto Factor II (i.e., items: 10, 11, 13, 14, 15, 16, 17, 18, 19, 20). The authors labeled Factor I Focusing (i.e., controlling attention by inhibiting distractors) and Factor II was labeled Shifting (i.e., controlling attention by shifting between tasks). Importantly, all of the reverse coded items loaded exclusively onto one of the two factors, which suggests the possibility that factor differentiation may be a function of a method effect. A second sample of Icelandic university students (n = 367) completed the 19 items identified via PCA and confirmatory factor analysis (CFA) was used to examine model fit. A correlated two-factor model evidenced better fit than a one-factor model; however, it did not meet a-priori standards of adequate fit until the error terms between four pairs of items were allowed to correlate (i.e.,

6 FACTOR STRUCTURE OF THE ACS 6 items: 4-5, 17-18, 3-6, 7-8). One of the most notable limitations of this study is that a crossloading rule was not applied (see Matsunaga, 2010). That is, items that simultaneously loaded onto both factors were retained (e.g., item 12 loaded on Factor I [Focusing] at.33 and on Factor II [Shifting] at.27, item 18 loaded on Factor I [Focusing] at.28 and Factor II [Shifting] at.35). This may explain why correlated error terms were required to obtain adequate fit via CFA. To our knowledge, the only study in which the structure of the English version ACS was examined was conducted by Judah and colleagues (2014). Specifically, a large sample of American university students (n = 499) completed the ACS and all 20 items were submitted to EFA with principal axis factoring and oblique rotation. Parallel analysis was also conducted (O Connor, 2000). Results from parallel analysis suggested retaining three factors. However, a two-factor solution was retained because relatively few items loaded onto the third factor at the a-priori identified threshold (i.e., factor loadings >.40). In total, 12 items were retained that did not load onto both factors (i.e., factor loading discrepancies of at least.28) and that had primary loadings >.40 on one of the factors. Seven items loaded onto Factor I (i.e., items: 1, 2, 3, 6, 7, 8, 12) and five items loaded onto Factor II (i.e., items: 10, 13, 17, 18, 19). Consistent with Ólafsson et al. (2011), Factor I was labelled Focusing and Factor II was labelled Shifting. Additionally, as was the case in Olafsson et al. s (2011) examination, all of the reverse-coded items loaded onto one of the two factors. Judah et al. (2014) used CFA to compare this 2-factor model made up of 12 items to Ólafsson et al. s (2011) 2-factor model made up of 19 items. Ólafsson et al. s (2011) model provided poor fit to the data, whereas Judah et al. s (2014) model evidenced adequate fit. In a recent study, undergraduate students (N = 125) completed the ACS and the fit indices of the two-factor models proposed by Ólafsson et al. (2011) and Judah et al. (2014) were

7 FACTOR STRUCTURE OF THE ACS 7 compared via CFA (Quigley et al., 2017). Although Judah et al. s (2014) model provided better fit than Ólafsson et al. s (2011) model, both models evidenced poor fit to the data. Quigley et al. (2017) found that adequate fit of Judah et al. s (2014) model could only be obtained in this relatively small sample by allowing the error terms for items one and six to correlate. Taken together, the extant literature raises concerns regarding the factor structure of the ACS and independent use of previously identified subscales. One of the most notable concerns with previously identified two-factor models of the ACS is that all of the reverse-coded items load onto one of the two factors. Although reverse-coded items might be sensitive to inattentive responding, their use has several psychometric disadvantages (Hughes, 2009; Rodebaugh, Woods, & Heimberg, 2007; Weems & Onwuegbuzie, 2001). For example, use of reverse-coded items may reduce scale validity by increasing the likelihood of systematic error (Rodebaugh et al., 2007). Additionally, reverse-coded items have a tendency of loading onto a separate factor, which can result in retention of a factor that is not conceptually meaningful (Dalal & Carter, 2015). This issue is particularly relevant to the ACS, as it is not clear whether one of the two factors identified via EFA is an artifact of reverse coding rather than being due to content specificity. Although relatively few investigations of the factor structure of the ACS have been conducted, the ACS total score and subscale scores are commonly used in empirical research (e.g., Cox, Cole, Kramer, & Olatunji, 2017; Taylor, Cross & Amir, 2016). Use of a total score assumes that each ACS domain-specific factor represents the same overarching attentional control construct, while use of subscale scores assumes that the domain-specific factors provide unique information beyond the total score. The tenability of these two approaches has yet to be addressed in the extant literature. To our knowledge, the correlated two-factor model is the only

8 FACTOR STRUCTURE OF THE ACS 8 model of the ACS to be examined in the extant literature. Examination of the two-factor model does not address the purported overarching attentional control construct. Additionally, the correlated two-factor model does not test the assumption that the domain-specific factors provide unique information beyond a higher-order attentional control construct (Reise, 2012). Because the ACS total and subscale scores are commonly used, it is important to use an analytic approach that can simultaneously address both of these assumptions (i.e., bifactor analysis). The unique contributions of both the general and domain-specific factors can be isolated by using a bifactor modeling approach (Reise, 2012). This approach allows for the simultaneous investigation of the general factor and the degree to which the domain-specific factors are distinct from this general factor. Additionally, we examined the incremental utility of the ACS domain-specific factors in predicting theoretically relevant constructs (i.e., general distress, executive functioning) after accounting for a general ACS factor. This factor analytic approach should allow us to determine the usefulness of subscale scores independent of the total score (Reise, 2012), which could have a significant impact on the ACS s use in future research and applied settings. Consistent with theory and previous research (e.g., Bardeen, Kumpula, & Orcutt, 2013, Derryberry & Reed, 2002; Mathews et al., 2004), we hypothesized that whose with relatively higher self-reported attentional control would also report (a) relatively lower general distress (a negative association between constructs), and (b) relatively higher self-reported executive functioning abilities (a positive association between constructs). Study 1 The factor structure of the ACS has been examined in relatively few studies and results between studies are inconsistent (Judah et al., 2014; Olafsson et al., 2011; Quigley et al., 2017). The purpose of Study 1 was twofold. First, EFA was used to identify the factor structure of the

9 FACTOR STRUCTURE OF THE ACS 9 ACS. Based on previous research (Judah et al., 2014; Olafsson et al., 2011), we predicted that two factors would be identified and retained. Second, CFA was used to compare the fit of the model identified via EFA to other theoretically relevant models (i.e., one-factor, bifactor). Consistent with evidence of multidimensionality (Judah et al., 2014; Olafsson et al., 2011), as well as theory suggesting an overarching attentional control construct (Derryberry & Reed, 2002), we expected that the bifactor model of the ACS would provide significantly better fit to the data than competing models. However, the utility of comparing bifactor models to alternative models using standard fit indices has been questioned because some have suggested that bifactor models provide better fit to the data because of their inherent qualities (e.g., high flexibility; Bonifay, Lane, & Reise, 2017; Reise, Kim, Mansolf, & Widaman, 2016). As such, a number of additional statistical indices, developed for use with the bifactor approach, were examined (Rodriguez, Reise, & Haviland, 2016). These statistics provide information central to the aims of the present study (e.g., determining the degree to which domain-specific factors have value beyond the general factor, determining the stability and replicability of the factors). Method Participants Participants were 1062 community adults recruited from an online crowd-sourcing site (see below); 9% of participants did not complete the ACS and were removed from further analyses. This larger sample (N = 956) was randomly split into two smaller samples of equal size (n = 478). This allowed us to conduct an EFA on the ACS in the first half of the sample, followed by model testing in the second half of the sample. Sample 1. The average age of participants was years (SD = 11.91, range = 19-65) and the majority of the sample was female (67.4%). The majority of the sample identified their

10 FACTOR STRUCTURE OF THE ACS 10 race as White (81.8%), followed by Asian (6.7%), Black (6.5%), other (3.8%), American Indian/Alaska Native (1.0%), and Native Hawaiian or Other Pacific Islander (.2%). Additionally, 6.5% of the sample identified their ethnicity as Hispanic or Latino. Sample 2. The average age of participants was years (SD = 11.31, range = 19-65) and the majority of the sample was female (67.6%). The majority of the sample identified their race as White (83.5%), followed by Black (8.4%), Asian (4.2%), other (3.1%), and American Indian/Alaska Native (.8%). Additionally, 6.7% of the sample identified their ethnicity as Hispanic or Latino. Measures Participants completed demographic information and the Attentional Control Scale (Derryberry & Reed, 2002). The 20-item ACS is scored on a 4-point scale from 1 (almost never) to 4 (always). The ACS has 11 reverse-coded items. Higher scores indicate higher levels of selfreported attentional control. Internal consistency for the ACS total score was adequate in Sample 1 (α =.90, M = 54.79, SD = 10.33, skew =.04, kurtosis = -.27) and Sample 2 (α =.88, M = 54.70, SD = 9.89, skew =.02, kurtosis =.07). Procedure All study procedures were approved by the local institutional review board. Participants were recruited via Amazon s Mechanical Turk (MTurk). MTurk samples are more diverse than typical American undergraduate samples, while still producing high quality data (Buhrmester, Kwang, & Gosling, 2011; Casler, Bickel, & Hackett, 2013). Participants were able to complete the study from any computer with Internet access and were compensated $1.50 for their participation. Data Analytic Strategy

11 FACTOR STRUCTURE OF THE ACS 11 Exploratory Factor Analysis. Analyses were conducted using IBM SPSS Statistics 24 and R Studio. In sample 1, we submitted the 20-item ACS to EFA (principal axis factoring) with oblique rotation. Initially, factors with eigenvalues > 1 were considered for retention (Kaiser, 1960). However, because the Kaiser-Guttman criterion has been shown to overestimate the number of factors extracted (Zwick & Velicer, 1986), we also conducted parallel analysis (O Connor, 2000) and Velicer s minimum average partial (MAP) test (Velicer, 1976). Research suggests that when used in conjunction, Velier s MAP and parallel analysis are the most accurate method of determining factor structure (Zwick & Velicer, 1986). O Connor s (2000) SPSS syntax was used to conduct parallel analysis, and the NFactors RStudio package was used to conduct the Velicer s MAP test. After factor identification, we followed Judah and colleague s (2014) criteria for item retention. Specifically, items with loadings >.4 were retained. We also employed Matsunaga s (2010) recommendation for cross-loaded items, and only retained items that had a difference of.4 or greater between the first highest and second highest loadings. Confirmatory Factor Analysis. The following three models were examined using Mplus 7.4 (Muthen & Muthen, 2015). Consistent with the results from the EFA above and Judah et al. (2014), the first model that was examined was a correlated two-factor model that consisted of seven items with primary loadings on Factor I (Focusing) and five items with primary loadings on Factor II (Shifting). No secondary loadings were modeled, but the factors were allowed to intercorrelate. Model two was a one-factor model; all 12 items loaded onto one factor. The fit of a second-order model (i.e., direct pathways added from Factors I and II to a higherorder general attentional control construct) was not tested because such a model would, with only two lower-order factors, (a) be under-identified and (b) exhibit the same fit as the correlated 2-factor model if additional constraints were added to the model to obtain fit statistics. However,

12 FACTOR STRUCTURE OF THE ACS 12 we did examine the described model while fixing the noted direct paths to one in order to obtain path coefficients for the lower-order factors on the second-order construct. Model three was a bifactor model in which all 12 items were simultaneously loaded onto a general factor and each of their respective domain-specific factors. All factor covariances were fixed to zero in the bifactor model (Brown, 2015). The following three fit indices were used to evaluate models: the Tucker-Lewis fit index (TLI), comparative fit index (CFI), and root mean square error of approximation (RMSEA; Brown, 2015; Hu & Bentler, 1999). The following guidelines were used to evaluate fit. For CFI and TLI, values >.90 indicate adequate fit (Bentler 1990; Jöreskog et al., 2000; Meyers, Gamst, & Guarino, 2006). For the RMSEA index, values <.05 indicate excellent fit,.05 to.08 indicate adequate fit (Browne & Cudeck 1993), and values from.08 to.1 indicate mediocre/acceptable fit, whereas values >.10 indicate inadequate fit (Meyers et al. 2006). Mean- and varianceadjusted weighted least squares (WLSMV) estimation was used to test all models because ACS item responses are ordered-categories (Asparouhov, 2005). The chi-square difference test was used to evaluate model comparisons (Kline, 2011). However, chi-square difference tests are strongly influenced by sample size, often indicating a significant difference when actual differences are trivial in magnitude (Cheung & Rensvold, 2002). As such, we also compared models using RMSEA 90% confidence intervals [CIs] (Brown, 2015; Kline, 2011). Differences in model fit are considered non-significant when models have overlapping 90% RMSEA CIs, (Wang & Russell, 2005). Bifactor Model Evaluation. The bifactor model was further examined using the following statistical indices (Dueber, 2017; Rodriguez et al., 2016). OmegaH (ωh) reflects the proportion of variance in ACS scores attributable to a single general factor. OmegaHS (ωhs)

13 FACTOR STRUCTURE OF THE ACS 13 reflects the proportion of variance in scores attributable to each domain-specific factor after removing the variance due to the general factor. Explained common variance (ECV) serves as a better index of the unidimensionality of a measure in comparison to ωh, which is best understood as a measure of general factor reliability. The amount of common variance for each ACS item attributable to the general factor is indicated by explained common variance (I-ECV). I-ECV values greater than.80 to.85 are indicative of unidimensionality at the item level (Gorsuch, 1983). Percentage of uncontaminated correlations (PUC) serves as an indicator of the percentage of ACS item correlations contaminated by variance attributed to the general and domain-specific factors. PUC is often interpreted in combination with ECV. When both values are greater than.70, common variance within a model can be regarded as essentially unidimensional. Average relative parameter bias (ARPB) serves as an indicator of the bias across parameters if items are forced into a unidimensional, versus multidimensional, structure. ARPB less than 0.10 or 0.15 suggests that multidimensionality within a measure is not substantial enough to preclude a unidimensional solution (Muthén, Kaplan, & Hollis, 1987; Rodriquez et al., 2016). The correlation between factors and factors scores (i.e., factor determinacy [FD]), serves as an indicator of the degree to which factor scores are of practical value and should be used in measurement models. Values greater than.90 are suggested (Gorsuch, 1983). Construct replicability (H) reflects the degree to which a factor is well defined by its indicators. H values greater than >.80 suggest that a latent variable will demonstrate sufficient stability across studies (Hancock & Mueller, 2001). Results Exploratory Factor Analysis. The assumptions of EFA were met. EFA indicated that four factors had eigenvalues greater than 1 (7.18, 2.11, 1.21, and 1.05). They accounted for

14 FACTOR STRUCTURE OF THE ACS %, 10.57%, 6.06% and 5.27% of the variance respectively. Parallel analysis indicated that only three factors should be retained, as the fourth did not account for more variance than its randomly generated counterpart. The MAP test indicated a two-factor solution; the mean squared partial correlation decreased from one factor to two, but increased once three factors were extracted. Judah et al. (2014) also used parallel analysis, which indicated a three-factor structure that was ultimately uninterpretable. As described by O Connor (2000), when discrepancies are observed, the tendency of parallel analysis to err on the side of over-extraction should be considered. Consistent with Judah et al. (2014), the identified three-factor model was uninterpretable. Specifically, only one item loaded onto Factor III, and it cross-loaded onto Factor I (Focusing). Therefore, the two-factor solution was retained for further testing (see Table 1 for factor loadings). Items 4, 9, 15, 16 and 20 had initial loadings less than.4 and were removed from further analysis. In addition, items 5 and 10 did not meet Matsunaga s (2010) criteria for having a discrepancy greater than.4 between the first and second loading, and thus, were removed from further analysis. This factor structure is nearly identical to that proposed by Judah et al. (2014) with the exception of item 14 which loaded onto Factor II (Shifting) at.56 in the present study, but exhibited loadings of.12 (Factor I: Focusing) and.25 (Factor II: Shifting) in Judah et al. s investigation. Similarly poor factor loadings were exhibited by item #14 in Olafsson et al. s (2011) psychometric investigation (i.e., Factor I [Focusing] =.13, Factor II [Shifting] =.37. As such, we removed item #14 from further analysis in the present study to test the established factor structure identified by Judah et al. (2014). Confirmatory Factor Analysis. Fit statistics for all three models are presented in Table 2. The correlated two-factor model provided adequate fit to the data, with RMSEA, CFI, and TLI

15 FACTOR STRUCTURE OF THE ACS 15 within specified guidelines. Additionally, the correlated two-factor model provided a significantly better fit to the data than the one-factor model which evidenced poor fit. The latent correlation between Factors I and 2 was medium to large in size (r =.45, p <.001). As described above, a second-order model, with the direct paths from the two lower-order factors to the higher-order general attentional construct fixed to one, was examined to obtain standardized coefficients for these direct paths. Both lower-order factors exhibited large magnitude factor loadings on the higher-order factor (Factor I [Focusing] =.67, Factor II [Shifting] =.67, ps <.001). The bifactor model provided the most favorable fit statistics. The significant Δχ2 supported the bifactor model as providing better fit than the correlated two-factor model, but overlapping RMSEA 90% CIs suggested that the difference in models may be trivial in magnitude. Factor loadings from the bifactor model are presented in Table 3. All items exhibited significant factor loadings (p <.001) on the general factor. Eight of the twelve items exhibited significant domain-specific factor loadings at p <.001. For Factor II (Shifting), two items did not exhibit significant loadings on this factor (i.e., Items 17 & 18) and two items (i.e., 13 & 19, ps <.01) exhibited significant loadings that were smaller in magnitude in comparison to the eight other domain-specific factor loadings. Bifactor Model Evaluation. Strong reliability was exhibited for the general (ω =.93) and domain-specific factors (Factor I [Focusing] ωs =.92; Factor II [Shifting] ωs =.89). The general factor accounted for a substantial amount of variance in ACS scores (ωh =.56). The majority of reliable variance in Factor I (Focusing), but not Factor II (Shifting), appears to be independent of the general factor (ωhs =.70 and.15, respectively). The ECV value of.47 does not support a unidimensional item pool. The ECV value for Factor I (Focusing) was.43 and Factor II s (Shifting) ECV value was.10, indicating that Factor II (Shifting) accounts for

16 FACTOR STRUCTURE OF THE ACS 16 relatively little variance that is unexplained by the general factor. I-ECV values ranging from.14 to.44 for eight of the 12 items suggests multidimensionality. However, I-ECV values for four of the five items from Factor II (Shifting) raise concerns regarding the importance of these items to their respective domain. Item s 13 and 19 are relatively less concerning with I-ECV values of.67 and.81, but the I-ECV value of.99 for items 17 and 18 suggest that responses to these items are accounted for by variation in the general factor alone. The PUC value for the ACS was.53, indicating that a small majority of item correlations of the ACS are attributable to the general factor. The APRB across ACS items was 62% which supports multidimensional modeling of the ACS and calls into question use of a total score. FD values for the general factor (.93) and Factor I (Focusing:.94) suggest adequate factor determinacy, whereas the FD value of.86 for Factor II (Shifting) indicates that this factor may not be suitable for use as a summed subscale score and as a latent variable in a SEM framework (Gorsuch, 1983). This hypothesis is further supported by an H value of.59 for Factor II (Shifting) which suggests inadequate construct replicability. The general factor and Factor I (Focusing) exhibited acceptable construct replicability (H =.89 and.87, respectively). Study 1 Summary In EFA, all of the reverse-coded items loaded onto one factor, while forward-coded items loaded onto the other factor. Results from CFA indicated that the correlated two-factor model (also tested by Judah et al., 2014) provided adequate fit to the data while the one-factor model did not. Additionally, both lower-order factors loaded well onto the higher-order attentional control construct. The bifactor model provided better fit in comparison to the one- and two-factor models. Of note, four of five items on Factor II (Shifting) exhibited nonsignificant or small factor loadings when the general factor was accounted for in the bifactor model. Bifactor model

17 FACTOR STRUCTURE OF THE ACS 17 evaluation resulted in several indices that suggest multidimensionality (e.g., ECV, APRB, and PUC values), while simultaneously calling into question the structural validity of Factor II (Shifting). Specifically, relatively little reliable variance in Factor II (Shifting) is independent of the general factor and I-ECV values for items 17 and 18 suggest that these items are better accounted for by the general factor. Interestingly, these two items share a theme that is distinct from all of the other ACS items (i.e., being able to shift attention easily after being distracted). Finally, H and FD values suggest that Factor II (Shifting) is poorly defined and may be of little practical value for use as a subscale score and in measurement models. Together, these results point to the possibility that the multidimensionality of the ACS might be a function of a method effect (i.e., reverse coding). Study 2 The primary purpose of Study 2 was to test the hypothesis that the multidimensional factor structure of the ACS is a function of reverse-coded items. To test this hypothesis, all reverse-coded items were reworded in a straightforward manner and subjected to EFA. If the previously identified two factors of the ACS contain distinct content, straightforward coding items should not alter the factor structure. Method Participants A total of 383 participants (216 female) completed the modified version of the ACS. The average age of participants was years (SD = 11.31, range = 18-65). The majority of participants reported their race as White (77.3%), followed by Black (12.5%), Asian (7.8%), American Indian or Alaska Native (1.3%), other (.8%), and Native Hawaiian or other Pacific Islander (.3%). In addition, 9.4% reported their ethnicity as Hispanic or Latino.

18 FACTOR STRUCTURE OF THE ACS 18 Measures Participants completed a demographics questionnaire and the modified version of the ACS. Items 1, 2, 3, 6, 7, 8, 11, 12, 15, 16, and 20 are the reverse-coded items that were forward coded in a straightforward manner for this study. For example, item seven was modified from When trying to focus my attention on something, I have difficulty blocking out distracting thoughts to When trying to focus my attention on something, it easy for me to block out distracting thoughts. Higher scores on the modified version of the ACS indicate greater selfreported attentional control. Procedure As in Study 1, participants were recruited via MTurk. They completed informed consent and the modified version of the ACS. Participants were paid $1.50 for study completion. Results EFA (principal axis factoring) with oblique rotation was used to identify the factor structure of the modified ACS. Consistent with Study 1, parallel analysis (O Connor, 2000) and Velicer s MAP (Velicer, 1976) were used to determine the optimal number of factors. Item loadings >.4 were considered interpretable (Judah et al., 2014). Items that did not have a difference of at least.4 between their first and second loadings were removed from further analysis (Matsunaga, 2010). The assumptions of EFA were met. Three eigenvalues were > 1 (9.71, 1.37, and 1.06). Factor I explained 48.56% of the variance, Factor II explained 6.86%, and Factor III explained 5.31%. In contrast, parallel analysis and Velicer s MAP indicated a two-factor solution. This suggests that the third factor did not account for more variance than its randomly generated counterpart, and that the mean squared partial correlation increased once three factors were

19 FACTOR STRUCTURE OF THE ACS 19 extracted. For these reasons, a two-factor model was retained for further testing (see Table 1 for factor loadings). Item nine was removed for exhibiting high communality, loadings on both factors were <.40 (i.e.,.31 and.22). Cross-loading was also observed for items 8, 12, 17 and 18; these items were also removed from further analysis (Matsunaga, 2010). Study 2 Summary EFA on the 20 forward-coded items resulted in a 2-factor structure that was similar to that observed by Ólafsson et al. (2011). Item nine s failure to adequately load on either factor is consistent with previous psychometric investigations of the ACS in which this item exhibited high communality and failed to provide an adequate factor loading (Judah et al., 2014; Ólafsson et al., 2011; Quigley et al., 2017). Had Ólafsson et al. (2011) used strict item retention criterion (i.e., values >.4; Judah et al., 2014), as well as a discrepancy rule to ensure content specificity (i.e., a discrepancy of.4 between the first and second highest loading; Matsunaga, 2010), items 8, 12, 17 and 18 would have been removed from the item pool in their study. Study 3 The purpose of Study 3 was to examine the factor structure of the modified version of the ACS identified in Study 2 via CFA. Specifically, we sought to determine whether a correlated two-factor model or the bifactor model, which simultaneously accounted for the domain-specific factors and the general factor, provided better fit to the data. As described, bifactor modeling allows for the investigation of the degree to which the domain-specific factors are distinct from the general factor. Additionally, structural regression modeling was used to examine the incremental utility of the ACS domain-specific factors in predicting theoretically relevant constructs after accounting for a general ACS factor. Measures of general distress and executive functioning were selected as criterion variables. General distress was selected because it is a

20 FACTOR STRUCTURE OF THE ACS 20 particularly relevant construct (Bardeen & Read, 2010; Derryberry & Reed, 2002; Healy, 2010; Reinholdt-Dunne et al., 2009). A broad measure of executive functioning was chosen as a criterion variable because evidence suggests that attentional control draws on some of the same cognitive processes as those encompassed by executive functioning (Spruijt, Dekker, Ziermans, & Swaab, 2018), and thus, should demonstrate meaningful but non-redundant associations. Method Participants Participants were 410 community adults (female = 235), whose average age was years (SD = 11.06, range = 18-65). The majority of participants indicated that their race was White (77.3%), followed by Black (9.8%), Asian (9.5%), other (2.4%), American Indian/Alaska Native (.7%), and Native Hawaiian or other Pacific Islander (.2%). In addition, 9 % of the sample identified their ethnicity as Hispanic or Latino. Measures and Procedure Participants provided informed consent and demographic information and then completed the 15-item modified version of the ACS. As in previous studies, participants were recruited via MTurk. Participants were paid $1.50 upon study completion. Descriptive statistics for the 15- item version of the ACS are as follows: ACS total score (M = 41.96, SD = 9.40, skew =.04, kurtosis = -.25), Factor I (Focusing: M = 19.90, SD = 4.93, skew = -.10, kurtosis = -.42), and Factor II (Shifting: M = 22.07, SD = 5.19, skew = -.01, kurtosis = -.28). Depression Anxiety Stress Scales-21 (DASS-21). The DASS-21 (Lovibond & Lovibond, 1995) is a 21-item measure with scales assessing depression, anxiety, and stress symptoms in the past week on a 4-point scale (0 = did not apply to me at all to 3 = applied to me very much, or most of the time). Higher scores indicate higher distress. The DASS-21 has

21 FACTOR STRUCTURE OF THE ACS 21 exhibited adequate psychometric properties, including construct validity and internal consistency (Antony, Bieling, Cox, Enss, & Swinson, 1998; Henry & Crawford, 2005; Osman et al., 2012). Internal consistency for the DASS-21 total score was adequate in the present sample, α =.97, M = 12.68, SD = 14.84, skew = 1.20, kurtosis =.71. Barkley Deficits in Executive Functioning Scale (BDEFS-SF; Barkley, 2011). The BDEFS-SF is a 20-item self-report measure designed to identify deficits in executive functioning. Executive functioning problems over the past six months are rated on a 4-point scale from 1 (never or rarely) to 4 (very often). The BDEFS-SF has exhibited adequate psychometric properties in previous research, including evidence of internal consistency and criterion-related validity (Feldman, Knouse & Robinson, 2013). Internal consistency for the BDEFS-SF total score was adequate in the present sample, α =.96, M = 32.24, SD = 12.91, skew = 1.00, kurtosis =.33. Results The data analytic strategy and evaluation guidelines used in Study 1 to test competing models via CFA and to examine the bifactor model were also used in this study. While the approaches were the same, factor structure (item content) differed as a function of the results of the EFA from Study 2. Specifically, Factor I (Focusing) consisted of seven forward-coded items (i.e., item #s: 1-7) and Factor II (Shifting) consisted of eight forward-coded items (i.e., item #s: 10, 11, 13-16, 19, 20). Confirmatory Factor Analysis. Fit statistics for all three models are presented in Table 2. Two of three fit indices suggested that the correlated two factor model provided acceptable fit to the data (i.e., CFI, TLI); however,: RMSEA did not meet specified guidelines for evaluating model fit (i.e.,.117). The correlated two-factor model provided a significantly better fit to the

22 FACTOR STRUCTURE OF THE ACS 22 data than the one-factor model which evidenced poor fit. The magnitude of the latent correlation between Factors I and II was large in size (r =.80, p <.001). For the second-order model, examined only to obtain factor loadings for the domain-specific factors on the higher-order factor, both lower-order factors exhibited large magnitude factor loadings on the higher-order factor (Factor I [Focusing] =.89, Factor II [Shifting] =.90, ps <.001). Of the three models, the bifactor model provided the most favorable fit statistics. Δχ 2 and non-overlapping RMSEA 90% CIs indicated that the bifactor model provided significantly better fit to the data than the twofactor model. Factor loadings from the bifactor model are presented in Table 4. All items exhibited significant factor loadings on the general factor (p <.001). The seven items of Factor I (Focusing) exhibited significant positive factor loadings on this domain-specific factor (p <.001). For Factor II (Shifting), only two of the eight items assigned to this domain exhibited significant positive factor loadings (i.e., Items 14, 15). Four items from Factor II (Shifting) did not exhibit significant loadings on this factor (i.e., Items 10, 16, 19, 20) and two items from this factor exhibited significant negative factor loadings (i.e., Items 11, 13), thus suggesting content redundancy from the simultaneous loading of all items on both the general domain-specific factor (i.e., net suppression). Bifactor Model Evaluation. Strong reliability was exhibited for the general (ω =.97) and domain-specific factors (Factor I [Focusing] ωs =.95; Factor II [Shifting] ωs =.95). The general factor accounted for the large majority of variance in ACS scores (ωh =.88). The minority of reliable variance in Factor I (Focusing) and Factor II (Shifting) was independent of the general factor (ωhs =.33 and.02, respectively). The ECV value of.76 for the general factor, in combination with ECV values of.17 for Factor I (Focusing) and.07 for Factor II (Shifting), are more indicative of a unidimensional, rather than multidimensional, model. For Factor I

23 FACTOR STRUCTURE OF THE ACS 23 (Focusing), I-ECV values ranged from.59 to.71, suggesting item content that may be domainspecific. In contrast, six of the eight items from Factor II (Shifting) had I-ECV values indicating that responses to these items are accounted for by the general factor (i.e., Items 10, 11, 13, 16, 19, 20; I-ECV values from.90 to.99). Items 14 and 15 from Factor II (Shifting) had I-ECV values of.62 and.54, respectively. The PUC value for the ACS was.53, indicating that a majority of item correlations of the ACS are attributable to the general factor. The APRB across ACS items was 12% suggesting that multidimensionality within the ACS may not be substantial enough to preclude a unidimensional solution (Muthén et al., 1987; Rodriquez et al., 2016). The FD value for the general factor (.98) suggests adequate factor determinacy, whereas the FD values of.89 and.88 for Factors I and II indicate that these factors may not be suitable for use as summed subscale scores and as a latent variables in a SEM framework (Gorsuch, 1983). This hypothesis is further supported by H values of.71 and.53 for Factors I and II which suggest inadequate construct replicability. The general factor exhibited acceptable construct replicability (H =.96). Structural Regression Models. Structural regression models were used to examine whether the two domain-specific factors of the ACS relate to relevant constructs (i.e., general distress, executive functioning) when simultaneously accounting for the general attentional control factor (i.e., incremental utility). Because the DASS-21 subscales are highly correlated, some have suggested that the subscales should be considered lower-order domains of a broader general distress construct (Clark & Watson, 1991; Moras, di Nardo, & Barlow, 1992). As such, and consistent with previous research (e.g., Bardeen et al., 2013; Bradbury et al., 2008; O Brien et al., 2016; Osman et al., 2012), the DASS-21 subscales were used as indicators of a general distress construct in the first structural regression model. Specifically, general distress was

24 FACTOR STRUCTURE OF THE ACS 24 modeled as a second-order factor; correlations between lower-order factors (i.e., anxiety, depression, stress) were omitted and direct pathways were modeled from the second-order factor (i.e., general distress) onto each lower-order factor. For the second structural regression, executive functioning served as the outcome variable and was modeled as a single factor; the 20 items of the BDEFS-SF served as indicators of the executive functioning construct. 1 For both models, the general factor and domain-specific factors from the bifactor model were simultaneously regressed onto the construct of interest (general distress [DASS] or executive functioning [BDEFS-SF]). Path coefficients from the general factor and domain-specific factors to each construct of interest were freely estimated. The general distress construct (i.e., DASS-21) described above evidenced adequate fit to the data, χ 2 (576) = , p <.001; RMSEA =.073 (90% CIs = ); CFI =.99; TLI =.99. When the general distress construct was modeled as the outcome variable in the first structural regression, model fit was adequate, χ 2 (576) = , p <.001; RMSEA =.049 (90% CIs = ); CFI =.99; TLI =.99. The ACS general factor significantly predicted general distress, β = -.23, p <.001. Factor I (Focusing) was not a significant predictor of general distress, β = -0.11, p =.11. Factor II (Shifting) significantly predicted general distress, β =.14, p =.001, but the direction of the relation was unexpected. To test for a possible suppression effect, the structural regression was modified such that general distress was only regressed onto Factor II (Shifting; i.e., the general factor and Factor II were modeled, but did not serve as predictor variables). In contrast to the first structural regression, the association between Factor II (Shifting) and general distress was in the theoretically expected direction, β = -.34, p <.001. For the executive functioning construct (i.e., BDEFS-SF) described above, two out of three fit statistics suggest adequate fit to the data, χ 2 (170) = , p <.001; RMSEA =.112

25 FACTOR STRUCTURE OF THE ACS 25 (90% CIs = ); CFI =.97; TLI =.97. In the second structural model, in which executive functioning served as the outcome of interest, model fit was adequate, χ 2 (542) = , p <.001; RMSEA =.058 (90% CIs = ); CFI =.98; TLI =.97. The ACS general factor and Factor I (Focusing) significantly predicted executive functioning, β = (p <.001) and (p =.005), respectively. When accounting for the general factor, Factor II (Shifting) did not significantly predict executive functioning, β = -0.10, p =.07. Study 3 Summary The bifactor model evidenced better fit in comparison to the one- and two-factor models. The one and two-factor models exhibited poor fit. Concern over the distinctiveness of the domain-specific factors was amplified by the results of Study 3, as forward coding items resulted in a large correlation between the two factors, as well as high loadings of each lower-order factor onto the higher-order attentional control factor. In contrast to results from Study 1, bifactor indices from this study, with only forward-coded items, are more indicative of a unidimensional model (i.e., ECV, PUC, APBR). Additionally, reliability was improved by forward coding items (i.e., ωh and ωhs). When accounting for the general factor, only two of eight items on Factor II (Shifting) exhibited significant, positive, factor loadings (i.e., 14, 15). Two of the remaining five items on Factor II (Shifting) exhibited significant negative factor loadings when accounting for the general factor. This may be a function of net suppression and indicative of content redundancy with the general factor. I-ECV values further suggest that Factor II (Shifting) is redundant with the general factor. In contrast to H and FD values for the general factor, H and FD values for Factors I and II suggest poorly defined factors that may be of little practical value. To test the predictive utility of the lower-order factors beyond the general factor, we conducted two structural regressions with self-report measures of general distress and executive

26 FACTOR STRUCTURE OF THE ACS 26 functioning serving as outcome variables. The ACS general factor significantly predicted both outcomes. After accounting for the ACS general factor, Factor I (Focusing) did not significantly predict general distress. Interestingly, Factor II (Shifting) significantly predicted general distress, but in a theoretically inconsistent direction. Suppression and/or redundancy effects can occur in regression when multiple predictor variables with strong correlations are entered simultaneously (Patrick, Hicks, Nichol, & Krueger, 2007; Paulhus, Robins, Trzesniewski, & Tracy, 2004). Results from a follow-up structural model suggest that the unexpected positive relationship between Factor II (Shifting) and general distress, when controlling for the ACS general factor, represents a suppression effect. This is consistent with the results of the bifactor analysis suggesting that the content of Factor II (Shifting) is redundant with the ACS general factor. In the second model, Factor I (Focusing) accounted for a small amount of unique variance in predicting executive functioning after accounting for the ACS general factor. General Discussion The factor structure of the ACS (Derryberry & Reed, 2002), in its original form and with forward coded items, was examined in the present set of studies. For both versions of the measure, the correlated two-factor structure, proposed in previous research (Judah et al., 2014; Ólafsson et al., 2011), was compared to a bifactor model. The bifactor model provided significantly better fit to the data compared to competing models and evidence from Study 3 suggests that the focusing factor may have limited utility beyond the general factor. However, examination of bifactor statistical indices beyond general model fit suggest that the domainspecific factors may be poorly defined, have poor construct replicability, and may not be sufficiently distinct from the general factor to warrant use as subscales. Among the domainspecific factors, the psychometric limitations of Factor II (Shifting) are particularly concerning.