Structure. Results: Evidence for Validity: Internal

Transcription

1 Results: Evidence for Validity: Internal Structure Majority of the respondent ability distributions are covered across their ranges by the item threshold locations. However, the threshold estimates fail to cover the lower end of the ability distributions.

2 Results: Evidence for Validity: Internal Structure For items 3, 7, 9, 11, 13, 17, 21, and 25 the hypothesized ordering of levels is observed for all of the categories. For items 1, 15, 19, 23, 27, 29, and 31 the hypothesized ordering was consistent with the empirical ordering for at least one pair of categories.

3 Results: Evidence for Validity: Internal Structure For items 2, 4, 10, 12, 14, 18, 20, 24 the hypothesized ordering of levels is observed for all of the categories. For items 8, 16, 22, 26, and 28 the hypothesized ordering was consistent with the empirical ordering for at least one pair of categories.

4 Results: Evidence for Validity: Internal Structure Three items fell outside the upper bound of this range (i.e., item 1 = 1.39, item 22 = 1.4, and item 25 = 1.5) and one item fell exactly on it (i.e., item 12 = 1.33).

5 Results: Evidence for Validity: Construct Representation -The saturated PCM model fit the data significantly better than The MFRM (no interactions model) (χ 2 (15) = 36.81, p =.001) -The saturated PCM model fit the data significantly better than The MFRM (interactions) (( χ 2 (9) = 31.16, p =.001) -The better fit of the saturated PCM model is also confirmed by the AIC values for each model.

6 Results: Evidence for Validity: Construct Representation -This result was not supported by the BIC values for each model. -When comparing between the two item explanatory models, the LR test was not significant as χ 2 (6) = 5.65 (p = 0.463), meaning that the interaction effects of the Resolution with the Emotion or the Language properties were not substantial. -Lower BIC values suggest that a model is considered to be more likely to be the true model.

7 Results: Evidence for Validity: Construct Representation Holding other properties constant, basic and both basic and complex emotions increased overall item difficulty by and logits, respectively

8 Results: Evidence for Validity: Relations to other Variables No significant relationships. It is observed, however, that within both the SPU and EI dimensions, the direction of the relationship between theta estimates and AQ score is as predicted (α = -.16 and α = -.15, respectively).

9 RQ # 1 Reliability Estimates of internal consistency within each of the dimensions on the test forms suggest that the SPU and EI items produced stable measurements along each of the measurement constructs. SEM(θ) estimates for each of the dimensions provides evidence of content coverage within each of the dimensions. The SER instrument did not function optimally for respondents with lower levels of proficiency. Therefore, the instrument will need to be augmented with more items targeting this section of the ability distribution.

10 RQ # 1 & RQ # 2 Validity Evidence for validity presented above suggests areas of strength and areas for improvement. Strength: Internal Structure The expected ordering of average θ estimates by score category for the SPU scale finds support in the literature, specifically when the considering the step transition from Basic Emotion SPU Detection to Complex Emotion SPU Detection (δ i3 ). This finding is consistent with the work of Baron-Cohen et al. (2001). The expected ordering of average θ estimates by score category for the EI scale also find support in the literature.

11 RQ # 1 Strength: Construct Representation The presence of Basic emotion SPUs and the presence of Both basic and complex emotion SPUs had statistically significant effects on the overall difficulty of correctly evaluating the outcome of the scenarios. This finding is consistent with the results of at least one study. Compared to samples of individuals without ASD, participants with ASD performed significantly more poorly on social perception questions relating to stories that contained multiple types of social cues (Pierce, Glad, & Schreibman, 1997).

12 RQ # 1 Strength: Construct Representation The lower BIC values for the MFRM (no interaction) and MFRM (interaction) relative to the saturated PCM model suggests that the item design according to the the emotion, language, and resolution factors worked well for item generation.

13 RQ # 1 Weakness: Relations to other Variables The relationships predicted to hold between the SER variables and the person properties investigated were not observed. Possibly, the person predictors in this analysis were the incorrect ones. More appropriate examples may include: 1) A measure of reading comprehension, or 2) number of comics read each month. Regarding the AQ-10, the extremely low estimates of person separation reliability (0.1) and internal consistency (0.16), in addition to the sparse coverage of the ability distribution by the item difficulty estimates indicate that this instrument was poorly suited for this sample.

14 RQ # 2 The superior fit of the two-dimensional model provides empirical support for explicitly distinguishing between the task of identifying instances of salient social information and the task of reasoning about the effectiveness of an individual s behavior in response to that information within a specific social context.

15 RQ # 2 In the literature on cognitive processing, the detection of SPUs is an example of a lower-level, local processing task. Local processing refers to the ability to identify the details of a stimulus (e.g., features, parts, or properties). Making accurate evaluative inferences, on the other hand, is an example of a higher-level, global processing task. Global processing refers to the ability to integrate the details of a stimulus into a coherent whole.

16 Problematic Testlet (θ 13 ) Variance estimate of this residual testlet dimension was 4.5 times that of the general factor.

17 Within each dimension, the range of θ encompassed by the first and last threshold estimates for each of the items associated with this scenario is the largest (# 25 = 3.5 logits, #26 = 2.56 logits). The WMSQ fit statistic for item 25 = 1.5

18 Additionally, regarding item #25, it was not possible to score into the highest level on the SPU construct, since only a single complex emotion is depicted. Comprehensive SPU Detection (level 4) category includes responses that identify basic and complex emotion SPUs or that accurately infer and attribute relevant qualifiers to the customer on the basis of the individual SPU(s)

19 A unique aspect of this scenario is that it is one of two in which the customer does not speak the complex emotion confusion is communicated nonverbally through bodily posture, facial expression, and by the thought bubble containing three question marks. Another unique aspect of this comic strip is the depiction of two coworkers communicating with one another in all of the others, communications are between employees and customers.

20 Removing items 25 and 26 result in a better model fit.

21 Next Steps Build the SPU and EI constructs into a larger Social Problem Solving framework based on the work of Collet-Klingenberg and Chadsey- Rusch (1991) and Park and Gaylord (1989). The Assessment of Workplace Solving Problem-Solving Ability in the Workplace (AWSPSA) includes two additional constructs: 1. Problem Definition 2. SPU Detection 3. EI 4. Solution Alternatives A Goal 5 grant has been submitted to IES in the topic area of Transition outcomes for secondary students with disabilities