Evaluating a Special Education Teacher Observation Tool through G-theory and MFRM Analysis

Transcription

1 Evaluating a Special Education Teacher Observation Tool through G-theory and MFRM Analysis Project RESET, Boise State University Evelyn Johnson, Angie Crawford, Yuzhu Zheng, & Laura Moylan NCME Annual Conference San Antonio, April 2017

2 RESET Purpose - Create a special education teacher (SET) observation tool that is: 1) aligned to evidence-based instructional practices (EBPs) 2)provides feedback to SETs to improve their implementation of EBPs 3)is connected to student outcomes

3 Evidence-Centered Design* *Mislevy, Steinberg & Almond, 2003

4 RESET: Conceptual Framework 1. Instructional Delivery Subscale Explicit Instruction Cognitive Strategy Instruction Peer-Mediated Instruction 2. Content Subscale Reading Math Writing 3. Individualization Subscale

5 Assessment Development To create the items for each rubric we: conducted a review of research on that practice synthesized across the common elements of the EBP across studies drafted rubric items that capture the common elements of that EBP

6 Explicit Instruction Rubric 3 - Implemented 2 - Partially Implemented 1 - Not Implemented 1 The goals of the lesson are clearly communicated to the students. The goals of the lesson are not clearly communicated to the students. The goals of the lesson are not communicated to the students. 2 The stated goal is specific. The stated goal is broad or vague. There is no stated goal. 3 The teacher clearly explains the relevance of the stated goal to the students. The teacher tries to explain the relevance of the stated goal to the students, but the explanation is unclear or lacks detail. The teacher does not explain the relevance of the stated goal to the students. 4 Instruction is completely aligned to the stated or implied goal. 5 All of the examples or materials selected are aligned to the stated or implied goal. Instruction is partially or loosely aligned to the stated or implied goal. Some of the examples or materials are aligned to the stated or implied goal; OR examples and materials are somewhat aligned to the stated or implied goal. Instruction is not aligned to the stated or implied goal. Examples or materials selected are not aligned to the stated or implied goal.

7 Assessment Delivery (Validation) There are numerous sources of error that influence the precision of observation results (Hill, Charalambous, & Kraft, 2012 ). Generalizability studies are useful for understanding the relative importance of various sources of error to assist in the design of more efficient measurement procedures (Brennan, 1992; Shavelson & Webb, 1991).

8 G-study purpose to estimate the variance attributable to different facets of measurement: teachers, lessons (occasions), raters, and items

9 Observation and Estimation Design Three facet, mixed model, partially nested design EduG v. 6.1 software Measurement Design: TL/RI

10 G-Study Table Grand mean for levels used: Variance error of the mean for levels used: Standard error of the grand mean: Minimum threshold (Nunnaly & Bernstein, 1994)

11 Analysis of Variance Source Variance SE % Teachers (T) Lessons (L) nested within (:) T Raters (R) Items (I) TR TI LR:T LI:T RI TRI LRI:T, Error

12 G-theory views the data largely from a group-level perspective - disentangling the sources of measurement error and estimating their magnitude - Many-Facets Rasch Measurement MFRM analysis focuses on individual level information and allows for substantive investigation into the individual element of the facets under consideration

13 Measr -Items +Teacher -Lesson -Raters Scale 2 (3) 3 Teacher 9 Teacher 2 Teacher 6 13 Teacher Teacher 7 Teacher , 8 Teacher 4 4 2, 9, 7 24 Teacher 1 Teacher 3 15, Teacher ,22 4, 17, , Figure 1. Variable map of the RESET facets items, teachers, lessons and raters.

14 Item Measure Report Item Number Difficulty (Logits) Model SE Infit MNSQ Outfit MNSQ Mean SD

15 Teacher Measure Report from Many-Facet Rasch Measurement Analysis Teacher Number Ability (Logits) Model SE Infit MNSQ Outfit MNSQ Mean (count = 10) SD Note. Root mean square error (model) =.08; adjusted SD =.61; separation = 7.39; reliability =.98; fixed chi-square = 492.7; df = 9; significance =.00.

16 Lesson Measure Report from Many- Facet Rasch Measurement Analysis Lesson Number Difficulty (Logits) Model SE Infit MNSQ Outfit MNSQ Mean (count = 4) SD Note. Root mean square error (model) =.05; adjusted SD =.19; separation = 3.72; reliability =.93; fixed chi-square = 43.4; df = 3; significance =.00.

17 Rater Measure Report from Many- Facet Rasch Measurement Analysis Rater Number Severity (Logits) Model SE Infit MNSQ Outfit MNSQ Mean (count = 4) SD Note. Root mean square error (model) =.05; adjusted SD =.47; separation = 9.07; reliability =.98; fixed chi-square = 232.7; df = 3; significance =.00.

18 THANK YOU