WWW.KEYDATASYS.COM ContactUs@KeyDataSys.com 951.245.0828
Table of Contents INSPECT Overview...3 What Comes with INSPECT?....4 Reliability and Validity of the INSPECT Item Bank. 5 The INSPECT Item Process......6 INSPECT FAQs......7 CCSS Item Count by Grade Level and Content Area.....8 CCSS Item Count by Item Type (ELA)... 9 CCSS Item Count by Item Type (Math)......10 2
INSPECT Overview INSPECT is an innovative, living test bank. The bank is composed of items that are written to assess the Common Core State Standards. Professional item writers focus on writing quality items that have content accuracy, range in cognitive complexity, and meet grade level standards. Items include clear rationales for why a student would pick each wrong answer as well as comprehensive rubrics and samples for all constructed response items. For this reason, all wrong answers are created based upon the most-likely cognitive disconnect in learning. Rigorous field-testing and item analyses are conducted to ensure the quality of the items in the INSPECT test bank. In addition to the test bank, INSPECT also includes a large library of pre-made CCSS, ELD, Algebra Readiness and CAHSEE assessments. Hundreds of new CCSS items and/or assessments are added monthly to the bank at no additional cost to provide educators the resources they need to address CCSS and 21 st century assessment demands. INSPECT has been developed in close partnership with hundreds of schools and educators. The INSPECT team quickly responds to any issue reports or item requests within a timely manner to ensure that support is available every step of the way. 3
What Comes With INSPECT? CCSS ITEMBANK 1. 26,000+ CCSS items (including non-tested grades) 2. Innovative SBAC-type items: constructed-response, non-traditional selected response, drop-down menus, multi-media, and more 3. Distractor rationale and rubric (constructed response questions) information to guide and inform instruction 4. Depth of Knowledge (DOK), Bloom s Taxonomy, and Difficulty level alignments 5. SBAC Claims and Targets scoring category alignments 6. Rigorous creation and vetting process, and regular validity and reliability review PREBUILT ASSESSMENTS Already built for your district 7. CCSS prebuilt assessments CCSS Interim Formative Assessments (Fall, Winter, and Spring) modeled after SBAC blueprints for math and ELA CCSS Summative Assessment (End of Year) developed using SBAC blueprints for math and ELA 8. 200+ prebuilt, comprehensive assessment suite, including ELLA (ELD Benchmarks) Algebra Readiness Assessment CAHSEE Diagnostic Assessment CAHSEE Blueprint Assessment CAHSEE 5 Standards Assessment EAP Diagnostic Assessment CCSS PERFORMANCE TASKS 9. CCSS Performance Tasks per tested grade level for ELA and math CCSS Standards Schedules 10. Suggested standards pacing for all grade levels, for math and ELA, following CCSS Interim and Summative Assessments (see above) REPORTING 11. Powerful reporting for targeted instruction by standard, by item, by SBAC and cognitive rigor scoring categories, and overall test, at the student, classroom, site and district levels 4
Reliability and Validity of the INSPECT Item Bank Multiple quality assurance measures are taken in order to ensure the reliability and validity of each INSPECT item. This level of quality control begins with the initial writing of items and continues well after each item has been administered. Our treatment of these issues includes quantitative and statistical approaches. When issues with an item are intercepted, steps are taken to remedy the issue. To examine reliability statistically, Kuder-Richardson formula 20 (KR20) is calculated. KR20 is a measure of internal consistency reliability for measures with dichotomous choices (e.g., correct versus incorrect). A high KR20 coefficient (around.90) indicates a test is measuring one construct (e.g., 3rd grade math). Our first efforts at establishing validity are centered around content validity. This is sometimes referred to as content definition (Messick, 1989) and is critical for score interpretation and item response validation (Haladyna, 1999). The best way to establish content validity is through the use of expert judges. For each INSPECT item, up to three Content Experts judge whether or not a particular item is aligned to the standard that the item writer has chosen. In addition to standard alignment, the cognitive complexity is assessed. Each item requires full consensus as to the accuracy of the standard alignment and cognitive complexity' before it makes it into the item bank. In addition to standard alignment, the level of difficulty (low, medium, and high) as well as cognitive complexity (DOK and Blooms level) are assessed. Each item requires full consensus as to the accuracy of the Common Core Standard alignment, difficulty, and cognitive complexity before it may become a part of the item bank. We also conduct focus groups with grade level teams of teachers. Information from these teams are compiled and used to modify items or other areas of concern. After assessment, item discrimination is also assessed to ensure that each item has the ability to sensitively measure individual differences in knowledge and ability among test-takers. Each item should be able to discriminate between high-performing students and lower-performing students. That is, a higher percent of students with more knowledge in a given subject area should answer those questions with greater accuracy than students with lower ability or knowledge in that area. Item discrimination is estimated using a point-biserial. This gives an estimate of the relationship between a given item and overall test performance. The point-biserial has a direct relationship to alpha (the reliability coefficient that is used to assess the reliability of a given test). Each summer all items are re-evaluated to ensure that the items are performing as they should. Any items that may cause concern are either revised or removed from the item bank and replaced. 5
The INSPECT Item Process INSPECT employs a professional team of item writers consisting of both current and former teachers and administrators. These highly-skilled item writers are based throughout the country and write INSPECT items specifically targeting the Common Core State Standards. In order to become an item writer for the INSPECT item bank, each qualifying applicant must go through a training process involving a number of informational webinars and documents. After training, they must submit sample items aligned to a particular set of guidelines. All sample items need to be accepted prior to becoming a contracted item writer. Even after the applicant becomes an official INSPECT item writer, they are held to a high standard. Each item writing month, an item writer must maintain an 85% acceptance rate to continue on. Throughout the item review process, each item writer maintains a focus on content accuracy, grade level, standard alignment, and cognitive complexity. For selected response items, the item writers must provide distractor options that reflect the most-likely wrong answers so that the most common cognitive disconnects may be revealed and targeted for intervention. The item writers must also provide a clear rationale for each wrong answer so these cognitive disconnects can be easily identified. For all constructed response items, item writers must provide comprehensive scoring rubrics and sample responses for ease of scoring. Once an item has been completed and submitted for evaluation, it goes through an intensive review process. A Content Expert evaluates each item and determines whether it should be accepted into the next level of evaluation or rejected; this determination is made by assessing the content, standard alignment, and cognitive complexity of each item. Rejected items are sent back to the writer and accepted items are sent to two additional Content Experts for further scrutiny. Only items that make it through all three levels of review receive final acceptance and are processed into the system for use in testing. 6
INSPECT FAQs What is INSPECT doing in preparation for the transition to the Common Core State Standards? Since 2010, professional item writers have been tasked with the duty of creating high quality items aligned to the Common Core State Standards. Each one of our professional item writers is expected to produce items that reflect content accuracy, range in cognitive complexity, and appropriate rigor expected of the Common Core State Standards. Every item received is reviewed by three content experts to verify that these goals are fully met. This extensive review process guarantees that our items follow an evidence-centered design in which each item provides evidence of the student's mastery of the concept. INSPECT currently features over 25,000 Common Core-aligned items with over 10,000 English language arts items and over 15,000 math items. Our item bank is constantly growing, increasing by several hundred new items each month. Does INSPECT feature new item types? With the ever-increasing expectations of the Common Core State Standards, new, innovative item types have become a major focus to be able to fully gauge students' mastery of the standards. As a result, professional INSPECT item writers have been developing these new types of items for assessment purposes. These item types include the following: Constructed response - open-ended items that use rubric scores Non-traditional selected response - items that feature anywhere from 3-6 options with up to 3 correct answer choices Technology-enhanced - items that require students to utilize a number of technology features to answer questions (drag and drop, selectable text, drop-down boxes, etc.) As Common Core develops, more item types will be added to the INSPECT item bank in order to ensure full alignment and development of these standards. Will INSPECT provide assessments for non-tested grade levels? In order to support non-tested grade levels, a full suite of assessments will be created. This includes: Interim Formative Assessments for grades K-2 and 9-10 Summative Assessments for grades 9 and 10 ELA Performance Tasks for grades 9 and 10 Do INSPECT CCSS items cover all skills within a standard with items at various levels of complexity? In order to fully assess college and career readiness students will need to be assessed at various levels of cognitive complexity. As a result, INSPECT 's focus is on providing items of various complexity levels in order to adequately measure students' deep conceptual understanding of the Common Core State Standards. All Common Core items include levels for both Webb's Depth of Knowledge and the Revised Bloom's Taxonomy. Furthermore, each mathematics common core item is aligned to mathematical practice standards. INSPECT s use of innovative item types also allows for the assessment of students at various levels of complexity. With the use of these innovative item types, many of the INSPECT CCSS items will be able to adequately assess many of the skills, if not all of the skills, within a single standard; however, there may be cases where multiple items are necessary in order to adequately assess the full range of skills within certain broader standards. Can I administer Paper Pencil and Online Tests? INSPECT assessments can be administered in either paper/pencil format via scanning software or web-based online testing. Keep in mind, certain technology-enhanced items cannot be administered with traditional paper/pencil assessments. 7
CCSS Item Count by Grade Level and Content Area INSPECT Item Count as of November 20th Hundreds of new items are added each month Common Core Items Grade English Language Arts Mathematics Total K 397 562 959 1 976 888 1864 2 888 1097 1985 3 1377 1567 2944 4 1691 2509 4200 5 1222 1947 3169 Elementary Total 6551 8570 15121 6 1047 2221 3268 7 703 1524 2227 8 669 1336 2005 9-10 1462 11-12 408 1540 3410 Secondary Total 4289 6621 10910 Total 10840 15191 26031 8
Item Count by Item Type (ELA) INSPECT Item Count as of November 20th Hundreds of new items are added each month English Language Arts Grade CR SR TE Total K 3 394 0 397 1 12 964 0 976 2 5 812 71 888 3 75 1205 97 1377 4 34 1547 110 1691 5 53 1107 62 1222 Elementary Total 182 6029 340 6551 6 101 901 45 1047 7 72 594 37 703 8 83 559 27 669 9-10 159 1233 70 1462 11-12 107 249 52 408 Secondary Total 522 3536 231 4289 Total 704 9565 571 10840 CR = constructed response NTSR = innovative selected response items with 3-6 options with up to 3 correct answers SR = traditional 4 option, 1 correct answer selected response items TE = technology-enhanced *NTSR and SR are combined in English Language Arts due to limited opportunity to utilize this item type with the English Language Arts Common Core State Standards. 9
Item Count by Item Type (Math) INSPECT Item Count as of November 20th Hundreds of new items are added each month Math Grade CR NTSR SR TE Total K 35 8 518 1 562 1 50 7 830 1 888 2 17 8 1070 2 1097 3 55 18 1367 127 1567 4 112 52 2242 103 2509 5 124 237 1529 57 1947 Elementary Total 393 330 7556 291 8570 6 282 221 1676 42 2221 7 330 88 1070 36 1524 8 67 173 1079 17 1336 9-10 11-12 55 140 1334 11 1540 Secondary Total 734 622 5159 106 6621 Total 1127 952 12715 397 15191 CR = constructed response NTSR = innovative selected response items with 3-6 options with up to 3 correct answers SR = traditional 4 option, 1 correct answer selected response items TE = technology-enhanced 10