Tips & Techniques for Writing Good Questions

Transcription

1 Tips & Techniques for Writing Good Questions Brian McNamara Marketing Director, Questionmark All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2 What we ll cover today Item writing within the test development process Item writing recommendations and common pitfalls Items that measure beyond recall Slide 2

3 To ask questions, use the Q&A feature Slide 3

4 About Questionmark Background Founded in 1988 Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes ISO/IEC Certified (Learn more at Customers Enterprise customers worldwide Business, Government; Higher Education; Certification Providers; Awarding Bodies

5 Item Writing within Assessment Development Process Planning assessment Authoring items Assembling assessment Pilot and review Delivery Analyze results Security

6 Assessment development process/cycle Plan / Blueprint Analyze results Author items Delivery Assemble assessment Pilot and review

7 Validity and Reliability Reliable: dependable, repeatable, consistent Valid: measures appropriate knowledge and skills Figure 1: Reliable but not Valid Figure 2: Not reliable, not Valid Figure 3: Reliable and Valid

8 8 Job Task Analysis (JTA) Surveys for content planning, content validity Survey SMEs, high performers: How difficult is it to do this task? How important is it to do this task? How frequently do this task? Job Task Analysis to determine content of test Test Blueprint built from JTA, which drives test item development. Brandon Hall Group 2016

9 Example of Assessment Blueprint Electrical Skills Test - Blueprint % of test # items Skill level Criticality Section 1: AC Motors 45% 34 1.a - Lorem ipsum dolor b - Duis aute irure c - Excepteur sint occaecat Section 2: High-voltage switchgear 25% 19 2.a - Neque porro quisquam b - Modi tempora incidunt Section 3: Synchronous Motors 30% 23 3.a - Totam rem aperiam b - Eaque ipsa quae

10 Terminology/Basic Structure of a Multiple Choice Item Stem / Stimulus Choices / Options / Alternatives What is the capital city of Peru? A. Santiago B. Bogota * C. Lima D. Montevideo Correct / Keyed Response Distractors Slide 10

11 Question-level feedback Question with participant s answer Corrective feedback Slide 11

12 Tips and Guidelines Slide 12

13 Yes or No: When loading your snarkleblaster, should you beedle the diggle or zix the frondle? o Yes Focus item on single question o No

14 When putting a child to bed for the night, you should: a. Spray him/her with a hose. b. Create a calm and relaxing atmosphere. c. Parade 3 elephants through the room. d. Paint yourself purple with pink polka dots and dance a jig. Ensure distractors are plausible

15 A chimpanzee s favorite treat is: a: a.apple a. An Apple b.banana b. A Banana c. Orange c. An Orange d.eggplant d. An Eggplant Avoid grammatical Cues

16 When replacing strings on a bass guitar: a. Change the pre-amp s battery. b. Seat the thick end of the string in the bridge, lightly tighten the string with the tuning machine in the headstock, then stretch the string before fully tightening and tuning it. c. Set the equalization. d. Adjust the strap. Wording provides cues gives away correct choice Aim to keep distractors consistent in length, homogenous as possible

17 Which factors impact a presidential election? Select a. Previous all that apply. president s performance, weather, local economy Previous president s performance b. Weather, local economy, national unemployment rate Weather c. c. Congress performance rating, weather, national unemployment rate Local economy d. Local economy, weather, Congress performance National unemployment rate rating

18 Which of the following is NOT not a characteristic of a friggle in a state of grex? a.hoomishness b.blue rickles c.excessive sninishness d.lack of nongs Avoid negatives in the stimulus If you must use a negative, provide emphasis make it clear to the participant

19 Arrange the parents of the following children with Down s syndrome in order of highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are: I: 46, XX, -14, +T(14q21q) pat II: 46, XX, -14, +T(14q21q) de novo III: 46, XX, -14, +T(14q21q) mut IV: 46, XX, -21, +T(14q21q) pat V: 47, XX, -21, +T (21q21q) (parents not karyotyped) A: III, IV, I, V, II B: IV, III, V, I, II C: III, I, IV, V, II D: IV, III, I, V, II E: III, IV, I, II, V Avoid unnecessarily complexity Source: National Board of Medical Examiners Constructing Written Test Questions for the Basic and Clinical Sciences

20 Which of the following describes a banana? a. It s a vegetable. b. It s yellow. c. It s blue. d. It s round. e. All of the above. Two distractors mutually exclusive quickly eliminates All of the above as a viable choice

21 Risks of AOTA and NOTA All of the Above If participant recognizes at least two distractors as correct can logically assume AOTA is correct Recognizing at least one of the distractors as incorrect quickly eliminates AOTA as plausible option None of the Above Defensibility if there are any issues with the keyed correct response, then NOTA could arguably be a correct response Slide 21

22 Well Written Stimulus Carefully worded Clear Concise avoid window dressing Avoid jargon Positive phrasing Specific focus Ask a single question Slide 22

23 Well Written Choices Only one correct answer (Multiple choice) Plausible distractors Roughly same length As short as possible Avoid keywords from within stimulus Avoid grammatical cues Slide 23

24 Avoiding bias and stereotyping Use neutral terms (sales agent instead of salesman) Standard, formal language Avoid references to race, ethnicity, gender, age, etc. Avoid idioms and colloquialisms Avoid obscure language, ambiguous acronyms, etc.

25 Using diagrams/illustrations You are operating a forklift rated for a 6000-pound capacity at a 24-inch load center. What is the maximum load capacity of the forklift to safely move a crate with a 36-inch load center? A pounds B pounds C pounds D pounds Answer requires participant to understand : Terminology (e.g. load center ) How to calculate the maximum weight: Divide truck's load center by the crate's load center Multiply this by the truck's maximum capacity. B pounds (24 / 36) * 6000 = 4000 pounds

26 You are operating a forklift rated for a 6000-pound capacity at a 24-inch load center. What is the maximum load capacity of the forklift to safely move a crate with a 36-inch load center? A pounds B pounds C pounds D pounds Does the illustration increase the fidelity? Or does it give away the answer? May make easier to guess addressable with a numeric question

27 Considerations for using images, When appropriate and/or potential to enhances fidelity Show figure with test item Ensure clarity, appropriate size and perspective Consider and address accessibility illustrations and diagrams Slide 27

28 Other Considerations. Develop a style guide Helps writers ensure consistency in Visual Composition Language Composition Documenting processes for Reproducibility Follow the same steps and use the same resources if you need to maintain the assessment or build a new assessment in the future Accountability Demonstrate that best-practices were followed in the even of a challenge or dispute Collect evidence needed for third party accreditation Organization Keep track of assessments or content that are in different phases of development Validity evidence Provide evidence to support validity arguments for the inferences of your assessment Slide 28

29 Questions that measure Beyond Recall Slide 29

30 Bloom s Taxonomy Classifies cognitive skills in levels Most real world jobs require many levels of taxonomy create evaluate analyze apply Cognitive Demand understand * Revised Bloom s Taxonomy. Original version from Bloom ordered levels as: Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation remember / recall Slide 30

31 Beyond Recall Advice from the experts: (William Coscarelli and Sharon Shrock) Knowledge (recall) level items are by far the easiest to write. Developing tests that truly reflect on-the-job performance requires writing items at the higher cognitive levels In general, the single most useful improvement you can make in writing test items is to write them above the memorization level. create evaluate analyze apply understand remember / recall

32 Some approaches to assessing higher level skills Writing conventional questions to assess beyond recall Use multiple choice, matching, ranking and other question types Techniques eliciting higher level cognition required to get the right answer Observational Assessment / Performance Testing An observer watches someone performing a practical task Answers questions about performance on a mobile device Situational Judgement Assessments Frame questions based on a dilemma that could be encountered in workplace Evaluate judgement of what course of action is taken Slide 32

33 Recall example What does a yellow traffic light mean? a) Stop b) Go c) Caution d) Look behind you Slide 33

34 Understand example If you are driving toward an intersection and the light turns from yellow to red, what should you do? a) Speed up and cross the intersection b) Continue at the same speed and cross the intersection c) Stop suddenly d) Stop gradually Slide 34

35 Evaluate or Judgement example You are giving a close friend a ride to work and because of traffic you are running 15 minutes late for an important appointment. You are driving toward an intersection with traffic lights on it. You can see the intersecting road to some extent and there seem to be no cars on it. The light turns to yellow as you approach. The car behind you sounds their horn and seems to be accelerating in the expectation that you are going to cross the intersection. Your friend tells you to put your foot down as you are running late and there are no cars visible on the intersection. What should you do? a) The road appears to be clear so speed up and cross the intersection to exit the intersection as soon as possible b) Stay at the same speed and continue to cross the intersection while keeping an eye out for crossing cars c) The light is about to turn red so you should stop immediately and ignore the car behind you as they will need to stop as well d) Put your hazard lights on to signal to the car behind you and stop at the intersection if the lights facing you turn red e) Ignore your friend s irritation and move across to the side of the road to let the other car pass Slide 35

36 Observational Assessment / Performance Testing Observer rates the participant Example Applications Sales Training Level 3 Evaluations Customer Service Equipment operation

37 Getting Objective Ratings, Reliable Results Context Can be delivered in Performance Environment Simulated Performance Environment (e.g. role playing) Items typically solicit ratings for: Performance of task(s) Demonstration of appropriate knowledge, skills, abilities, behaviors Goal is to ensure ratings are as objective as possible Keys to ensuring reliability of observational assessment results: Establish consistent rating scales & scoring rubrics Educate raters to ensure consistent understanding and application of rating scales Slide 37

38 Assessing Judgement and Decision Making Rules immensely important in defining the parameters within which teams and individuals should operate Rules alone are not enough Whether and how they are adopted by people when making decisions that matters Judgement is at the heart of nearly every business scandal that ever occurred (Ernst & Young) Slide 38

39 Assessing for Situational Judgement Presenting work-relevant dilemmas with questions that require judgements about possible responses SJAs Assess how well participants can: Recognize critical aspects of a situation and how they should be prioritized Identify which actions/options are more vs. less likely to achieve objective Determine which responses are more vs. less aligned with organizational expectations Display soft skills required work collaboratively with colleagues to overcome obstacles Deal with dilemmas with no clear positive outcomes or where available information may be ambiguous Slide 39

40 Key Aspects of a Dilemma Choice needs to be made Using judgement for the choice is clearly linked to a needed domain of knowledge, skill or competency in the job role Requires professional judgement to answer well. (Question discriminates between more and less capable people)

41 Example SJA question - compliance You work in the back office in the team approving new customers ensuring that the organization s procedures have been followed (such as credit rating and know your customer). Your manager is away on holiday this week. A senior manager in the company (three levels above you) comes into your office and says that there is an important new customer who needs to be approved today as they want to place a big order, and that he can vouch that the customer is good. You review the customer details, and one piece of information required by your procedures is not present. You tell the senior manager and he says not to worry, he is vouching for the customer. You know this senior manager by reputation and have heard that he got a colleague fired a few months ago when she didn t do what he asked. Select the most and least effective of the following options: Take the senior manager s word and approve the customer Call your manager s cellphone and interrupt her holiday to get advice Tell the manager you cannot approve the customer without the information needed Make a judgement Ask the manager for signed written instructions to override standard procedures to allow you to approve the customer Slide 41

42 Steps to construct an SJA item Define competencies Build a dilemma for the situation Survey experts to refine Survey SMEs to identify situations Develop response options Item analysis Types of response options: Select most effective Select most effective and least effective Rank options (e.g. most effective to least effective ) Multiple response Survey ratings Slide 42

43 See our White Paper for more guidance: Assessing for Situational Judgement Assessing for Situational Judgment Designing, deploying and getting value from Situational Judgment Assessments By Eugene Burke & John Kleeman 40-page white paper provides: Overview of Situational Judgment Assessments (SJAs) Benefits and applications of SJAs Techniques for creating and validating SJAs Example SJA items to illustrate key concepts Slide 43

44 Recommended Resources Short Videos: Item writing tutorial Getting started with Authoring Assessments White papers and ebooks 5 Steps to better tests Item Analysis Analytics Assessing for Situational Judgement Assessments through the Learning Process Slide 44

45 More good practice Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training Sharon Shrock and William Coscarelli Developing and Validating Multiple-Choice Test Items Thomas Haladyna Slide 45

46 Considerations for choosing a tool to develop high-quality items Slide 46

47 Authoring Delivery Reporting Questions to ask when choosing a tool to develop high-quality items Collaborative? (Easy for multiple SMEs to be involved?) Security? Item bank organization and management Shuffle/Randomize choices Version control? (Easy to maintain item version history) Feedback Is it easy to deliver assessments for trial purposes? Can I include experimental questions without impacting scores? Can I capture people s comments during pilots to improve validity? Can I run reports on trial assessments? Item analysis to ensure reliability and spot possible problem items? Item bank reports to ensure blueprint coverage?

48 How to Evaluate Request a one-on-one demo The Questionmark team will contact you to arrange a demonstration tailored to your needs and questions Request a 30-day trial of Questionmark OnDemand Complimentary technical support during trial Slide 49

49 Questionmark Conference 2019 Assess for Success SAVE THE DATE! February 26th-March 1st Hard Rock Hotel San Diego "My team always walks away with new knowledge - and we have been using the product for over 12 years!" Slide 50

50 Thank you for attending! We hope to see you at a future webinar Keep up to date at blog.questionmark.com! Follow us All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.