An accessible delivery mode for face-to-face speaking tests: enhancing mobility of professionals/students

Transcription

1 An accessible delivery mode for face-to-face speaking tests: enhancing mobility of professionals/students Vivien Berry (British Council) Fumiyo Nakatsuhara, Chihiro Inoue (CRELLA, University of Bedfordshire) Evelina Galaczi (Cambridge English Language Assessment) 14 th EALTA conference, Sèvres, France, 1-3 June 2017

2 RESEARCH BACKGROUND

3 Rationale for the Project Ensuring accessibility and fairness in remote or politically sensitive areas where frequent delivery of face-to-face tests is not possible. Making use of recent advances in technology to offer computer-delivered tests without losing interactional features / responding to the broadening construct of speaking (increasing use of online video communication). 3

4 Background Different speaking test delivery modes f2f mode: Capable of eliciting reciprocal interaction Computer-delivered mode (administered by computer only): Practical How can we combine the advantages of both f2f and computer-delivered modes? Video-conferencing (VC) delivery mode: Exploratory study by the US Defense Language Institute Foreign Language Center (Clark & Hooshmand, 1992, 1992) Craig and Kim (2010) and Kim and Craig (2012) Davis, Timpe-Laughlin, Gu and Ockey (forthcoming) 4

5 3 Phases of the Project Phase 1 (2014): 32 test-takers, 4 examiners in London To explore similarities and differences in 2 modes in terms of scores, linguistic output, examiner behaviour and test-takers and examiners perceptions Phase 2 (2015): 99 test-takers, 10 examiners in Shanghai To confirm the Phase 1 results, after providing examiner/test-taker training. Focus on more thorough statistical analysis of scores, effects of training, the impact of sound quality Phase 3 (2016): 89 test-takers, 8 examiners in Bogotá, Buenos Aires, Caracas, Medellín & Mexico City To confirm the validity of scores awarded on VC tests, after developing a bespoke platform. Focus on confirming further effects of training, and perceptions of on-screen prompts and effect of sound quality on scores 5

6 Main findings/suggestions from Phases 1&2 Mode of delivery (f2f vs VC) has no significant impact on the scores achieved by test-takers. Comparable language functions elicited from testtakers except for asking for clarification. Mixed, but generally positive perceptions of the VC test by examiners and test-takers. We need: A bespoke technological solution to deliver the specific VC test Further training of examiners and test-takers with the new delivery platform 6

7 Research Questions (Phase 3) RQ1: How well is the scoring validity of the videoconferencing tests supported by the four-facet MFRM (i.e. test-taker, rater, test version and rating category)? RQ2: To what extent did sound quality affect performance on the video-conferencing test (as perceived by examiners, as perceived by test-takers, as observed in test scores)? RQ3: How did the test-takers perceive the VC test, the new platform and training for the VC test? RQ4: How did the examiners perceive the VC test, the new platform and training for the VC test? 7

8 RESEARCH DESIGN mixed-methods approach

9 Data Collection 25 City 4 Examiners@Bogotá 20 test-takers@medellín 44 test-takers@caracas 89 Test-takers & 8 Examiners (ID: K-R) 4 examiners@buenos Aires 9

10 Participants familiarity with the internet & VC How often do you use? Almost everyday Almost everyday 1-2/ week /week Examiners 1.63 Test-takers Never 1-2/month 1.63 Q1 THE INTERNET SOCIALLY TO GET IN TOUCH WITH PEOPLE? Q2 THE INTERNET TO TEACH/FOR YOUR STUDIES? 1=Never, 3=Once or twice a week, 5=Everyday Q3 VIDEO- CONFERENCING SOCIALLY TO COMMUNICATE WITH PEOPLE? Q4 VIDEO- CONFERENCING TO TEACH/FOR YOUR STUDIES? 10

11 Test structure: IELTS Speaking Test Part 1 Intro. and interview 4-5 mins. 5 test versions Part 2 Individual long turn 3-4 mins. (randomly selected) Part 3 Two-way discussion 4-5 mins. Rating criteria: 1) Fluency and coherence; 2) Lexical resource; 3) Grammatical range and accuracy; 4) Pronunciation Double rating: All test sessions double-rated with videorecorded performances; Sufficient connectivity achieved through an examiner rating matrix 11

12 Outline of Training for VC New VC Examiner training package: What does not change and what does btw f2f & VC VC-specific interview techniques (e.g. active listening) Rating under VC (rating practice with 3 video-recorded tests) Peer practice (taking turns to do examiner, candidate and observer) Feedback on group practice sessions and discussion Candidate guidelines: Bilingual information sheet with clear instructions, what to expect 12

13 A bespoke platform Part 2 prompt card with examiner face Examiner control panel 13

14 Data Analysis: Mixed-methods approach Test score analysis (RQ1) Analysis of sound quality perception (RQ2) a) Test scores awarded by the examiners b) Test-takers responses to a feedback questionnaire & interview c) Examiners responses to a feedback questionnaire d) Examiners focus group discussions Analysis of test-takers (RQ3) and examiners (RQ4) perceptions of the test, new platform and training for the VC test 14

15 RESULTS

16 Test Scores (RQ1) FACETS map Satisfactory fit statistics for all facets Comparable test versions Most difficult - Grammar; Easiest - Fluency Differences in severity among raters (Fair ave. K=5.74, Q=6.50 Difference of 0.76 of a band) 16

17 Sound Quality Perceptions (RQ2) The same 2 questions to Examiners and Test-takers Q1. The quality of the sound in the VC test [1. Not clear at all - 5. Very clear] Examiners (M=3.98 Clear) > Test-takers (M=3.86 Clear) Q2. The effect of sound quality on test-takers performance [1. No - 5. Very much] Test-takers (M=1.87 Not much) > Examiners (M=1.74 Not much) Small but sig. negative correlation b/w test-takers perception of the impact of sound quality & their prof. level (r=-0.283, p=0.009) Sig. difference in ANOVA (F=3.19, p=0.046) Low (Band 5.5 ) M=2.33 > Mid (Bands 6&6.5) M=1.69 > High (Band 7& ) M=1.58 [Generally good sound but some problems reported in 70 of the 89 tests] Slight 2 sec delay but didn't interfere with process (Examiner Q), Some delay in sound which caused overlapping (Examiner K), Bleeps during the intro part, the screen went green for a second (Examiner O), The screen froze for 20 seconds (Examiner M). 17

18 Test-taker Feedback (RQ3) Q1 CANDIDATE GUIDELINES WERE (1.NOT USEFUL - 5.V USEFUL) Q2 PICTURES IN THE GUIDELINES WERE (1.NOT HELPFUL - 5.V HELPFUL) Q3 UNDERSTOOD THE EXAMINER? (1.NEVER - 5.ALWAYS) Useful Helpful Almost always Q4 VC TEST MADE ME FEEL (1.V NERVOUS - 5.V COMFORTABLE) OK - Comfortable 3.46 Q5 VC TEST WAS (1.V DIFFICULT - 5.V EASY) Q6 ENOUGH OPPORTUNITY TO DEMONSTRATE SPEAKING ABILITY? (1.NOT AT ALL - 5.V MUCH) Q7 PART 2 PROMPT ON THE SCREEN WAS (1.NOT CLEAR AT ALL - 5.V CLEAR) Easy Much Clear

19 Selected test-taker comments It was a good experience, modern and useful (C002). I d like to do this frequently (C049). It was an excellent opportunity to be evaluated by the expert examiner, as the economical situation in my country does not allow us to have native speaker English teachers (C090). The guidelines had a lot of information, and I didn t read some parts properly (C007). In Part 2, it would be good to have a timer to manage my speech (C009). In part 2, a bigger prompt on the screen would be better (C068). It would be better with headphones (C086). There were three moments where the transmission froze (C094). I really like the idea but you need to take into consideration the problems with the Internet connection in Venezuela. Very good sound quality must be guaranteed (C091). 19

20 Examiner Training (RQ4) The VC examiner training [1. strongly disagree 5. strongly agree] Mean (SD) N=8 adequately prepared me for administering the test 4.63 (0.74) gave me confidence in handling the interlocutor frame 4.50 (1.41) adequately prepared me for rating candidate performance 4.75 (0.46) gave me confidence in the accuracy of my ratings 4.13 (0.99) I found it a little bit difficult to rate the pronunciation, although the sound was clear. (Examiner P) 20

21 Examiner Training per Part (RQ4) I found it straightforward to administer [1. strongly disagree 5. strongly agree] Mean (SD) N=8 Part 1 (Interview) 4.50 (0.52) Part 2 (Long turn) 4.25 (0.71) Part 3 (Discussion) 4.50 (0.53) I found it easy to handle task prompts on the screen in Part (0.74) You might have to delay two seconds to get the card up (Examiner N) We have to pause while the invigilator hands over the paper and pencil so that eats into four minutes (Examiner L) 21

22 Comparison of two modes (RQ4) F2F VC No difference Missing DO YOU PREFER? 37.5% 50.0% 12.5% GAVE A BETTER CHANCE TO DEMONSTRATE 12.5% 62.5% 25.0% EASIER TO RATE 25.0% 75.0% EASIER TO ADMINISTER 37.5% 12.5% 37.5% 12.5% FEEL MORE COMFORTABLE 50.0% 37.5% 12.5% 22

23 Suggestions from focus groups (RQ4) Slight change to the Interlocutor Frame for example, at the end of Part 2 where it says please give back the paper and pencil, I would add a now which I said it s just little things that make the whole thing a bit smoother (Examiner L). This has so many possibilities of failing, poor internet connection, whatever, might be an area has a power cut, so I think the administrators need to have a plan B of what they re going to do with a candidate who couldn t be tested on that day for whatever reason (Examiner N). Maybe the tests can be carried out early in the day so that they have another chance later in the day (Examiner L). The importance of the invigilators and the IT support in not only in dealing with delays but the whole process (Examiner O). 23

24 CONCLUSIONS

25 Summary: Central & South America Study Participants: Use of the Internet almost everyday and Video-conferencing 1-2 a week ( computer literate) Scores: No systematic inconsistency, but more training needed to standardise rater severity Sound quality: Generally clear, but some (minor to major) problems reported in almost 80% of the test sessions; Lower-scoring test-takers felt that their performance was more affected by the sound quality Perceptions: Both test-takers and examiners were positive about the VC test, new platform and training, AS LONG AS INTERNET CONNECTIVITY IS ADEQUATE! 25

26 Implications from Phase 1-3 studies Scores: The comparability of test scores in the two delivery modes is established. Test construct and Interactiveness: The VC test may NOT offer the same level of subtlety as in the f2f test. Its communicativeness and interactiveness seem to be operationalised in the form of more explicit negotiation of meaning. ( In line with changes in the speaking construct in real-life communication) Constructive feedback and suggestions provided by test-takers & examiners To be thoroughly considered and fully addressed. 26

27 Thank you! This research was funded and supported by the IELTS Partners: British Council, Cambridge English Language Assessment and IDP (IELTS), Australia. Nakatsuhara, F., Inoue, C., Berry, V. and Galaczi, E. (2017) Exploring the use of videoconferencing technology in the assessment of spoken language: A mixed-methods study, Language Assessment Quarterly, 14 (1): Phase 1 Research Report (2015), Phase 2 Research Report (forthcoming), available online, IELTS Partnership Research Papers,