Construct(s) measured in face-to-face and video-conferencing delivered speaking tests

Transcription

1 Construct(s) measured in face-to-face and video-conferencing delivered speaking tests Fumiyo Nakatsuhara, Chihiro Inoue (CRELLA, University of Bedfordshire) Vivien Berry (British Council) Evelina Galaczi (Cambridge English Language Assessment)

2 RESEARCH BACKGROUND

3 Background Different speaking test delivery modes f2f mode: Cambridge English General English tests, IELTS Computer-delivered mode (administered by computer only): Aptis, PTE, TOEFL ibt Both modes: BULATS What about a computer-delivered mode administered by an examiner on the other side of the screen, to combine advantages of both these modes? Video-conferencing (VC) delivery mode: Exploratory study by the US Defense Language Institute Foreign Language Center (Clark & Hooshmand, 1992, 1992) Operational VC test: Computer-based Standard Speaking Test (ALC, 1999 but no longer available) More recent research by Craig and Kim (2010) and Kim and Craig (2012) 3

4 Background (cont.) Rapid advances in online video communication technology: e.g. Skype, Facetime, Zoom Zoom: online video conferencing program ( which offers high definition video conferencing and desktop sharing 4

5 Rationale Ensuring accessibility and fairness in remote areas where frequent delivery of f2f tests is not possible Making use of recent advances in technology to offer computer-delivered tests without losing interactional features / responding to the broadening construct of speaking (increasing use of online video communication) 5

6 3 Phases of the Project Phase 1 (2014): 32 test-takers, 4 examiners in London To explore similarities and differences in 2 modes in terms of scores, linguistic output, examiner behaviour and test-takers and examiners perceptions Phase 2 (2015): 99 test-takers, 10 examiners in Shanghai To confirm the Phase 1 results, after providing examiner/test-taker training. Focus on more thorough statistical analysis of scores, effects of training, the impact of sound quality Phase 3 (in progress): c.100 test-takers, 10 examiners in Bogota, Buenos Aires, Caracas and Mexico City To confirm the Phase 1 and 2 results, after developing a bespoke platform 6

7 Research Questions (Phase 1) Comparing the standard f2f and VC mode: RQ1: Are there any statistically significant differences in test-takers scores? RQ2: Are there any differences in linguistic output, specifically types of language function, elicited from test-takers? RQ3: Are there any differences in test-takers perceptions of their test-taking experience? RQ4: Are there any perceived differences in examiners test administration behaviour? RQ5: Are there any perceived differences in examiners rating behaviour? 7

8 RESEARCH DESIGN - mixed methods approach -

9 Test-takers: 32 test-takers (CEFR B1 C1) taking both f2f and VC-delivered tests Examiners: 4 trained examiners (Examiners A D) Test structure: Data Collection Part 1 Intro. and interview 4-5 mins. 2 test versions Part 2 Individual long turn 3-4 mins. - Travelling - Success Part 3 Two-way discussion 4-5 mins. Counter-balanced: delivery mode x test version Rating criteria: 1) Fluency and coherence; 2) Lexical resource; 3) Grammatical range and accuracy; 4) Pronunciation 9

10 Gathered Data f2f Room a) Video-recorded test performances b) Scores (by Examiner) on both modes & written comments c) Test-takers responses to feedback questionnaire & interview d) Examiners responses to feedback questionnaire e) Examiners verbal reports on selected videos f) Researchers observations (field notes) VC: Candidate Room VC: Examiner Room 10

11 Language functions using O Sullivan et al. s (2002) Observation Checklist (RQ2) Data Analysis Score comparison (RQ1) Test-takers perceptions of the 2 delivery modes (RQ3) a) Video-recorded test performance b) Scores (by Examiner) on both modes & written comments c) Candidate s responses to feedback questionnaire & interview d) Examiner s responses to feedback questionnaire e) Examiner s verbal report on selected videos f) Researchers observation (field notes) Examiners test administration behaviour (as interlocutor) (RQ4) Examiners rating behaviour under the 2 conditions (RQ5) Examiners perceptions of examining under the 2 conditions (RQ4/5) 11

12 RESULTS

13 Test Scores (RQ1) - Wilcoxon Signed Rank Test - Category Test mode Median Mean (SD) Z Sig Fluency f2f (1.04) VC (1.16) Lexis f2f (1.05) VC (1.14) Grammar f2f (1.01) VC (1.10) Pronunciation f2f (.78) VC (.88) Overall (mean) f2f (.83) Overall (rounded) VC (.98) f2f (.88) VC (.991)

14 FACETS Variable map Q1. No sig. difference in scores between 2 modes 14

15 Language Functions (RQ2) 5 functions that test-takers used sig. differently under the 2 conditions. Asking for clarification (Parts 1 & 3) VC > f2f Elaborating (Part 3) VC > f2f Comparing (Part 3) f2f > VC Suggesting (Part 3) f2f > VC Modifying (Part 3) f2f > VC No sig. differences in Part 2 Q2. Some differences in linguistic output 15

16 Test-takers Perceptions of 2 Modes (RQ3) Understand the examiner? Taking the test was (1. V difficult 5. V easy) Test mode Mean (SD) t (df = 31) Sig f2f 4.72 (.46) VC 3.72 (.77) f2f 3.84 (.85) VC 3.13 (.83) f2f VC No diff Which test made you more nervous? 28.1% 46.9% 25.0% Which test was more difficult? 12.5% 65.6% 21.9% Which test gave you more opportunity to speak English? 50.0% 18.8% 31.3% Which test did you prefer? 84.4% 9.4% 6.3% Q3. More positive perception towards f2f 16

17 Selected comments from test-takers Having good eye contact and seeing the examiner s facial expressions and postures in the f2f mode helped me to understand him better. (S06) I felt that I was able to speak more f2f, because I was able to understand when to speak and chip in. (S17) I m shy and I always become nervous when I speak to a person f2f. Instead, the computer test made me more relaxed. (S07) I preferred the f2f mode. The examiner gave me lively encouragement. (S10) I felt the computer-delivery mode gave me more opportunities to speak. (S27) On the computer, it doesn t matter what I say as the interviewer is not in front of me. I felt freer and was able to speak more. Perhaps I made more mistakes, though. (S05) The both modes were the same. It s still speaking to someone. (S20) 17

18 Examiners Test Administration/Rating Behaviour (RQ4/5) Role and frequency of examiner response tokens Rate and articulation of examiner speech, effect of examiner intonation Use of gestures by examiners Issues related to turn-taking management Requests for clarification, Need to control the direction of the interview Examiner D With VC, I couldn t really hear S20 s word endings or other microphonologic features. I didn t want to give the benefit of the doubt, so I gave lower scores for Pronunciation and Grammar. I wasn t sure whether I needed extra efforts due to S07 s pronunciation or due to the technology. Also, visual information is very important to rate Pronunciation, but due to an unclear view of the candidate and her picture and sound not exactly synchronised, I felt that I perhaps overrated S07 s pronunciation under the VC condition. Examiner B Q4/5. Some differences in examiner behaviour

19 Examiners Perceptions of 2 modes (RQ4/5) Comparisons of 2 modes Which mode of speaking test did you feel more comfortable with? Which mode of speaking test did you feel was easier for you to administer? Which mode of speaking test did you feel was easier for you to rate? Which mode of speaking test do you think gave a better chance for the testtaker to demonstrate their level of English language proficiency? f2f VC NO DIFFERENCE Examiner comments suggest that exposure to the VC delivery mode encourages familiarity and confidence with that mode. Which speaking test did you prefer?

20 CONCLUSIONS

21 Main Findings and Phase 2 Research Essentially the same speaking test construct across the 2 modes The 2 modes were generally comparable in terms of scores Some differences in elicited language functions and examiner behaviour Need for appropriate examiner and test-taker training for the VC mode. Phase 2 (2015) with 99 test-takers and 10 examiners in Shanghai: - Training for the VC delivery tests - More thorough statistical analysis on scores - Effects of training (on test-taker output, examiner behaviour, testtaker and examiner perception) - Sound quality perceptions 21

22 Some of Phase 2 Findings Score: No sig. difference between 2 modes Examiner & test-taker training: very positively received For Linguistic more details analysis: on Phase more 2, negotiation please come of meaning and see our on presentation VC mode at LTRC 2016 in Palermo! Observers notes: good examiner techniques to bring test-takers back to the topic on VC Examiner/test-taker feedback questionnaires: more positive responses on VC than Phase 1 Examiner focus group: Test-takers comfort level higher on VC; Amazed at the excellent sound quality Perceived sound quality: Examiners > Observers > Test-takers (but no difference between stronger and weaker test-takers) 22

23 Thank you! This research was funded and supported by the IELTS Partners: British Council, Cambridge English Language Assessment and IDP (IELTS) Australia.