Clinical Applicability of Adaptive Speech Testing:

Transcription

1 Clinical Applicability of Adaptive Speech Testing: A comparison of the administration time, accuracy, efficiency and reliability of adaptive speech tests with conventional speech audiometry Greg A. O Beirne 1, Brenna P. Sincock 1, Christina Starfinger 3 1 Department of Communication Disorders 2 Centre of Bioengineering, Department of Mechanical Engineering University of Canterbury, Christchurch, New Zealand Good afternoon. My name is Greg O Beirne. I m from the University of Canterbury in Christchurch in New Zealand. The work I am presenting today was carried out in conjunction with Brenna Sincock and was assisted by Christina Starfinger. In this research we compared adaptive speech tests with what I have called conventional speech audiometry.

2 What is conventional speech audiometry? I should clarify what I mean by that. In the talks we ve heard today, we ve seen the state of the art in speech testing, but that standard of testing is not usually what s done.

3 Standard practice in Australia and New Zealand PTA + 5 PTA + 20 Meaningful consonant-vowelconsonant (CVC) words on CD, delivered through audiometer. Performance/intensity function calculated from three lists of 10 words. Phonemic scoring PTA % SRT at 8 db HL In many countries, the standard, everyday speech test typically consists of three lists of 10 meaningful CVC words in New Zealand, or AB words in Australia, delivered at three intensity levels relative to the pure-tone average, and scored phonemically to produce a performance/intensity, or PI, function.

4 Standard practice in Australia and New Zealand PTA + 5 PTA + 20 Meaningful consonant-vowelconsonant (CVC) words on CD, delivered through audiometer. Performance/intensity function calculated from three lists of 10 words. Phonemic scoring PTA + 35 In fact, it s usually just drawn by eye, so looks more like this. The word lists are usually played from a CD.

5 Basic clinical speech audiometry is essentially unchanged since the 1950s Lilly DJ, Franzen RL. (1968). Reproducing Styli for Speech Audiometry. Journal of Speech and Hearing Research 11: In the 1950s, they were played from vinyl records...

6 Basic clinical speech audiometry is essentially unchanged since the 1950s Lilly DJ, Franzen RL. (1968). Reproducing Styli for Speech Audiometry. Journal of Speech and Hearing Research 11: Now they re played from a CD, but apart from that, the basic method the method of constant stimuli has not changed all that much for the average clinician.

7 Adaptive speech tests More efficient than methods of constant stimuli Saberi & Green, Percept Psychophys 59: ; Watson & Fitzhugh, Percept Psychophys 47: Comparable accuracy to methods of constant stimuli Taylor, Forbes, & Creelman, J Acoust Soc Am 74: ; Kollmeier, Gilkey, & Sieben, J Acoust Soc Am 83: ; He, Dubno, & Mills, J Acoust Soc Am 103: ; Buss, Hall, Grose, & Dev, J Acoust Soc Am 109: Adaptive tests, as you know, change the presentation level based on whether an item was scored correctly or incorrectly, thereby avoiding floor and ceiling effects. Adaptive methods have been shown repeatedly to be i) more efficient, and ii) about as accurate as constant stimuli methods. Their use is extremely widespread in all facets of psychophysical research, so apart from a few notable exceptions, why are they not used more often clinically? There are computers are in almost every clinic, after all.

8 Research questions What are the optimal parameters for the implementation of adaptive procedures in a clinical speech test? Can adaptive procedures improve clinical speech audiometry in terms of: administration time? accuracy? efficiency? reliability? Our main research questions were these: We wanted to find out what the most suitable parameters were for the clinical use of our test (which I ll describe shortly). And most importantly, we wanted to find out whether adaptive procedures could improve clinical speech audiometry in terms of i) administration time, ii) accuracy, iii) efficiency, and iv) reliability.

9 UC MAST Closed-set 4AFC speech test using NU-CHIPs stimuli. Text version Picture version Using the work of Mackie and Dermody as a starting point, we developed a test platform that we called the University of Canterbury Monosyllabic Adaptive Speech Test, or UC MAST for short, and on it implemented a closed-set four-alternative forced choice test using the NU-CHIPS word list. The items can be text or pictures, but the version we used in this study was the text version.

10 UC MAST insert earphones Closed-set 4AFC speech test using NU-CHIPs stimuli. Participant uses touch-screen. Automated scoring and tracking AUDIOMETER TOUCH SCREEN LAPTOP Open-set variant using CVC words. Tester scores correctness of response Automated tracking Score by word (less accurate) Score by phoneme (preferred) The participant hears the word through their headphones, selects their response using a touch screen connected to our laptop, and the software adjusts the stimulus level to maintain their performance at a certain percentage of words correct. We also created an open-set variant of this test using the same CVC words we use in the conventional speech procedure. While the software still controls the stimulus level and adaptively tracks their threshold, the correctness of the response is judged manually by the audiologist, either by word, or preferably by phoneme.

11 UC MAST test results We ve all seen adaptive tracks before, but I never get tired of it. Starting at 40 db HL, we approach threshold fairly quickly. After a certain number of reversals, we can average the midpoints of the excursions and calculate their SRT, which in this case was 5.6 db HL. This track of 50 responses took 2 minutes to obtain.

12 Participants Preliminary Phase: Refinement of the Adaptive Procedures 20 normal hearing adults (10 males, 10 females) One 90-minute session using 6 different procedures (5 adaptive, 1 non-adaptive). Clinical Testing Phase 46 adults (19 males, 27 females) tested using the optimal closed-set and open-set adaptive procedures. Age range from years (mean 65 ± 14 years) Pure-tone average (PTA) range from db (mean 22.9 ± 13.5 db) Audiogram and conventional speech audiometry Pre-test self-report questionnaire to gauge participant s age, visual acuity, degree of literacy, expressive language skills, English proficiency All testing carried out in double-walled sound proof booth at the University of Canterbury Speech and Hearing Clinic We fine tuned our test parameters on 20 normally-hearing adults, and then tested it in the clinic using a sample of 46 adults of varying ages and hearing levels. The participants formed a representative sample of those who would typically attend our Audiology clinic. We measured their audiogram and carried out the conventional speech test I showed you earlier, but also tested them with our adaptive speech tests.

13 Adaptive algorithms tested Open-set task: Simple up/down staircase procedure (Dixon & Mood, J Am Stats Assoc 43: ) Up:down ratio = 1:1 Tracking the 50% level midpoint between incorrect (0%) and perfect (100%) scores Closed-set task: Weighted up/down staircase procedure (Kaernbach, Percept Psychophys 49: ) Up:down ratio = 1.0:0.6 Tracking the 62.5% level midpoint between chance (25%) and perfect (100%) scores 100% 50% 0% level (db) 100% 50% 50% 62.5% 0% level (db) 4 alternatives chance = 25% The open-set task tracked the 50% correct level using a simple up/down staircase procedure. The closed-set task used a weighted up/down staircase procedure to track the 62.5% correct level, which is mid-way between 100% and the 25% score due to chance on a 4-alternative forced choice test.

14 Adaptive algorithms tested Two alternative strategies: i) constant step size throughout ii) larger step size at the beginning Presentation Level (dbhl) Presentation level (db HL) trials Estimated SRT = 3.4 db HL larger steps 57 trials Estimated SRT = 4.8 db HL trial Trial trial Trial We also tested whether it was better to use a constant step size, as advocated by Miguel García-Pérez, or to use larger step sizes at the beginning of the test to approach the threshold more rapidly, as time is of the essence in clinical situations (as it is in this talk!).

15 How were the criteria tested? Test time Timed length of test, including instructions to participant. Time of each response recorded by software to allow post-hoc analysis for different end-point criteria. Accuracy Consistency between SRT and PTA (Brandy, 2002). Efficiency Time/number of trials taken to reach 5 db range around final end point. Reliability Correlation between test and retest scores. We tested the criteria as follows: We recorded the time that it took to administer the test, including the time taken to give the instructions. For the computer-based tests, every response was time-stamped so we could go back afterwards and analyse the timing using different end-point criteria. With regard to the accuracy of the test, there is no gold standard, so we used the proximity of the SRT to the pure-tone average. To measure the efficiency of each adaptive test, we ran them for more trials than we would do under normal clinical circumstances (we stopped after 22 reversals), and then went back and determined post-hoc at which point the test came within a 5 db window around the final estimate for 95% of participants. And finally, to assess the test-retest reliability, we took repeated measurements from 30% of our participants and compared their two scores.

16 Results Our results on these four measures were as follows:

17 Results: Test time No significant difference in test time between conventional speech audiometry and closed-set adaptive tests. Open-set adaptive tests took significantly longer (p < 0.001) Closed-set adaptive Open-set adaptive test time (s) Conventional 50 0 a a a b c Conventional N Adaptive Cl-c Adaptive Cl-l Adaptive Op-l QUEST Q speech Closed-set Closed-set Open-set audiometry constant SS larger SS larger SS We found no significant difference in the time taken to perform the conventional and the closed-set adaptive tests, despite the adaptive tests presenting around 70% more test items. The open-set tests took about a minute longer, due to the fact that the audiologist had to score the numbers of correct phonemes themselves.

18 Results: Accuracy All tests showed high correlation with PTA. Correlations with the pure-tone average: PTA Conventional SRT 50% correct threshold Termination criteria- Proximity to final threshold of: Adaptive Closed-Set Constant SS Larger SS Adaptive Open-Set 62.5% correct threshold 50% correct threshold. within +4/-1 db 54 trials. within +3/-2 db 52 trials Larger SS within ±2.5 db 48 trials Correlations with conventional speech reception threshold: PTA Adaptive Closed-Set Adaptive Open-Set 62.5% correct threshold 50% correct threshold Constant SS Larger SS Larger SS Conventional SRT 50% correct threshold Termination criteria- Proximity to final threshold of:. within +4/-1 db 54 trials. within +3/-2 db 52 trials All correlations significant at p < 0.05 level. within ±2.5 db 48 trials In terms of accuracy, all of the speech tests were very highly correlated with the pure-tone average, and also with the results of the conventional speech test, indicating that all of these tests were sensitive to differences in the PTA.

19 Results: Efficiency (trials) For adaptive staircase tests, larger step sizes were more efficient than constant step sizes throughout. Efficiency in terms of number of trials: trials Number of trials to reach same accuracy level (re: PTA) * Closed-set adaptive Open-set adaptive Constant step size a b b Adaptive Closed-set constant SS Larger step sizes Adaptive Closed-set larger SS Larger step sizes Adaptive Open-set larger SS p < df = 2 * chi-square = Because of the inherent differences between the two types of procedures, we were unable to compare the efficiency of the conventional tests with the adaptive tests directly, but of the staircase tests we used, the closed-set tests with larger initial steps were the most efficient, both in terms of the number of trials

20 Results: Efficiency (time) For adaptive staircase tests, the closed-set test with larger step sizes was most efficient. Efficiency in terms of time taken: Time taken to reach same accuracy level (re: PTA) Closed-set adaptive Open-set adaptive test time (s) Constant step size c c a b b Adaptive Closed-set constant SS * Larger step sizes Adaptive Closed-set larger SS Larger step sizes Adaptive Open-set larger SS p < df = 2 * chi-square = and the time taken to achieve the same level of accuracy with regard to the PTA.

21 Results: Reliability Good test-retest reliability for pure-tone average. Good test-retest reliability for adaptive tests. Poorer test-retest reliability for conventional speech audiometry. Correlations between the first and second threshold estimates of 30% of the participants (n = 6 of 20) on each speech test: Mean difference = 5 db Test Conventional SRT Correlation coefficient Mean difference >2 db Pure-tone average Adaptive closed set constant SS larger SS * * * * Significant at p<0.05level Adaptive open set larger SS * We retested 30% of the participants at least two weeks after their first test, and found that the test-retest reliability was good for the pure-tone average and for the adaptive tests, but poorer for conventional speech audiometry. It is not certain whether this increase in reliability is a feature of the adaptive procedures themselves, or whether it is simply a function of an increase in the number of scorable items. The mean test-retest difference in SRT in the two adaptive trials was less than 2 db, compared to 5 db for the conventional test.

22 Summary: Adaptive versus conventional Administration time No significant difference in administration time between conventional and closed-set adaptive tests (including instructions to participants). Open-set adaptive test took about 30% longer. Greater number of trials presented in adaptive tests. Note: A computer-based method of constant stimuli test with flexible stimulus timing would also have speed advantages. Accuracy No gold standard, however SRT values obtained with the adaptive staircase tests were consistently lower and closer to the PTA than those obtained with conventional speech audiometry. Consistent with Kollmeier et al. (1988), Saberi & Green (1997) and Dubno & Mills (1998). So to sum up There was no significant difference in administration time overall, despite there being a greater number of trials presented in adaptive tests. The open-set adaptive test took about 30% longer. It s important to note that one of the reasons for the shorter time per trial of the adaptive tests is that the next stimulus is delivered as soon as the previous one is answered, rather than having to wait for the CD recording, so a computer-based method of constant stimuli test with similarly flexible timing would also have speed advantages. Consistent with previous reports, the SRT values obtained with the adaptive staircase tests were consistently lower in absolute terms and closer to the PTA than those obtained with conventional speech audiometry.

23 Summary: Adaptive versus conventional Efficiency Unable to evaluate comparison to conventional tests directly (however, see admin time, accuracy & reliability) Of the staircase tests, closed-set & larger initial steps most efficient. Reliability Adaptive staircase tests produced more reliable threshold estimates than conventional speech audiometry. Highlighted lower test-retest reliability of conventional speech audiometry as currently carried out in clinics. Adaptive staircase SRTs showed a higher correlation and less variance, suggesting they are more suited for use in a clinical Audiology setting. The closed-set test with larger initial steps was the most efficient, both in terms of the number of trials, and the time taken. Most importantly, the adaptive staircase tests produced threshold estimates that were much more reliable that those from conventional speech audiometry. They showed a higher correlation and less variance, suggesting they are more suited for use in a clinical Audiology setting than the conventional method.

24 Slope estimates UC MAST also fits a psychometric function to the adaptive track data. However, slope estimates much better when two points (the pair of compromise ) tracked rather than just SRT point. (Brand and Kollmeier, 2002) proportion correct proportion correct Participant 1 Weibull fit stimulus level (db HL) Participant 4 Weibull fit stimulus level (db HL) proportion correct proportion correct Participant 2 Weibull fit stimulus level (db HL) Participant 5 Weibull fit stimulus level (db HL) proportion correct proportion correct Participant 3 Weibull fit stimulus level (db HL) Participant 6 Weibull fit stimulus level (db HL) Of course, so far I have concentrated on the speech-reception threshold, but the slope of the psychometric function is also very important, as is the participants speech performance at supra-threshold levels. The software also does a fit of a psychometric function to the adaptive track data and obtains the slope that way, either using nonlinear regression as shown here, or using the QUEST maximum likelihood estimation algorithm. However, as Brand and Kollmeier showed, to obtain a concurrent estimate of the slope of the psychometric function it is optimal to track two points on either side of the SRT the so-called pair of compromise, rather than just aim for the SRT itself. This, of course, would take slightly longer, but what s another two minutes? As you can see, some of these discrimination functions are rather shallow, but

25 Next Sentence-based variant in progress See also our poster (P109) on adaptive LPF tests it is trivial to modify the software to use sentence stimuli, and we plan that for a future version. I d also like to direct you to our poster which is on a related topic.

26 Thank you! We would like to thank: Mr Andrew McGaffin & Dr Emily Lin University of Canterbury Dr Sarah F. Poissant University of Massachusetts and our participants Thank you very much!