APPLICATION OF FUZZY SIGNAL DETECTION THEORY TO VIGILANCE: THE EFFECT OF CRITERION SHIFTS

Transcription

1 i.-, 1678 APPLICATION OF FUZZY SIGNAL DETECTION THEORY TO VIGILANCE: THE EFFECT OF CRITERION SHIFTS Shawn C. Stafford, James L. Szalma, 2 Peter A. Hancock,,* & Mustapha Mouloua I Department of Psychology Institute for Simulation and Training University of Central Florida 2 A recent advance on Signal Detection Theory (SDT) promises to enhance measurement of performance in complex real world domains. This development, Fuzzy Signal Detection Theory (FSDT), combines traditional SDT with Fuzzy Set Theory to extend signal detection analysis beyond the traditional crisp, categorical model. FSDT permits events to simultaneously be in more than one state category (e.g. signal and non-signal), so that the stimulus and response dimensions can be continuous rather than categorical. Consequently, FSDT can be employed in settings where the degree to which an event is a signal for detection may vary. This study is an initial test of application of FSDT to vigilance, a domain in which SDT has been widely applied. Results indicate that manipulations of stimulus probability impacts response bias in a fuzzy vigilance task, but that these effects differ somewhat from tasks employing traditional signal detection. INTRODUCTION Signal detection theory (SDT) has been extensively applied to vigilance, and research has indicated that perceptual sensitivity declines over the course of a vigil (See, Howe, Warm, & Dember, 1995). In addition, response criterion shifts to more conservative levels with time on watch (Davies & Parasuraman, 1982; See, Warm, Dember, & Howe, 1997). Note, however, that the traditional SDT model assumes that inspected events fall into discrete mutually exclusive categories (i.e. signal or non-signal). A recent advance in SDT promises to enhance measurement of performance in domains where this form of categorization is either not desirable or simply not possible. This development, Fuzzy Signal Detection Theory (FSDT; Hancock, Masalonis, & Parasuraman, 2000; Parasuraman, Masalonis, & Hancock, 2000), combines traditional SDT with Fuzzy Set Theory to extend signal detection analysis beyond the traditional crisp, categorical decision-making model. In contrast to the mutually exclusive categories of signal and non-signal in traditional SDT, FSDT allows for events to simultaneously be in more than one state category (e.g. signal and non-signal), so that the stimulus and response dimensions can be continuous. FSDT allows stimuli to vary in signalness from 0-1 by deriving mapping functions between the stimulus dimension and the state of the world (characteristics of the real-world objects that are relevant dimensions for target detection). This process is referred to as fuzzification (see Parasuraman et al., 2000). This permits the quantification of uncertainty inherent in the stimulus and response dimensions through the calculation of fuzzy hit and false alarm rates, from which standard signal detection theory measures can be computed. Consequently, FSDT can be employed in applied settings where the degree to which an event is a signal for detection may vary. For instance, the appearance of an oblong object in airline baggage can vary in the degree to which it represents a threat. Traditional SDT would force categorization of the object into either signal or non-signal, thereby restricting the possible range of response. In contrast, FSDT captures the fuzziness inherent in the stimulus itself and thus uses uncertainly as a powerfbl source of information. The current study applied this new methodology to the vigilance problem as part of a general effort to understand the factors that control the performance, workload, and stress of tasks requiring sustained attention. Criterion Setting in Vigilance The FSDT methodology provides a new approach to assessing vigilance performance that incorporates aspects of the task not captured in traditional SDT. For instance, in vigilance the response criterion is especially likely to be high when signal probability is low. However, when signal probability shifts from high to low during a vigil, observers maintain a level of bias that

2 1679 is more lenient than a control group that experiences a constant signal probability (see Parsons et al., 2000; Vickers & Leary, 1983). These effects occur without changes in sensitivity. It may be that application of Fuzzy Signal Detection Theory may result in performance changes that are different from those obtained using the traditional SDT model, since FSDT better captures the uncertainty inherent in the stimulus. Therefore, the current study investigated the role of signal probability in controlling the criterion increment in the context of fuzzified stimuli. This was accomplished by varying the frequency distribution of stimulus uncertainty and evaluating the effect of changes in that distribution on the response criterion observers adopt (see Figures 1 and 2). METHOD ci rc I I Degree of Signalness (s) Figure 1. Relative frequency ofpresentation as a function of signalness (High-signal probability conditions) I J Sixteen male and sixteen female undergraduate at the University of Central Florida participated in the experiment. Eight students were randomly assigned to one of four groups defined by the nature of the transition in stimulus probability. 1) Low-Low (LL): participants experienced the distribution of stimuli displayed in Figure 1, in which more non-signal-like stimuli were presented than signal-like stimuli; 2) High-High (HH): participants experienced the distribution of stimuli displayed in Figure 2, in which more signal-like stimuli were presented than non-signal-like stimuli; 3) High-Low (HL): Participants experienced the high probability distribution in session one and the low probability distribution in session 2. 4) Low-High (LH): Participants experienced the low-probability distribution in session 1 and the high probability session in session two. Note that participants were not informed of the probability distribution they would receive in either session. Prior to the first session participants engaged in a 5-minute practice vigil. The practice session was followed immediately by a two separate 20 minute vigils, each divided into four continuous five-minute periods of watch. A brief interval separated the two vigils. The stimuli employed in this experiment consisted of two vertically oriented 1.64 cm by cm rectangular bars separated by 1.26 cm. Each rectangle was made out of a pattern of 20,828 squares each measuring.04 cm x.04 cm. Seven different images were presented during the experiment, although participants were informed that the stimuli would vary between two extremes in color !?! 0.2 L A 0.15 Q) degree of signalness (s) Figure 2. Relative frequency of presentation as a function of Signalness (Low-signal probability conditions) Stimuli were presented on a Dell 1701FP monitor through an NVIDIA G-Force 3 TI-500 video card. In each image one of the two rectangles always remained a checkered pattern of squares consisting of the following two colors in equal proportions; violet squares with RGB value; Red 204, Green 0, Blue 255 and Grey squares with RGB value; Red 127, Green 127, Blue 127. The 2nd vertical rectangle consisted of checkered pattern of violet squares with RGB value; Red 204, Green 0, Blue 255, and also one of seven saturated versions of yellow squares. Using a variation of a psychophysical function developed by Indow and Stevens (1966) yellow (RGB value; Red 255, Green 255, Blue 0) was varied in its saturation level in images one thru seven for the second vertical rectangle only. The event rate was 21 events per minute, with a stimulus duration of 200msec. Participants were 1 1

3 1680 instructed to respond to stimuli by rating the degree to which each stimulus was a signal (the rectangles differed in saturation) versus a non-signal (the two bars did not differ) by pressing keys 1 through 7, with the response 7 indicating that the stimulus was definitely a signal. No other information regarding the number of possible stimulus categories or the relative frequencies of those categories was provided to the observers. those of observers in the other conditions. Thus, observers who experienced the low-probability conditions in the second session (i.e., LL and HL) were significantly more lenient than observers who experienced a high-probability distribution in both sessions. RESULTS Responses by participants were used to compute fuzzy signal detection hit and false alarm rates using formulas obtained from Parasuraman et al. (2000), which were used to compute sensitivity (d ) and response bias (c) using standard formulas (see MacMillan & Creelman, 1991). ANOVA s revealed that perceptual sensitivity did not vary as a function of stimulus probability or time on task in either session (pb.05 in each case). Mean response bias scores are plotted as a function of periods of watch for each session in Figure 3. An ANOVA of response bias (the index c) revealed a significant effect for probability condition in the first session, F (3,28)=19.1, pc.01. Post-hoc tests using the Bonferroni correction (alpha=.05) indicated that the two conditions receiving the high signal probability distribution (HH and HL) did not differ from one another in response bias in the first session, but that observers in both conditions were significantly more conservative in responding compared with observers in the two low signal probability conditions (LL and LH). Note that the typical signal probability finding in vigilance, that increased signal probability decreases conservatism in responding (see Warm & Jerison, 1984; but see also Parsons et al., 2000), was not observed in the context of a fuzzy signal detection task. Instead, increased probability of signal-like stimuli increased conservatism, while decreased probability of signallike stimuli induced more lenient responding. An ANOVA of response bias in the second session indicated a significant effect for probability condition, F(3,28)=3.08, p<.05. Post-hoc tests using the Bonferroni correction indicated that observers who shifted from high to low (HL) were significantly more lenient in responding during the second session compared to those who remained on high schedule in both sessions (HH), and that observers who experienced the low probability distribution in both sessions (LL) differed significantly from those who experienced the HH condition. The bias scores of individuals who shifted from the low to the high probability distribution (LH) did not differ from Session 1 Session 2 Periods of Watch Figure 3. Response bias as a function of 5-min periods of watch in each session. DISCUSSION The current study was designed to evaluate the fuzzy signal detection theory model by investigating whether response bias effects observed in vigilance using traditional tasks and analyses would also be found using an FSDT approach. No differences in sensitivity were observed across conditions, consistent with prior findings by Parsons et al. (2000). However, in this experiment the vigilance decrement was not observed. This may be due to the moderate event rate (21 events per min) and the brief interval between the two twenty minute sessions. In general, however, results indicated that the fuzzy detection model was sensitive to changes in response bias due to shifting stimulus probabilities. Observers in the HL condition showed the expected shift in response bias toward more lenient responding, consistent with the findings reported by Parsons et al. (2000; but see also Vickers & Leary, 1983). However, contrary to the findings of Parsons and her colleagues, observers in this study who experienced increasing signal probability (LH) did not significantly rise in conservatism relative to those in the LL condition. Moreover, Parsons et al. (2000) observed effects for time on task, while no such effects were observed herein. Specifically, Parsons and

4 1681 her colleagues observed that in the decreasing condition (analogous to the HL in this experiment), responding remained at a constant, relatively unbiased level throughout the vigil, while in the increasing condition observers became more conservative over time. The differences between the findings of the current study and those of Parsons and her colleagues (2000) may be a result of the incorporation of uncertainty regarding target and non-target classification into the stimulus dimension in the FSDT model, accomplished by fuzzifying the stimulus dimension (see Parasuraman et al., 2000). The resulting demands such uncertainty places upon operators, who in this experiment were compelled to rate the degree of signalness rather than make discrete yesho responses, induces bias effects different from those in more traditional experiments requiring yesho decisions. Whether these bias effects would be observed with a fuzzy stimulus dimension but with a binary (yesho) decision axis is a matter for future experimental work. In addition to the differences noted above, there were also differences between studies in how probabilities were shifted. In the experiments by Parsons et al. (2000) and Vickers and Leary (1983) signal probability was shifted within a session, while in this study probabilities were shifted between sessions. This was necessary due to the fuzzification of the stimulus dimension. That is, it was not possible to define a discrete signal probability in a fuzzy task, since five of the seven stimuli were defined as having various degrees of signalness and non-signalness. This difference in task characteristics may also have contributed to the differential findings between this study and that of Parsons et al. (2000). The current results reveal, however, that the explanation for the increased leniency resulting from a shift from high to low probability(c0ntrary to the ideal observer hypothesis; see Green & Swets, 1966), may need to be reconsidered. As discussed by Parsons and her colleagues (2000; see also Parsons, 2001; Vickers & Leary, 1983), the increased leniency may be attributed to self-doubt by the observers regarding the accuracy of their responses. According to this view, rather than adjust their responding to the distribution of signals, observers attribute their change in response rate during a session to their own limitations in sensitivity and adjust their responding accordingly to compensate. However, in the present experiment, probabilities associated with each stimulus remained constant and were shifted between sessions. Hence, the leniency in responding to stimuli that are primarily non-signal like may be due to processes other than self-doubt in response to shifting signal probabilities. The mechanism for this effect is a matter for future investigation. Note that regardless of whether an observer experienced a change in probability distribution, response bias was more lenient when the distribution of stimuli contained a larger proportion of stimuli toward the non-signal end of the continuum. That is, observers shifted their criteria to match the probability distribution they were currently experiencing, but only in the groups with the low signal probability distribution in the second session (although there was a non-significant trend for the LH group to increase in conservatism in the second session to match their cohorts in the HH group). The observation that a greater proportion of signal-like stimuli induce conservatism and a smaller proportion of signal-like stimuli induces leniency is inconsistent with the traditional finding in vigilance that conservatism varies inversely with signal probability. These results indicate that the conservatism in responding in a vigilance task may differ depending on whether the task is fuzzy or discrete in nature. Alternatively, it may be that criterion setting in target detection is fundamentally different in fuzzy detection tasks compared to traditional signalhoke tasks. Current empirical work suggests that fuzzy and traditional signal detection models may capture different aspects of sensitivity and response bias (Murphy, Szalma, & Hancock, 2003). The current study has implications for real-world target detection, since in many applications the discrete categorization of events as critical events and noncritical events is not possible. For instance, in both law enforcement and combat environments, it is not unusual for signals to have characteristics of signal and nonsignal, which has led to the development of guidelines for levels of threat and levels of force. In addition, it is not unusual for personnel in these environments to experience prolonged periods of low signal appearance, but to rapidly transition to periods of high signal probability (c.f. Warm, 1993). The results of this experiment suggest systems be designed to accommodate the shifts in response bias that occur with such transitions. ACKNOWLEDGEMENTS This work was supported by the Department of Defense Multidisciplinary University Research Initiative (MURI) program, P.A. Hancock, Principal Investigator, administered by the Army Research Office under grant DAAD The views expressed in this work are those of the authors and do not necessarily reflect official Army policy. The authors wish to thank Dr.

5 1682 Sherry Tove, Dr. Elmar Schmeisser, and Dr. Mike Drillings for providing administrative and technical direction for the Grant. REFERENCES Green, D.M. Swets, J.A. (1966). Signal detection theory and psychophysics. New York: Wiley. Hancock, P.A., Masalonis, A.J., & Parasuraman. (2000). On the theory of fuzzy signal detection: Theoretical and practical considerations. Theoretical Issues in Ergonomic Science, 1, (3), Indow, T. & Stevens, S.S. (1966). Scaling of saturation and hue. Perception h Psychophysics, 1, Macmillan, N.A., & Creelman, C.D. (1991). Detection theory: A user s guide. New York: Cambridge University Press. Murphy, L.L., Szalma, J.L., & Hancock, P.A. (2003). Comparison of fuzzy signal detection and traditional signal detection theory approaches to performance measurement. Proceedings of the Human Factors and Ergonomics Society, 41, in press. Parasuraman, R., Masalonis, A.J., & Hancock, P.A. (2000). Fuzzy signal detection theory: Basic postulates and formulas for analyzing human and machine performance. Human Factors, 42, Parsons, K.S. (2001). Changes in signalprobability and response bias in vigilance. Unpublished Master s Thesis, University of Cincinnati, Cincinnati, OH. Parsons, K.S., Warm, J.S., Matthews, G., Dember, W.N., Galinsky, T.L., & Hitchcock, E.M. (2000, November). Changes in signal probability and vigilance performance. Poster presented at the meeting of the Psychonomic Society, New Orleans, LA. See, J.E., Howe, S. R., Warm, J.S. &Dember, W.N. (1995). A meta-analysis of the sensitivity decrement in vigilance. Psychological Bulletin, 117, See, J.E., Warm, J.S., Dember, W.N., & Howe, S.R. (1997). Vigilance and signal detection theory: An empirical evaluation of five measures of response bias. Human Factors, 39, Vickers, D., & Leary, J.N. (1983). Criterion control in signal detection. Human Factors, 25, Warm, J.S. (1993). Vigilance and target detection. In B.M. Huey, & C.D. Wickens, (Eds.), Workload transition: Implications for individual and team performance (pp ). Washington, DC: National Academy press. Warm, J.S. & Jerison, H.J. (1984). The psychophysics of vigilance. In J.S. Warm (Ed.), Sustained attention in human performance (pp ). Chichester, UK: Wiley.