Signal detectability in visual nonsense forms as a function of familiarity and knowledge of results

Transcription

1 Signal detectability in visual nonsense forms as a function of familiarity and knowledge of results Item type text; Thesis-Reproduction (electronic) Authors Seeley, George William, Publisher Rights The University of Arizona. Copyright is held by the author. Digital access to this material is made possible by the University Libraries, University of Arizona. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author. Downloaded 6-Mar :13:36 Link to item

2 SIGNAL DETECTABILITY IN VISUAL NONSENSE FORMS AS A FUNCTION OF FAMILIARITY AND KNOWLEDGE OF RESULTS by George William Seeley A Thesis Submitted to the Faculty of the DEPARTMENT OF PSYCHOLOGY In Partial Fulfillment of the Requirements For the Degree of MASTER, OF ARTS In the Graduate College THE UNIVERSITY OF ARIZONA 1972

3 STATEMENT BY AUTHOR This thesis has been submitted in partial fulfillment of requirements for an advanced degree at The University of Arizona and is deposited in the University Library to be made available to borrowers under rules of the Library. Brief quotations from this thesis are allowable without special permission, provided that accurate acknowledgment of source is made. Requests for permission for extended quotation from or reproduction of this manuscript in whole or in part may be granted by the head of the major department or the Dean of the Graduate College when in his judgment the proposed use of the material is in the interests of scholarship. In all other instances, however, permission must be obtained from the author. SIGNED: APPROVAL BY THESIS DIRECTOR This thesis has been approved on the date shown below: LAWRENCE WHEELER Professor of Psychology Date

4 ACKNOWLEDGMENTS Acknowledgment is made to Dre Lawrence Wheeler9 thesis director, for his assistance, direction, and coordination of this study Thanks go to Dr. Terry C. Daniel for his direction in statistical pro cedures and the theoretical framework that formed the basis of this study. Appreciation also goes to Dr. Neil R. Bartlett without whose guidance and suggestions this study would have been impossible.

5 TABLE OF CONTENTS Page LIST OF ILLUSTRATIONS e. o o <> v LIST O F TABLES o o0 0 o 00 o vi ABSTRACT vii INTRODUCTION 1 METHOD o o o o o o a 4 Subjects o o o o o.o o 4 Stimuli and Apparatus 4 Procedure 3 Method of Analysis o o, 7 RESULTS o , DIS CUS S ION o. o o o 17 APPENDIX As DEGREES OF CERTAINTY CHART 19 APPENDIX B: INSTRUCTIONS (KR AND FAMILIARITY STUDY) 20 LIST OF REFERENCES <> 22 iv

6 LIST OF ILLUSTRATIONS Figure Page 1. Mean confidence ratings for pointed [%] and rounded ^ stimuli in the test phase of the experiment, demonstrating each group s general performance Test phase d^ demonstrating sensitivity of groups Mean confidence ratings for pointed [%] and rounded ^ stimuli from practice phase of the experiment, demonstrating general performance of each group The d values from the practice phase of the experiment, demonstrating the sensitivity of the groups The d sensitivity values for NKR [^] and KR ^ during practice and test p h a s e s... 15

7 ! I LIST OF TABLES i- Table to 2. Page Analysis of variance with respect to subjects and treatments for test trials only.... i..... «10 Analysis of variance with respect to subjects and treatments during practice :««..««. <,» «. «13 Percentages of hits and false alarms for each group during test trials, o * *. «- <. o o.» «. o 16 vi

8 I ABSTRACT! Knowledge of results (KR) and stimulus familiarity were examined in a transfer-of-training experiment on the accuracy of signal detectione A signal-detectability sensitivity measure* d^» was compared with an ordinary performance analysis«kr produced significant differences in performance but not in sensitivity. The major effect was a reduction of false-alarm rates in the KR groups Thus a reexamination of previous KR studies is called for because of the possibility that performance increments might sometimes have been criterion shift effects.rather than discrimination or sensitivity effects. vii

9 INTRODUCTION Observer responses to degraded photographic stimuli have been examined recently by Wheeler et al. (1971). They employed a signaldetection mode of observer response, and their stimuli were computergenerated, pointed, abstract forms each having four "arms'' (quad- rigons). They varied signal-to-noise ratio by differential rounding of the interior and exterior angles of each form. Linear blur, grain magnification, and figure-surround contrast ratio were also independent variables in their factorial design. Observers gave scaled expressions of confidence that the objects depicted in the photographic stimuli had or had not been pointed ones originally. Receiver operating characteristic (ROC) functions provided quantitative indices of the differences in discriminability of the signal condition (pointed objects) for each combination of stimulus degradation. Signal discriminability decreased systematically as the extent of degradation was increased, with certain exceptions. The present experiment extends the work.of Wheeler et al. (1971) by exploring the effects of familiarity and information feedback on accuracy of signal detection. Since the stimuli (quadrigons) were nonsense forms, the confounding effects of meaning were greatly reduced, and because the pointedness or roundedness parameter was analogous to certain important characteristics of natural images, the results should have considerable generality. 1

10 It has been the general finding since Thorndike (1931) that neither enhanced performance nor learning usually take place without some kind of feedback or knowledge of results (KR) 0 KR can be divided into two types; intrinsic (stimulus inherent) and extrinsic (experimenter controlled) It is the common practice of experimenters to use some form of extrinsic KR to train Ss in more effective use of intrinsic stimulus cues If KR is withdrawn and Ss maintain a high level of performance in relation to a no-feedback (NKR) group, then learning is assumed to have occurred. In many experimental procedures, this has generally been the finding. Ammons (1956), for example, reviewed the KR literature up to that time and concluded that KR does affect both the rate and level of learning. When one group is given KR and another is not, the KR group usually learns more rapidly and reaches a higher proficiency than the NKR group. More recent research has continued to demonstrate that KR enhances and stabilizes performance (Weidenfeller, Baker, and Ware, Nash and Adamson, 1968; Wiener, 1969), and in addition, that the higher the ratio of trials containing KR, the better the performance (Howell and Emanuel, 1968). The present study was designed to ascertain the effects of KR on performance in the signal-detection task developed by Wheeler et al. (1971) and also to clarify the effects of stimulus familiarity. A transfer paradigm, in which all groups were equated in the test phase, provided an answer to the question of familiarity; the signaldetection mode of response increased the sensitivity of the analysis

11 by showing whether KR principally affected Ss*8 guessing criterion (by aiding S to adjust his decision criterion toward the actual ratio of signal to nonsignal stimuli), or whether S actually learned to distinguish among the stimuli more accurately.

12 METHOD Subjects Twenty-four University of Arizona Optical Sciences Center graduate students and secretaries served as Ss (16 male, 8 female) and were assigned at random to four groups of equal size. Three Ss were paid. All Ss had at least 20/20 distance vision (Snellen letters). The Ss* mean age was 25% years. Stimuli and Apparatus One hundred and eight stimuli were selected from the 500 degraded quadrigons generated by Wheeler et al. (1971). Each stimulus was a 1%-in. by 2-in. photographic print mounted on 4-in. by 4-in. gray cardboard (35% reflectance). levels of quadrigon roundedness; The stimulus set contained four 0, %-in., %-in.» and %-in., as defined in Wheeler et al. (1971, p. 3); three levels of figure-ground contrast; 5:95, 25:75, and 40:60 (the ratio of figure and ground per centage reflectance)$ three grain magnification factors: 0, 8, and 16; and three levels of linear blur: 0, 5, and 9 mm. Each stimulus represented one level of each of the four independent variables simultaneously. A more detailed description of the stimuli may be found in Wheeler et al. (1971, pp ). The 108 stimuli were assigned at random to a set of 72 stimuli or to a set of 36, with the restriction that 18 stimuli of zero 4

13 roundedness (signals) were included in the set of 72, and 9 such stimuli in the set of 36. In the practice or training phase the 72 stimuli were presented to each S one at a time. A separate random sequence was used for each S, and stimuli Were viewed under C illuminant (Macbeth s daylight lamp) at S s normal viewing distance (about 18 inches). The same conditions existed during the test phase. During training trials, the feedback information was visual, consisting of a red or a green light (red = incorrect5 green = correct), presented for 1 second. The light exposure was controlled by an Industrial Corporation timer started by the experimenter immediately after each response. A 10-point degree-of-certainty chart (Appendix A), ranging from 0 through 9, governed S s response optionss 0 = <SI am absolutely certain - rounded 5 4» <SI am just guessing rounded ; 5 = <<I am just guessing pointed ; and 9 = 661 am absolutely certain - pointed. The other numbers represented appropriately graded levels of certainty. To equate exposure between KR and other groups, each S was allowed approximately 1% seconds of continuous observation of the stimulus after each response. Procedure Ss were asked to decide whether the original objects (not the photographed images) had been pointed or not. The apparent roundedness or pointedness of the present stimulus-figure was not to be considered. S s binary decisions permitted analysis by means of the theory of signal detection (TSD).

14 The transfer design included three control groups5 test-only, pre-exposure and then test, and NKR and then test; and one experimental groups KR and then test. The test-only group saw only the 36 test stimuli 5 their data acted as a baseline with respect to the other groups. The pre-exposute ^igb@8$p controlled for the possibility that mere exposure to, or familiarity with, the stimuli might affect performance during the test phase. This group saw the 72 stimuli one at a time, but were instructed to make no responses; task instructions were given just before the 36 test stimuli were presented. The NKR Ss were instructed about their task before any stimuli were given but were given no feedback at any time during the practice or test phases. This group replicated the Wheeler et al. (1971) experiment and was also a control group that could be compared directly with the KR group in both pretest and test phases. Thus, each group had a different amount of information to work with. The experimental (KR) group had the most information; its treatment was identical to that of the NKR group except that KR was given after every response during the practice phase. No group received KR feedback during the test phase. A standard set of taped instructions was presented to each S (Appendix B), with additional verbal instructions depending upon the S s group. The Ss in the pre-exposure group were told to look at each stimulus carefully, to make no response, and that further instructions would follow; they were then given appropriate response

15 instructions just prior to the test phase. After each KR group S had heard the taped instructions, he was told that he would receive visual feedback concerning his performance accuracy on every trial. Each S was run individually and was requested to refrain from discussing his task with the other Ss. Method of Analysis Analysis of responses was based on the theory of signal detectability (TSD)> which divides performance into two parts I sensitivity (d/) and criterion state, or response bias (Green and Swets, 1966). By statistically removing the decision or response bias component from S s performance, it is possible to assess the effects of the experimental conditions on S's true sensitivity to stimulus characteristics. A recently developed detection measure, d, (Grice, 1968) increased the statistical sensitivity of the TSD analysis in the present study. This measure is the mean distance between distributions of * noise5' and signal plus noise'' represented in the stimulus situation. The distance between signal-plus-noise and noise distributions at any given decision criterion point is termed d^. The arithmetic mean of the d^'s, calculated from the actual values to which the receiver operating characteristic (ROC) function is fitted, is called d. ~m The d^ measure is a more satisfactory index of subject sensi- tivity than are the traditional d and dg measure of TSD, which are usually measured directly from plotted ROC functions. Fluctuations

16 in the ROC slope affect d7 and d even though the area under the ROC s curve may remain constant. Such changes in slope, however, leave d^ relatively unchanged. In addition, reflects all the available data, is easy to compute, and correlates highly with the area under the ROC curve. For a more complete discussion of the rationale and development of this analytic procedure see Wheeler e al. (1971, pp ).

17 RESULTS Separate analyses of variance were performed on the mean rat ing test scores for pointed and for rounded stimuli, Parallel analy ses were carried out for the values calculated in the signal- detection analyses. Each mean rating analysis showed significant treatment effects, F (3,20) = 7.668, P < 0.05, and F (3,20) = 5.886, P 0.05, for the pointed and rounded conditions, respectively. Stimulus and stimulus-by-treatment sources were partitioned in order to increase the precision of the analysis of treatment effects. Stimulus effects were significant in each analysis (Table 1). The stimulus-by-treatment effect was not significant for the pointed objects but reached significance for the rounded ones. The d analysis in Table 1 is an analysis of variance (model I) of the ~tn d^s mentioned previously and indicates the effect of treatment on sensitivity. Thus, there was a difference in outcome between the mean rating and the analyses of treatment effects. Graphs of the mean ratings for the pointed and rounded stimulus values and of the d^ values demonstrate the power of the TSD method and the associated d index in this situation. As seen in Fig. 1, the pre-exposure group performed more poorly than any other group during the test trials; the test only and the NKR groups were better in performance; the KR group gave the best performance of all groups.,

18 10 Table 1 Analysis of variance with respect to subjects and treatments for test trials only Mean Rating Analysis Pointed Source df Mean square F P Treatment <0.05 Ss/Treatment Stimuli <0.05 Stimuli X Treatment NS (0.20) Stimuli X Ss/Treatment Mean Rating Analysis - Rounded Source df Mean square F P Treatment ,886 <0.05 Ss/Treatment Stimuli <0.05 Stimuli X Treatment <0.05 Stimuli X Ss/Treatment d Analysis Source df Mean square F P Treatment NS* d./ss X Treatment *Compare with F (3,30) = 2.92 at P = 0.05.

19 : 11 The plot of d values» Fig, 2, showing observer sensitivity, provides a different picture. With response bias removed, the test- only group proves to have been the least sensitive. The pre-exposure and the NKR groups were almost equivalent in sensitivity and were both better than the test-only group, while the KR group was the most sensitive. A possible inference from the two analyses is that KR generally improved performance but increased sensitivity only slightly. The d analysis (Fig. 2) indicates that familiarity also had ro some effect on sensitivity because the pre-exposure and the NKR test groups were each more sensitive than the test-only group. Analyses of variance were also performed on data from the practice phase? only NKR and KR groups were compared. The mean rating analyses of pointed and rounded stimuli again yielded significant treatment and stimulus effects as indicated in Table 2. Stimulus and stimulus~by=treatment effects were again partitioned to increase the precision of the analysis. In the-d analysis, the treatment effect again failed to reach significance. A comparison of Figs. 3 and 4, the plots of practice-phase mean ratings of pointed and rounded stimuli and the associated d^ values, demonstrates that during the practice phase the KR group performed differently from the NKR group but that the two groups were almost identical in sensitivity. Figure 5 represents the d^ values for KR and NKR groups during practice and test trials. It indicates that the KR group increased markedly in sensitivity during the test phase.

20 9 CERTAIN POINTED 8 7 OQ GUESSING CERTAIN 0 ROUNDED 0 Test Pre- NKR KR NKR KR only exposure only exposure TREATMENT GROUPS Fig. 1. Mean confidence ratings for Fig. 2. Text phase d^, demonstrating pointed C H and rounded stimuli in the test phase of the experiment, demonstrating each group's general performance. sensitivity of groups.

21 13 Table 2 Analysis of variance with respect to subjects and treatments during practice Mean Rating Analysis - Pointed Source df Mean Square F P Treatment <0.05 Ss/Treatment Stimuli <0.05 Stimuli X Treatment NS Stimuli X Ss/Treatment Mean Rating Analysis Rounded Source df Mean square F P ' ' Treatment <0.05 Ss/Treatment Stimuli <0.05 Stimuli X Treatment NS Stimuli X Ss/Treatment m Analysis Source df Mean square F P Treatment NS d./ss X Treatment i

22 CERTAIN POINTED s z < DC UJ u z o u z< Ui 5 GUESSING CERTAIN ROUNDED NKR KR NKR TREATMENT GROUPS KR Fig. 3. Mean confidence ratings for pointed and rounded stimuli from practice phase of the experiment, demonstrating general performance of each group. Fig. 4. The d values from the practice m phase of the experiment, demonstrating the sensitivity of the groups.

23 NKR KR PRACTICE NKR KR TEST Fig. 5. The d sensitivity values for -m NKR Q and KR ^ during practice and test phases.

24 16 Table 3, showing percentages of false alarms and hits for each group during the test trials, demonstrates that KR affected accuracy primarily by reducing the false-alarm rate. Table 3 Percentages of hits and false alarms for each group during test trials Groups ~ Hits False alarms Test-only 0, Pre-exposure NKR KR

25 ! DISCUSSION " ; : i In this experiment the quadrigon stimuli and the TSD methodology proved to be useful tools for examination of the effects,of knowledge of results. The sensitivity analysis, as opposed to the mean rating analysis, gave a more accurate,picture of the effects of the independent variables.. Analyses, of correct pointed judgments (hit rate) alone might have led to erroneous conclusions concerning effects of the independent variables, but the TSD correction for guessing bias produced a clear picture of the effects of KR on performance in the detection task. The strongest implication of the present study is that much previous KR research should be. reexamined in terms of the possibility that apparent increments in performance might, in fact, have been due mainly to criterion shifts. Knowledge of results produced more accurate performance, and there was a hint that the KR group increased in sensitivity, but this effect was not statistically significant. In general, KR helped keep false alarms at a minimum while hits were maintained at a relatively high level. 'The three control groups also had high hit rates, but these were associated with relatively high false-alarm rates. The higher false-alarm rates for the control or familiarity groups indicate that these groups had much looser criterion levels than the KR group. In addition, the KR group tended to become more conservative in its 66pointed59 judgments. 17

26 .... is The results of the d analysis, for the practice phase, present a much clearer picture of Ss9 discrimination behavior than does the mean rating analysis, demonstrating an important difference between performance and. seiisitivity«with regard to general performance (mean ratings) the KR group performed better than the NKR groupe. On the other hand, the d analysis indicates no difference in sensi- -m,. tivity between these groups» This outcome implies that although the KR group improved in performance, its sensitivity to stimulus cues remained essentially the same as.that of the NKR group* There was some evidence of improved sensitivity in the KR group in the d analysis of the test phase performance, This effect might become more pronounced if more training trials were given; such a study is now in progress * The question cannot be answered adequately on the basis of the present experiment *

27 APPENDIX A DEGREES OF CERTAINTY CHART The following chart shows the numbers used by the Ss to respond to the quadrigon presentations. Absolutely certain- Pointed More than fairly certain 8 Pointed Fairly certain Pointed 5 More than just guessing Pointed Just guessing Pointed Just guessing Rounded O More than just guessing ^ Rounded 2 Fairly certain Rounded More than Fairly certain Rounded Absolutely certain Rounded 19

28 I APPENDIX B I INSTRUCTIONS (KR AND FAMILIARITY STUDY) Accurate detection and recognition of objects on the ground, by means of photographs taken from aircraft or spacecraft, is an important problem today. The present experiment is a simplified version of this problem* We are going to show you, one at a time, some photographs of single objects,* Each object was different from the others, and we will ask you to make one decision about each object * Some of the objects had pointed angles, both interior and exterior,. and looked like this (demonstrate with pointed model)* Some of the objects were rounded off at all corners, like this (demonstrate with rounded model)* Even a very slight rounding means the object was not pointed* For each photograph that we show you, we want you to tell us how certain you are that the original object was a pointed one, or not* Many of the photographs will be poor, just as they might be in real life* They may be blurred, grainy, low in contrast, and so on, but you should pay no attention to this* You should think about the object that was photographed, not about the resulting picture. To make your replies as rapid and accurate as possible, we ask you to report one of the numbers shown on this card (give card to subject) when we show you each photograph * For instance, 9 means 6*1 am absolutely certain that the photographed object was pointed. It 20

29 was not roundedo59 7 means 6 am fairly certain that the object was 21 pointed* not rounded»99 5 means * Ijust guessing* but the object might have been a pointed one. ** And 0 means 6<?I am absolutely certain that the photographed object was not pointed. It was rounded.95 In other words we are asking you to put a numerical value on your own degree of certainty that each object was a pointed one, or not. Please use the full range of certainty numbers * if you possibly can. This will produce much better results for the experiment as a wholeo Do you have any questions before we start? (Do not tell subjects the frequency of pointed objects or the total number of objects.) Just give me one number* from 9 through 0* that shows how certain you are that the original object was pointed or not.

30 LIST OF REFERENCES Ammons, R. B Effects of knowledge of performance; a survey and tentative theoretical formulation. The Journal of General Psychology, 1956, 54, Green, D. M., and Swets, J. A. Signal detection theory and psychophysics. New York; John Wiley and Sons, Inc., Grice, G. R. Computations in the use of TSD. Unpublished laboratory note, University of Illinois, Howell, W. C., and Emanuel, J. T. Information feedback, instructions and incentives in the guidance of human choice behavior. Journal of Experimental Psychology, 1968, 78(3), Nash, A., and Adamson, R. Effects of information feedback and reference tones on signal detection. Psychonomic Science, 1968, 13, " Thorndike, E, L. Human learning. New York; Century, Weidenfeller, E. W., Baker, R. A., and Ware, J. R. Effects of knowledge of results (true and false) on vigilance performance. Perceptual and Motor Skills, 1962, 1_4, Wheeler, L., Daniel, T., Seeley, G., and Swindell, W. Detectability of degraded visual signals; a basis for evaluating, imageretrieval programs. Optical Sciences Center Technical Report 73, University of Arizona, December Wiener, E. L. Money and the monitor. Perceptual and Motor Skills, 1969, 29,

31 r'n ^., fid e.: BaSii iiiiai (T:^T _ 1* :...V/"^ t # l '-sag, 1 ii< iliil. a T * msm ^ix-, o 6 79