TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK. John T. Cockrell System Development Corporation. and

Transcription

1 fto 7V7?*7 Technical Research Reprt 1171 LIBRARY TECHNICAL REPORT SECTION NAVAL POSTGRADUATE SCHOOt MONTEREY, CALIFORNIA AD TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK Jhn T. Cckrell System Develpment Crpratin and Rbert Sadacca Behavir and Systems Research Labratry U. S. Army Behavir and Systems Research Labratry June 1971 Apprved fr public release; distributin unlimited.

2 BEHAVIOR AND SYSTEMS RESEARCH LABORATORY An activity f the Chief, Research and Develpment J. E. UHLANER Directr Research accmplished Under Cntract t Department f the Army System Develpment Crpratin NOTICES DISTRIBUTION: Primary distributin f this reprt has been made by BESRL. Please address crrespndence cncerning distributin f reprts t: U. S. Army Behavir and Systems Research Labratry. Attn: RDMR-BLZ, 1300 Wilsn Bulevard, Arlingtn, Virginia FINAL DISPOSITION: This reprt may be destryed when it is n lnger needed. Please d nt return it t the Behavir and Systems Research Labratry. NOTE: The findings in this reprt are nt t be cnstrued as an fficial Department f the Army psitin, unless s designated by ther authrized dcuments.

3 Technical Research Reprt 1171 AD TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK Jhn T. Cckrell System Develpment Crpratin and Rbert Sadacca Behavir and Systems Research Labratry Abraham H. Birnbaum, Prgram Directr SUPPORT SYSTEMS RESEARCH DIVISION Jseph Zeidner, Chief BEHAVIOR AND SYSTEMS RESEARCH LABORATORY Office, Chief f Research and Develpment Department f the Army 1300 Wilsn Bulevard, Arlingtn, Virginia June 1971 Army Prject Number Cntract Number A721 DAHC19-67-C-0040 Surveillance Systems c-21 Apprved fr public release; distributin unlimited.

4 BESRL Technical Research Reprts and Technical Research Ntes are intended fr spnsrs f R&D tasks and ther research and military agencies. Any findings ready fr implementatin at the time f publicatin are presented in the latter part f the Brief. Upn cmpletin f a majr phase f the task, frmal recmmendatins fr fficial actin nrmally are cnveyed t apprpriate military agencies by briefing r Dispsitin Frm.

5 FOREWORD The SURVEILLANCE SYSTEMS research prgram f the U.S. Army Behavir and Systems Research Labratry has as its bjective the prductin f scientific data bearing n the extractin f infrmatin frm surveillance displays, and the efficient strage, retrieval, and transmissin f this infrmatin within an advanced cmputerized interpretatin facility. Results are used in the develpment f techniques t enhance all phases f the image interpretatin prcess and t prvide findings fr use in future systems design. Research is cnducted under RDT&E Prject 2Q662704A721, Surveillance Systems, FY 1971 Wrk Prgram. The ADVANCED SURVEILLANCE SYSTEMS Wrk Unit cntinues BESRL research previusly cnducted as the COMPONENT INTEGRATION Wrk Unit, with emphasis n the human factr requirements f tactical interpretatin facilities, as these facilities are currently evlving, particularly with regard t the critical analysis and evaluatin f the functins f the interpreter, his display needs, and the effectiveness f the ttal system. The present publicatin summarizes fur experiments leading t mre effective methds f maintaining the skills f image interpreters within an advanced interpretatin system thrugh feedback furnished by team members. BESRL research in this area is cnducted as an in-huse research effrt augmented by cntracts with rganizatins selected as having unique capabilities and facilities fr research in aerial surveillance. The present research was cnducted jintly by persnnel f BESRL and f the System Develpment Crpratin. J. E. UHLANER, Directr Behavir and Systems Research Labratry

6 TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK BRIEF Requirement: T explre the effectiveness f a lw cst prficiency imprvement and maintenance prgram fr image interpreters; specifically, t determine the effect n interpreter detectin and identificatin skills f a training methd using feedback arising frm the agreement (cnsensus) which team members achieve when they cmpare and discuss their interpretatins. Prcedure: Fur experiments were cnducted, explring varius team cnsensus feedback methds with varying cnditins f team size and cmpsitin. In Experiment I, team cnsensus feedback was cmpared with precise feedback (prir identificatins made by experienced interpreters) and n feedback in effectiveness in imprving bth target detectin and target identificatin. Experiment II studied the effectiveness f team cnsensus feedback in imprving target detectin perfrmance nly. Experiment III evaluated mdified team cnsensus feedback techniques designed t minimize feedback delays. In Experiment IV, the nature f learning curves using team cnsensus feedback techniques was investigated and the mst prmising prcedures identified in the first three experiments were cmpared in a thrugh analysis f the team cnsensus methd. Findings: Team cnsensus feedback, requiring nly that interpreters cmpare their answers in the detectin and identificatin f tactical targets n previusly interpreted r uninterpreted aerial imagery, was demnstrated t prvide greater perfrmance imprvement than a cntrl cnditin where interpreters practiced individually with n feedback. The greatest perfrmance gains were made in target identificatin. The number f inventive errrs was reduced using team cnsensus feedback training; imprvement in the number f targets crrectly detected was fund nly in the furth experiment in Which the mst prmising techniques f the first three experiments were cmpared. The results further shwed that interpreters wh were initially lw in prficiency achieved the mst significant gains in perfrmance. Interpreters assigned t teams that were hetergeneus in terms f initial prficiency made greater gains than interpreters that were members f teams hmgeneus in terms f initial prficiency. Lw prficiency interpreters were shwn t make greater gains in prficiency than interpreters with high initial prficiency. The results indicate that lw prficiency interpreters learned frm, r benefited thrugh their cllabratin with, higher prficiency interpreters.

7 Team discussin and team size did nt significantly influence interpreter perfrmance; hwever, there was sme indicatin that these variables cmbine t prduce an verall effect. The greatest perfrmance gains were achieved with a methd using three-man teams, hetergeneus in terms f initial prficiency, in which individuals made their initial interpretatins individually but were permitted t discuss their identificatins freely. Team methds requiring the cntinued clse assciatin f interpreters frm the beginning f search did nt lead t verall perfrmance imprvement, pssibly because high prficiency interpreters precluded effrt n the part f less prficient interpreters. Utilizatin f Findings: Team cnsensus feedback was demnstrated t be an effective methd f imprving and maintaining image interpreter prficiency in an peratinal image interpretatin facility. The methd requires n specialized training materials r instructr preparatin. Tw r three interpreters, using a rll f tactical imagery, may imprve their perfrmance, particularly in identificatin, by individually interpreting frames f imagery and then discussing their detectins and identificatins. Mre prficient interpreters shuld be included in each team t effect the greatest imprvement in the less prficient interpreters. This methd culd als prfitably be used in the classrm during team exercises.

8 TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK CONTENTS Page BACKGROUND AND PURPOSE 1 Cntext f the Study 1 OBJECTIVES 2 FRAMEWORK FOR THE EXPERIMENTATION 3 FINDINGS 3 Effects f Different Types f Feedback 3 Effects f Cnsensus Feedback n Target Detectin 5 Use f Jint Search Team Prcedures t Minimize Delay in Cnsensus Feedback 5 Nature and Scpe f Learning During Team Cnsensus Feedback 6 IMPLICATION OF THE FINDINGS 7 TECHNICAL SUPPLEMENT 9 APPENDIXES 41 DISTRIBUTION 45 DD FORM 1473 (Dcument Cntrl Data - R&D) 47

9 TABLES Page Table 1. Mean differences between pre-test and pst-test scres fr three measures f image interpreter perfrmance--experiment I Mean perfrmance gains in identificatin scre between pre-test and pst-test--experiment I Mean perfrmance gains in crrect detectin scre between pre-test and pst-test Experiment I Mean perfrmance gains in false target detectin scre between pre-test and pst-test--experiment I 18 5«F-ratis fr difference scres--experiment I Experimental design fr Experiment II Cmpsitin by prficiency f teams in Experiment II Mean gains between pre-test and pst-test in detectin scres Experiment II Difference scre F-ratis fr detectin scres- Experiment II Cmparisns between experimental grups and cntrl grups fr mean gains in detectin scres- Experiment II F-Ratis fr cmparisn between cllapsed experimental grups and cntrl grups in detectin scrers--experiment II Arrangement fr analysis f variance f difference scres between initial and final prficiency tests- Experiment III Mean percentage scres fr detectin cmpleteness n pre-test and pst test Experiment III Mean percentage scres fr detectin accuracy n pre-test and pst-test Experiment III F-ratis fr difference scres between pre-test and pst-test Experiment III 30

10 Table Table 16, , 19. Experimental arrangement fr Experiment IV Mean scres fr identificatin, crrect detectin, and inventive errrs acrss test administratins--experiment IV Analysis f variance f identificatin, psitive detectin, and false target detectin Experiment IV P-values fr the Newman Keuls test fr the three sessins separately fr all scres- Experiment IV Page FIGURES Figure 1, Identificatin learning curves fr three methds used by interpreters f high, medium, and lw prficiency Detectin skill learning curves fr three ~ methds used by interpreters f high, medium, and lw prficiency Inventive errr learning curves fr three methds used by interpreters f high, medium, and lw prficiency

11 TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK BACKGROUND AND PURPOSE Cntext f the Study A series f experiments has been cnducted ver the past several years by the Behavir and Systems Research Labratry (BESRL) t develp and test the team cnsensus feedback methd as a technique fr maintaining and enhancing the prficiency f individual image interpreters. The essential feature f the methd is that interpreters practice in teams, arriving at decisins with regard t target detectin and identificatin by a cnsensus f the team members. Generally, a cnsensus has been defined as either a tw-thirds vte r a unanimus vte f the team members. The nly feedback which the interpreters receive with regard t the accuracy f their decisins they prvide fr themselves during team discussin and cmparisn. Image interpreters wrking alne n a missin are ften unaware when they are ding a pr jb f detecting r identifying targets. Seldm d they receive any feedback, and if they d, it is generally t late t be effective. In teams, hwever, interpreters can be frced cntinually t take stck f themselves, since their teammates are finding targets and making identificatins which disagree with their wn. In any cnflict regarding identificatin f a specific target, smene has t be wrng. This awareness f disagreement nt nly frces team members t take a lng hard lk at the target, but als allws less prficient interpreters t becme aware f sme f their wn deficiencies and t learn frm the mre prficient interpreters. The team cnsensus feedback methd is based n prir team studies cnducted at BESRL ( x, s, 3 ), which demnstrated that image interpreters wrking in teams can prduce mre cmplete and accurate intelligence infrmatin frm aerial recnnaissance imagery than interpreters wrking alne. The cnsensual judgment f team members was fund t be especially effective in reducing the number f identificatin errrs made by single interpreters. Since teams generally prduce better reprts than Sadacca, R., H. Martinek, and A. I. Schwartz. Image Interpretatin Task--Status Reprt. Technical Research Reprt 1129, June 1962 (AD ). Behavir and Systems Research Labratry, Arlingtn, Va. 2 3 Blin, S. F., R. Sadacca, and H. Martinek. Team prcedures in image interpretatin. Technical Research Nte 164, December 1965 (AD ). Behavir and Systems Research Labratry, Arlingtn, Va. Dten, G. W., J. T. Cckrell, and R. Sadacca. The use f teams in image interpretatin: Infrmatin exchange, cnfidence, and reslving disagreements. Technical Research Reprt 1151, Octber I966 (AD ). Behavir and Systems Research Labratry, Arlingtn, Va.

12 individuals, interpreters training in teams shuld receive mre accurate knwledge f results, r feedback, than interpreters practicing alne. The research apprach fllwed has been t evaluate empirically varius team cnsensus feedback prcedures t determine which variatins are mst effective in training individual interpreters. Factrs ther than wrk prcedures which have been varied in these experiments are delay f feedback, manner in which the feedback is presented, type f feedback, size f team, cmpsitin f the team in terms f initial prficiency f team members, and the effect f initial prficiency n subsequent learning. In the final experiment, reprted here as Experiment IV, an attempt was made t investigate the nature and extent f the learning which ccurs and the shape f the learning curve. OBJECTIVES Mst field interpretatin units typically have a relatively large number f inexperienced persnnel and a relatively small number f experienced persnnel. Sme prficiency imprvement and maintenance prgram is necessary fr these persnnel, especially fr recent graduates f interpretatin schls and transferees. The team cnsensus feedback methd, if prved feasible, wuld ffer a relatively simple and inexpensive methd f prviding this practice. The advantages f the methd are that n elabrate and expensive materials are needed, and practice sessins can be initiated during any slack perid by simply taking a rll f imagery "ff the shelf" and presenting it t interpreter teams in an image interpretatin facility. The fur experiments reprted here were cnducted in an effrt t develp team practice methds which will lead t the greatest perfrmance gains by individual interpreters. The first experiment was designed t btain a general assessment f the usefulness f the cnsensus feedback prcess. In this experiment, team feedback was cntrasted with n feed-- back n the ne hand and precise feedback n the ther. In the n-feedback cnditin, interpreters simply practiced by themselves. Precise feedback cnsisted f the "schl slutin" derived by a team f expert interpreters wrking in part with grund survey data. It was assumed that the quality f team cnsensus feedback wuld lie smewhere between n feedback and precise feedback, depending upn the effectiveness f the team prcedure being used. The secnd experiment sught t determine if team cnsensus feedback wuld be effective in enhancing detectin perfrmance. In the first experiment, mst f the team practice had been cncentrated n the identificatin f targets. In the secnd experiment, emphasis was shifted simply by eliminating the requirement fr identificatin, thereby greatly increasing the amunt f team practice in target detectin. In the third experiment, emphasis was cncentrated nce again n target detectin, but team prcedures were changed t minimize delay f feedback. This experiment was als structured t allw team utput during

13 the practice sessins t be measured. This latter step was taken in rder t determine if teams were detecting mre targets during practice than were the interpreters wrking alne. The furth experiment was cncerned mainly with the nature f the learning whish ccurs during the team cnsensus feedback practice sessins and als with the shape f the learning curve. Since this experiment was intended t be the final experiment in the series, it was cnsidered a replicatin f sme f the preceding experiments. T this end, the mst effective prcedures identified in the previus three experiments were evaluated. FRAMEWORK FOR THE EXPERIMENTATION The same general sequence f events was fllwed in all fur experiments. Each grup f subjects spent five days in labratry experimentatin. The first day was taken up with instructins and an individually administered pre-training test. The secnd, third, and furth days were used fr team practice sessins, and the fifth day was used fr the psttraining test. In the last experiment, a third individual test was intrduced in the middle f the third day. Fr all experiments, a cntrl grup f individual interpreters practiced alne during the secnd, third, and furth days n the same material used by the teams. The imagery used in the experiments cnsisted f aerial phtgraphs f Army field maneuvers. The imagery was arranged in stere pairs, and 150 f these pairs were spliced tgether in a rll f 9" x 9" psitive transparencies. Each interpreter had his wn viewing device--a light table--and ther basic interpreter equipment. The task cnsisted f searching each stere pair fr designated types f military targets, anntating the imagery by circling and numbering the targets, and then identifying the targets where identificatin was required. The interpreters then recrded this infrmatin n data sheets. The experimental sample was cmpsed f enlisted men wh had just cmpleted the image interpretatin curse at the U. S. Army Intelligence Schl, Frt Hlabird, Maryland. These relatively inexperienced interpreters were judged t have prficiency cnsnant with the prficiency levels f interpreters wh might benefit frm cnsensus feedback exercises in the field. Fifty-fur interpreters were used in the first experiment, 60 in the secnd, 48 in the third, and 54 in the final experiment. Effects f Different Types f Feedback (Experiment I) FINDINGS In the first experiment, team cnsensus feedback was cmpared t precise feedback and n feedback. All teams in this experiment had three men assigned t each team, ne man being high in prficiency. The remaining team members were either bth medium in prficiency, bth lw in 3 -

14 prficiency, r ne medium and ne lw. Prficiency was determined by means f the pre-training test. Three different cnsensus-feedback prcedures were used in this experiment: Serial Cnsensus. Each man initially wrked independently n a different stere pair. The men anntated all targets fund, identified the targets, and recrded the targets n data frms. The team members then rtated seats and checked their teammates' wrk. Checking cnsisted f agreeing r disagreeing with the targets fund by the previus interpreters and then lking fr additinal targets missed by the previus interpreters. Fllwing checking, the three men gt tgether and discussed all cnflicts n each stere pair. Immediate Cnsensus. Each man wrked n a cpy f the same stere pair during initial interpretatin. The men then went directly t the discussin phase and attempted t reslve all cnflicts. Delayed Cnsensus. This methd was the same as immediate cnsensus except that each man cmpleted three stere pairs befre discussin was begun. Fr the precise feedback cnditin, tw wrk prcedures were used: 1. Precise Team Prcedure--The teams first went thrugh the immediate cnsensus prcedure and then were given the crrect answers t cmpare with their team sheet. 2. Precise Individual Prcedure--The men wrked independently. Fllwing interpretatin f each stere pair, each interpreter was given the crrect answers t cmpare with his wn. Fr the n-feedback cntrl cnditin, each man wrked by himself and n feedback was received at any time. In this first experiment, interpreters practicing under team cnsensus feedback made greater gains than interpreters in the n-feedback cnditin. As expected, thse interpreters prvided precise feedback shwed the greatest gain. Anther finding indicating the imprtance f feedback was that the delayed cnsensus prcedure shwed the least gain amng the feedback prcedures. This finding is in keeping with the general results in psychlgical experiments dealing with delay f reinfrcement r feedback. A breakdwn f the results revealed that mst f the gains in favr f the cnsensus feedback prcedures ccurred in identificatin f targets. N significant gains were btained in the number f targets detected r in the number f false targets eliminated.

15 Effects f Cnsensus Feedback n Target Detectin (Experiment II) The secnd experiment explred the effect f cnsensus feedback n target detectin. The immediate cnsensus prcedure and the n feedback cntrl prcedure were used, but the requirement t identify targets was eliminated. This change reduced the amunt f time required fr each stere pair and greatly increased the number f stere pairs which culd be cvered during team practice, thereby presumably increasing the amunt f detectin practice. Anther feature intrduced in the secnd experiment was the use f cnfidence statements. Interpreters were tld that if they recrded dubtful targets with a cnfidence f 49$ r less, they wuld receive ne-half credit if they were crrect and wuld lse nthing if they were incrrect. This technique greatly encuraged respnding during team practice and brught many mre dubtful discriminatins befre the team fr reslutin. In additin, this experiment explred the effects f difference in size f team (2- and 3-man teams), the use f discussin versus n discussin, and team cmpsitin based n whether the team members were hmgeneus r hetergeneus in initial ability. Team cnsensus feedback greatly reduced the number f inventive 'false alarm' errrs hut did nt significantly increase the number f targets crrectly detected. Results with regard t team size and discussin were nn-significant, but results with regard t team type were significant. Interpreters wh trained in hetergeneus teams shwed greater gains than the hmgeneus teams in bth number f targets detected and reductin f inventive errrs. Interpreters with initial lw prficiency shwed the greatest imprvement in the reductin f inventive errrs. Prficiency differences were significant fr the reductin f inventive errrs but nt fr the number f targets crrectly detected. Use f Jint Search Team Prcedures t Minimize Delay in Cnsensus Feedback (Experiment III) The third experiment explred tw jint-search prcedures designed t minimize the time interval between initial detectin f a target and team cnsensus feedback. An equally imprtant purpse f the jintsearch prcedures was t increase the number f targets detected by each team. In the jint-search prcedure with the least delay, each man had a separate light table and a cpy f each stere pair. As sn as ne man detected a target, he wuld lean ver and pint t the target n the ther man's light table. The men wuld then stp searching and discuss the target. After cnsensus had been reached, they wuld resume search until the next target was fund. Generally, the first few easy targets were fund and agreed upn in a very shrt time; increasing time intervals and lnger discussins were nted as the targets became mre dubtful.

16 In the secnd jint-search prcedure, the men pinted ut targets t each ther as sn as they were fund, but n discussin tk place until bth men had finished searching the stere pair. At this time, all lcated targets were discussed. In additin t these tw wrk prcedures, the serial search prcedure and a n-feedback cnditin used in the first experiment were included. All teams in the third experiment were cmpsed f tw men, ne high in initial prficiency and ne lw. Cnfidence estimates and scring were as in the secnd experiment. During the teamwrk sessins, half the teams were required t identify the targets at a grss name levelj the ther half had nly t detect the targets. During the individual preand pst-training tests, nly detectin was required. N significant imprvement in perfrmance was achieved either in number f targets crrectly detected r in reductin f inventive errrs. Thus, it wuld appear that the ntin that mre immediate feedback increases learning was nt brne ut by this experiment. Hwever, practice decreased the number f inventive errrs made by the initially less prficient interpreters. Nature and Scpe f Learning During Team Cnsensus Feedback (Experiment IV) In the first three experiments, n attempt was made t determine directly if learning was ccurring during the three days f cnsensus feedback training. Learning was inferred frm differences between experimental and cntrl grups. In the final experiment, an attempt was made t determine the nature and shape f the learning curve invlved in team cnsensus feedback. An effrt was made t chse the team prcedures and cnditins which had led t the largest gains relative t the cntrl grup in the first three experiments. Tw team prcedures used in Experiment I, serial cnsensus and immediate cnsensus, were selected as the mst prmising f the prcedures. In chsing team size, cnsideratin was given t trends and ther indicatins which seemed t indicate that threeman teams are preferable t tw-men teams. Althugh n beneficial effects had been demnstrated fr discussin, team members were allwed t discuss their target identificatins. The teams were required t identify all targets at grss name level. All teams were hetergeneus in terms f initial prficiency, each team cnsisting f ne high, ne medium, and ne lw prficiency interpreter. Three individual tests were used t assess learning; pre-training, intermediate, and pst-training. The three tests were rughly equated as t difficulty, but t insure that difficulty did nt bear n the results, a Latin square arrangement was used t cunterbalance tests, sessins, and training prcedures.

17 Results f the final experiment shwed very definitely that learning ccurred during the three-day practice perid and that this learning was significantly greater fr the team cnsensus feedback grups. Learning was demnstrated fr all perfrmance measures--target identificatin, number f targets crrectly detected, and number f inventive errrs. The immediate cnsensus feedback prcedure seemed slightly superir t the serial cnsensus prcedure. With serial cnsensus feedback prcedure, learning was rapid at first and then slwed sharply; learning with the immediate cnsensus feedback prcedure was steady at a medium rate. Fr the cntrl grup, learning ccurred at a lw steady rate. IMPLICATION OF THE FINDINGS The general cnclusin frm the fur experiments is that the team cnsensus feedback is an effective methd fr maintaining and imprving image interpreter prficiency. Althugh precise feedback leads t greater learning, precise feedback is seldm available, especially in field situatins. Cnsensus feedback training can be readily implemented where tw r mre interpreters have free time and a rll f imagery. Overall results fr the fur experiments indicate that team cnsensus feedback practice leads t greater learning than individual practicein target detectin and identificatin. The methd has been shwn t be mst effective in increasing identificatin prficiency. The team cnsensus feedback methd was als effective in reducing number f false targets reprted. Only with the mst effective team prcedures (in Experiment IV) was there a significant increase in number f targets crrectly detected. Team cmpsitin in terms f prficiency seems highly imprtant in the team cnsensus feedback prcedure. Where teams are hetergeneus in prficiency, learning is substantially greater than where teams are hmgeneus in prficiency. This fact seems t indicate that team members learn frm each ther, and unless at least ne member f the team is mre prficient than the thers, very little learning will ccur. This cnclusin was clearly brne ut in Experiment IV where the highly prficient interpreters in the team practice grups learned nly a small amunt mre than the cntrl interpreters, whereas the interpreters f medium and lw prficiency learned much mre than the cntrl interpreters. The ideal prcedure seems t be t have three-man teams whse members are hetergeneus in prficiency using the immediate cnsensus wrk prcedure. In this prcedure, all interpreters first perfrm initial interpretatin n an individual basis and then get tgether t decide n the team reprt. The crucial factrs seem t be the individualized initial interpretatin and the pprtunity fr immediate feedback thrugh cmparisn f respnses and discussin with teammates.

18

19 TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK TECHNICAL SUPPLEMENT - 9

20

21 EXPERIMENTAL DESIGN AND RESULTS OF FOUR EXPERIMENTS ON TEAM CONSENSUS FEEDBACK Orientatin and Instructins METHOD In all fur experiments, the first half-day was taken up with rientatin and instructins. Fllwing the first presentatin f the instructins, the interpreters were given practice frames cntaining targets that culd easily be identified. Each f the practice data sheets was checked individually t determine if the interpreters understd the instructins. The instructins were then repeated fr any prcedure which had been misunderstd by ne f the interpreters. During this perid and subsequently thrughut the experiments, each interpreter was given a set f phtgraphic keys which cntained phtgraphs, scale drawings, and measurements fr each target n the target list. The phtgraphic keys als cntained vertical phtgraphs f each target in stere at a scale similar t thse used in the experiments. T insure that subjects were thrughly familiar with the keys, an rientatin test cnsisting f several frames cntaining anntated targets was given the subjects. These rientatin tests cntained ne r mre targets fr each type f target n the target list. T identify these targets, it was necessary t study the keys thrughly. Fllwing practice with the keys, subjects were given anther rientatin test cnsisting f several frames f imagery similar t that used during the main experiment t familiarize them with the target detectin prblems they wuld encunter during the pre-training test. During the instructin perid, n feedback f any kind was given with reference t any f the targets n the imagery. Pre-training Test This test was given individually t all interpreters in all experiments n the secnd half f the first day. Generally, the test cnsisted f 9"12 stere pairs cntaining a ttal f targets. The stere pairs were selected arbitrarily frm the pl f I50 stere pairs. An effrt was made t have all scales represented and t have frames with varying numbers f targets. The tests were different fr each experiment althugh there was sme verlap. Ten minutes were allwed fr cmpletin f each stere pair, and interpreters were started and stpped in unisn. In the event an interpreter finished prir t the end f the 10-minute perid, he was required t sit quietly until the next stere pair was started. N feedback was given during the test, and interpreters were nt permitted t talk abut the test during breaks nr t cmpare data sheets

22 Team Practice Team practice was cnducted during the secnd, third, and furth experimental days. The cntrl grup practiced as individuals n cpies f the imagery used by the teams. In Experiment I, all teams were required t start and stp each set f three stere pairs at the same time t cntrl the amunt f material cvered by each team. During Experiments II, III, and IV, the teams were allwed t prceed at their wn pace t cntrl the ttal amunt f time spent in practice. Sme f the wrk prcedures used by the teams were cmmn t tw r mre experiments and sme were unique. The prcedures were: 1. Serial Cnsensus--This prcedure was used in Experiments I, III, and IV. Each man perfrms initial interpretatin n a different stere pair. The men then rtate chairs, crrecting and adding targets t their teammates' wrk. The men then discuss all cnflicts. 2. Immediate Cnsensus--This prcedure was used in Experiments I, II, and IV. Each man perfrms initial interpretatin n a cpy f the same stere pair. The men then discuss all cnflicts. 3. Delayed Cnsensus--This prcedure was nly used in Experiment I and is the same as immediate cnsensus except that team members individually perfrm initial interpretatin n three stere pairs and then discuss all three at nce. 4. Jint-Search--This prcedure was nly used in Experiment III. Team members perfrm initial interpretatin tgether, pinting ut targets t each ther as they are fund. In ne variatin, the teammates discuss each target as it is fund; in the ther, the teammates wait until they have cmpleted the stere pair t discuss the targets. 5. Cntrl Prcedure--Each man perfrms the interpretatin by himself and des nt discuss his targets r cmpare respnse sheets with ther interpreters. The cntrl prcedure was the same in all experiments. Pst-training and Intermediate Tests A pst-training test was given in all experiments. This test was given n the fifth day and generally lasted mst f the day. The test cnsisted f 9-I7 stere pairs cntaining a ttal f targets. The nature f the pst-training test and the manner f administratin were similar t the pre-training test. In Experiment IV, an intermediate test was given during the middle f the third day. 12

23 Experimental Subjects Image interpreter trainees just graduated frm the image interpretatin curse at the U. S. Army Intelligence Schl, Frt Hlabird, Maryland, were the subjects fr the fur experiments. Experimental Imagery The imagery used in the experiments were aerial phtgraphs f Army field maneuvers. The phts were arranged in stere pairs; 150 f these pairs were spliced tgether in a 9" x 9" rll f psitive transparencies. There was a stere verlap f 40-60$ n each stere pair, and scales ranged frm 1:1000-1:5000. Each stere pair cntained frm 0-19 targets. Cnfidence Statement The interpreters in all fur experiments were required t place a cnfidence estimate ranging frm 0-100$ after each target. In Experiment I, cnfidence did nt enter int the scring, but in Experiments II, III, and IV the interpreters were tld that cnfidence wuld affect their scres. Cnfidence was nly used with reference t target detectin and did nt pertain t identificatin. Fr Experiments II and III, cnfidence estimates ranging frm $ were scred with a weight f +1 fr real targets and -1 fr false targets. Cnfidence estimates ranging frm 0-49$ were scred with a weight f +1/2 fr real targets and 0 fr false targets. In Experiment IV, an attempt was made t expand the weights and use the full scale f cnfidence as fllws: Cnfidence Real Targets False Targets Dependent Variables The three basic scres used were based n number f crrect identificatins, number f targets crrectly detected, and number f inventive errrs (false targets). All scres were weighted, the weights being based either n target pririty (Experiment i) r cnfidence estimates 13

24 (Experiments II, III, and IV). In Experiment I, ffensive tracked vehicles and artillery were given greater weight than ther targets. (This weighting scheme had little abslute effect and n differential effect n the results, and the scheme was drpped in later experiments.) In Experiments II, III, and IV, weights were based n the cnfidence estimate fr each target reprted. This weighting scheme als had little differential effect n the scres; hwever, it was retained in rder t encurage as much respnding as pssible. The identificatin scre was cmputed by adding up pints fr each target crrectly identified and subtracting a pint fr each target incrrectly identified. Targets detected but nt identified did nt enter int the identificatin scre. The crrect detectin scre was cmputed by summing pints fr each target n the target list crrectly detected. The inventive errr scre was cmputed by summing the pints fr all false targets reprted. In Experiment III, these last tw scres were cnverted t percentage scres as shwn belw t allw cmparisn between individual scres and team scre: Detectin cmpleteness = Detectin accuracy Sum f pints fr targets crrectly reprted x 100 Sum f pints fr all pssible targets Sum f pints fr targets crrectly reprted x 100 Sum f pints fr targets crrectly reprted + pints fr false targets reprted EXPERIMENT I. EFFECTS OF DIFFERENT TYPES OF FEEDBACK* The primary bjective f the first experiment was t determine the effectiveness f team cnsensus feedback fr imprving interpreter prficiency. Team cnsensus feedback was cmpared with precise feedback and n feedback. The experiment invlved six experimental cnditins: Team Cnsensus Feedback Variatins 1. Serial cnsensus feedback 2. Immediate cnsensus feedback 3. Delayed cnsensus feedback 4 A mre cmplete descriptin f this experiment may be fund in Cckrell, J. T. Maintaining image interpreter prficiency thrugh team cnsensus feedback. Technical Research Nte 195. April 1968 (AD ). Behavir and Systems Research Labratry. Arlingtn, VA. 14

25 Precise Feedback Variatins 4. Precise team feedback 5«Precise individual feedback Cntrl Grup 6. N feedback Experimental Design The six experimental grups were cmpared by means f difference scres measuring gain in individual prficiency ver pre-training perfrmance. Nine interpreters were assigned t each methd. The 54 interpreters were categrized as t prficiency n the basis f their initial perfrmance scres. A randmized-blcks analysis f variance was used t assess the significance f the differences between initial and final perfrmance scres acrss methds and prficiency grups. The nine interpreters assigned t each f the fur team methds were frmed int three-man teams f three types: high-medium-medium, highmedium-lw, and high-lw-lw. Assignment f men at each prficiency level t a team was n a randm basis. One high prficiency man was assigned t each team type n the hypthesis that less prficient interpreters wuld benefit frm wrking with mre prficient interpreters in the cnsensus feedback methds. Results fr Experiment I The main results shwing the effect f the three different types f feedback n interpreter prficiency are presented in Table 1. The scres are generally in the predicted directin; namely, the precise feedback methds shwed the greatest verall gain, the cnsensus feedback methd shwed intermediate gain, and the n-feedback methd shwed the least gain. Tables 2, 3, and 4 shw the three sets f detectin and identificatin scres arranged accrding t prficiency level, Table 5 shws the F-ratis fr each f the scres. Practice invlving feedback imprved nly target identificatin significantly and nt detectin. Further analysis f the identificatin scres (using Dunnett's test) indicated that perfrmance under the precise team feedback cnditin was significantly better than fr the nfeedback cnditin at the.01 level and that perfrmance under the precise individual feedback and serial cnsensus feedback cnditins was better than that under the n-feedback cnditin at the.05 level. Differences amng the varius team cnsensus and precise feedback cnditins were nt significant. 15

26 Table 1 MEAN DIFFERENCES BETWEEN PRE-TEST AND POST-TEST SCORES FOR THREE MEASURES OF IMAGE INTERPRETER PERFORMANCE--EXPERIMENT I Scres Crrect False Target Methd Identificatin Detectin Detectin* Precise Feedback Cnsensus Feedback Cntrl Psitive value represents a decrease in number f false target respnses. EXPERIMENT II. EFFECTS OF CONSENSUS FEEDBACK ON TARGET DETECTION 6 The primary bjective f the secnd experiment was t determine the effect f team cnsensus feedback n target detectin. In Experiment I, there was n significant difference between the cntrl grup and experimental grups n the detectin skill. It was felt that this may have been due t the lack f sufficient practice n this skill. Accrdingly, in Experiment II, the requirement fr target identificatin was eliminated, and interpreters were required nly t indicate the lcatin f each target and their cnfidence that it was truly a target. Each team in Experiment II cmpleted apprximately fur times as many practice stere pairs as the teams in Experiment I and thereby presumably increased amunt f detectin practice. 5 A mre cmplete descriptin f this experiment may be fund in Cckrell, J. T. Maintaining Target Detectin Prficiency Thrugh Team Cnsensus Feedback. Technical Research Nte 219. Octber I969 (AD ). Behavir and Systems Research Labratry. Arlingtn, VA. 16

27 w T3 I O -" X! < U en ih in CO v m CM O OO 00 CM v vo vo CT> vo vo ON c CM CO ON uo 00 O W)X1 T3 CD fa a) CM 00 r-h CO r-l vo vo VO vo <u a r-. r-» vo vo eg X> H 3 in CO -I in -jr«. 3 O e CO CM H > VO CM vo 00 i-l IT) vo CM <r c M fi P^ r~ CO CM «<u r^ r- r^ <t H CO LO vo U0.M «u 3 (fl T3 O CO i-i XI rj T3 > i-l c 00 CD r- p^ <D fa c M CD W 1-1 y fi O r»- a» «CD m vt > H r-» vo r». vo u a CD ' i-l U VH, u fa X! -H 60 "O Ü 1) > 3 >1 u C <D T-< 1-4 CD u > H fa ja 00 % <D > CD - 17

28 CO T3.-I O < «CO CM CM ON CO CM X I-l crj Ü XI M 4J CD c (1) Ü In O CM u 0 z -a 4J a c H CO M H CD >S CO i-l CD p I»«CO CM CO CM I*«c CO CM 40 CO H Cfl r^ I s» CO H J- m 00 CD!-) c CO CM W ^ CO CJ 3 CO T3 43 H T3 > CD -,-< a) In (D CO H a H s CD CO H CD S H CO 00 CNJ CM CO CM CM uo CM CM 8 0) F-l r^ CD U > i-l CD "+H l-j O H 4= 00 I i-l -a CD CD > CD i-l

29 Table 5 F-RATIOS FOR DIFI RENCE SCORES--EXPERIMENT I Scre Identifica- Crrect False Target Surce df tin Detectin Detectin Methds (M) * Prficiency (P) MP Within (Mean Square) Ttal 53 >p < J05. Experimental Design In Experiment II, in additin t a detailed study f the effects f team cnsensus feedback n target detectin, fur ther factrs were studied: 1) team size (2-man r 3-man),2) team cmpsitin (team members hmgeneus in initial ability versus hetergeneus), 3) discussin versus n discussin, and 4) interpreter prficiency (high, medium, and lw). The design used fr this experiment was a 2x2x2x3 factrial with a cntrl grup. The detailed breakdwn is shwn in Table 6. Perfrmance amng the eight experimental grups and the n-feedback (cntrl) grup, shwn in the clumns in Table 6, was cmpared by means f difference scres btained by subtracting pre-training test scres frm pst-training test scres. A randmized-blcks analysis f variance was used t assess the significance f differences amng the experimental cnditins. The three blcks were based n initial prficiency as measured by the pre-training test. Frm each blck, tw men were drawn randmly fr each f the experimental grups. The six interpreters assigned t each f the eight experimental grups were assigned t hmgeneus r hetergeneus teams and t tw-man r three-man teams as shwn in Table 7. The cntrl grup included 12 interpreters. 19

30 t-4 c u - <r «* es t U O H 00 H v vo vo 00 i-i i-i i-i «a- CN CN CN VO en 4J u 3 CN CN CN VO vo u s a X w u ^ CN CN CN vo CO H O»H M <U W O w Q nd CN CN CN VO U ij CO -< w I CO Pi U K X 0) W VJ a u 9) c CN II en R CO a) H c 2 I m c H 0 a en 0 en as 3 en X H Q BC C H H s en en 3 e> en s H O Q pd CN CN CN VO CN CN CN vo CN CN CN v CN CN CN VO c QJ (X H 3 O O H h 14-1 CJ O >- D-, J2 I -H CO ^ aj 5 as s J H 20

31 Table 7 COMPOSITION BY PROFICIENCY OF TEAMS IN EXPERIMENT II Team Type 3-Man Teams 2-Man Teams Hmgeneus Hetergeneus 1. High, High, Medium 2. Medium, Lw, Lw 1. High, Medium, Lw 2. High, Medium, Lw 1. High, High 2. Medium, Medium 3. Lw, Lw 1. High, Lw 2. High, Medium 3. Med ium, Lw Team Prcedures Team prcedures used in Experiment II were tw variatins n the immediate cnsensus prcedure. Where discussin was used, the prcedure was the same as in Experiment I. Where discussin was nt used, the teammates merely passed their cmpleted data frms t each ther s that cmparisns culd be made. Results fr Experiment II The results fr Experiment II were analyzed in tw steps. The first step cmpared the differences amng the varius experimental factrs; the secnd step cmpared the team cnsensus feedback grups t the cntrl gr.up. Table 8 shws the mean perfrmance gains fr the varius factrs in terms f scres based n number f crrect detectins and number f inventive errrs. This table shws a substantial difference in favr f hetergeneus teams fr bth scres, and a large reductin in the number f inventive errrs by lw and medium prficiency interpreters. Table 9 presents the analysis f variance results fr the fur factrs cnsidered. Team size and discussin versus n-discussin cmparisns shwed n significant perfrmance increases. Interpreter prficiency (high, medium, and lw) shwed a significant difference in the reductin f inventive errrs; lw prficiency interpreters shwed a marked reductin in inventive errrs, while mre prficient interpreters 21

32 shwed less marked imprvement. The perfrmance f hetergeneus teams imprved significantly mre than that f hmgeneus teams bth in number f targets crrectly detected and number f inventive errrs. Frm these last tw sets f results, there is the indicatin that lw prficiency interpreters gain frm the team cnsensus feedback methd and that they benefit frm their assciatin with a mre prficient team member. In rder t analyze the difference between the n-feedback cntrl grup and the cnsensus feedback experimental grups, the discussin and team-size factrs were cllapsed, leaving nly team type and prficiency factrs. Table 10 shws the mean difference scres fr this analysis in bth detectin scres. Table 11 shws the analysis f variance fr grups with unequal numbers f subjects. Team cnsensus feedback training resulted in a significant reductin in the number f inventive errrs, when cmpared with perfrmance f the cntrl grup. Team cnsensus feedback, hwever, did nt imprve detectin perfrmance in terms f number f targets detected. The results in Table 11 als repeat the finding f significant gains in errr reductin fr the lw prficiency interpreters. EXPERIMENT III. JOINT SEARCH PROCEDURES TO MINIMIZE DELAY IN CONSENSUS FEEDBACK In Experiment III, tw new teamwrk prcedures were designed t minimize the time delay between initial interpretatin and cnsensus feedback. These prcedures used jint target search rather than individual search. A reductin in feedback time delay was felt t be imprtant in view f the results frm Experiment I with delayed cnsensus feedback. Team Methds The tw-man team prcedures were as fllws: Jint Search -- One-Respnse Delay. Interpreters search tgether and discuss each target as it is detected. Jint Search -- One-Frame Delay. Interpreters search tgether but discuss all targets at end f frame. Serial Search -- Tw-Frame Delay. Interpreters search individually n different frames, crss-check by swapping seats, then discuss each frame in turn. Independent Search -- N Feedback. In this cntrl grup, interpreters search individually with n feedback at any time. During team practice sessins, ne half the teams were required t make a grss identificatin f all targets (e.g., tracked vehicle, truck, shelter). This grss identificatin was nt required during the pretraining and pst-training tests. The purpse f the identificatin 22 -

33 Table 8 MEAN GAINS BETWEEN PRE-TEST AND POST-TEST IN DETECTION SCORES--EXPERIMENT II Experimental Crrect False Target Factrs Detectin Detectin Team Type Hetergeneus 66.4* 9.7* Hmgeneus Discussin Yes N Team Size 3-Man Man Initial Prficiency High ** Medium Lw ** P <.05. P <

34 Table 9 DIFFERENCE SCORE F-RATIOS FOR DETECTION SCORES--EXPERIMENT II Surce df Team Type (T) 1 Prficiency (P) 2 Discussin (D) 1 Team Size (S) 1 TP 2 TD 1 TS 1 PD 2 PS 2 DS 1 TPD 1 TPS 2 TDS 1 PDS 2 TPDS 2 Crrect Detectin F- Ratis False Target Detectin 4.34* 6.89* ** * Within (Mean Square) Ttal 47 P <.06. p <

35 Table 10 COMPARISONS BETWEEN EXPERIMENTAL GROUPS AND CONTROL GROUPS FOR MEAN GAINS IN DETECTION SCORES--EXPERIMENT II Crrect Detectin Prficiency Hetergeneus Hmgeneus Grup Teams Teams Cntrl High Medium Lw All Grups False Target Detectin* Hetergeneus Hmgeneus Teams Teams Cntrl High Medium Lw All Grups Psitive scre represents decrease In the number f false target respnses. Table 11 F-RATI0S FOR COMPARISON BETWEEN COLLAPSED EXPERIMENTAL GROUPS AND CONTROL GROUPS IN DETECTION SCORES--EXPERIMENT II Surce Methd (M) Prfic iency (P) M x P Within (Mean Square) Ttal df Crrect Detectin F- Ratis False Target Detectin ** * P <.06. p <

36 requirement was t determine if team utput wuld be affected in terms f amunt and accuracy. A cunt f the number f stere pairs cmpleted during the teamwrk sessins shwed that the identificatin requirement had n effect n amunt f interpretatin accmplished. Experimental Design Fr Experiment III, the variables studied were team methd, requirement fr identificatin (required versus nt required), and initial prficiency (high versus lw). The experimental design is a 4x2x2 factrial with pre-pst test difference scres as dependent variable. Table 12 shws a detailed breakdwn. Results fr Experiment III While it culd be expected frm the ratinale fr the team cnsensus feedback methd that variatins prviding mre immediate feedback--such as the jint-search methds--wuld result in imprved perfrmance, this ntin was nt brne ut by the results. Tables 15 and 14 shw the pretest, pst-test, and difference percentage scres based n number f crrect detectins and number f inventive errrs, respectively. Difference scres between initial and final prficiency--in terms f mean percentage gain--were used fr the analysis f means, the results f which are shwn in Table 15. There were n significant differences fr feedback methd, i.e., n demnstrated superirity f the mre immediate feedback methds ver ther feedback methds. The nly significant difference fund was fr initial prficiency; lw prficiency interpreters shwed significant reductin in the number f inventive errrs made. EXPERIMENT IV. NATURE OF LEARNING DURING TEAM CONSENSUS FEEDBACK The purpse f Experiment IV was t determine the nature f the learning which ccurs during the teamwrk sessins, and t sme extent, the shape f the learning curve. This experiment differed frm the preceding experiments in that it permitted measurement f amunt f learning independent f test difficulty. Three prficiency tests were used which were cunterbalanced t eliminate the effects f test difficulty. The prcedures and team cmpsitin which had been fund t be best in previus experiments were used -- three-man teams, hetergeneus teams in terms f initial ability, and discussin as team members wished. The wrk prcedures were serial cnsensus and immediate cnsensus plus a cntrl grup. 26

37 O H <r <r O» CN «tf 42 u VJ y-s CO OJ a 9 M CO ü cj M CO u CD 43 d X) a; ih <u n O Cl) c W b a) u IX C <u O z -a U c «H CO Cl vo CO r vo 42 u IJ Cl) CO e a) CO > CO u CO fc i-t.-h i <u CO <U CO n 5 m n v CO CO VO CO H a> Q H M CO >- (j en O i-i a i CD a) Q c H r f) v p") c v <U en c a p> en CO <u ^ «! a) i a a) ä m c v c en \ u c <u a -i 3 H H 14-1 H 42 Cd H O O SC HJ H 27

38 Table 13 MEAN PERCENTAGE SCORES FOR DETECTION COMPLETENESS ON PRE-TEST AND POST-TEST--EXPERIMENT III Prf : iciency Tests Factrs Initial Test Final Test Difference Team Methd Jint-Search-One Respnse Jint-Search-One Pair Serial Search-Tw Pairs Independent Search Identificatin Yes N Initial Prficiency High Lw

39 Table 14 MEAN PERCENTAGE SCORES FOR DETECTION ACCURACY ON PRE-TEST AND POST-TEST--EXPERIMENT III Prficiency Tests Factrs Initial Test Final Test Difference Team Methd Jint-Search-One Respnse Jint-Search-One Pair Serial Search Independent Search Identificatin Yes N Initial Prficiency High Lw

40 Table 15 F-RATIOS FOR DIFFERENCE SCORES BETWEEN PRE-TESTS AND POST-TEST--EXPERIMENT III Detectin Detectin Surce df Cmpleteness Accuracy Team Methd (M) Identificatin (I) Initial Prficiency (P) ** MI MP ,32 IP MIP Within (Mean Square) Ttal 47»P <

41 Experimental Design Each subject was given ne f the three individual perfrmance tests as a pre-training test. Based n scres n this test, interpreters were divided int three prficiency blcks -- high, medium, and lw. One man was randmly drawn frm each blck t frm three-man teams and these teams were assigned randmly t ne f the three wrk prcedures. Table 16 shws the design fr the experiment. Results Results fr Experiment IV are summarized in Table 17 In general, imprved perfrmance was fund acrss the three sessins, including that fr the n-feedback (cntrl) grup. Hwever, the tw cnsensus feedback grups shwed greater learning than the cntrl grup n all three perfrmance measures -- identificatin, crrect target detectin, and inventive errrs. Thus, the tw cnsensus feedback methds were effective in imprving interpreter prficiency. Figures 1, 2, and 3 present a breakdwn f results by prficiency level. Fr all three perfrmance variables, the greatest perfrmance gains were achieved by lw prficiency interpreters. Medium prficiency interpreters gained less, and high prficiency interpreters gained little mre than interpreters in the cntrl (n feedback) cnditin. Table 18 shws the results f the analysis f variance fr factrs f primary experimental cncern -- feedback methd, initial prficiency, and sessins -- as well as fr rder and test. The analysis f variance was carried ut n perfrmance scre (nt perfrmance gain as in prir experiments) fr identificatin, crrect detectins, and inventive errrs. The results presented in Table 18 shw clearly that (initial) prficiency is an imprtant factr, the lw prficiency interpreters benefiting mst frm team cnsensus feedback training. The results als shw that learning ccurred, as evidenced by the significant sessins effect. The apparent lack f significant results fr methd (except fr the crrect detectin scre) requires a wrd f explanatin. As subjects were assigned t experimental methds n the basis f initial test scres, differences amng methds acrss the three tests (pre-test, intermediate test, and pst-test) wuld be reduced. Fr this reasn, additinal analyses were carried ut t lk in mre detail at differences amng feedback methds. The Newman-Keuls test (6) was used t cmpare the tw feedback methds against the cntrl grup and against each ther fr each f the three tests separately as shwn in Table 19. The applicatin f the Newman-Keuls test t scres n the pre-test shws that, as desired, there were n significant differences amng the the tw feedback cnditins and 6 Winer, B. J. Statistical Principles in Experimental Design, McGraw- Hill Bk Cmpany, New Yrk,

42 Table 16 EXPERIMENTAL ARRANGEMENT FOR EXPERIMENT IV (N = 54 Subjects, 2 Per Cell) Methd Order Prficiency Sessins H M L Test A Test B Test C Immediate Cnsensus II H M L B III H M L H M L B Serial Cnsensus II H M L B III H M L H M L Cntrl II H M L B III H M L B

43 the cntrl cnditin and between the feedback cnditins, since feedback was nt intrduced in the pre-test cnditin. Hwever, Table 19 des shw significant differences between bth feedback grups and the cntrl grup in bth identificatin and crrect detectins in subsequent sessins. N significant differences were fund fr inventive errrs. These results, then, again shw that team cnsensus feedback is effective in achieving increased interpretatin prficiency, particularly in target identificatin. Table 17 MEAN SCORES FOR IDENTIFICATION, CORRECT DETECTION, AND INVENTIVE ERRORS ACROSS TEST ADMINISTRATIONS EXPERIMENT IV Dependent Test Administratin Variable Methd Identificatin Immediate Cnsensus Serial Cnsensus Cntrl Average Crrect Detectin Immediate Cnsensus Serial Cnsensus Cntrl 83a Average Inventive Errrs Immediate Cnsensus Serial Cnsensus Cntrl Average

44 Table 18 ANALYSIS OF VARIANCE OF IDENTIFICATION, POSITIVE DETECTION, AND FALSE TARGET DETECTION--EXPERIMENT IV F-Rat is Crrect False Target Surce f Variance df Identificatin Detectin Detectin Between Subjects Methd (M) * 1.79 Order (0) ** 5.90** 5.23* Prficiency (P) ** 15.83* 4.05* MO MP OP MOP Subjects Within Grups (Mean Sq.) Within Subjects Sessins (S) ** * Test (T) MS * 4.39** 3.07* MT ** PS ,94** PT MPS MPT Residual * Errr (Mean Sq.) Ttal 108 *P <.05. P <

45 IDENTIFICATION SCORE uj 100 OE O O p 80 < 2 in 60 t HIGH -S 2 3 SESSIONS 9 0 SERIAL CONSENSUS IMMEDIATE CONSENSUS & CONTROL ui g 80 in 7- MEDIUM I 2 3 SESSIONS r V) O < LOW ^ I 2 3 SESSIONS Figure 1. Identificatin learning curves fr three methds used by interpreters f high, medium, and lw prficiency. - 55

46 CORRECT DETECTION SCORE 140 tr O in z tu 100 ^.t HIGH SESSIONS SERIAL CONSENSUS IMMEDIATE CONSENSUS A A CONTROL 120 UJ <r O w 100 z MEDIUM L; 6.? 2 3 SESSIONS 120 LOW 100 z p UJ 80 A CO t I 2 SESSIONS Figure 2. Detectin skill learning curves fr three methds used by interpreters f high, medium, and lw prficiency 36

47 INVENTIVE ERROR SCORE 25 cr 20 c e w 15 > z' > HIGH SERIAL CONSENSUS IMMEDIATE CONSENSUS -- A CONTROL 25 c 20 c a: UJ > Z 5 MEDIUM i 2 3 SESSIONS *: - k - -Li -^ c c 15 u I > z l0 5 - LOW SESSIONS I 2 SFSSIONS Figure 3. Inventive errr learning curves fr three methds used by interpreters f high, medium, and lw prficiency. 37 -

48 Table 19 P-VALUES FOR THE NEWMAN KEULS-TEST FOR THE THREE SESSIONS SEPARATELY FOR ALL SCORES--EXPERIMENT IV Scre Cmparisn IC vs Cntrl Identificatin SC vs Cntrl SC vs IC Sessin ns ns ns ns ns IC vs Cntrl Crrect Detectin SC vs Cntrl SC vs IC ns ns.05 ns ns.05 ns False Target Detectin IC vs Cntrl SC vs Cntrl SC vs IC ns ns ns ns ns ns ns ns ns 38 -

49 APPENDIXES Page Appendix A. Target List 43 B. Data Sheet 44-39

50

51 APPENDIX A TARGET LIST T TRACKED VEHICLES TT Tanks TS SP (Guns, Hwitzers, Mrtars, Anti-aircraft) TA APC's TB Armred Bridge Launchers TR Recvery Vehicles TP Prime Mver/Tractr A ARTILLERY AT Twed Hwitzers AM Mrtar AA Anti-aircraft AK Anti-tank M MISSILES MS Surface-t-Surface Missile ML Missile Launcher/Transprter MT Missile Transprter MA SAM MM SAM Launcher/Transprter W WHEELED AND CONSTRUCTION VEHICLES WL Light Carg Trucks, 1/4 Tn, 5/4 Tn, Ambulance WH Heavy Carg Trucks, 2-1/2 Tn, 5 Tn, 10 Tn WK Tank Trucks (Water, Fuel) WW Wrecker Trucks WT Truck Tractr (List Separate frm Trailer) WV Van Trucks (Generatr, Shp, Cmmunicatin, Radar) WD Dump Truck WC Cnstructin Vehicles (Bulldzers, Cranes, Shvels, Scps, etc.) L TRAILERS (ANNOTATE SEPARATELY FROM TRUCKS EVEN IF ATTACHED) LL Light Carg, 1/4 Tn, 3/4 Tn LH Heavy Carg, 1-1/2 Tn LS Small Special Purpse (Amm, Generatr, Water, Fuel) LR Large Special Purpse (L By, Tank Transprter, Van, Tanker) LE Huse Trailers (Military) C CANVAS SHELTER CS Small Persnnel Tents (Pup, Wall) CM Medium Special Purpse Tents (CP, Hex, Kitchen) CL Large Tents (GP, Maintenance, Hspital) CC Miscellaneus (Latrine, Canvas Shelter, Canvas Water Tank, Canvas Cvered Supplies, Canvas Cvered Garbage Pits, Flys) 41

52

53 - 45 -

54

55 DISTRIBUTION BESRL Distributin List l OASDIM&RA) 1 HQDA(DACS-ZC) *1 HQDA (DAPE-MPC) 1 HQDA (DAPE-CPS) *1 HQDA (DAPE-DW) 2 HQDA (DARD-ARZ-B) 1 HQDA (DARDARB) *1 HQDA (DARD-ARP) 2 HQDA (DARD-OT) 1 HQDA (DARD-IST-P) 5 HQDA (DAFD-DC) 1 HQDA (DAFD-ZBD) 1 HQDA (DAFD-MF) 1 HQDA (DAFD-CN) 1 HQDA (DAMI DCS) 5 HQDA(DAPQPM) 1 HQDA (DAIO-ZC) 1 HQDA (DAAG-PSS-A) 1 HQDA (DAAG-ASL) 1 HQDA (DAPM-MSB) *1 HQDA (DAPM-PL) *1 HQDA (SGRD-MDB) 3 HQDAJEPEEC» 4 HQDA(RDSE) 1 HQDA(RDRE) 1 HQDA(RDRD) *1 HQ USAMC, ATTN: AMCRD-TE 2 HQ USCONARC, ATTN: GI-ATPER 10 HQ USCONARC, ATTN: ATIT-STM 2 HQ First US Army 2 HQ Third US Army 2 HQ Fifth US Army 2 HQ Sixth US Army 1 HQ Sixth US Army, ATTN: Library 6 HQ USACDC, ATTN: DCS/Opns QAD 1 HQ USACDC, ATTN: Library 1 HQ USASA, ATTN: IARD-T *1 HQ USA Cmputer SysCmd, ATTN: CSCS-TMT-SD *1 HQ USA Recruiting Cmd, ATTN: USARCO 1 Dir, Prject MASSTE R, Ft Hd *1 USA Electrnics Cmd, ATTN: AMSEL-CT-A *1 USA Electrnics Cmd, ATTN: AMSEL-CB-AS *1 USA Electrnics Cmd, ATTN: AMSEL-PA-AH 1 USACDC Exper Cmd, Ft Ord, ATTN: Plans Officer *1 USACDC Exper Cmd, Ft Ord, ATTN: Library *1 USACDC, Cbt Sys Gp, ATTN: CSYGC-S *1 USACDC Institute f Land Cmbat. ATTN: Library 1 USACDC Armr Agency 1 USACDC Artillery Agency 1 USACDC Aviatin Agency 1 USACDC Infantry Agency, ATTN: CAGIN-OA 1 USACDC Infantry Agency, ATTN: Central Files 1 USACDC SW Agency 1 USACDC Pers & Adm Svcs Agency 1 USACDC Liaisn Ofc, USAF, Tact Air Warfare Cen, Eglin AFB 1 USA Adv Mat Cncepts Agency, ATTN: AMXAM-TS 1 WRAIR, ATTN: Neurpsychiatry Div. *1 WRAIR, ATTN: Dept f Exper Psych-physilgy *1 USA Med Res Lab, Ft Knx, ATTN: Exper Psychl *1 USA Natick Labs, ATTN: Library 1 USA Natick Labs, ATTN: Psychl Labs 1 USA Natick Labs, ATTN: Assc Dir, Beh Sei, Pin Div *1 USA Mbility Eq. R&D Cen, ATTN: HFE *1 USA Mbility Eq. R&D Cen, ATTN: Library *1 USA Electrnic PrvgGrd, ATTN: STEEP-SU-I *1 USA Electrnic Prvg Grd, ATTN: STEEP-T-BI 2 USA Aberdeen Prvg Grd, ATTN: Dir, HEL *1 USA Aberdeen Prvg Grd, ATTN: HEL, R. A. Mnty *1 USA Edgewd Arsenal, ATTN: Hum Factrs Gp, Bimed Lab 1 USA Trpic Test Cen, Ft Claytn, ATTN: STETC-PD-M 2 USA Arctic Test Cen, Alaska *1 USA Gen Eq. Test Activ., Ft Lee, ATTN: HF Div, MED *1 USA Avn Mat Lab, ATTN: Library *1 EWL, Ft Hlabird, ATTN: Intel Mat. Dev. Off. *2 USA Garrisn, Ft Huachuca, ATTN: Tech Ref Off 1 USA C&GSC, ATTN: Educ Adv 6 USA C&GSC, ATTN: Library 1 USA C&GSC, ATTN: Beh Sei Rep, Dir Grad Stud & Res 1 USA C&GSC, ATTN: DOI "2 USMA, Mil Psych & Ldrshp *1 USMA, Dir f Institutinal Res 1 USMA, Library 1 USA Ordn Cen & Sch, Aberdeen PG, ATTN: DOI 1 USA Ordn Cen & Sch. Aberdeen PG, ATTN: ATSOR-1R 2 USA Engr Sch, Ft Belvir, ATTN: DOI *2 USA Mgt Sch, Ft Belvir 1 USA Inf Sch, Ft Benning, ATTN: DOI *1 USA Inf Sch, Ft Benning, ATTN: AJIGP *1 USA Inf Sch, Ft Benning, ATTN: AJIIS-H 1 USA Inf Sch, Ft Benning, ATTN: Library 1 USA Inf Bd, Ft Benning, ATTN: Secy ATBD 1 USA Air Def Sch, Ft Bliss, ATTN: DOI *2 USA Air Def Sch, Ft Bliss. ATTN: Library 1 USA Air Def Sch, Ft Bliss, ATTN: ATSAD-DL-C *1 USA Air Def Bd, Ft Bliss 1 USA Inst Mil Asst, Ft Bragg, ATTN: DOI 1 USA Inst Mil Asst, Ft Bragg. ATTN: ATSMA-ID 1 USASA Tng Cen & Sch, Ft Devens 2 USA Transprt Sch, Ft Eustis, ATTN: DOI 2 USA MP Sch, Ft Grdn, ATTN: DOI 'Des nt receive reprts fr all research areas

56 USA SE Sig Seh, Ft Grdn, ATTN: DOI USA SE Sig Seh, Ft Grdn, ATTN: Ed Advsr USA Chaplains Seh, Ft Hamiltn, ATTN: DOI USA AG Seh, Ft Benjamin Harrisn, ATTN: DOI USA Fin Crps Seh, Ft Benjamin Harrisn, ATTN: DOI USA Med Fld Svc Seh, Ft Sam Hustn, ATTN: Stimsn Library USA Cbt Survl Seh Tng Cen, Ft Huachuca, ATTN: DOI USA Int Cen & Seh, Ft Huachuca, ATTN: DOI USA Int Cen & Seh, Ft Huachuca, ATTN: AHBH-S-CB-(ASRC) USA Int Cen & Seh, Ft Huachuca, ATTN: Library USA Armr Seh, Ft Knx, ATTN: DOI USA Armr Seh, Ft Knx, ATTN: AHBAAS-DMP USA QM Seh, Ft Lee USA Chem Crps Seh, Ft McCIellan USWAC Seh, Ft McCIellan, ATTN: DOI USA Sig Cen & Seh, Ft Mnmuth, ATTN: DOI USA Sig Cen & Seh, Ft Mnmuth, ATTN: Educ Advsr USA Msl & Mun Cen & Seh, Redstne Arsnl, ATTN: DOI USA Msl & Mun Cen & Seh, Redstne Arsnl, ATTN: Lib USA Avn Seh, Ft Rucker, ATTN: DOI USA Avn Seh, Ft Rucker, ATTN: Library USA Avn Test Bd, Ft Rucker. ATTN: STEBG-TP-V USA Fid Arty Seh, Ft Sill, ATTN: DOI (Ed Svcs Div) USA Prim Hei CerySch, Ft Wlters, ATTN: DOI ONR, Cde 458 US Navy, BuPers (Pers A3) US Navy, BuM&S, Cde 513 US Navy, BuM&S, Cde 713 Nav Pers Tng Rsch Lab, San Dieg, ATTN: Library Nav Pers Tng Rsch Lab, San Dieg, ATTN: Stat Dept Nav Pers Tng Rsch Lab, San Dieg, ATTN: Pers Meas Rsch Dept Nav Pers R&D Lab, Washingtn Nav Med NA Rsch Unit, San Dieg Nav Rsch Lab, Library, Cde 2029 Nav Rsch Lab, Cde 2027 Nav Tng Dev Cen, Orland, Cde 55 Nav Aerspace Med Inst, Pensacla, Cde 31 Nav Pstgraduate Seh, Mnterey, ATTN: Library HQUSMC, ATTN: ADIB HQUSAF (AF/DPXYA) HO USAF (INYMA) HQAFSC/DLSL *3 AFHRL (PD), Lackland AFB *1 CINCPACAF (IGSAG) 2 US Armed Frces Institute *1 American Embassy NATO Internatinal Staff *1 US Dept f Agriculture, Pers Rsch Staff *1 US Dept f Labr, Inf Cen fr Mpr Rsch 2 US Cast Guard Hq, C/Psychl Rsch *1 US Cast Guard Acad., Directr f Admissins *1 US Cast Guard Tng Cen., Gvernrs Island, Educ Advis. *1 Natinal Security Agency, ATTN: M13 *1 Central Intelligence Agency, ATTN: OCR/ADD/Publicatins *1 Nat Cen fr Child Health and Human Dev 1 NIMHOCNCMHI *1 FASEB, Life Sciences Rsch Ofc *1 BLM, Denver 2 HumRRO, Alex., ATTN Library 2 HumRRO Div 2, Ft Knx 1 HumRRO Div 3, Presidi f Mnterey 1 HumRRO Div 4, Ft Benning *1 HumRRO Div 5, Ft Bliss 1 HumRRO Div 6, Ft Rucker 1 Inst. fr Def Anal., Arlingtn 1 Rsch Anal Crp, McLean, ATTN: Library 1 RAND Crp, Santa Mnica, ATTN: Library 98D *1 Cen fr Nav Anal, Arlingtn, ATTN: Dcument Center *1 AIR CRESS, Kensingtn, ATTN: Inf Sys Br 1 Hum Engr Inf & Anal Svc, Tufts Univ *1 Bilgical Abstracts 1 Univ Illinis, Aviatin Rsch Lab *1 Human Factrs Rsch Inc. Gleta 1 Littn Sei Sup Lab, Ft Ord, Behaviral Sciences 140 Library f Cngress, Unit Dc Exped Prj 6 Library f Cngress, Exchange & Gift Div 12 Defense Dcumentatin Center *4 Dir Psych, Army Hq, Russell Ofcs, Canberra 1 Scientific Advsr, Mil Bd, Army Hq, Russell Ofcs, Canberra 1 Mil and Air Attache, Austrian Embassy *1 Centre de Recherche Des Facteurs, Humaine de la Defense Natinale, Brussels 4 British Defense Staffs Washingtn, ATTN: Scientific Officer 2 Canadian Jint Staff Washingtn 3 C/Canadian Defense Rsch Staff, C/CRDS(W) 1 C/Air Staff, Ryal Canadian AF, ATTN: Pers Std Anal Br *1 Def & Civ Inst f Envirn Med, Canada "1 Militaerpsyklgisk Tjeneste, Cpenhagen *1 Military Attache, French Embassy. ATTN: Dc Sec *1 Medecin Chef, C.E.R.P.A.-Arsenal, Tuln/Naval France 1 Prin Scientific Off, Appl Hum Engr Rsch Div, Ministry f Defense, New Delhi 1 Pers Rsch Ofc Library, AKA Irgun, Israel Defense Frces *1 Ministerie van Defensie, Oper Officer Persneel KL, Afd Sciaal Psychlgische Zaken, The Hague, Netherlands 'Des nt receive reprts fr all research areas. k6 -

57 Unclassified Security Clarificatin DOCUMENT CONTROL DATA -R&D (Security clmttllicmtin f fit/», bdy f mbmttmel and Indmxlnj annfaffn mum I b»»nimmt whmn Ihm»wall rmptt It elm filledj I. ORIGINATING ACTIVITY (Crprmlm mulhr) 2a. REPORT SECURITY CLASSIFICATION System Develpment Crpratin, Falls Church, Virginia Unclassified and Behavir and Systems Research Labratry, 26. CROUP Arlingtn, Virginia. 3. REPORT TITLE TRAINING INDIVIDUAL IMAGE INTERPRETERS USING TEAM CONSENSUS FEEDBACK 4. DESCRIPTIVE NO T ES (Typt f rmprl mnd lnclu.lv, dmtem) S. AUTHORIS) (Flrtl nam. mlddlm Initial, /«./ nmmm) Jhn T. Cckrell (SDC) and Rbert Sadacca (BESRL) 6. REPORT OATE June 1971 Sa. CONTRACT OR 6RANT NO. DAHC C-0040 b. PROJECT NO. DA R&D PJ N. 2Q662704A721 Surveillance Systems * e DISTRIBUTION STATEMENT 7a. TOTAL NO. OF PASES 54»a. ORIGINATOR'S REPORT NUMBER«) Apprved fr public release; distributin unlimited. 7b. NO. OF REFS 6 Technical Research Reprt II7I b. OTHER REPORT NOW (Any ft«niattn Him I may ba aaaffviad Oil» rmprl) II. SUPPLEMENTARY NOTES 12. SPONSORING MILITARY ACTIVITY Asst Chief f Staff fr Intelligence, Wash. D. C. Office, Chief f Research and Develpment, DA, Wash., D. C. IS. ABSTRACT In a series f experiments cnducted ver the past several years, the ADVANCED SURVEILLANCE SYSTEMS Wrk Unit f the Behavir and Systems Research Labratry (BESRL) has sught t develp and test the team cnsensus feedback methd as a technique fr maintaining and enhancing the prficiency f individual image interpreters. In the present research, cnducted jintly by the BESRL unit and persnnel f the System Develpment Crpratin, fur experiments were undertaken t explre varius team cnsensus feedback methds with varying cnditins f team size and cmpsitin. Specifically, the experiments were directed t develping team practice methds which wuld lead t the greatest perfrmance gains by individual interpreters in an advanced interpretatin system. The first experiment was designed t btain a general assessment f the usefulness f the cnsensus feedback prcess in bth target detectin and target identificatin. The secnd experiment sught t determine if team cnsensus feedback wuld be effective in enhancing target detectin perfrmance nly. Experiment III evaluated mdified team cnsensus feedback techniques designed t minimize feedback delays. Experiment IV was cncerned mainly with the nature f learning curves using team cnsensus feedback techniques and with evaluatin f the mst effective prcedures Identified in the first three experiments. Team cnsensus feedback was demnstrated t be an effective methd f maintaining and imprving image interpreter prficiency in an peratinal image interpretatin facility. Greatest perfrmance gains were made in target identificatin. Results DD "OftM Unclassified Security Classificatin

58 Unclassified Security Classificatin KEY WORD* Surveillance systems research Image Interpreter skills Feedback techniques Target detectin Cnsensus Target identificatin Experimentatin Image interpreter prficiency Team cnsensus feedback methd Team discussin Team size Team cmpsitin Operatinal image interpretatin facility Tactical imagery Intelligence infrmatin Military psychlgy Labratry facilities 48 - Unclassified Security Classificatin

59 DD Frm ABSTRACT cntinued shwed that interpreters wh were initially lw in prficiency achieved the mst significant gains in perfrmance. Lw prficiency interpreters learned frm, r benefited thrugh their cllabratin with, higher prficiency interpreters. Neither team discussin nr team size significantly affected interpreter perfrmance; hwever, these variables did cmbine t prduce an verall effect. The ideal prcedure seems t be ne using three-man teams, hetergeneus in terms f initial prficiency, in which individuals first perfrm initial interpretatin n an individual basis and then discuss their identificatins freely in deciding n the team reprt. 49 -

60 U1503