KTu DISTRIBUTED BY: National Technical Information Service U. S. DEPARTMENT OF COMMERCE 5285 Port Royal Road, Springfield Va.

Transcription

1 ' H 11 " ' " AD RESEARCH ON THE TECHNOLOGY OF NFERENCE AND DECSON Ward Edwards Mchgan Unversty Prepared for: Offce of Naval Research Advanced Research Projects Agenjy 30 November 1973 DSTRBUTED BY: KTu Natonal Techncal nformaton Servce U. S. DEPARTMENT OF COMMERCE 5285 Port Royal Road, Sprngfeld Va ^M>MaM ~J

2 ß Wrv '''J' """"'«."{y' "" > 'j V- 1."!'^l' l. l ", opnp F T THE UNVERSTY ÖF MCHGAN H ENGNEERNG PSYCHOLOGY LABORATORY Fnal Techncal Report Research on the Technology of nference and Decson 5 WARD EDWARDS Prncpal nvestgator Sponsored by: Advanced Research Proectk Agency ARPA ORDER No o o c \- -4"-l DEC» KC yljssarrslljj Mon/fored by.' Engneerng Psychology Programs Offce of Naval Research Contract No. N000K-67-A NR Contract Perod: 1 June fhay 1974 M. Approved for Publc Release; Dstrbuton Unlmted Reproducton n whole or n part s permtted for any use of the U.S. Government Adm/nfterrd through. Reproduced by NATONAL TECHNCAL NFORMATON SERVCE US Depertment of Commerce Sprngfeld, VA November 1973 ^-OFFC^ OF RESEARCH ADMNSTRATON : ANN ARBOR The vews antf^tabdaxfons contaned n ths document are those of the authors and should rot be nterpreted as necessarly representng the offcal polces, ether expressed or mpled, of the Advanced Research Projects Agency or the US. Government. n * n n» y

3 BMHMH m F RESEARCH ON THE TECHNOLOGY OF NFERENCE AND DECSON Fnal Techncal Report 30 November 1973 ward Edwards Engneerng Psychology Laboratory The Unversty of Mchgan Ann Arbor, Mchgan Ths s a fnal report of research supported by the Advanced Research Projects Agency of the Department of Defense and was montored by the Engneerng Psychology Programs, Offce of Naval Research under Contract No. N A , Work Unt Number No Approved for Publc Release; Dstrbuton Unlmted -»^^^^^^-^-^» ^^^ ^.^.----.^ -^ -. ^.. -,...-..

4 MMBkMMMBB. m ^m! SECURTY CLASSFCATON OF THS Pa.CZ ( Uhm Dctu tnttred) REPORT DOCUMENTATON PAGE READ NSTRUCTONS. REPORT SUMa R BEFORE COMPLETNG FORM 2. GOVT ACCESSON NO. T-rtEClPENrS CATALOG NUMBER F 4. TTLE (and uoutlt) RESEARCH ON THE TECHNOLOGY OF NFERENCE AND DECSON 7. AUTHOR^/" Ward Edwards 9. PERFORMNG ORGANZATON NAME AND ADDRESS Engneerng Psychology Laboratory nsttute of Scence & Technology Unversty of Mchgan Ann Arbor. Mchgan «1. CONTROLLNG OFFCE NAM AND ADDRESS Advanced Research Projects Agency 1400 Wlson Boulevard Arlngton, Vrgna 14. MONTORL.r AGENCV NAM AND ADDRESS (l djjermt Jnm Cwrolmg O'/CfJ Engneerng Psychology Programs Offce of Naval Research Department of the Navy Arlngton, Vrgna»6. DSrRlS'JTlÜN STATEMENT fr.f,-,-«r c port) S. TYPE OF REPORT & PEROD COVERED Fnal Techncal 1 June November PERFORMNG ORG. REPORT NUMBER None 8. CONTRACT OR GRANT NUMBER ^ N A JO. Pr<OGRAM ELEMENT. PROJECT TASK AREA at WORK UNT NUMBERS NR ARPA Order No \2. REPORT DATE 30 November NUMBER OF PAGES SECURTY CLASS. fo/zä» r^ort) Unclassfed \Sa DECLASSFCATON/ OOWNr ''VoNG SCHEDULE ^ Approved for publc release; dstrbuton unlmted. 17. DSTRBUTON STATEV '(oj:he aatract mthmä n: Block 20, fäj/mmt/mm Rport} *' s. SUPFLE* J! :\TAF;V KO.«" None 1 1 Multattrbute utlty theory Elctaton nformaton processng 20 AS^ "" """ Ths report summarzes sxteen months of research on the technology of nference and decson. Efforts n arrangng a conference nvolvng the natonal securty communty and research personnel n the su.mer of 1973 are detaled. The frst major research result, flat maxma n decson analyss, s suttrarzed and ts mplcatons for socal and psychologcal theory as well as decson technology are dscussed. The major conclusons from ths area of research s that the structurng of the decson problem may be much more mportant than the elctaton M.'JSS^UJJ EO.rON JF 1 NOV 65 S OBSOLETE // JtCURTY L.l A Unclassfed -:<; t-'a. - 11,... "r.rrr

5 r SECUPTY CLASS FX ATO», OF THS, PACE/11 A,?/ Data Etlttr.d) of specfc parameter values. The second major topc s a revew of mulffffhhs Ut J ty f^^ement and an expenment s outlned whch may crcumvent the problem of evaluatng utlty elctaton methodologes wthout hav>q an external standard for utlty. Elctaton technology for probabltes consttutes the thrd research area. Extensve revew of data collected n fve cntcal experments ndcated that feedback may be more mportant than aggregaton as a source of human conservatsm n nference. The mportance of ths fndng s stressed The rourth and last area sutmarzes the fnal three of nne Techncal Reports to stem from ths contract. These concern Bayesan statutca" analys " for comparng determnstc models, a crtque of the Bayesan foundatons of decson theory, and error analyss n Probablstc nfbrmaton Processng systems -r-rr-»ecurtv CL. - S'FC.ATON '- THS P«GE Whn >>uu>:.r. t!l -. _-^ >,. -*

6 MUtMMttaMaM ' TABLE OF CONTENTS NTRODUCTON 1 A TECHNCAL OVERVEW Applcaton of decson technology to actual mltary decsons 2 Flat maxma n decson analyss 5 Multattrbute ut 1 ty measurement 11 Elctaton technology for probabltes 16 Assorted crtcsms 20 MANAGEMENT NFORMATON 21 REFERENCES 24 ABSTRACTS //

7 - - "- mmmmm *-., ntroducton On 18 January 1972 the Unversty of Mchgan submtted to the Advanced Research Projects Agency a proposal for Research on the Technology of nference and üecson. The proposal called for. 5 years of research at a total cost of $400,000. The Prncpal nvestgator was Professor Ward Edwards. A one-year contract was awarded, N y , and research began on 10 June 1972, the date on whch funds became avalable. Montorng responsblty for the contract was undertaken by Dr. Martn A Tnlcott of the Offce of Naval Research. n January n* 1973 a proposal for contnaton of the program was submtted to ARPA, and was funded. n March of 1973 t began to seer, lkely that tdwards would leave the Unversty of Mchgan, and n May he accepted an offer to become Drector of the Socal Scence Research nsttute, Unversty of Southern Calforna. Plans were then made to termnate the contract wth Mchgan. Consequently, although ths s formally a fnal report of the work at Mchgan, t s n fact a progress report coverng about 1 1/3 years of a fve-year program. The man products of the program so far have been nne techncal reports whch are recevng dstrbuton ndependently of ths Fnal Report. They stand alone ana speak for themselves. Consequently the purpose of ths Fnal Report wll be to present the overall concepton nto whch the techncal reports ft, to report on ncomplete actvte: that wll contnue at USC, and to summarze

8 HHMRP^ ' " l " "" l " ' w^w^^pwww some fnancal and personnel nformaton. Absttcts of the nne techncal reports are also ncluded. A Techncal Overvew The orgnal proposal that led to ths program called for ' research bearng on the topcs of nformaton processng (especall. 1 n ntellgence systems), of tactcal acton selecton, and of nformaton acquston. Research was to be a mxture of theoretcal work, laboratory work, and work growng out of contacts wth real mltary envronments. As the program developed, four man themes emerged, and n addton some other actvtes less closely related to these also occured. So ths revew wll be organzed under fve headngs, of whch the last s somewhat of a catch-all. Applcaton of decson technology to actual mltary decsons n late 1972 and early 1973, Col. Kbler, of ARPA, and Edwards had several conversatons about how ARPA should go about encouragng the development of decson technology n paths relevant to mltary problems and ts applcaton at relatvely hgh levels wthn the natonal securty communty. Evdence of applcablty, obtaned mostly wthn the ntellgence commu r n»,y, exsts n reasonable abundanre. But the nature of t^e technology s such as to produce relatvely hgh levels of resstance to applcaton, anu so evdence of successful applcaton wthn one agency s not eno-gh to produce,. ^^ ' -, n -- _

9 ^ ^WwnMW^M^v^n - 3 efforts at applcaton wthn other agences (or evrn elsewhere wthn the same agency). The upshot of these dscussons was a decson to hold a summer study n the summer of 1973, under ARPA sponsorshp. Attendees would be fvgh-level members of the natonal :ecurty communty, who would bref the study group about the knds of decson problems ther organzatons encounter and the methods they use to solve them, and key ndvduals wthn the academc and scentfc communty, who would bref the study group about technologcal tools avalable or n prospect, and about research needs. The goal would be the producton of a document that would suggest approprate drectons for subsequent ARPA actvtes n the feld. Col. Kbler asked Dr. Davs Bobrow, of the Unversty of Mnnesota, and Edwards to take jont responsblty for the scentfc leadershp of the study. Edwards's actvtes pror to the conference were sponsored by ths contract, although the conference cself (and Edwards's actvtes whle attendng t) were supported ndependertly. Ths turned out to be a substantal job. t requred Edwards :; to travel to Washngton for conferences wth Kbler anh Bobrow and others on February 9, January 10, March 20 and May 1, to travel to Mnneapols on June 14, and to receve a number of vsts n Ann Arbor. nteractons wth nvtees took tme. Most mportant..j.-_..^->.- ^t._m»»^...^. - - ' ' -t ^ w^.... _.,^.-. ^..

10 - - " -- n^nmm^fcama n however, was the wrtng, jo ntly wth Bobrow, of a "straw man" study report. Ths document was dstrbuted to the partcpants pror to or at the begnnng of the study as a gude to what Dcbrow nd Edwards consdered mportant and approprate. Whle the fnal study report was qute dfferent from the "straw man" verson, t seems safe to say that a good deal of the agenda of the study reflected varous knds of reactons to the straw man, and con- sequently that that document played * sgnfcant role n engen- derng the fnal Output of the study. t s no approprate to dstrbute the "straw man" report as a techncal report of ths contract, snce t was not ntended for such general dstrbuton. Nor s t approprate to nclude t n ths fnal report, both because t s too long and because t was not ntended to have that knd of performance. Nevertheless, n terms of ts nfluence t seems possble that t s the most sgnfcant output of ths frst 1 1/3 years of ths program. The summer study tself n varous ways furthered the objectves of ths contract. t establshed relatons between Edwards and a number of members of the natonal securty communty whom he had not prevously known, many n a poston to provde access to decson settngs and systems of ronsderable natonal mportance. n partcular, t hghlghted the nature and mportance of WHNCCS, and permtted Edwards to become acquanted wth at least a few of.

11 -.* J 1-5 those nvolved. On the bass of what was learned at that study, WWNCCS seems lke a Suffcently promsng locus for the applcaton of decson technology to deserve a much closer look; such a look shold be taken early n Flat maxma n uecson analyss. The phenomenon of flat maxma has been notced by decson analysts lookng at a number of specfc contexts, rangng from the use of proper scorng rules n weather forecastng (Murphy and Wnkler, 1970), to nformaton purchase wth optonal stoppng (Edwards, 1965). Whle everyone has assumed that these were specal nstances of a much more genedl phenomenon, no one has attemp^d to defne the general phenomenon or to look at the broad range of ts mplcatons. As a frst step n lookng nto the queston, von Wnterfeldt and Edwards looked at the applcatons of decson-theoretcal thnkng to sensory processes and to probablty estmaton. The result was the frst Technval Report of ths program; ts abstract appears n the secton at the end of ths fnal report that contans the abstracts of all techncal reports so far produced. (They a-'e lsted there n the order n whch they are dscussed here.) The method used by von Wnterfeldt and Edwards n that study was mathematcal, but rather specfc. Whle t dd not work wth only specfc examples, t proved no general theorems. Nevertheless,

12 ^^^^^^^«" ^ ^ ^ ' ' -. _,.- t was able to reach some general conclusons. The basc concluson s that everywhere one looks, decson-theoretcal maxma are flat; that s, sgnfcant devatons from optmal strategy lead to relatvely nsgnfcant percentage reductons n expected payoff. After draftng that report, von Wnterfelot and Edwards contnued to gnaw away at the problem. How can the flat-maxmum problem be formulated generally, rather than as a set of specfc examples? The soluton had to le n the convexty property of decson-theoretcal payoff functons and t dd. Ultmately, von Wnterfeldt and Edwards were able to formulate and prove the general theorem whch the examples all exemplfy. (See Techncal Report Abstract No. 2.) What does the flatness property mean? t s easy to overnterpret t. The dfference n expected payoff between an optmal strategy and a non-adjacent suboptmal one can be made as large as desred by smply ncreasng the magntudes of all payoffs. f the expected payoff assocated wth an optmal strategy s a bllon dollars, then a 1% reducton n that expected payoff s ten mllon dollars. Flatness s meanngful only when consdered n percentage terms, as the frst von Wnterfeldt-Edwards report makes clear. The mplcatons of flatness can be looked at n two ways: from the pont of vew of the decson analyst, or from the pont '

13 of vew of the socal phlosopher. Consder the latter frst. 1 From ths pont of vew, the relatve nsenstvty of payoffs W to sgnfcant but not monstrous devatons from optmal behavor s a knd of glue that permts :>Lcety to hold together. Suppose that the consequences of even mnor devatons from complete ratonalty were grossly dsast'ous--how long could a socety of only-partly-ratonal men survve? But the fact that mnor devatons are almost costless leaves some room for both error and ndvdual dfferences, whle the fact that they are not completely costless makes analyss and ntellgence worth brngng to bear on decsons. The decson analyst must face much more specfc consequences of flatness. These consequences fall most heavly on elctaton technology. For example, the use of proper scorng rules has been assumed to motvate probablty estü v o-s to produce "good" estmates. But analyss shows that relatvely large devatons from the optmal probablty estmate produce only relatvely small reductons n expected payoff. Consequently, the motvatng effect of proper scorng rules revewed below shows exactly that. Experments have shown that proper scorng rules mprove probablty estmates but they certanly have not establshed that t s the rules themselves, rather than the ndoctrnaton and practce that goes wth the r use, that cause tne mprovement. On the other hand, the fa. of flat maxma makes precse probablty and value estmates of less mportance than they mght otherwse be. f a }0% error n an estmated quantty produces - - ^ - -

14 MM.* only a 2% decrease n expected value, perhaps that 10«error s tolerable--certanly more tolerable than f t produced a 302 decrease n expected value. The mplcaton of ths argument s that the must mportant aspect of decson analyss s the structurng of the problem for analyss, not the elctaton of numbers and computatonal processes that follow. Unfortunately, ths process of structurng the problem s least amenable to formal prescrpton. t seems to be mostly a matter of wsdom, experence, and ablty to tolerate confuson, ambguty, and conflct. For psychologcal theory, the phenomenon of flat maxma has yet another mplcaton. Psychologcal models, such as probablstc learnng models, ncorporate two classes of parameters: known parameters, such as the number of stmul used n an experment, and parameters to be estmated from data, such as learnng rates. Known parameters are errorless. But estmated parameters are always estmated va decson-theoretcal procedures such as maxmum lkelhood, least squares, etc. Thesa procedures formally have the propertes of all decson-theoretcal flat maxma. Consequently, the appearance of precson gven by, say, a lea-.t squares estmate of a parameter s somewhat spurous substantally dfferent Values of the parameter woula produce only modest ncreases n Lhs sum of squares that was used as crteron for parameter estmaton.

15 _ n an attempt to evalute the effect of ths decsontheoretcal phenomenon on learnng models, von Wnterfeldt and Edwards have compared errors produced by naccurate parameter estmates wth errors produc.y erroneous values of known stmulus parameters. Very generally, the models are far more senstve to the latter knos of errors than to the former. To put f another way, learnng models depend very senstvely on numbers that descrbe the envronment, but only very weakly on numbers that descrbe the organsm. To what extent, then, are they models of organsms? The tradtonal dstncton between normatve and descrptve models :; mm- s that normatve models descrbe tasks, whle descrptve models descrbe what organsms do n tasks. But f the descrpton of what organsms do n tasks s vague, n the sense that a dfferent descrpton produces almost the same result, then why should theorsts r ;: accept relatvely poor descrptons of tasks nstead of smply usng the approprate normatve models? The argument sketched above s not yet fully formed. From here on, the ssues get more phlosophcal and less technologcal. They have to do wth model success. What s a mode, for? How can you tell when t s dong what t should do? A substantal phlosophcal lterature exsts on ths queston, and we have explored t, but found t unhelpful. n our mathematcal study

16 _^ of the parameter estmaton process, however, we fnd the curous result that whle least squares, ch square, and smlar estmato.«procedures have the flatness that we expect, maxmum lkelhood procedures do not. n fact, estmates based on maxmum lkelhood procedures seem to be unreasonably sharp mplyng more precson than the data could ev:r n fact yeld. We had hoped to have a techncal report ready o. ths set of topcs by the end of ths contract, and n fact a draft verson exsts. But these ssues reman suffcently unclear to us that we have decded not to ssue t. n Los Angeles, we hope to get some advce from R. Duncan Luce about these dffcult questons. We have pursued one further lne of thought n ths area. Some nondecson-theoretcal transformaton clearly can re.tore sharpness to flat maxma. An obvous example mght be cat.ed '.he wnner transformaton. Whle the loss functon assocated wth devaton from optmal probablty estmates when a proper scorng rule s beng used s nevtably flat, sharpness can be restored by turnng the stuaton nto a contest. For example, weather forecasters mght compete for a week, and whchever one had the hghest cumulatve Brer score at the end of the week mght wn a prze. The trouble wth the wnner transformaton, unfortunately, s that ts decson-theoretcal propertes are unpleasant. t s

17 **Xä 11 an nstance of a wdespread phenomenon of real lfe--the mplct lnear scorng rule. n a sngle-estmate contest, for example, the optmal strategy under the wnner transformaton s to estmate the most lkely event as havng probablty 1, and all ts compettors as havng probablty 0 (unless the estmator can know hs compettors' estmates when he makes hs own). Strateges get much more complcated n a multple-estmate contests, but they all have ths sort of flavor.. Stll, these unpleasant formal propertes may not be suf- fcently good reason to reject the wnner transformaton as a practcal tool. Wendt planned such an experment, n a Bayesan revson task of the two-normal-dstrbutons type. However, as a result of the decson to move to USC, ths experment has been postponed untl after the move. Ths experment wll probably be a part of a larger experment on response modes and tranng technques. Multattrbute Utlty Measurement. The lterature and the technology of multattrbute utlty measurement have grown very rapdly n the last few years. (See for example Raffa, 1968, 1969; - Keeney, 1971; Edwards, 1971.) Our own vew s that ths technology stands now roughly where the Bayesan technology stood n but has a brghter future, because the topc of values and evaluaton s nherently more mportant than the topc of dag- nostc nference.

18 - 12 The technque for multattrbute utlty measurement, n ts ratng-scale verson, noks deceptvely smple (Edwards, 1971). But n fact some rather sophstcated mathematcs and some rather strong assumptons le behnd t. n partcular, the dstncton between rskless and rsky multattrbute utlty measures, almost meanngless n practcal applcatons (because all practcal stuatons nvolve some rsk, yet t s often not worth Whle to take t explctly nto account), s strong and mportant n the underlyng theory. Von Wnterfeldt and Fscher (see Techncal Report Abstract No. 3) have revewed the lterature bearng on the assumptons that underle multattrbute utlty measurement and the relaton of those assumptons to the choce of an elctaton technque. A. pont that emerges from the revew, not so much as a concluson but as a fact of lfe, s that the elctaton technques that are formally justfed by the assumptons are far clumser and more unpleasant to use than one would wsh, whle the smplest elctaton technques requre very strong assumptons, and even then are less strongly related to the model than mght be dtsred. The obvous consequence of ths state of affars s that a more serous study should be made than has been of the degree to whch multattrbute utlty measures based on smple elctaton

19 13 technques agree wth those based on more complex technques. Tfat s, nstead of treatng utltes as formal numbers, ether exactly ".OTcct or else useless, one should thnk of them as approxmatons, and explore how good varous approxmatons are. Numercal exploratons of ths queston are -l^arly called for, but wll not occur untl after the move to USC. Von Wnterfeldt and Edwards (see Techncal Report Abstract No. 4) performed an expermental study of several approaches to multattrbute utlty measurement. The man fndng was that elctaton methods based on gambles were preferable to other elctaton methods. Ths s a surprsng concluson, :nce the whole thrust o' the cho ces-among-bets lterature s that such choces are poorly lnked to the nput parameters. t wll need further examnaton. The most urgent task n the study of utlty measurement s the development of what, n our laboratory slang, we call God's utlty functon--that s, an objectve standard wth whch to compare elcted utltes. Most of the most mportant conclusons.. n the area of probablty elctaton have emerged from comparson of elcted probabltes wth calculated ones, n stuatons n whch such calculatons are possble. n the absence of God's utlty functon, su':h comparsons are not possble for utltes-- and research s severely handc3pped

20 14 An extreme subjectvst would assert that one cannot dspute, or prescrbe, tastes--that the goal of fndng a stuaton n whch such an external standard can be defned s unatt«nable. (He mght, n fact, make exactly the same argument about probablteswhy are opnons any more prescrbable than tastes?) We thnk that utltes are contextual, and ndeed are often nterpersonal, and thus are sometmes subject to prescrpton. Several approaches to the problem can be conceved of. We have explored one n consderable detal. Damonds are extremely nterestng stmul for use n utlty experments. They are valuable, and the value s rather precsely reflected n the prce, whch can be taken as an ndex of overall utlty. The dmensons of value are extremely well specfer and understood. They are cut, clarty, color and carats. Of these, all (except perhaps clarty) are n prncple objectvely measureable--bjt n practce an appraser of damonds works wth only a scale, a par of lockng tweezers, a jeweler's loupe, a good, unform lght source, and hs hghly educated eyes and bran. These experts are extremely hghly practced; a typcal wholesale damond merchant wll apprase many thousands of damonds n the course of a year. We are under the mpresson that the results of such apprasal show a great deal of nter-expert agreement, though the evdence on the pont f less abundant than we mght lke (see Bruton, 1970).

21 - - :5 We plan, and have conducted extensve prelmnary work lookng toward, an experment on elctaton technology for multattrbu^ Jtlty, usng damond wholesalers as our experts and wholstc judgments of prce as the standard of God's utlty. The argument wll be that th, technque that most nearly reproduces (up to a lnear transformaton) those prces s the best technque. We have arranged for cooperaton from a group of damond wholesalers n New York. (We may be _.le to obtan judgments n Los Angeles also.) Wendt, who s collaboratng wth Edwards on ths, has returned to Hamburg, and beleves he may be able to obtan judgments from damond wholesalers n Amsterdam, the center of the world damond market. The damond study s the best approach we have yet found to God's utlty functon. But the stmul are hard to obtan, and access to the experts s a problem. So we are also explorng the possblty of usng the addtve nature of certan knds of objects as the bass for such experments. A market basket full of groceres s, n a sense, a commodty. But ts utlty would be conceded by most to be an addtve aggregate of the utltes of the objects n the basket. Gven the utltes of these separate objects, ther sum specfes a form of God's utlty functon for such baskets. Ths can be compared wth judgmental utltes obtaned n one way or another. There are some serous dffcultes wth ths dea. ^^^^a - ^.

22 MMuMaa.1 16 and t wll take consderable further thnkng to refne t to the pont at whch an experment can grow out of t.. Elctaton techno'ogy for probabltes. The topc ot u'lctaton of probabltes has been a major theme of ths laboratory's work fo; mo-e than ten years and s stll by no means a closed topc. The alternatves are pretty well understood, but by no means s the»? enough nformaton about them, especally wth populatons other than college students and stmul other than the typcal bookbags-and-poker-chps or pck-up stcks, to permt unhestatng choce among them. Goodman (see Techncal Report Abstract No. 5) has performed very extensve further analyses of the data from the fve key experments done n ths laboratory on the tooc. Her statstcal technque was a form of regresson analyss that has not been used n öhs context, before, and the ndependent varables she studed we^-e: the response mode tself, the scale (log or lnear) on whch t was expressed, whether or not the subject had to aggregate evdence n hs head, and whether or not feedback concernng the meanng of the response was present at the tme of response. Her man conclusons are summarzed n the abstract. The mportance of feedback n producng conservatsm had been suspected before-- but ths analyss s surprsng n ndcatng that that s more

23 mportant than whether or not the subject aggregates evdence n hs head. The sgnfcance or ths study of Goodman's to the desgners of probablstc nformaton processng systems wojld be dffcult to overestmate. 1 : Goodmön's paper nvtes an edtoral comment. t s reasonably short--and qute dffcult to read, mostly because t s full of jargon and very tghtly reasoned arguments. Only someone thoroughly steeped n the Bayesan pont of vew and the Bayesan lterature wll fnd t easy to get through, And the cost of wrtng t n such a way as to make t ntellgble to those not already famlar wth Bayesan deas s prohbtve the length of the paper would trple or quadruple, and much of t would then consst of revewng famlar deas. Ths, of course, s the normal course of development of a feld of scence but ths report dramatzes the fact that the Bayesan pont of vew has moved a * long way n ten years. The standard decson-analytc technology of elctaton en.phaszes nternal consstency. f a subject makes nconsstent judgments, the nconsstency s called to hs attenton and he s nvted to revse any or all judgments to elmnate t (see Raffa, 1968). r v.

24 - 18 Ths would be fne f only consstency were mportant. But, especally when the judgments concern probabltes, verdcalty s more so. People tend to be much more secure and confdent about posteror odds judgments than about lkelhood rato judgments (Edwards, Plllps, Hays, and Goodman, 19r3)-- yet the evdence s abundant that lkelhood rato estmates are usually much more accurate than are posteror odds (see Edwards, 1968). Consequently, f a subject judges both, he s lkely to be nconsstent (.e. volate Bayes's theorem). f he s then nvted to revse for consstency, t seems qute possble that he wll revse, not the odds, but the lkelhood ratos, and therefore revse them away from verdcalty. An experment was desgned to explore ths hypothess. The standard pck-up-stck task was used. Subjects frst estmated sngle-stck lkelhoo ratos, and then estmated posteror odds for four-stck sequences. Then they were taught about Bayes's theorem, ther nconsstences were exhbted, and they were nvted to make whatever revsons seemed approprate. The data from 15 subjects were hghly unsatsfactory. Half of the subjects were more verdcal after revson than before; half were less. The problem, we now suspect, s that four-stck sequences are too short to produce suffcently conservatve posteror odds. We plan to start ths experment all over agan at

25 USC, usng longer sequences and perhaps a larger value of d'. Seghers (see Techncal Report Abstract No. 6) has conducted an experment on proper scorng rules taken as bets. Do subjects maxmzed expected monetary value n such stuatons? The experment farly conclusvely says no. Ths fndng means that the assumpton that subjects wll maxmze expected monetary \alue, usually taken as the bass for use of proper scorng rules, s smply not approprate. Proper scorng rules probably help to nstruct subjects about the meanng of probablty estmates, but by themselves they do not constran the subjects to produce such estmates as ther formal nature would prescrbe. (The facts about flat maxma rake ths concluson ^11 the more reasonable.) We have been thnkng about tranng, though the thnkng s as yet too unstructured to lead to experments or theory. The frst and most mportant pont s that tranng for probablty estmators mght be called The llusory Panacea. Whenever some pecularty of human behavor n probablty estmaton stuatons s notced, the explanaton always s that the estmators were untraned or undertraned. But what consttutes correct and suffcent tranng? When professonal probablsts and Bayesans so often fnd themselves caught n logcal errors, can any lesser standard of tranng be called enough?

26 - 20! We are comng to thnk tht all ths emphass on tranng : s msplaced. nstead, what no^ seems mportant s the packagng of decson technology that s. specfcaton of a combnaton of analytc technques, elctaton methods, and tranng for judgtj that wll permt ts applcaton n practcal stuatons. n short, rather than tranng everyone to be Jula Chlds we are comng to feel that t would be better to have a cookbook--wth pretested recpes, please. Assorted crtcsms. Three other Techncal Reports are n varous ways auxlary to or crtcal of work revewed above. Perhaps the most useful s a report by Wendt (see Techncal Report Abstract No. 7) that provdes practcal technques for applyng the technques of Bayesan statstcal analyss to comparsons among models especally among determnstc models. Ths paper grew out of thought about data analyss problems that arose n Seghers' experment, and s appled n that Report, but the technques presented by Wendt are far more wdely useful than that. Wendt (see Techncal Report Abstract No. 8) s crtcal of many of the assumptons and workng hypotheses of the Bayesan pont of vew, and has attempted to assemble hs crtcsms n a coherent

27 MMMHM mmmmmm* form. Ths Report s frankly a thnk-pece. Few decson theorsts would agree wth all of ts arguments: some would dsagree wth all of them. But foundatons should not be allowed to reman unexar,ned, and crtcsms of ths sort often lead to later constructve work. Fryback and Edwards (see Techncal Report Abstract No. 9) have looked nto the problem of errors produced by varablty of lkelhood rato estmates n a probablstc nformaton processng system. That error can n prncple be of substantal sze, as a model based on the assumpton that such errors are normally dstrbuted makes clear. But actual examnaton of test-retest relabltes n a major earler experment (Edwards. Phllps, Hays, and Goodman, 1968) makes clear that n fact such relabltes are very hgh ndeed, and consequently that the effect (at least n that experment) was relatvely small. Management nformaton The followng Table lsts those who have worked on ths contract for sgnfcant fractons of tme.

28 Personnel (Based on 16-month perod) Name Ward Edwards Barbara C. Goodman Drk Wendt Gregoary W. Fscher Kurt Snapper Detlof von Wnterfeldt Denns G. Fryback Rayrnond C. Seghers Patrca Homan Annette Johnson Ttle Prncpal nvestgator Assocate Research Psychologst Vstng Research Psychologst Assstant Research Psychologst Research Assstant Research Assstant Research Assstant Research Assstant Secretary Admnstratve Assstant nrl > no charge.15 ' to ARPA * ^-JC_ _

29 M - r 23 the Contract. The followng presents fnancal facts about the work so far on Balance as of 1 October 1973: $63, Antcpated Expendtures: October Salares & Wages: (EPL) Tech. typsts Graphcs.C. (58.2%) Total $ 3, , , $ 6,521.00! Drect Costs: Supples: Telephone: $ 6, November Salares & Wages Graphcs Reproducton.C. (58.2%) Total , Drect Costs: Reproducton supples Telephone Postage ^ Balance: $ 63, Antcpated expendtures: 8, Estmated balance: $ 54, «

30 w ~"-" 24 References Bruton, E. Damonds. N. A. G. Press Ltd., London, Edwards, W. Optmal strateges for seekng nformaton: Models for statstcs, choce reacton tmes, and human nformaton processng. J. math. Psychol., 1965, 2, Edwards, W. Conservatsm n human nformaton processng. n Klenmuntz, B. (Ed.) Formal representaton of human judgmer.t. New York:Wley, Edwards, W. Socal utltes. n Decson and Rsk Analyss; > - D owerful new tools for management. Proceedngs of the sxth trennal Symposum, June, 1971 Edwards, W., Phllps, L.D., Hays, W.L., & Goodman, B.C. Probablstc nformaton processng systems: Desgn and evaluaton. EEE Trans. Syst. Se. Cybernetcs, 1968, Keeney, R. Utlty ndependence and preference for multattrbuted consequences. Operatons Research, 1971, 19 Murphy, A.H., and Wnkler, R.L. Scorng rules n probablty assessment evaluaton. Acta Psychologca, 1970, 34,

31 L. 25 Ra1ffa ' H - Decson Analyss. Bo3ton:Addson We-.ay, Raffa, H. Preferences for mult-attrbuted alternatves. RAND Memorandum No The RAND Corporaton, Santa Monca, Calforna, : T

32 «^ ^-^ «^»^«^^^^w.^ a Report Abstract Costs and Payoffs n Perceptual Research Detlof v. Wnterfeldt and Ward Edwards Abstract A persstent problem n any knd of psychologcal research that reaches conclusons about naccessble processes or experences nsde a subject's head s tc valdate those conclusons-that s, to exhbt persuasve reasons to beleve that emtted behavor s some sense fathfully reports naccessble processes. n the md-1950s, perceptual researchers wdely adopted an approach that mqht be called valdaton by cupdty. f the expermenter s wllng to defne a correct response, he can reward the subject for correct responses and not for wrong ones; sutable reward schemes combne wth an assumpton of ratonal behavor on tne subject's part to permt drect nference of nternal processes. However, decson-theoretcal maxma are flat, n the sense that serously napproprate behavor produces relatvely lttle reducton n the subject's expected payoff. Ths means that costs and payoffs are rather feeble means of nstructng subjects what to do, or of ensurng that he does t. Ths argument s made specfc n examples drawn from three knds of perceptual experments. n some tasks, such as probablty estmaton, subjects drectly estmate subjectve quanttes, and receve rewards for accuracy of estmate. An analyss of proper scorng rules for probablty estmaton shows that ther maxma are nevtably qute flat. An analyss of a yes-no decson task shows that the ncorrect answer produces flat maxma; whle the payoff functon can be sharpened by ncreasng the magntudes of all payoffs, a sutable relatve payoff functon s ntractable. n such yes-no tasks, crteron varablty produces even more flatness, so much so that t would be surprsng f such varaton dd not,

33 -^. ^...,^-.^ ^7 1 occur n most real experments. Crteron varablty suffcent to produce a 30% reducton n estmates of d' produce only 5% to 8% reductons n expected wnnngs. mplcatons of these results for expermental desgn, for nterpretng expermental results, and for more general decson-theoretcal thnkng are dscussed.»"-..^^_ -'-^^--" *- J

34 ' -" - r n Report Abstract T FLAT MAXMA N LNEAR OPTMZATON MODELS Detlof v. Wnterfeldt and Ward Edwards Abstract Expected value functons as functons of decsons and decson strateges are flat around ther maxma. Ths so called flat maxmum phenomenon has been dscovered n senstvty analyses n vrtually all decson theoretc paradgms. But untl now most of the research on flat maxma explored more or less general examples and lmtng consderatons. Two basc questons remaned unanswered: what are the mathematcal reasons for the restrcted shape of the evaluaton functons; and can these restrctons be nterpreted as flatness n a psychologcal sense? Whle the second queston calls for psychologcal expermentaton, the frst queston can be answered wth mathematcal tools. The present artcle shows that the mathematcal characterstcs of lnear optmzaton models mpose severe restrctons on the functons evaluatng choce alternatves such as gambles, mult-attrbuted outcomes, or consumpton streams. The course of proof of ths argument provdes a helpful tool for senstvty analyses n decson theory. The concepts and methods are demonstrated n examples from statstcal decson theory, psychologcal modelng, and appled decson theory.

35 n Report Abstract 3 MULT-ATTRBUTE UTLTY THEORY : MODELS AND ASSESSMENT PROCEDURES Detlof v. Wnterfeldt and Gregory W. Fscher Unversty of Mchgan Abstract Ths artcle revews mult-attrbute utlty theory from a measurement theoretc perspectve. t descrbes and classfes decson stuatons accordng to three salent aspects of choce : uncertanty, tme-varablty, and mult-dmensonalty. For each choce stuaton the man mathematcal representatons, ther nter- relatons and dfferences are dscussed. Measurement theoretc tests are descrbed whch separate between mult-attrbute utlty models» n rskless and rsky tme nvarant choce stuatons. Assessment procedures are outlned to encode utlty functons for the representatons developed, and expermental applcatons of mult-attrbute utlty theory are brefly revewed. 1

36 - M^MMMMte a o Report Abstract Evaluaton of Complex Stmul Usng Mult-attrbute Utlty Procedures Detlof v. Wnterfeldt and Ward bdwards Three procedures for constructng addtve mult-attrbute utlty theory (MAUT) models were tested for ther dfferental valdty: a probablstc procedure, a smple drect ratng procedure, and a modfed drect ratng procedure. Valdaton crtera were ratngs and smple : :; choces. Procedures were evaluated after an ntensve tranng SPoSon n whch subjects learned to adopt an evaluaton strategy wth whch they felt most comfortable and whch best reflected ther preferences. The results of a correlatonal analyss ndcated that MAUT can mprove upon the decson maker's own unaded ntuton. The probablstc procedure.. was found to be the superor method for predctng smple choces between stmul.

37 t^aüm^a ^M^^ mm*m***mrv - -- \ Report Abstract T r Drect Estmaton Procedures for Elctng Judgments about Uncertan Events Barbara C. Goodman Ths report re-analyses data from fve studes concerned wth methods for elctng judgments about uncertan events. t focusses on response modes, such as odds, lkelhood ratos, etc.; whether or not the response requred the subject to aggregate tems of data n hs head; whether the scale on whch the response was made logarthmc or lnear; and whether the subject receved addtonal feedback about the mplcatons of hs estmates n the course of makng them. Whle no sngle experment studes all these ssues smultaneously, combnaton of the data from the fve experments permts some strong conclusons: 1. Presence of addtonal feedback about the mplcatons of estmates s probably the most powerful varable controllng the extremeness of these estmates; feedback makes estmates less extreme. Whether the less extreme estmates are closer to or further from correct Bayesan values depends on stmulus condtons. 2. Aggregated responses are consstently less extreme than nonagqregated responses. 3. Lnear scales produce less extreme responses than logarthmc scales. 4. Lkelhood rato estmates are sometmes less extreme than odds estmates. Other conclusons are also revewed. mplcatons of these conclusons for the desgn of probablstc nformaton processng systems and for further research on response modes for nformaton processng are dscussed.

38 1 Report Abstract 6 Relatve Varance Preferences n a Choce-Among-Bets Paradgm Raymond C. Seghers, Denns G. Fryback, and Barbara C. Goodman Abstract T The valdty of the prme assumpton of proper scorng rules (PSP), that people maxmze subjectvely expected value (SEV), was tested n the case where SEV was assumed to equal EV. A choce-among-bets paradgm was used mm n whch the lsts of bets conformed to the requrements of a PSR. Both real and hypothetcal payoff condtons were used, and n addton, EV, varance, and!s of the garrbles were systematcally "a^-'ed. Of the 12 Ss only 3 tended to maxmze EV under both real and nypothetcal payoff condtons, whle relaf-wo w ar 2r)Ce preferences can account for the decson strat.*»nes of the other Ss. nferred strateges were smpler and more consstent durng the real payoff sessons. The effect of the gambles' propertes was dosyncratc and no overall conclusons v/ere drawn. The use of the lst of bets generated by a PSR as a response mode for nferrng subjectve probabltes s questoned because of the weakness of the SEV maxmzaton assumpton n ths context.

39 1 mm - UMHM s Report Abstract 7 BAYESAN DATA ANALYSS OF GAMBLNG PREFERENCES Drk Wendt Unversty of Mchgan Abstract Ths paper emphaszes the use of Bayesan data analyss for experments wth choces among gambles. n an ntroductory example, the method s llustrated by a comparson of two learnng theores. Specal problems arse wth the analyss of data from decson makng experments whch assume determnstc choce models whch cannot be handled by Bayesan analyses. Several ways around these dffcultes are suggested, dscussed, and demonstrated on two sets of data from choce-among-gambles experments.

40 ! fllll «! Ml ^ M M^^MMM lawn M« Report Abstract 8 Some Crtcsms of the General Models Used n Decson Makng Experments Drk Wendt Abstract The general normatve model of expectaton maxmzaton :s outlned and crtczed for several reasons. t may not be approprate as a normatve model a varety of stuatons where t s assumed to be ratonal. Some of ts condtons, e. g., ndependence of evaluaton-of-aspects and probablty-revson cues, and correctness of the smple addtve utlty model, may not be met. Moreover, determnstc models may be too strong to predct human behavor properly. Perhaps they should be replaced by probablstc ones. The emphass of ths paper, however, s not to doubt the applcablty of the model n prncple but rather to pont at some problems where more research s needed.

41 .--^, r T Report Abstract 9 TOWARD AN ERROR THEORY FOR PP: NFERENCE BASED ON AN ALTERNATVE FORMULATON OF THE DATA SPACE Denns G. Fryback and Ward Edwards Abstract Probablstc nformaton Processng (PP) systems, as currently conceved, use experts' ntutve judgments about the dagnostc mpact of ndvdual data as nputs for mechancal aggregaton by Bayes's theorem. Past research has shown that the posteror odds output by PP are much more extreme than those arrved at va human aggregaton. Because of ths superor effcency PP-t.ype processng of fallble data has been recommended as an mportant tool for decson makng. The present paper questons the uncrtcal use n PP of estmated lkelhood ratos as f they were verdcal. A theory s developed whch ncorporates nto the nferental process the nherent var- ablty of human judgment. The resultng effect s a decrease n the posteror odds gven by PP. Employng specfc dstrbutonal assumptons, a numercal example s gven that shows the possble magntude of ths decrease. Applcaton of the present results and ther mplcatons for further theoretcal and emprcal research are dscussed.