Statistical Techniques For Comparing ACT-R Models of Cognitive Performance

Transcription

1 Statistical Tchniqus For Comparing ACT-R Modls of Cognitiv Prformanc Ryan Shaun Bakr Albrt T. Corbtt Knnth R. Kodingr Human-Computr Intraction Institut, Carngi Mllon Univrsity 5000 Forbs Av. Pittsburgh, PA USA Abstract W discuss how to apply statistical tsts to compar diffrnt ACT-R modls, by trating th ACT-R modls as approximations of th mathmatical modls undrlying thm. To this nd, w propos a mthod for dciding how many tims to run a modl, and a mthod for dtrmining how many fr paramtrs ach modl has. Introduction Usually, thr is mor than on possibl account or modl for a phnomna, or for a st of phnomna. Somtims a rsarch group crats multipl modls varying along som dimnsion of thortical intrst at othr tims, on group of rsarchrs wants to compar thir modl to a prviously publishd modl. In ithr cas, w must dtrmin which modl bttr fits th data. Two broad approachs hav bn takn to compar th quality of diffrnt modls. Th first is to compar th two on nw data. This can b a scond data st takn for th sam task and gnral population, or data from a modratly distant transfr task tsting ach modl s gnrality. Choosing a transfr task can b tricky. If a spcific transfr task is chosn, and modl A is found to b bttr than modl B, thr is no guarant that th opposit would not b tru on anothr transfr task. Nonthlss, it can b an ffctiv mthod for comparing how wll th modls gnraliz, and has bn usd in comptitions btwn diffrnt cognitiv architcturs. (Gluck and Pw 2002) Altrnativly, modls can b compard within th original data st. In comparing two modls, both th absolut fit to th data and th flxibility of th fitting tchniqus must b takn into account. Computational modls hav bn criticizd for not taking full account of th rlativ complxity of th modls bing compard (Robrts and Pashlr 2000) and to addrss this concrn, cognitiv modlrs hav bn paying incrasing attntion to modl slction formulas dvlopd by th statistical community to xplicitly dal with th numbr of fr paramtrs, (Zucchini 2000) and th cross-validation tchniqus usd in th machin larning community. (Mitchll 1997). Statistical Tchniqus For Modl Comparison In thir xtnsiv rviw of mthods for valuating goodnss of fit, Schunn and Wallach (in prss) conclud that modl slction formulas, dspit thir many advantags, ar too difficult for most computational modlrs to us to compar currnt computational modls. Thy point in particular to th difficulty of dvloping closd-form quations for a computational modl, and th difficulty of dtrmining th propr numbr of fr paramtrs and thir rlativ impact, in th absnc of closd-form quations. On answr to th nd for closd-form quations is to dvlop thm. Dvloping such quations is possibl for ACT-R modls of prformanc that always trminat in a spcific st of bhaviors, as ACT-R s bhavior is basd on a spcific st of wll-dfind closd-form quations. Th spcific quations xplaining th proportions of bhaviors in a givn modl can thrfor b dvlopd using ths quations and th productions inputs and outputs. This approach has bn conductd in th past (Andrson and Matssa 1997), but Schunn and Wallach ar corrct that it bcoms xtrmly difficult and tim-consuming for vn modratly complx modls. For xampl, 55 quations ar ndd to rprsnt a modratly complx modl of arly algbra problm-solving, with as many as 44 trms in a singl quation. (Kodingr and MacLarn 2002) In this papr, w discuss how it may not b ncssary to actually driv quations in ordr to conduct statistical analysis on ACT-R modls, and prsnt a cas-study of using ANOVA and BiC (th Baysian Information Critrion) to compar computational modls. In ordr to prsnt a tractabl first discussion of this topic, w will limit th scop of this papr to modls of cognitiv (as opposd to prcptual and motor) prformanc at a spcific stag in th larning procss, rfrrd to as trminal modls by Salvucci and Andrson (1997). Such modls rflct th assumption that th bhavior bing studid is sufficintly dvlopd that it is not changing during th cours of invstigation an assumption Andrson, Lbir, and Lovtt (1998) rfr to as a typical assumption in much of th xprimntal rsarch on human cognition, ustifid in cass whr th bhavior undr study is at som rlativly asymptotic lvl or th crticial factors bing invstigatd do not chang ovr th rang of xprincs ncountrd in th xprimnt. In othr words, ths modls may involv th cration of nw mmory chunks, but do not involv th cration of nw productions or changs in productions utilitis. A sizabl minority of ACT-R 4.0 and 5.0 modls fit this assumption, including modls of prformanc at th Towr of Hanoi task (Andrson and Lbir 1998), modls of th fan ffct (Andrson and Rdr 1999), and modls of studnt prformanc at ducational tasks (Kodingr and MacLarn 2002), (Noks, Ohlsson, and Corrigan-Halprn 2002). W bliv that th mthods prsntd hr can b xtndd in fairly straightforward ways to modls rlying upon ACT- R s prcptual and motor moduls, modls whr utility paramtrs shift ovr tim, and vn modls whr nw productions ar cratd but w lav this for futur work.

2 Computational Modls Approximat Mathmatical Modls In ffct, whn a computational modl is run onc, it givs an approximation of th mathmatical quations that can b usd to dscrib it. (Simon 1992) By running it a gratr numbr of tims, it producs a mor accurat approximation of th solution of thos quations. As th numbr of runs approachs infinity, th rror of th approximation will rach 0. Th qustion thn bcoms how many tims th modl should b run to appropriatly approximat th mathmatical stimats of th data. On consrvativ stratgy is to mak sur th modl s rsults will fall within a crtain confidnc intrval a pr-slctd prcntag of th tim, givn th worstpossibl standard dviation s of rsults (which will b th squar root of half th th rang of th possibl valus of th data which is 0. 5 for data consisting of proportions of rsults xprssd btwn 0 and 1). Th quation for computing th dsird sampl siz (n) is xprssd in trms of th dsird prcntag of th tim th modl will b within th givn rang (α), th valu of th t- distribution corrsponding to α, givn th sampl siz ( t n ( α / 2)), th distanc allowd in ithr dirction from th actual proportion (d), and s. Using th standard quation for confidnc intrvals for a t-distribution 1, w find: 2 s d ( tn ( α / 2)) or s n ( tn ( α / 2) ) n d In choosing how tight to mak th confidnc intrvals, it is worth considring how tight th confidnc intrvals of th data st ar. Thr is no pnalty for making th modl s stimats arbitrarily tight (xcpt for tim), but thr is also not much nd to mak thos stimats ordrs of magnitud tightr than th stimats in th original data st. If that lvl of prcision is ndd to compar diffrnt candidat modls, thr is considrabl risk of dtrmining which modl bttr fits th rror in th data rathr than which modl bttr fits th data itslf. So, for xampl, if it was udgd appropriat to mak sur that vry proportion will b found within 5% in ithr dirction 95% of th tim (p0.05 that it will b outsid that rang), thn using th valus (d 0.05, α0.05, s 0. 5 ), n should qual at last 778. Hnc, w rcommnd modlrs dsiring this lvl of prcision run thir modl 778 tims whn making final calculations of th modl s fit to th data. In principl, a lowr minimum n might b found by taking into account th lvl of stochasticity in th modl to dtrmin a lowr valu for s, but in practic this is 1 By th cntral limit thorm, th stimats of th man and varianc should b approximatly normally distributd for larg sampls, vn if th population (of th rsults of th ACT-R modl) has a vry diffrnt distribution, (Stilson 1966) allowing us to us th t distribution. In cass whr ACT-R s bhavior is xtrmly skwd and long-taild, a phnomnon obsrvd in ACT-R modls involving utility larning (Young and Cox 2002), transformation mthods can b usd to incras th sampl s normality. (Ramsy and Schafr 1997) This should not b a substantial problm, howvr, for th prformanc modls discussd hr. not ncssary, sinc th mthod prsntd hr producs consrvativ but tractabl minimum bounds on th numbr of runs ncssary. Aftr running th modl an appropriat numbr of tims to closly approximat th closd-form quations, w can trat th modl s rsults th sam way w would trat what would rsult from closd-form quations. By taking th diffrnc btwn its prdictions and th data valus, w can comput rsiduals. 2 Ths can thn b usd to mak modl comparisons. Th othr pic of information which will b ndd to conduct ths analyss is th numbr of fr paramtrs ach modl uss, which will b discussd in th following sction. Assssing Modl Complxity: How Many Fr Paramtrs Ar Ndd? Whn comparing two modls of data, it is important not ust to compar th closnss of thir fit to th data st but thir comparativ complxity. Th mor complx a modl is, th mor likly it can closly fit an arbitrary data st, or th rror in that data st, by chanc. This limits that modl s gnralizability, a phnomnon usually trmd ovrfitting. To addrss this, svral mthods hav bn dvlopd for assssing th comparativ complxity of diffrnt modls, and th intraction btwn this and thir goodnss-of-fit. Som approachs to computing complxity, such as Minimum Dscription Lngth (MDL), tak into account th rlativ influnc of diffrnt factors on th numbr of potntial fits th modl can mak (Pitt t al 2002). Othr mthods, such as th Baysian Information Critrion (BiC), us a mor approximat masur of complxity, by idntically trating ach factor (trmd a paramtr) that can affct th modl s rsults, and counting th numbr of ths paramtrs for ach candidat modl (Raftry 1995). Th dbat btwn ths two stratgis for complxity analysis is currntly vry activ in th statistical community. In this papr, w will b following th paramtr-counting approach, as it offrs substantial information and is much asir to conduct in th absnc of closd-form quations. Thr ar thr potntial sourcs of fr paramtrs in an ACT-R modl: its productions, its chunks, and its ACT-R global paramtrs. In th following sctions, w will discuss how to count th paramtrs from ach of ths sourcs. Productions In ordr to dtrmin how many fr paramtrs can b accountd for from th productions, w nd to analyz th quations that undrli ACT-R 5.0. (ACT-R Rsarch Group 2002) In ACT-R, th liklihood that any production will b usd is basd on th production s utility, U i ρ i G - C i + ε, whr ρ i stands for th (xpctd) probability that firing th production will lad to corrctly complting th currnt obctiv, G stands for th valu of th obctiv, and C i stands for th xpctd cost of accomplishing th obctiv. 2 It would b valuabl to rlat th uncrtainty in ths rsiduals to th uncrtainty capturd by th various goodnss-of-fit/flxibility-of-fit critria w discuss latr in th papr, and this is an ara w intnd to invstigat. At this point, though, it is sufficint to not that th uncrtainty of ths rsiduals can b mad substantially smallr than th diffrnc in uncrtainty btwn th modls.

3 ε stands for th nois addd to th rsult (in ordr to dtrmin what th proportions of diffrnt rsults will b), and is calculatd using a logistic distribution with a man of 0 and a varianc dtrmind using a global paramtr, s. Givn xpctd utility U i, th probability that a givn production will fir at any givn point is computd as follows, whr rangs ovr all productions that could fir at this point: P( Pi) Ui For xampl, whn thr ar two productions that could fir, th following quations ar usd: P( P1) U 2, P( P2) U U In gnral, whn computing th probability a spcific bhavior will b xprssd, it is ncssary to multiply togthr th probabilitis of ach production in ach chain of productions that producs that bhavior. Thus, if rsult A is producd solly by production P1, P(A) P(P1). If rsult A is producd by productions P1 and P3 in combination, or by productions P2 and P4 in combination, thn P(A) P(P1)*P(P3 P1) + P(P2)*P(P4 P2). Thrfor, th probability of bhavior A dpnds at last on vry production that could hav fird to produc bhavior A. It also dpnds on all of th othr productions that could hav fird at thos stps and producd a diffrnt rsult, as thos productions utilitis ar usd in th dnominator of th probability of ach production. Hnc, ach of th productions that could hav fird at thos stps must b countd as a fr paramtr. Givn this, our stratgy for stimating th numbr of paramtrs mor or lss follows Simon s (1992) suggstion that vry production b countd as a fr paramtr. Howvr, w rcommnd a fw rfinmnts on this gnral approach. For instanc, som productions do not nd to b countd as fr paramtrs. Such productions fall into two catgoris: First, productions which do not affct th rsults which will b compard to data. Almost vry modl will hav a fw productions that ar ssntial to th implmntation but ar not part of th modl of knowldg: productions that handl book-kping, productions that prpar th modl for anothr run, and so on. Gnrally, ths productions occur vry run or cycl, or always co-occur with othr productions lading us to our scond catgory. If two productions P1 and P2 always co-occur (aftr P1 firs, P2 always firs it nvr fails to fir, and thr is no othr production that could fir in its stad), thy can and should b countd as only on fr paramtr -- vn if thr is a st of productions that fir in btwn P1 and P2. Co-occurrnc can b dtrmind during modl dsign, by inspction, or via posthoc snsitivity analysis. (Kodingr and MacLarn 2002) Byond ths cass, vry production should b countd as at last on fr paramtr. Evn if two productions ar yokd togthr to hav xactly th sam ρ, ach production s xistnc can produc qualitativly diffrnt bhavior and affcts th utility of th othr productions. Thr ar vn cass whr a production should b countd as two paramtrs. If both of a production s trms involvd in computing utility -- ρ i and C i -- ar allowd to float, thn that production should b countd as two fr paramtrs. If only ρ i or C i, or nithr of th two, is dfind for th production, thn it will count as on fr paramtr. Mmory Similar analysis can b applid to calculating th numbr of fr paramtrs accounting for th dclarativ chunks within a givn modl. During dclarativ rtrival, th activation of any givn chunk i quals its Bas-Lvl Activation (B i ) plus th total sprading activation givn by othr chunks. Th sprading activation of a givn chunk, writtn W S i in th quation blow, is th product of W, which quals th global activation paramtr Ga, dividd by th numbr of chunks that rfrncs. A B W S i i + i This formula is thn usd to comput th probability of rtrival and th latncy takn to rtriv th chunk. Ai P( i), RT ( i) F Ai A As can b sn, th formula for th probability of rtriving a chunk is th sam as th formula of th probability of choosing a matching production, xcpt for th substitution of activation for utility. Calculations of latncy rquir th sam information as calculations of probability of rtrival, as such calculations rly upon activation and th production having alrady bn rtrivd. Dtrmining th numbr of fr paramtrs givn by th dclarativ chunks thus rlis on th fr paramtrs usd in dtrmining activation, which includs all of th chunks that could hav bn rcalld. Sinc activation includs sprading activation, it also includs vry chunk that sprads activation to on of thos chunks. Hnc, vry chunk that can b rtrivd, or that sprads activation to a chunk that can b rtrivd, should b countd as a fr paramtr. Not that this dos not ust apply to chunks that xistd at th bginning of th modl s run. If th modl crats dclarativ chunks during its run, ths chunks nd to also b includd in th counts of fr paramtrs. In gnral, vry uniqu chunk that is cratd on any run should b countd as a fr paramtr ach chunk s xistnc or non-xistnc on any spcific run crtainly affcts th quations that dscrib th modl s prformanc. Two chunks can b considrd uniqu if thr ar any situations whr on would b rtrivd or sprad activation, and th othr would not. As with productions, som chunks do not nd to b countd. If thr is a chunk which is only usd for information storag rathr than to produc th pattrn of rsults in th modl, and it sprads no activation, it can b xcludd. Such a chunk must ncssarily fulfill two conditions: its rtrival nvr fails, and thr is nvr a cas whr it is compting with anothr chunk for rtrival. Additionally, a chunk can b rmovd from considration as a paramtr if its slots do not chang and it is associatd on-to-on with a spcific prcding production th production always lads to th chunk bing rtrivd, not to a failur or anothr chunk.

4 ACT-R Global Paramtrs Th third sourc of fr paramtrs is ACT-R s global paramtrs. W propos hr that ACT-R global paramtrs b tratd as fr paramtrs and givn th sam wight as productions and chunks (and will discuss th limitations to this approach in th nxt sction). Howvr, not all ACT-R global paramtrs that xist nd to b countd as fr paramtrs. Any paramtr chosn bfor any modl-fitting is attmptd can b tratd a constant rathr than as a fr paramtr. Th fact that som global paramtrs can b xcludd from considration ncssarily calls for honsty on th part of modlrs as to what paramtrs wr allowd to vary at any point, and which wr chosn bforhand; but this should b asy to discrn. In practic, if a paramtr is lft at ACT-R dfault valus, at 0, or at a wll-known paramtr drivd from prvious xprimnts (as in Lbir and Wst, 1999 and Lbir, Wallach, and Wst, 2000) and its valu was nvr manipulatd, than it can b omittd from th list of fr paramtrs. But if it was vr twakd, it should b tratd as a fr paramtr. Summary: Computing th numbr of fr paramtrs For trminal ACT-R modls, w rcommnd th following approach (givn th cavats discussd in th sction abov): Us a minimum of on paramtr pr production usd during th stps of intrst. If both Pi and Ci vary, us two. Us on paramtr pr mmory lmnt which is usd in th stps of intrst, and which ithr compts with anothr chunk or can fail to b rtrivd. Includ a paramtr for any othr mmory lmnt that sprads activation to on of th mmory lmnts that can b rtrivd in th stps of intrst. Us on paramtr pr ACT-R global paramtr allowd to vary. Again, w bliv it is both possibl and dsirabl to xtnd this approach both to modls which larn, and modls with radically diffrnt ratios of diffrnt typs of paramtrs. W lav this to futur work. W conclud by again rminding our radrs to carfully documnt what productions and mmory chunks ar tratd as fr paramtrs. Whn comparing two modls, spcially thos producd by diffrnt rsarchrs, it is of paramount importanc that fr paramtrs ar countd in th sam fashion for ach modl. A Cas Study in Modl Comparison W hav had th opportunity to xplor som of ths idas in comparing computational modls of studnt rrors in constructing scattrplots of data (Bakr, Corbtt, and Kodingr 2001,2002a), in ordr to inform th dsign of a cognitiv tutor (Bakr t al 2003) Scattrplots should contain th rlationship btwn two quantitativ variabls, but whn studnts wr givn two such variabls, plus a catgorical variabl as a distractor, studnts frquntly committd two concptually similar rrors. Whn givn no advic on which variabls to plac in thir graph, 15% mad what w call th variabl choic rror, incorrctly choosing a catgorical variabl for th X -- 0% usd th corrct variabls. Naming th variabls to us in th qustion did not liminat this rror, but 77% usd th corrct variabls. 13% of thos studnts, howvr, thn mad what w trm th nominalization rror: trating th valus of th quantitativ X variabl as if thy wr catgorical. Thy wrot th variabl s valus along th axis in th ordr thy appard in th data tabl, rathr than numrical ordr,.g., placing along th axis rathr than It was also found that labling th axis variabls for th studnt did not significantly rduc th rprsntation rror s frquncy. Th frquncy of ths rrors is shown in Tabl 1. Givn th concptual similarity btwn ths two rrors, w wondrd if thy could b xplaind as xcution of th sam stratgy or as th xcution of diffrnt stratgis producing similar rsults. W wr also intrstd in dtrmining what typ of bhavior undrlid corrct prformanc in this domain, and what th rol of factors such as th variabls in th qustion was. Fitting and comparing modls of th data W cratd a st of ACT-R modls that rprsntd this data, and compard thir ability to fit th data. For ach modl, w usd multipl runs with diffrnt starting points of an itrativ gradint dscnt algorithm (courtously providd to us by Christian Lbir) to find th bst possibl paramtrs. During runs of IGD, w minimizd a function combining r 2 and th Man Absolut Dviation (MAD). To comput ach modl s prdictions at ach stp of th procss, w ran vry condition of ach modl 778 tims. Th data and th prdictions of our modls wr rprsntd as th proportions of occurrnc of ach bhavior, with th probabilitis of th vnts of th scond stp as probabilitis contingnt on corrct bhavior on stp 1. (This rvald that thr wr no obsrvations for stp 2 in th no prompts condition, bcaus no studnt mad it to stp 2. Thus, w xcludd thos clls during data fitting.) Only 3 global ACT-R paramtrs wr allowd to vary: th utility (:ut) and rtrival (:rt) thrsholds, and th xpctd gain (:gs). W had 6 dclarativ chunks that could b somtims rtrivd in plac of on anothr, giving six mor paramtrs. Sinc bas-lvl activation was st much highr than th rtrival thrshold, failur to rtriv a mmory chunk did not occur, and th othr chunks did not nd to b countd as paramtrs. W allowd th ρ of th productions which producd stratgic dcisions to vary, but in accordanc with th policy dcidd on arlir, countd vry production usd in th stps of intrst as a paramtr, xcpt for productions which always fird and only fird aftr anothr spcific production had fird.

5 Variabl choic rror Corrct axis variabls (CAV) Givn CAV, nominalization rror on X axis only Givn CAV, nominalization rror on Y axis only Givn CAV, nominalization rror on both axs No prompts No labls X variabl labld Y variabl labld Both variabls labld n/a n/a n/a Givn CAV, corrct variabl rprsntation on ach axis n/a Tabl 1: Frquncy of diffrnt bhaviors in (Bakr, Corbtt, and Kodingr 2001,2002) Givn this, th total numbr of paramtrs fll btwn 28 and 34 for th diffrnt modls. 3 It is rlvant to not that, by comparison, a prior modl of th sam phnomna which usd ACT-R 4.0-styl rtrivals (Bakr, Corbtt, and Kodingr 2002b) usd 18 paramtrs in th modl corrsponding to our currnt 34 paramtr modl. This is bcaus ACT-R 5.0 modls ar of substantially finr granularity than ACT-R 4.0 modls, and suggsts that modls in th two architcturs should not b dirctly compard using th mthods prsntd hr. In th long trm, an architctur that compils dirctly btwn diffrnt grain-sizs, such as ACT-Simpl (Salvucci and L 2003), may rndr this limitation lss rlvant. W usd th xtra-sums-of-squars-f-tst and BiC, th Baysian Information Critrion (Raftry 1995), for our modl comparisons. Th F-tst was usd to dtrmin whthr thr was a statistically significant diffrnc btwn two of th modls which appard to xplain substantially diffrnt amounts of th data for situations whr on modl was a subst of th othr, and th BiC was usd to compar th rlativ probability of modls whr ithr of ths conditions did not hold. In ordr to us ths mthods, w ndd ach modl s rsiduals compard to th original data, computd by subtracting th matrix of modl prdictions from th matrix of valus in th original data (shown in Tabl 1), and ach modl s dgrs of frdom. 3 Th low ratio btwn numbr of proportions in our data and numbr of fr paramtrs might suggst our modls ar ovr-fit, but th proportions ar basd on th prformanc of 146 studnts, and th modl s prformanc could thrfor b r-analyzd as th rsiduals on ach of th 3,796 cass. Sinc this would only affct assssmnts of th ovrall quality of th modls, th simplr charactrization of th data is prfrabl, bing asir to us to compar th two modls. Additionally, a low ratio btwn data st siz and fr paramtrs dos not ngativly affct ithr of th mthods w us in th nxt sction. Modl Comparisons Th first issu w studid through modl comparison was whthr thr was vidnc that any of th studnts in th original study, who had compltd a unit of traditional classroom instruction on scattrplots in th prvious yar, had any undrstanding of scattrplots at all. W compard a modl whr som studnts undrstood what typ of information was usd in scattrplots and othr studnts undrstood how to rprsnt quantitativ variabls proprly (KNOW-IT-ALL) to a modl whr studnts undrstood th information usd in scattrplots but knw nothing about quantitativ variabls outsid that contxt (KNOW-SCATTERPLOTS), and to a modl whr studnts did not know anything about scattrplots but knw how to rprsnt quantitativ variabls proprly (KNOW-QUANTITATIVES). KNOW-IT-ALL achivd an xcllnt fit to th data st, with an r 2 of 0.972, but dspit having fwr paramtrs, KNOW-QUANTITATIVES achivd an vn bttr fit to th data, with an r 2 of Givn this it was unsurprising that thr was vry strong vidnc that KNOW- QUANTITATIVES was mor probabl (BiC181.8) than KNOW-IT-ALL (BiC194.1) 4 KNOW-SCATTERPLOTS achivd substantially poorr fit to th data than ithr of ths modls, with an r 2 of Th diffrnc btwn KNOW-SCATTERPLOTS and KNOW-IT-ALL was significant, F(26,1)659.6, p0.03, and thr was vry strong vidnc that KNOW- QUANTITATIVES (BiC181.8) was mor probabl than KNOW-SCATTERPLOTS (BiC270.0). Ths modl comparisons dmonstrat that thr is no vidnc that ths studnts knw anything about scattrplots at all -- th modl whr no studnts knw anything about scattrplots was found to b th most probabl. On th othr hand, if studnts did not undrstand quantitativ variabls in and of thmslvs, it substantially rducd th modl s fit. A scond issu w invstigatd through modl comparison was whthr th studnts usd th information givn in th qustion (which implicitly indicatd which variabl to plac on ach axis). W compard modl KNOW-IT-ALL to a modl whr studnts could not us th information givn in th qustion (CAN T-USE-QUESTION). CAN T-USE- QUESTION had a considrably wors fit on th surfac, with r , and fit th data significantly lss wll, F(26,2)105.1, p0.01. Hnc, our modling providd vidnc that many studnts wr using th information in th qustion to gt corrct rsults. A third issu w invstigatd through modl comparison was whthr th variabl choic rror and nominalization rror stmmd from studnts randomly choosing variabls and randomly choosing how to rprsnt th givn variabls, or from inappropriat transfr of knowldg of how to choos and rprsnt information in bar graphs. Modl KNOW-IT- ALL modld som studnts as knowing bar graphs and attmpting to crat thm in th task at hand, whras modl DON T-KNOW-BAR-GRAPHS liminatd all such skill hnc, any instancs of th variabl choic rror or nominalization rror would occur bcaus of random choic 4 Whn intrprting valus of BiC, th absolut valus of BiC for ach modl ar unimportant compard to th valus of th modls vis-à-vis ach othr. A diffrnc of mor than 6 indicats strong vidnc, and mor than 10 indicats vry strong vidnc. (Raftry 1995)

6 (though th dgr of prfrnc for quantitativ or nominal variabls could still b othr than 50/50 at ach stp). DON T- KNOW-BAR-GRAPHS had poorr surfac fit, with r , and fit th data significantly lss wll, F(26,6)16.94, p< Hnc, it sms most likly that studnt prformanc was affctd by transfr of pr-xisting knowldg about bar graphs. Conclusion In this papr, w prsntd a st of tchniqus for making a principld comparison of th goodnss of fit of two computational modls without dvloping closd-form quations for thos modls. W prsntd a procdur that trats th computational modls as approximations of th closd-form quations which can b drivd from thm, and showd how to dtrmin a rasonably fair numbr of fr paramtrs for thos modls. W thn showd how this procdur was usd in conducting statistical tsts to compar diffrnt modls of studnt rrors in scattrplot gnration. Acknowldgmnts This rsarch was supportd by an NDSEG (National Dfns Scinc and Enginring Graduat) Fllowship, by a rsarch contract from Carngi Larning Inc: "Cognitiv Tutors for Middl School Mathmatics", and by NSF grant numbr to "CIRCLE: Cntr for Intrdisciplinary Rsarch in Constructiv Larning Environmnts". W would lik to warmly thank Brian Junkr, Christian Schunn, and Rhiannon Wavr for hlping us rfin many of th idas in this papr, and Christian Lbir for providing us with th implmntation of itrativ gradint dscnt w usd to rfin our modl s prdictions. W would lik to also thank Bnoit Hudson, Samul Bakr, Adam Fass, John Graham, Andrw Ko, Bnamin MacLarn, Hddrik van Rin, Irina Shklovski, Atsushi Trao, and othrs for hlpful discussions and suggstions. Rfrncs ACT-R Rsarch Group, (2002) ACT-R 5.0 Tutorial Units. r.psy.cmu.du/tutorials/ Andrson, J.R. & Lbir, C. (1998) Knowldg Rprsntation. In Andrson, J.R. & Lbir, C. (Ed.) Atomic Componnts of Thought. Mahwah, NJ: Lawrnc Erlbaum Associats. Andrson, J.R. & Matssa, M.P. (1997) A production systm thory of srial mmory. Psychological Rviw, 104, Andrson, J.R. & Rdr, L.M. (1999) Th fan ffct: Nw rsults and nw thoris. Journal of Exprimntal Psychology: Gnral, 128, Bakr R.S., Corbtt A.T., Kodingr K.R. (2002a) Th Rsilinc of Ovrgnralization of Knowldg about Data Rprsntations. Prsntd at Amrican Educational Rsarch Association Confrnc. Bakr R.S., Corbtt A.T., Kodingr K.R. (2002b) Distinct Errors Arising From a Singl Misconcption. Publishd as abstract, Procdings of th Cognitiv Scinc Socity Confrnc, p. 990, Bakr R.S., Corbtt A.T., Kodingr K.R. (2001) Toward a Modl of Larning Data Rprsntations. Procdings of th Cognitiv Scinc Socity Confrnc, Bakr, R.S., Corbtt, A.T., Kodingr, K.R., Schnidr, M.P. (2003) A Formativ Evaluation of a Tutor for Scattrplot Gnration: Evidnc on Difficulty Factors. To Appar At Confrnc on Artificial Intllignc in Education. Gluck, K. A., Pw, R. W. (2002) Th AMBR Modl Comparison Proct: Round III Modling Catgory Larning. Procdings of th Cognitiv Scinc Socity Confrnc, 24, Kodingr, K.R., MacLarn, B.A. (2002) Dvloping a Pdagogical Domain Thory of Early Algbra Problm Solving. Tchnical Rport CMU-HCII , Carngi Mllon Univrsity, Pittsburgh, PA. Lbir, C., Wallach, D., & Wst, R. L. (2000). A mmorybasd account of th prisonr's dilmma and othr 2x2 gams. In Procdings of Intrnational Confrnc on Cognitiv Modling, NL: Univrsal Prss. Lbir, C., & Wst, R. L. (1999). A dynamic ACT-R modl of simpl gams. In Procdings of th Twnty-first Confrnc of th Cognitiv Scinc Socity, pp Mahwah, NJ: Erlbaum. Mitchll, T. (1997) Machin Larning. Boston, MA: WCB/McGraw-Hill. Noks, T., Ohlsson, S., and Corrigan-Halprn, A. (2002) Larning by analogy vs larning by instruction: Sam knowldg, diffrnt rprsntations. Procdings of th 9 th Annual ACT-R Workshop. Pitt, M.A., Myung, I.J., & Zhang, S. (2002) Toward a mthod of slcting among computational modls of cognition. Psychological Rviw, 109, Raftry, A.E. (1995) Baysian Modl Slction. Sociological Mthodology, Robrts, S. & Pashlr, H. (2000) How Prsuasiv Is a Good Fit? A Commnt on Thory Tsting. Psychological Rviw, 107, 2, Salvucci, D. & Andrson, J.R. (1998) Analogy. In Andrson, J.R. & Lbir, C. (Eds.) Th Atomic Componnts of Thought, Mahwah, NJ: Erlbaum. Salvucci, D.D. & L, F.J. (2003) Simpl Cognitiv Modling in a Complx Cognitiv Architctur. Procdings of th Association of Computing Machinry Confrnc on Computr-Human Intraction (CHI 2003), Schunn, C. D. & Wallach, D. (2001) Evaluating Goodnss-of- Fit in Comparison of Modls to Data. Onlin Manuscript. Simon, H.A. (1992) What Is an Explanation of Bhavior? Psychological Scinc, 3 (3), Stilson, D.W. (1966) Probability and Statistics in Psychological Rsarch and Thory. San Francisco, CA: Holdn-Day. Zucchini, W. (2000) An Introduction to Modl Slction. Journal of Mathmatical Psychology, 44, Young, R.M. and Cox, A. (2002) Random walk procsss in ACT-R mchanisms lad to a wild distribution of larning tims. Papr prsntd at th Eighth Annual ACT-R Workshop, Pittsburgh, PA.