A Numerical Analysis of the Effect of Sampling of Alternatives in Discrete Choice Models

Transcription

1 A Numrical Analysis of th Effct of Sampling of Altrnativs in Discrt Choic Modls Sriharsha Nrlla and Chandra R. Bhat Th Univrsity of Txas at Austin, Dpartmnt of Civil Enginring 1 Univrsity Station C1761, Austin, Txas, Phon: , Fax: hi_harsha@yahoo.com, bhat@mail.utxas.du TRB 2004: FOR PRESENTATION AND PUBLICATION TRB Papr # Final Submission Dat: March 30, 2004 Word Count: 7,690

2 Nrlla and Bhat ABSTRACT A larg numbr of altrnativs charactriz th choic st in many activity and travl choic contxts. Analysts gnrally sampl altrnativs from th choic st in such situations bcaus stimating modls from th full choic st can b vry xpnsiv or vn prohibitiv. This papr undrtaks numrical xprimnts to xamin th ffct of th sampl siz of altrnativs on modl prformanc for both an MNL modl (for which consistncy with a subst of altrnativs is guarantd) and a mixd multinomial logit modl (for which no consistncy rsult holds).

3 Nrlla and Bhat 1 1. INTRODUCTION Svral of th activity and travl dcisions mad by individuals, such as travl mod choic, activity participation location choic, rsidntial location choic, and rout choic, ar discrt in natur. This rcognition has ld to th widsprad us of discrt choic modls in travl dmand modling. Almost all of ths discrt choic modls ar basd on th Random Utility Maximization (RUM) hypothsis, which assums that a dcision-making agnt s choic is a rflction of undrlying prfrncs for ach of th availabl altrnativs, and that th agnt slcts th altrnativ with th highst prfrnc or utility. Th undrlying prfrncs ar random to th analyst, bcaus s/h dos not obsrv all th factors considrd by th dcisionmakr in th choic procss. An issu that ariss in th RUM-basd discrt choic modling of many activity and travl rlatd dimnsions is th larg numbr of altrnativs in th choic st. For xampl, in an activity participation location or rsidntial choic situation, a dcision-makr can potntially hav anywhr btwn a fw hundrds of choic altrnativs (if an aggrgat spatial unit such as nighborhoods or traffic analysis zons is usd to charactriz th altrnativs) to hundrds of thousands of choic altrnativs (if a fin spatial rsolution such as land parcls is usd to charactriz th altrnativs). Similarly, in a rout choic dcision contxt, a travlr potntially has an infinit numbr of routs to choos from to travl to his/hr dsird location for activity participation. In such larg choic st situations, it is challnging to considr all th altrnativs during stimation bcaus of th substantial ffort that would b ntaild in assmbling th rlvant datast. Th computational burdn can also b an important considration in stimation with a vry larg st of altrnativs. 1 Th challng of stimating choic modls with a hug st of altrnativs has ld rsarchrs to xplor and apply mthods to nabl consistnt stimation with only a subst of altrnativs (s Tabl 1 for a list of studis that hav usd a subst of altrnativs rathr than th complt choic st). McFaddn (3) provd that, in th cas of th multinomial logit modl (MNL), it is straightforward to consistntly stimat paramtrs from a sampl of altrnativs by maximizing a conditional liklihood function which also has an MNL form. This is a nat thortical rsult and is associatd with th indpndnc from irrlvant altrnativs (IIA) proprty of th multinomial logit modl. Howvr, thr has bn no systmatic numrical analysis, to our knowldg, xamining how th sampl siz of altrnativs affcts th mpirical accuracy and fficincy of th stimatd paramtrs. Anothr issu in choic situations with a larg numbr of altrnativs is th cas whn non-mnl modls ar usd. Th MNL modl, whil simpl and lgant in structur, is saddld with th IIA proprty, which can b bhaviorally unralistic in many choic situations. For xampl, in an activity participation location or rsidntial choic situation, it is possibl (if not vry likly) that th utility of spatial altrnativs clos to ach othr will hav a highr dgr of snsitivity du to common unobsrvd spatial lmnts. A common spcification in th spatial 1 On a philosophical not, on could argu that individuals ar limitd information procssors, and do not considr mor than a fw altrnativs in any choic situation. Clarly, this is an important rsarch issu within th broad ara of undrstanding th choic st gnration procss. Howvr, in th absnc of a clar undrstanding of th choic st gnration procss, th most common practical assumption is that all altrnativs in th univrsal choic st ar availabl. Th currnt papr is positiond within th framwork of this assumption of full choic st considration. Howvr, it should b mphasizd that th papr provids guidanc vn for modling framworks that incorporat choic st formation xplicitly [for xampl, s Swait (1); Basar and Bhat (2)]. In such framworks, th modl taks a non-mnl form with th full choic st rprsnting th univrsal choic st from which som altrnativs ar considrd by th dcision makr. To th xtnt that th univrsal choic st is vry larg, th analyst may want to rduc th univrsal choic st siz to somthing managabl for ach dcision makr. This situation is mimickd by our analysis in this papr with non-mnl modls.

4 Nrlla and Bhat 2 analysis litratur for capturing such spatial corrlation is to allow contiguous altrnativs to b corrlatd (4). Similarly, in a rout choic contxt, routs with ovrlapping links ar likly to hav a highr snsitivity btwn ach othr compard to paths with littl or no ovrlap. A common spcification, thrfor, in rout choic modls is to assum that th covarianc of path utilitis is proportional to th ovrlap lngth (5). In ths and othr choic situations, th us of th MNL modl is clarly not appropriat, though th analytic lganc and ability to sampl altrnativs within th MNL framwork has ld to its continud us in th litratur. Rcnt simulation-rlatd and GEV-basd modl dvlopmnts, howvr, ar vry rapidly librating th analyst from using rstrictiv modl forms such as th MNL. But, thortically spaking, sampling of altrnativs dos not provid consistnt paramtr stimats in ths mor advancd modl forms. Thus, th dilmma for th analyst is whthr to impos th unralistic MNL structur at th outst or us a mor ralistic structur and thn potntially undo th advantag of th richr structur by sampling of altrnativs. Th discussion abov provids th motivation for th currnt rsarch. Spcifically, this papr has two objctivs. Th first objctiv is to xamin th ffct of th sampling siz of altrnativs on th mpirical accuracy and fficincy of stimatd paramtrs (and othr rlvant fit statistics) in th contxt of th MNL modl. Whil McFaddn s (3) rsult shows thortically that any sampl siz of altrnativs will provid consistnt stimats in th MNL framwork, th ustion of how many altrnativs to slct is still an mpirical on. Th scond objctiv is to assss th impact of th sampling siz of altrnativs on th mpirical accuracy and fficincy of paramtr and fit statistics in th contxt of non-mnl modls. In such modls, it is thortically known that sampling of altrnativs dos not work, but th ustion is: Is thr a crtain siz of altrnativs that maks th rsults from th sampl of altrnativs clos nough (mpirically spaking) to th tru valus obtaind from th full choic st? A fw nots ar in ordr bfor w procd. First, w us th mixd multinomial logit (MMNL) form as th rprsntativ structur for th non-mnl forms in this papr. This is bcaus th MMNL modl is a vry flxibl discrt choic structur, is asy to stimat, and is bcoming th mthod of prfrnc for accommodating bhaviorally ralistic structurs. Scond, our assssmnt of th ffct of sampl siz of altrnativs on modl prformanc is basd on numrical xprimnts. Third, th rsults from this papr should b viwd as providing guidanc to th analyst whn confrontd with a choic situation with a larg numbr of altrnativs. Th rsults should not b viwd as absolut ruls sinc ach mpirical contxt is likly to b uniu and diffrnt from othrs. It is simply impossibl in a numrical xprimnt to considr all th situations that may aris in rality, including combinations of diffrnt sampl sizs of obsrvations, diffrnt numbrs of altrnativs in th univrsal choic st, diffrnt lvls of snsitivity btwn pairs of altrnativs, diffrnt numbrs of variabls usd in th spcification and thir momnt valus, and th varying distributions of th rspons pattrns to variabls in th population. Th rst of th papr is organizd as follows. Sction 2 discusss th MNL and MMNL structurs and th issus involvd in sampling of altrnativs. Sction 3 dscribs th dsign of th numrical xprimnts. Sction 4 prsnts th mpirical rsults and discusss th important findings. Th final sction concluds th papr.

5 Nrlla and Bhat 3 2. THE MODELS 2.1 Th MNL Modl (MNL) Th MNL modl taks th following familiar form for th probability that individual slcts altrnativ i from th st of all availabl altrnativs C. P i = j C β ' X i β ' X j (1) whr X i is a vctor of obsrvd variabls spcific to individual and altrnativ i, and β is a corrsponding fixd paramtr vctor of cofficints. Now, considr that th analyst dcids to us only a subst of altrnativs, D, for individual. Lt ( i) b th probability undr th rsarchr s slction mchanism of π D choosing subst D givn that altrnativ i is chosn by individual. For stimation purposs, D should includ th chosn altrnativ, so that π ( i) = 0 for any D that dos not includ i. Th conditional probability of individual choosing altrnativ i conditional on th rsarchr sampling th subst D for th individual may b drivd in a straightforward mannr using Bays thorm as (6, p.68): P ( i D D Piπ ( D i) Piπ ( D i) ) = = (2) P π ( D j) P π ( D j) j C j j D j Th simplification in th dnominator on th right sid in th uation abov is basd on th fact that π ( D j) = 0 for j not in D. Nxt, for th MNL modl, w can us Euation (1) in Euation (2) to writ: β ' X i β ' X i + ln π ( D i) P π ( D i) ( i D ) = = β ' X j β ' X j + ln π ( D j π ( D j) ) j D j D (3) Th simplification in going from Euation (2) to Euation (3) is basd on th cancllation of th dnominators of P i in th MNL modl (this cancllation is also fundamntally rsponsibl for th IIA proprty). Th analyst can us Euation (3) with any sampling mchanism s/h chooss, and only has to incorporat an additional variabl ln π ( i) in th utility of ach altrnativ. Th cofficint on this variabl is rstrictd to 1 during stimation, which is basd on maximizing th following conditional liklihood function: D C (β) = i D y ln P ( i β, D ) i (4)

6 Nrlla and Bhat 4 McFaddn (3) provs that maximizing th abov function provids consistnt stimats of β. In th typical cas whn th analyst uss a random sampling approach, th following uniform conditioning proprty holds: π ( D i) = π ( D j) i, j D (5) Using this uniform conditioning proprty, Euation (3) collapss to a standard logit modl with a choic st D (a subst of C) for individual. Thus, a random sampling of altrnativs allows consistnt paramtr stimation in th standard multinomial logit modl. 2.2 Th Mixd Multinomial Logit Modl (MMNL) Th MMNL modl is a gnralization of th multinomial logit (MNL) modl. Spcifically, it involvs th intgration of th MNL formula ovr th distribution of random paramtrs. It taks th structur shown blow: P + i Li β ) f ( β θ ) β ' X i = ( dβ, whr L (6) i = j C β ' X j. Th us of th xprssion abov in Euation (2) for th conditional probability of choosing altrnativ i givn subst D immdiatly indicats that thr is no simplification whn sampling altrnativs for th MMNL as for th MNL modl in Euation (3). Th rason is that, for th MNL cas, a cancllation of th dnominators in th probability xprssion taks plac, putting th conditional probability back into th form of a tractabl MNL xprssion. No such simplification occurs for th non-mnl modls, bcaus vn undr th assumptions of a uniform conditioning sampling approach, Euation (2) simplifis only to: P ( i D ) = j P D i ( θ ) P j ( θ ) (7) Th uation abov ruirs th probability of ach altrnativ to b computd with rspct to all altrnativs in th choic st. Thus, no sampling stratgy will work in th cas of th MMNL modl (and mor gnrally, in th cas of othr non-mnl modls too such as th GEV class of modls). But, an approximation in Euation (6) simplifis th xprssion in Euation (7). Spcifically, on can approximat L i in Euation (6) as: L i N S * β ' X i j D β ' X j, (8)

7 Nrlla and Bhat 5 whr S is th numbr of altrnativs in D (i.., th numbr of sampld altrnativs) and N is th numbr of altrnativs in C (i.., th numbr of altrnativs in th univrsal choic st). Th trm (N/S) is a factor that xpands th sum of th dnominator from th sampld altrnativs to th full choic st. Thn, on can writ: β S X i Pi ( θ ) * f ( β θ ) dβ β X i N. (9) j D Euation (7) thn collapss to: P ( i D ) = = k D k D β X β X i β X j β X k k f ( β θ ) dβ f ( β θ ) dβ j k D β X j β X k f ( β θ ) dβ (10) Th simplification abov occurs bcaus th dnominator in th first xprssion of Euation (10) is ual to 1. Thus, with th approximation in (9), th conditional probability is put back into a simpl MMNL xprssion within th st of sampld altrnativs. Of cours, th approximation in (9) is th rason for th simplification. In gnral, th xprssion on th right sid of Euation (9) is not a consistnt stimator of P i (θ). Furthr thortical xploration of this approximation is an important ara for futur rsarch. In th currnt papr, w mpirically tst th ability to rcovr th undrlying paramtrs and othr rlvant statistics using an MMNL modl with a sampl of altrnativs and th xprssion in Euation (10). 3. EXPERIMENTAL DESIGN In th numrical xprimnts of our study, w gnrat two datasts, on for th multinomial logit modl and th othr for th mixd multinomial logit modl. Each datast includs fiv indpndnt variabls for 200 altrnativs for ach of 750 obsrvations. Th valus of th fiv indpndnt variabls for ach of th 200 altrnativs ar drawn from a standard normal univariat distribution with th variabls of th first 100 altrnativs having a man of 1 and th variabls of th othr 100 altrnativs having a man of 0.5. For th multinomial logit datast, th cofficints applid to ach indpndnt variabl for ach obsrvation is takn as 1. Th dtrministic componnt of th utility is thn calculatd. Th rror trm for ach altrnativ and ach obsrvation is drawn indpndntly from a typ I xtrm valu distribution. This is achivd by obtaining draws from th uniform random distribution and applying th transformation -ln(-ln(u)) whr u is a random numbr drawn from th uniform distribution btwn 0 and 1. Th dtrministic and th probabilistic componnts of th utilitis for ach altrnativ and ach obsrvation ar addd nxt to obtain th total utility for

8 Nrlla and Bhat 6 ach altrnativ. Finally, for ach obsrvation, th altrnativ with th highst utility is idntifid as th chosn altrnativ. Th stps involvd in th gnration of th datast for th MMNL modl ar vry similar to thos usd in gnrating th datast for th MNL cas. Th only diffrnc is that two of th fiv indpndnt variabls ar assumd to hav random cofficints. Th random cofficints ar assumd to b distributd univariat normal. As for th MNL data gnration, th man of th cofficints on all fiv indpndnt variabls is takn as 1. Howvr, for two of ths cofficints, w allow randomnss across obsrvations by drawing th cofficint from a univariat normal distribution with a man valu of 1 and a varianc of 1 (this is, of cours, achivd by drawing from a standard univariat normal distribution and adding 1). Th rror trms for th utilitis ar calculatd in th sam way as th MNL modl, and th altrnativ with th highst utility is idntifid as th chosn altrnativ. 4. COMPUTATIONAL RESULTS 4.1 Estimation Issus All th modls wr stimatd using th GAUSS matrix programming languag. Th logliklihood function and th gradint function for both th MNL and MMNL structurs wr codd. Th Halton suncs ruird to simulat th probabilitis in th mixd multinomial logit cas wr also gnratd using GAUSS. In th first st of stimations involving th MNL modl, th cofficints on th fiv indpndnt variabls in th simulatd datast wr first stimatd considring th full choic st of 200 altrnativs. Ths rsults srvd as th bnchmark to valuat th prformanc of th random sampling of altrnativs procdur. Nxt, w considrd 6 diffrnt sampl sizs for th numbr of altrnativs in th random sampling: 5, 10, 25, 50, 100, and 150. For ach siz, th sampling was achivd through a GAUSS cod that, for ach obsrvation, randomly slctd (M- 1) altrnativs (without rplacmnt) from th full choic st xcpt th chosn altrnativ, and thn addd th chosn altrnativ to achiv th dsird siz M. Furthr, for ach sampl siz, th sampling procdur just discussd was rpatd 10 tims using diffrnt random sds to stimat th varianc du to th sampling of altrnativs. In th scond st of stimations involving th MMNL modl, th sam procdur as for th MNL was usd in sampling altrnativs. Unlik th MNL modl, howvr, th maximum liklihood stimation of th MMNL modl ruirs th valuation of an analytically-intractabl intgral. Th stimation is accomplishd through a maximum simulatd liklihood (MSL) approach using scrambld Halton draws with prims of 2 and 3 as th bass for th suncs (7). An important issu hr is th numbr of Halton draws to us pr obsrvation. It is critical that th two-dimnsional intgral in th probability xprssions of th MMNL modl b valuatd accuratly, so that th diffrnc in modl paramtrs btwn using a sampl of altrnativs and th full choic st can b attributd solly to th sampling of altrnativs. In our MSL stimation of th MMNL modl, w usd 200 scrambld Halton draws basd on xtnsiv tsting with diffrnt numbrs of scrambld Halton draws. Spcifically, w stimatd an MMNL modl using th MMNL datast with 5 randomly sampld altrnativs and th full choic st to rprsnt th rang of sampl sizs of altrnativs usd in th xprimnts. For ach of ths two stimations, w stimatd th modl with diffrnt numbrs of Halton draws, and found that th modl paramtrs wr basically indistinguishabl byond 200 Halton draws.

9 Nrlla and Bhat Evaluation Critria Th focus of th valuation ffort is to assss th prformanc of th modls stimatd with a sampl of altrnativs rlativ to th modl stimatd with th full choic st. This valuation was basd on four critria: (a) Ability to rcovr modl paramtrs, (b) Ability to stimat th ovrall log-liklihood function accuratly, (c) Ability to rplicat th choic probability of th chosn altrnativ for ach obsrvation (i.., ability to rproduc th individual liklihood function valus), and (d) Ability to rproduc th aggrgat shars of th altrnativs. For th valuation basd on th lattr thr critria, w applid th stimatd paramtr valus from ach stimation to th full choic st to comput th stimatd choic probabilitis for ach of th 200 altrnativs for ach obsrvation. Th rlvant valus for th thr critria ar thn basd on comparing th prformanc of ach numbr of sampld altrnativs on th full choic st with th tru valus computd from modl stimation using th full choic st. This procdur brings th stimations with diffrnt sampl sizs to a common platform and nabls maningful comparisons of modl prformanc. For ach of th four critria idntifid abov, th valuation of th proximity of th stimatd and tru valus was basd on two prformanc masurs: (a) Root man suar rror and (b) Man absolut prcntag rror. Furthr, for ach critrion-prformanc masur combination, w computd thr proprtis: (a), or th diffrnc btwn th man of stimats for ach sampl siz of altrnativs across th 10 runs and th tru valus, (b) Simulation varianc, or th varianc in th rlvant paramtrs across th 10 runs for ach sampl siz of altrnativs, and (c) Total rror, or th diffrnc btwn th stimatd and th tru valus across all 10 runs for ach sampl siz of altrnativs. Th prformanc statistics wr compard across th diffrnt sampl sizs to undrstand th ffct of random sampling in ach of th two modl structurs (MNL and MMNL), and across th two modl structurs to undrstand th diffrncs of th ffcts of random sampling btwn thm. 4.3 Prformanc Rsults Tabls 2 through 5 prsnt th computational rsults. In ach tabl, th rror masurs dcras in magnitud as th sampl siz incrass, xcpt for som minor abrrations in th bias masur for small sampl sizs. Furthr, in ach tabl, th rror masurs ar largr for th MMNL modl compard to th MNL modl for ach sampl siz of altrnativs. This is to b xpctd, bcaus of th thortical rsult that random sampling is consistnt in th MNL cas whil no such rsult holds for th MMNL cas. W nxt discuss th important rsults from ach tabl in turn. Tabl 2 provids th masurs of th ability to rcovr th modl paramtrs. Svral obsrvations may b mad from th tabl. First, for th MNL modl, a doubling of th sampl siz of altrnativs rducs th RMSE by about a fourth for sampl sizs lss than 50 (xcpt for th dcras btwn 25 and 50 altrnativs), and rducs th RMSE by about a half byond sampl sizs of 50. Similarly, for th MNL modl, a doubling of th sampl siz of altrnativs rducs th MAPE by about a third to a half as th sampl siz of altrnativs is doubld, with th improvmnts in prformanc bing stpr at highr sampl sizs. Ths pattrns ar rflctd in th bias, varianc, and total rror masurs. Scond, for th MMNL modl, a doubling of th sampl siz rducs th bias and th total rror of both th RMSE and MAPE by a fourth to a half up to a sampl siz of 50, but rducs ths masurs by half or mor byond a sampl siz of 50. Th rduction in simulation varianc in th MMNL modl du to incrasing sampl siz is mor dramatic than th rduction in bias and total rror masurs and is rathr consistnt with a 50% rduction or mor for a doubling in sampl siz (xcpt for th rsults

10 Nrlla and Bhat 8 corrsponding to an incras from 50 to 100 altrnativs). Third, th rror masurs for th MMNL modl at a sampl siz of 5 ar about % highr than for th MNL. Howvr, th disparity btwn th MNL and th MMNL masurs rducs to about % for sampl sizs of 10, 25, and 50, and rducs furthr to about 40% byond a sampl siz of 50. Furthr, th MAPE rror is vry high in th MMNL modl for small sizs, and rducs uit substantially at highr sampl sizs. Ovrall, th rsults suggst that it is vry important to us high sampl sizs for th MMNL modl; at last a uartr of th full choic st and idally a half of th full choic st or mor. Tabl 3 provids th rsults for th ovrall log-liklihood function valu. Th tabl shows that th MMNL RMSE rror masurs ar larg at small sampl sizs in both absolut trms and rlativ to th MNL rror masurs. Howvr, at sampl sizs of 100 altrnativs or mor, th rror masurs bcom comparabl btwn th MNL and MMNL modls. Th RMSE bias in this tabl is ngativ bcaus th ovrall data fit from applying th paramtrs stimatd from th sampling of altrnativs procdur to th full st of altrnativs can only b wors than th fit obtaind by using th full choic st in stimation (which is th tru convrgnt log-liklihood function valu). Also, th MAPE bias and rror masurs in Tabl 2 ar xactly th sam bcaus th log-liklihood is ovrstimatd in magnitud (rlativ to th tru valu) by ach of th 10 runs for ach sampl siz. Tabl 4 indicats that th MNL rror masurs ar uit substantial at vry small sampl sizs for th individual log-liklihood function valus (i.. th probability of th chosn altrnativ). For xampl, th man absolut prcntag rror is about 12% whn a sampl siz of 5 is considrd in th MNL. Th prcntag rror is, as xpctd, much highr for th MMNL at small sampl sizs. But, at largr sampl sizs, it is rmarkabl that th MMNL rror masurs ar vry comparabl to thos from th MNL. Again, th rsults show that a sampl siz of 100 or mor altrnativs (or half th full choic st of altrnativs or mor) in th MMNL modl provids good accuracy. Tabl 5 mirrors th rsults from th arlir tabls. Th valus in this tabl for th RMSE ar smallr than for th othr tabls bcaus th shars ar computd at an aggrgat lvl. Th MAPE masur provids a bttr prspctiv hr. To summariz, thr important obsrvations may b drawn from Tabls 2 through 5. First, random sampling of altrnativs provids good stimats vn for small sampl sizs (i.., small numbr of randomly sampld altrnativs) in th MNL modl. Howvr, thr is ithr a constant or incrasing rturns to scal in trms of accuracy and prcision as th sampl siz is incrasd in th MNL modl. Consuntly, for th MNL modl, th analyst would do wll not to sttl for vry small sampl sizs. Our rsults suggst a sampl siz of on-ighth of th full choic st as a minimum, and on-fourth of th full choic st as a good numbr of altrnativs to targt. Scond, and as xpctd, th prformanc of th mixd multinomial logit modl is vry poor at small sampl sizs. Th good nws for th analyst, howvr, is that th rturns from incrasing th sampl siz ar much mor dramatic in th MMNL modl compard to th MNL modl. In fact, at vry high sampl sizs, th accuracy of random sampling is comparabl to th accuracy from th MNL modl. As ovrall guidanc, our rcommndation basd on th rsults would b that th analyst considr a sampl siz no lss than a fourth of th full choic st and prfrably half or mor of th full choic st. Th radr will not that vn using half of th full choic st, though computationally xpnsiv, can still lad to uit considrabl savings in computational tim compard to using th full choic st in th MMNL modl. On th othr hand, using a vry small sampl siz may b good for computational tim, but is litrally

11 Nrlla and Bhat 9 garbag from an accuracy prspctiv. Third, a comparison of th MAPE valus from Tabls 2 and 3 show that th ovrall log-liklihood valu is mor accuratly stimatd than th paramtr valus, spcially for th MMNL modl. This suggsts a rathr flat log-liklihood function nar th optimum; that is, closnss to th log-liklihood function dos not ncssarily imply closnss in modl paramtrs too. In ordr to undrstand th ffct of random sampling for unrstrictd choic sts of smallr sizs than 200, w also gnratd datasts with 100 altrnativs and 50 altrnativs and undrtook th sam kind of analysis as just discussd abov for th cas with 200 altrnativs. Tabls 6 through 9 prsnt th simulation rsults for datasts with 100 altrnativs (w ar not prsnting th tabls for th simulation rsults with 50 altrnativs du to spac constraints). Th pattrns of th rsults ar similar to th cas with 200 altrnativs, and th sam ovrall conclusions may b drawn. 5. SUMMARY AND CONCLUSIONS Many activity and travl choic dcisions mad by individuals involv a larg numbr of choic altrnativs. Exampls includ activity participation location choic, rsidntial location choic, and rout choic. McFaddn (3) provd that, if th analyst is willing to assum a simpl multinomial logit (MNL) formulation for th bhavior undrlying th choic procss, a sampling of altrnativs schm will provid consistnt modl paramtrs. Svral rsarchrs hav xploitd this rsult for stimations in diffrnt mpirical contxts (8-12). Howvr, thr has bn no systmatic study (until this papr) of th ffct of sampl siz on th mpirical accuracy and fficincy of th stimatd paramtrs. Furthr, with rcnt advancs in th fild, rsarchrs ar incrasingly turning to mor bhaviorally ralistic discrt choic modls in analysis for which McFaddn s (3) rsult dos not hold. At th sam tim, sampling of altrnativs can rduc computational tim uit substantially compard to using th full choic st in ths advancd modls. Thus, it is of valu to study th ffct of sampl siz on modl prformanc in ths non-mnl modls. This papr dvlops an valuation framwork for xamining th ffct of th sampl siz of altrnativs on modl prformanc both in an MNL contxt and a mixd multinomial logit (MMNL) contxt. Th rsults from this papr show th good numrical prformanc of th MNL modl vn with vry small sampl sizs. Howvr, sinc th bang for th buck is high as on procds to largr sampl sizs of altrnativs, it is advisabl to considr sampl sizs that ar not too small. Basd on our rsults, w rcommnd th us of an ighth of th siz of th full choic st as a minimum, and suggst a fourth of th full choic st as a dsirabl targt. Th prformanc of th MMNL modl, on th othr hand, is vry poor at small sampl sizs. Howvr, th bang for buck is vn bttr for th MMNL modl with incrasing sampl sizs compard to th MNL cas. At a minimum, w suggst using a fourth of th full choic st. Howvr, w strongly suggst using on-half of th full choic st or mor basd on th numrical xrciss in this papr. As with any numrical xrcis, th usual cautions for gnralizing th rsults apply to this papr too. Thr is crtainly a nd for mor computational and mpirical rsarch on th topic of sampling of altrnativs in diffrnt sttings (such as diffrnt pattrns of corrlation among xognous variabls, diffrnt lvls of snsitivity and htrognity in th snsitivity to variabls, diffrnt numbrs of variabls with random cofficints, and diffrnt numbrs of dcision makrs in th sampl) to draw mor dfinitiv conclusions. In th mantim, th rsults

12 Nrlla and Bhat 10 of this papr should srv as a good guid to th analyst facd with modling choic situations with larg choic sts. A final not bfor closing. This papr should not b viwd as ncouraging sampling of altrnativs in a non-mnl stting. In such sttings, it is always most idal to considr th full choic st. But, if considring th full choic st is difficult for th MNL modl, it is substantially mor difficult for th non-mnl modls. Th purpos of this numrical analysis is to provid som guidanc to analysts wanting to us non-mnl modls, but ar simply unabl to considr th full choic st. Th rsults hr should b viwd as an ffort to st minimum sampl siz guidlins for th MMNL modl whn th full choic st cannot b considrd. ACKNOWLEDGEMENT Th commnts from thr anonymous rfrs ar gratly apprciatd.

13 Nrlla and Bhat 11 REFERENCES 1. Swait, J. Choic St Gnration Within th Gnralizd Extrm Valu Family of Discrt Choic Modls, Transportation Rsarch Part B, Vol. 35, No. 7, 2001, pp Basar, G., and C.R. Bhat. A Paramtrizd Considration St Modl for Airport Choic: An Application to th San Francisco bay Ara. Transportation Rsarch Part B, forthcoming, McFaddn, D. Modling th Choic of Rsidntial Location. In Transportation Rsarch Rcord 673, TRB, National Rsarch Council, Washington D.C., 1978, pp Bhat, C.R., and J.Y. Guo. A Mixd Spatially Corrlatd Modl: Formulation and Application to Rsidntial Choic Modling. Transportation Rsarch Part B, Vol. 38, No. 2, 2004, pp Bkhor, S., Bn-Akiva, M., and M.S. Ramming. Adaptation of Logit Krnl to Rout Choic Situation. In Transportation Rsarch Rcord 1805, TRB, National Rsarch Council, Washington D.C., 2002, pp Train, K. Discrt Choic Mthods with Simulation. Cambridg Univrsity Prss, Cambridg, UK, Bhat, C.R. Simulation Estimation of Mixd Discrt Choic Modls Using Randomizd and Scrambld Halton Suncs. Transportation Rsarch Part B, Vol. 37, No. 9, 2003, pp Srmons, M.W., and F.S. Kopplman. Rprsnting Diffrncs Btwn Fmal and Mal Commut Bhavior in Rsidntial Location Choic Modls. Journal of Transport Gography, Vol. 9, No. 2, 2001, pp Waddll, P. Accssibility and Rsidntial Location: Th Intraction of Workplac, Rsidntial Mobility, Tnur, and Location Choics. Prsntd at th Lincoln Land Institut TRED Confrnc, 1996 ( 10. Bhat, C.R., Govindarajan, A., and V. Pulugurtha. Disaggrgat Attraction-End Choic modling: Formulation and Empirical Analysis. In Transportation Rsarch Rcord 1645, TRB, National Rsarch Council, Washington D.C., 1998, pp Guo, J.Y., and C.R. Bhat. Rsidntial Location Choic Modling: A Multinomial Logit Approach. Tchnical Papr, Dpartmnt of Civil Enginring, Th Univrsity of Txas at Austin, Bn-Akiva, M.E., and J.L. Bowman. Intgration of an Activity-Basd Modl Systm and a Rsidntial Location Modl. Urban Studis, Vol. 35, No. 7, 1998, pp

14 Nrlla and Bhat Srour, I.M., Kocklman, K.M., and T.P. Dunn. Accssibility Indics: A Connction to Rsidntial Land Prics and Location Choics. In Transportation Rsarch Rcord 1805, TRB, National Rsarch Council, Washington D.C., 2002, pp Schlich, R., Simma, A., and K.W. Axhausn. Dstination Choic Modling for Diffrnt Lisur Activitis. 2 nd Swiss Transport Rsarch Confrnc 2002, Ascona, March Pozsgay, M.A., and C.R. Bhat. Dstination Choic Modling for Hom-Basd Rcrational Trips: Analysis and Implications for Land-Us, Transportation, and Air Quality Planning. In Transportation Rsarch Rcord 1777, TRB, National Rsarch Council, Washington D.C., 2002, pp

15 Nrlla and Bhat 13 LIST OF TABLES TABLE 1 Earlir Studis Using a Subst of Choic Altrnativs TABLE 2 Evaluation of Ability to Rcovr Modl Paramtrs (with a choic st of 200 altrnativs) TABLE 3 Evaluation of Ability to Estimat Ovrall Log-Liklihood Function Valu (with a choic st of 200 altrnativs) TABLE 4 Evaluation of Ability to Estimat Individual Choic Probabilitis (with a choic st of 200 altrnativs) TABLE 5 Evaluation of Ability to Estimat Aggrgat Shars of Altrnativs (with a choic st of 200 altrnativs) TABLE 6 Evaluation of Ability to Rcovr Modl Paramtrs (with a choic st of 100 altrnativs) TABLE 7 Evaluation of Ability to Estimat Ovrall Log-Liklihood Function Valu (with a choic st of 100 altrnativs) TABLE 8 Evaluation of Ability to Estimat Individual Choic Probabilitis (with a choic st of 100 altrnativs) TABLE 9 Evaluation of Ability to Estimat Aggrgat Shars of Altrnativs (with a choic st of 100 altrnativs)

16 Nrlla and Bhat 14 Study TABLE 1 Earlir Studis Using a Subst of Choic Altrnativs Choic Modld Altrnativs Considrd (Total Numbr) Sampling Mchanism Guo and Bhat (11) Rsidntial Location TAP Zons (900) Simpl Random Sampling MNL Srmons and Kopplman (8) Rsidntial Location Cnsus Tracts (1099) Simpl Random Sampling MNL Modl Structur Bn-Akiva and Bowman (12) Rsidntial Location TAZ (787) Stratifid Importanc Sampling Nstd Logit Bkhor, S. t al. (5) Rout Choic Routs (50) Simpl Random Sampling Krnl Logit Waddll (9) Rsidntial Location TAZ (761) Simpl Random Sampling Nstd Logit Schlich t al. (14) Dstination Choic Municipal Lvl (555) Simpl Random Sampling MNL Pozsgay and Bhat (15) Dstination Choic TSZ (919) Simpl Random Sampling MNL Bhat t al. (10) Attraction-nd Choic TAZ (858) Importanc Sampling MNL

17 Nrlla and Bhat 15 TABLE 2 Evaluation of Ability to Rcovr Modl Paramtrs (with a choic st of 200 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl RMSE MNL MAPE RMSE MMNL MAPE RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

18 Nrlla and Bhat 16 TABLE 3 Evaluation of Ability to Estimat Ovrall Log-Liklihood Function Valu (with a choic st of 200 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl MNL MMNL RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

19 Nrlla and Bhat 17 TABLE 4 Evaluation of Ability to Estimat Individual Choic Probabilitis (with a choic st of 200 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl MNL MMNL RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

20 Nrlla and Bhat 18 TABLE 5 Evaluation of Ability to Estimat Aggrgat Shars of Altrnativs (with a choic st of 200 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl E E E E E E E E E E E E-05 MNL E E E E E E E E E E E E E E E E E E-05 MMNL E E E E E E RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

21 Nrlla and Bhat 19 TABLE 6 Evaluation of Ability to Rcovr Modl Paramtrs (with a choic st of 100 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl MNL MMNL RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

22 Nrlla and Bhat 20 TABLE 7 Evaluation of Ability to Estimat Ovrall Log-Liklihood Function Valu (with a choic st of 100 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl MNL MMNL RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

23 Nrlla and Bhat 21 TABLE 8 Evaluation of Ability to Estimat Individual Choic Probabilitis (with a choic st of 100 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl MNL MMNL RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error

24 Nrlla and Bhat 22 TABLE 9 Evaluation of Ability to Estimat Aggrgat Shars of Altrnativs (with a choic st of 100 altrnativs) Modl Prformanc Masur Estimator Proprty 5 Numbr of altrnativs considrd in th random sampl E E E E E E E E E E-05 MNL E E E E E E E E E E E E E E E-05 MMNL E E E E E RMSE: Root Man Suar Error 2 MAPE: Man Absolut Prcntag Error