HEURISTICS-BASED PROTOTYPING STRATEGY FORMATION: DEVELOPMENT AND TESTING OF A NEW PROTOTYPING PLANNING TOOL

Transcription

1 DRAFT Proceedings of the ASME 2014 Interntionl Mechnicl Engineering Congress & Exposition IMECE2014 Novemer 14-20, 2014, Montrel, Queec, Cnd IMECE HEURISTICS-BASED PROTOTYPING STRATEGY FORMATION: DEVELOPMENT AND TESTING OF A NEW PROTOTYPING PLANNING TOOL Brock U. Dunlp University of Texs Austin, TX, USA Christopher L. Hmon University of Texs Austin, TX, USA Brdley A. Cmurn Singpore University of Technology nd Design, Singpore Richrd H. Crwford University of Texs Austin, TX, USA Dn D. Jensen US Air Force Acdemy Colordo Springs, CO, USA Mthew G. Green LeTourneu University Longview, TX, USA Kevin Otto Singpore University of Technology nd Design, Singpore Kristin L. Wood Singpore University of Technology nd Design, Singpore ABSTRACT This work seeks to introduce nd evlute effects of novel method for designing prototyping strtegies. This newly developed heuristics-sed tool guides designers in plnning prototyping strtegy sed on nswers to Likert-scle questions tht emody empiriclly vlidted heuristics. We creted this tool to ugment prior work in the development of prototyping plnning methods. The new tool guides designers through six criticl prototype strtegy choices: (1) How mny concepts should e prototyped? (2) How mny itertions of concept should e uilt? (3) Should the prototype e virtul or physicl? (4) Should susystems e isolted? (5) Should the prototype e scled? (6) Should the design requirements e temporrily relxed? We ssessed the new plnning tool in two environments: (1) controlled experiment in which volunteers completed prototyping design chllenge, nd (2) cpstone design clss with diverse rnge of open-ended sponsored design projects. In oth cses, students received trining for the method nd then employed it in their own efforts. In our study the new tool cused student tems to employ significntly more efficient nd effective prototyping strtegies, such s prototyping erly nd often. The results indicte higher functionl performnce of prototypes from groups using the new plnning tool compred to control groups. This pper descries the new prototyping strtegy plnning tool, detils oth sets of experiments, nd discusses results. 1. BACKGROUND Prototyping is promising frontier for design methodology reserch dvnces. Reserch shows tht prototyping decisions re often sed on prcticl knowledge nd mngement pproches rther thn experimentlly tested methods [1,2,3,4,5,6,7,8]. Simultneously it hs een shown tht perhps the gretest portion of sunken costs in new product development occurs during the prototyping process [9]. Therefore, methodicl tool to help tems direct their prototyping efforts could e gret sset to mitigting improper use of time, money, nd other resources. 1.1 HEURISTICS FOR PROTOTYPING STRATEGIES A numer of projects hve explored heuristic oservtions of etter prctices in prototyping. Viswnthn, et l. conducted n in-depth trcking study of grdute design students to determine eneficil prctices of prototyping [10]. Their experiment involved dt collection over three semesters of grdute design course. These results include foundtionl open-ended heuristics such s use stndrdized prts nd support uilding with nlyticl clcultions. An in-depth DoD study mkes the following oservtions on est prctices over forty yers of prototyping: [11]: 1. Mke sure the (finl) prototype meets the minimum design requirements; 2. The gol of prototype is to prove tht the finl product is vile in the rel world; 3. Prototypes re intended to e focused on determining unknown quntities; therefore, void dding noncriticl fetures; 4. During prototyping there should e no commitment to production; 5. Once the design process is

2 underwy, do not dd design requirements or performnce expecttions. Another set of reserch efforts explored modeling techniques to hypothesize the numer of prototypes to increse profit nd decrese risk. Dhn nd Mendelson find tht sequentil designs succeed in cost-constrined environments while prllel designs succeed in time-constrined environments [12]. Thomke [13] nd Thomke nd Bell [14] dd tht significnt svings cn e chieved through multiple low fidelity prototypes. Dhn s [12] equtions leverge sic ssumptions for the uncertinty of success of prototyping effort nd the mrginl increse in profit tht results from tht effort. An dditionl set of empiricl studies evlutes the effects of controlling these strtegy vriles one t time nd mesuring design outcomes. Hggmn, et l. [15] trcked the ctivities of mid-creer professionl grdute students during the preliminry design phse, exmining vrious correltions etween throwwy rpid prototyping nd performnce metrics. They found tht uilding prototypes erly in the design process correlted positively with success, while the totl mount of time spent did not. Similrly, the lower performing tems prototyped lter in the process. Kershw, et l. [16] found tht tems which developed prototypes erlier identified nd positively rected to flws in their designs, nd developed countermesures or improvements compred to tems tht prototyped lter in the process or did not develop multiple prototypes. Yng 2004, furthermore, shows tht time spent testing is positively correlted with outcome nd conversely, time spent fricting is negtively correlted with outcome [17]. Jng confirms in nother, independent empiricl study tht more successful tems prototype erlier nd more often throughout the entire process [18]. Dow [19] conducted controlled study requiring hlf the prticipnts to iterte nd requiring the other hlf to focus ll ville time on one prototype without itertion. This study empiriclly confirms tht, in the circumstnces tested, pursuing t lest three dditionl itertions eyond development of single prototype significntly improved finl design performnce. Most of the studies reviewed ove re of one of two types, either oservtion of designers prctice without externl control or evlution of strictly enforced single strtegy vrile studies. This first type is criticl to identify est prctices. The second type is criticl to determine if the prctices nd their results re repetle. We developed n experimentl pproch to explore (1) if designers will ctully pply these heuristics in their own prctice when provided with method t the outset of prototyping; nd (2) if these tems will in fct outperform control groups tht do not employ the method. 1.2 PRIOR HEURISTICS-BASED PROTOTYPING STRATEGY PLANNING A previously demonstrted prototyping strtegy plnning method [20] emodied heuristics such s those descried ove into equtions to ssist designer nswer questions such s How mny concepts should e prototyped? How mny itertions of concept should e uilt? Should the prototype e scled? This method involved using context vriles with hlf-dozen equtions to clculte quntittive vlues used to guide decisions. This previously reported prototyping strtegy methodology [20] used the independent context vriles of design prolem in order to determine vlues for dependent prototyping strtegy vriles (e.g. numer of prototypes to uild, prototype scling, nd susystem isoltion). These dependent strtegy vriles were mlgmted from heuristics for prototyping est prctices outlined y Moe [21], Christie [22], nd Viswnthn [10]. These est prctices re summrized s follows: Successful tems often initilly prototype three or more different concepts. Prototype erly nd often. Consider low-resolution prototypes to explore mny concepts quickly nd economiclly. Keep prototypes s simple s possile while yielding the needed informtion, therey sving time nd money. Allocte dequte time to the engineering process for uilding nd testing. Prototyping nd engineering nlysis need to work together for mximum effectiveness. This prototyping strtegy formtion methodology provides frmework tht ddresses key prototyping decisions designers fce y employing hierrchicl structure of generlized decision vriles extrcted from the heuristics referenced ove. The experimentl results pulished y Cmurn, et l. [20] indicte using this prototyping strtegy formtion method is positively correlted with erly-stge design success. Accordingly, employing this method cn potentilly improve design performnce while incresing the likelihood of stying within udget nd time constrints. Cmurn s reserch on n engineering pproch to prototyping strtegy implementtion nd the need for more comprehensive prototyping plnning form the foundtion for the dvnces reported here. However, this method proved not s stright-forwrd, time-efficient, nd intuitive s desired. Efforts to significntly improve upon this pulished method led to the significntly simplified nd stremlined pproch presented here. 2. NEW HEURISTICS-BASED PROTOTYPING STRATEGY PLANNING METHOD The process of trnsforming design concept into prototype, either physicl or virtul, is highly dependent upon the specific circumstnces of design prolem. Since every design prolem (nd design tem) is unique, generlly pplicle methodology for prototyping is needed. This pper presents prototyping strtegy utilizing heuristics-sed pproch tht is reltively esy to implement nd is pplicle cross multiple design prolems. Here we define prototyping strtegy s plnned set of ctions the design tem will tke in the upcoming prototyping effort. This novel method enles more flexile pproch to developing strtegy on ehlf of the designer nd tkes significntly less time to implement thn the strtegy descried ove. The next section lso shows tht the method increses efficiency nd effectiveness of the prototyping process. It is therefore pplicle to roder rnge of design prolems. Additionlly, this study lso

3 provides much more detiled investigtion of quntified effects of the strtegy formtion method on prototyping efforts. Specificlly, this tool contriutes Likert-scle ssessment of context to guide designers in trnslting context vriles into prototyping strtegy vriles. The method guides strtegy tht covers rod rnge of concerns designer my hve when prototyping. These strtegy vriles re descried elow: 1. Numer of design concepts simultneously prototyped - Prllel prototyping occurs when multiple concepts re uilt t the sme time, unlike seril prototyping in which one prototype is followed y nother (e.g. competitive prototyping y su-groups t design firm). 2. Numer of itertions for ech concept - Building prototype, testing nd evluting the prototype, refining the design concept, nd re-uilding nother prototype of tht sme concept is clled iterting (e.g. the progression from initil to finl form models for cr ody design). 3. Scling - Prototype size cn e either lrger or smller thn the plnned finl design size; however, the prototype retins reltive chrcteristics of the full-size form (e.g. Nvy ship uilt 1/100 scle for initil wter-tunnel testing). 4. Susystem isoltion - Often susystem of design concept cn e prototyped nd evluted in isoltion (e.g. testing of LCD components, without csing, for monitor design project). 5. Relxtion of design requirements - Prototypes my e uilt with relxed design requirements to simplify the process (e.g. n engine tht runs t prtil torque vlues to initilly reduce mjor dmping modes in engine lock design). 6. Physicl vs. virtul models A physicl prototype is tngile, mteril model of product or susystem, wheres virtul prototype is computer-sed model (CAD model, motion nlysis, FEA, CFD, etc.) of product (e.g. rchitecturl CAD models of skyscrppers). The heuristics for ech strtegy vrile in this methodology re sed on synthesis of the empiricl nd theoreticl reserch findings tht were discussed in the literture review section. Numer of concepts in prllel Empiricl reserch hs shown tht pursuit of multiple design concepts in prllel cn increse performnce of the finl prototype [19, 15]. Beyond this, Dhn nd Mendelson [12] investigte more nlyticl pproch to the prototyping process sed on n ssumption of profit from the design solution. This reserch investigted the trde-offs of prllel concept testing sed on the roles of profit distriution (uncertinty), cost of testing, nd totl udget. Prllel concepts permit more rpid redth explortion of the design spce nd my e explored when cost permits. These findings re reflected in the Likert scle prompts of the method in formt tht retins the nlyticl findings nd still llows considertion of the designer s experientil knowledge. Numer of itertions Empiricl studies hve determined tht pursuing itertion [19] correltes with incresed performnce outcome in the finl prototype. We lso dpt the theoreticl findings of Thomke & Bell, [13, 14] who use n uncertinty minimiztion pproch to determine the numer of itertions to develop. Thomke nd Bell conclude tht svings could e chieved through multiple low fidelity prototypes, which is lso supported y the empiricl reserch [13, 14]. Finlly Dhn nd Mendelson conclude tht itertions succeed when cost is constrined s itertion is lower cost tht prllel testing [12]. The strtegy encourges the designer to explore multiple itertions when fesile. Scled or full system prototyping Previous empiricl reserch studies found tht when fullsystem model is very costly, time consuming or imprcticl to uild for verifiction purposes s prototype, scled prototype cn e very useful for testing the system [11]. This my lso e true when full-size system is fesile, ut scled model is much lower in cost nd llows rpid itertions. Susystem isoltion or integrtion The empiricl reserch identifies tht prototype my emody susystem or the full system [11]. The indictions for pursuing susystem re similr to those for scling. When it is reltively difficult to construct the full system nd designer is confident tht sufficient informtion is otinle from uilding nd testing n isolted susystem, susystem prototype cn e used. Prticulrly, susystem isoltion is useful when it llows rpid cycles of uild nd test for complex susystem. Requirement relxtion or full requirement testing Prototypes my or my not meet the finl design requirements [11]. By crefully constructing test tht my not meet full system requirements, ut does in fct cpture some criticl spects of system function, the designer cn determine potentil enefits or drwcks of design without investing n unnecessry mount of effort or resources to the uild. Virtul versus physicl prototyping Finlly, previous studies [10, 11] lso identify tht prototype my e either physicl or virtul. In recent puliction, [23] we lso find tht virtul prototypes re eneficil when the cost of virtul prototype is lower nd llows for more rpid itertion. Figure 1 elow contins the six multi-point prompts of the design tool. Ech strtegy vrile is determined y verging the Likert response to the multi-point prompts. Completing this process cn drsticlly lter designer s proposed prototyping strtegy. For exmple, one of the tems in the inclss study (discussed elow) ws designing mteril corrosion prevention system nd used the plnning tool to formulte their prototyping strtegy. Prior to using the tool they identified two concepts tht ppered to e promising: (1) n impressed current nd scrificil node monitoring system, nd (2) four-point node monitoring system. As they reviewed the mtrix in Figure 1, question 1, they found tht there ws enough mteril to uild multiple design concepts, two concepts showed promise, nd ech would e quick to

4 uild. Therefore they decided to pursue construction of oth of these design concepts, n ide tht previously ws not considered. As cn e seen in Figure 1, the method is designed to llow considertion of the designer s experience with strtegic reserch-sed heuristics. This ddresses the fct tht mteril nd time llotments re not lwys explicit or pre-determined nd thus llow for humn discretion in these choices, while t the sme time providing guide sed on known est prctices. Figure 1. LIKERT SCALE MATRICES FOR DETERMINING THE SIX PROTOTYPING STRATEGY VARIABLES. Compute the verge response to the prompts under ech ctegory to determine strtegy. 1 For high vg, develop multiple concepts; else, uild one only. c There re sufficient mterils to prototype multiple concepts. There is sufficient time to prototype multiple concepts. Rnkings of severl concepts re very close (e.g. from Pugh chrt). 2 For high vg, iterte; else, uild once. c The difficulty of meeting the requirements will necessitte itertion. The difficulty of mnufcturing will necessitte itertive prototyping. My tem hs miniml prototyping experience. Strongly Disgree. Disgree. Neutrl. Agree. Strongly Agree One concept No not iterte Multiple concepts Iterte 3 For high vg, use virtul prototype; else, use physicl models. Physicl Virtul c d Virtul prototype(s) will require less time thn physicl one(s). Virtul modeling will vlidte: physics, interfces nd/or requirements. A CAD model is needed for nlysis (FEA, CFD, etc.) or mnufcture. Time & udget llow pursuit of oth virtul nd physicl prototypes. 4 For high vg, isolte susystems.; else, integrte the system. c d Interfces etween susystems re predictle nd/or re NOT criticl. 1 or 2 susystems emody criticl design requirements & need itertion. A susystem uild would significntly reduce time, cost or complexity. An isolted susystem cn e properly tested. 5 For high vg, use scled model; else, use full size model. Scling lw(s) will permit ccurte system modeling vi scled uild. Scling will significntly simplify the prototype. 6 For high vg, relx requirements.; else, pursue full requirements. Requirement flexiility llows significnt results from relxed model. Requirement relxtion will significntly simplify the prototype. Integrte Susystems Do not scle Do not relx Isolte Susystems Scle Relx design requirements 3. EXPERIMENTAL ASSESSMENT OF THE METHOD We ssessed the new tool in two environments: (1) controlled experiment in which volunteers completed prototyping design chllenge, nd (2) n open-ended cpstone design clss with vriety of sponsored design projects. In oth cses students received trining nd employed the newly creted prototyping strtegy formtion tool CONTROLLED STUDY EXPERIMENT In this experiment 64 students from senior level mechnicl engineering design clss t The University of Texs t Austin were divided into 32 two-person tems. The tems were split eqully into control nd experimentl groups. Our previous reserch hs indicted tht mny designers do not consciously consider the prototyping strtegy vriles we introduce; therefore, the control is defined s the group solving the sme design prolem in the sme llotment of time nd resources, ut without ccess to the strtegy decision mtrices. The design prolem prompted tems to uild freestnding triggered device to propel n 8.5x11 inch sheet of pper the

5 frthest distnce with the gretest mount of repetility. Students were instructed to mximize the ojective function: Prototype Performnce 2 (EQ. 1) Experimentl Control p= Although our nlysis weights distnce nd repetility in vrious wys, this ojective eqution clerly guided tems to mximize distnce while minimizing vrince. The design criteri lso specified minimum score of 25 feet to successfully complete the design chllenge. This design prolem ws chosen ecuse it cn e solved in two hours of prototyping time, prototyping efforts re trctle, nd there re multiple design solutions. Feet p= Performnce Score Avg Performnce Distnce Figure 3. COMPARISON OF EXPERIMENTAL AND CONTROL GROUPS OVERALL PERFORMANCE. Figure 2. EXAMPLE SOLUTION TO THE GIVEN DESIGN PROBLEM. To simplify the experiment nd reduce the noise induced y concept genertion, we provided the tems with four rudimentry design concept sketches to se their prototyping upon: (A) sling shot, (B) wheeled vehicle, (C) rolling cylinder, (D) ctpult. Figure 2 depicts n exmple of one solution tem designed, using concept C, to successfully complete the chllenge. All tems kept running log of time spent testing, concept tested, design chnge mde since previous test, nd distnce reched. At the end of the two-hour prototyping period, ech tem mde five lunches nd ws evluted y the reserchers ccording to the three lunches tht gve the est performnce score ccording to EQ. 1. One possile explntion for this result is tht this performnce mesure overly penlizes design vrince nd inherently mplifies the stndrd devition. Therefore, second metric considers only distnce. In this cse the performnce metric ws clculted y verging ll five tries from ech tem within the experimentl nd control groups respectively (Figure 3) This distnce-only ssessment yielded similr results to our previous evlution, indicting the higher verge of the experimentl group ut lso proved to e sttisticlly significnt (p =.036, t-test). This nlysis indictes there is performnce dvntge gined from the prototyping strtegy formtion method for this experiment. Furthermore, the dt lso justifies the incresed performnce y showing remrkle increse in prototyping est prctices such s prototyping erlier nd iterting more s cn e seen elow. Figure 4 shows the experimentl group time-to-first-prototype is drmticlly lower (p=0.03, t-test), nd Figure 5 shows the experimentl group outperformed the control group t the end of one hour (p=0.00, t-test) y verging distnces nerly 10 feet more thn the control CONTROLLED STUDY RESULTS Using EQ. 1 s the performnce ojective, the three est finl lunches from ech tem were verged into tem score. A sttisticl nlysis ws performed using two-tiled t-test. Figure 3 depicts the verge overll performnce rting for the 16 experimentl nd 16 control tems. The experimentl group shows higher verge performnce score, ut not to level of sttisticl significnce (p=0.43, t-test). Time (mins) Time of 1st Prototype Test Experimentl Control p = Avg Time of 1st Prototype Figure 4. AVERAGE TIME OF FIRST PROTOTYPE TEST FOR EXPERIMENTAL AND CONTROL GROUPS

6 250 Numer of Prototype Tests VS Time 200 Feet :11 0:25 0:40 0:54 1:09 1:23 1:37 1:52 2:06 Time Experimentl Control Figure 7. CUMULATIVE NUMBER OF PROTOTYPE TESTS FOR ALL EXPERIMENTAL AND CONTROL GROUPS. Figure 5. AVERAGE DISTANCE REACHED BY EXPERIMENTAL AND CONTROL GROUPS PROTOTYPES FOR FIRST AND SECOND HOUR. Figure 6 further demonstrtes tht the experimentl group chieves performnce more rpidly. It cn lso e seen tht performnce for oth conditions strts to level out fter out n hour nd hlf. This indictes tht the prolem lent itself to certin level of design performnce sturtion, nd tht for complex prolems the strtegy my e even more pplicle. Figure 8 dds further insight to the oservtion of testing over time y depicting only tests tht were prototype itertions. Here prototype itertion is defined s fundmentl chnge to the physicl model. The grph gin shows tht the control group ws slow to strt in the first hlf hour. This reltively slow progress my represent intuitive minor djustments to their designs, rther thn explicit itertion. In contrst, the experimentl group ws quick to strt nd consistently iterted throughout the experiment. 160 Best Tril VS Time Feet :11 0:25 0:40 0:54 1:09 1:23 1:37 1:52 2:06 TIme Experimentl Control Figure 6. BEST PROTOTYPE TEST VS TIME FOR EXPERIMENTAL AND CONTROL GROUPS. Figure 7 grphiclly depicts how ech group tested over time, nd shows the experimentl tems prototyped nd tested t consistent rte throughout the llotted time, suggesting etter time mngement nd etter prototyping prctices. The control tems were slow to egin prototyping nd hd to drmticlly increse their efforts towrds the end to finish. Figure 8. NUMBER OF PROTOTYPE ITERATIONS FOR EXPERIMENTAL AND CONTROL GROUPS. A post-experiment survey ws dministered in which tems recorded how mny itertions they completed efore reching their finished product. We found tht the experimentl tems reported n verge of 11.5 itertions for their prototypes compred to the 7 itertions for the control tems. This result represents confidence intervl greter thn 95% tht experimentl tems completed more itertions. The more detiled prototype trcking logs showed tht the experimentl group hd n verge consistent with wht they originlly reported (12.31, 6.5% difference). In contrst the control group misreported their itertions y nerly 30% (see Figure 9), suggesting the experimentl group showed mrked improvement in record keeping ccurcy. This likely indictes tht close ttention to strtegy lso llows for more ccurte self-ssessment s choices were explicit rther thn intuited.

7 designers cross suite of design prolems, tem sizes nd tem reltionl dynmics. The Likert scle method is wellestlished reserch tool in design science [24]. This section does not ddress technicl or performnce effects of the method, s discussed in section 3.2. However, it does demonstrte the integrtion of the strtegy method into longterm multi-system projects tht llow evlution of the method on more complex design prolems. Tle 1. CAPSTONE DESIGN CLASS PROTOTYPE STRATEGY DEVELOPMENT GUIDE EVALUATION Figure 9. COMPARISON OF TOTAL NUMBER OF ITERATIONS AS REPORTED BY EACH TEAM AND AS REVIEWED IN THE DATA. The experimentl tems were informed of the reserch showing successful tems often initilly prototype three or more different concepts, nd ech experimentl tem did initilly pln to test two or more concepts. However, only seven of the sixteen tems ctully ttempted more thn one concept. This ws more thn the control group, ut less thn expected. Mny tems tht did not ttempt the second concept reported their first prototype proved promising enough tht they did not see the need to prototype second concept. The prototyping strtegy formtion method clerly prompted tems to consider multiple concepts; however, in this prticulr experiment it my hve een likely tht tems did not hve sufficient mterils. This result is wht would e expected from Dhn nd Mendelson s experimentl model [12]. Further, the performnce sturtion curve, Figure 6, indictes tht time ws not the driving constrint (for which prllel prototyping is more criticl). Therefore we do not expect tht this oservtion would recur in different design prolems, s it is specific to these conditions rther thn indictive of the method CAPSTONE DESIGN CLASS EXPERIMENT Senior engineering students t the United Sttes Air Force Acdemy enroll in two semester cpstone design course nd work on wide vriety of corporte sponsored projects. As prt of their curriculum we introduced the method to the design tems during the first semester of the clss. Ech tem then plnned to implement the method once they reched the prototyping phse of their project. At the eginning of the second semester ll tems hd egun their prototyping. We then gthered feedck from the students nd instructors to ssess the usefulness of the method for formulting effective prototyping strtegies CAPSTONE DESIGN CLASS EXPERIMENT RESULTS After implementing the new tool, one student nd the instructor from ech of the seven tems evluted the method on Likert scle of 1-5 sed on the following criteri: esy to follow, useful, efficient, nd helped them consider spects of prototyping they hd not thought of efore. As shown in Tle 1, ech of these criteri hd n verge rting of 4.0 or greter. These results vlidte tht the methodology hs positive experientil enefit on the perception of the prticipnt The prototype strtegy development guide: Avg Stdev...is esy to follow is useful in helping my tem formulte prototyping strtegy helped my tem consider spects of prototyping tht would hve otherwise een overlooked is n efficient tool for formlizing prototyping strtegy is n importnt prt of the design process Likert scle: 1 (completely disgree) 5 (completely gree), Smple size: N=14 Along with these positive responses to the method, we lso received invlule feedck to improve the method, including: We should clrify whether the strtegy must e followed strictly or is re-workle fter initil efforts. The generlizility of the criteri should e further explored. Alternte orders for the strtegy should e considered. 4. FUTURE RESEARCH Results from oth the controlled nd cpstone studies re encourging. Future reserch should consider severl dditions s well s ttempts to ddress the feedck. These my e guided y the following questions: 1. Implementtion of prototyping effort often ppers to e very multi-fceted nd dynmic process mening tht plns chnge sed on new informtion gined t ech stge of the prototyping effort. How cn the method e ugmented to ddress time-dependent context vriles tht influence the preferred strtegy? 2. The strtegy development tool (Figure 1) hs multiple criteri for ech of the strtegy vriles. Wht lterntives to the Likert scle verging method my e effective? 3. Is the order of the strtegy development correct? Tht is, cn tem decide how mny concepts to do in prllel efore considering how mny itertions my e necessry to crry out ech concept? We lso intend to continue to follow the cpstone design tems throughout their prototyping processes. We will gin follow up with the USAFA design students t the end of the course nd evlute how the method ws used nd mesure its effectiveness.

8 While the current method is limited in its scope, the initil results re quite exciting nd illuminting, nd we im to expnd the method to include more prototyping decisions mde y design tems, such s mteril choices (using the sme mterils in prototypes s plnned in production), mnufcturing techniques, etc. 5. CONCLUSION This pper reports on newly developed heuristics-sed tool tht guides designers in plnning prototyping strtegy sed on nswers to Likert-scle questions tht emody empiriclly vlidted heuristics. Results from controlled study indicte the method did improve students performnce cross numer of ssessment metrics. We found tht tems who use the method tend to iterte erlier nd more often thn those tht did not use the method. Furthermore, those who used the method mnged their time etter nd were le to improve performnce t fster rte. As shown in vrious other experiments, these vriles re directly correlted with higher success. Bsed upon our results we know the method enhnces performnce, nd these dditionl metrics show us how. In conjunction with the controlled study, the method ws introduced to cpstone design clss t the US Air Force Acdemy with diverse rnge of open-ended sponsored design projects. The students nd fculty rected positively towrds the method, indicting tht it ws esy to follow, useful, efficient, nd helped them consider spects of prototyping they hd not thought of efore. 6. ACKNOWLEDGMENTS This mteril is sed on reserch sponsored y the United Stes Air Force Acdemy under greement numer FA The US Government is uthorized to reproduce nd distriute reprints for Government purposes notwithstnding ny copyright nottion thereon. The reserch is lso supported in prt y the SUTD-MIT Interntionl Design Centre (IDC, idc.sutd.edu.sg) t the Singpore University of Technology nd Design (SUTD). The views nd conclusions contined herein re those of the uthors nd should not e interpreted s necessrily representing the officil policies or endorsements, either expressed or implied, of the United Sttes Air Force Acdemy or the US Government. 7. BIBLIOGRAPHY [1] Christensen, C. M, 2003, The Innovtor s Dilemm: The Revolutionry Book Tht Will Chnge the Wy You Do Business, Hrper Business. New York. [2] Dodgson, M., 2005, Think, Ply, Do: Technology, Innovtion, nd Orgniztion, OUP, Oxford, New York. [3] Vergnti, R., 2009, Design-Driven Innovtion: Chnging the Rules of Competition y Rdiclly Innovting Wht Things Men, Hrvrd Business Press, Boston, MA. [4] Mrtin, R., 2009, The Design of Business: Why Design Thinking Is the Next Competitive Advntge, Hrvrd Business Press, Boston, MA. [5] Thomke, S. H., 2003, Experimenttion Mtters: Unlocking the Potentil of New Technologies for Innovtion, Hrvrd Business Press, Boston, MA. [6] Schrge, M., 2000, Serious Ply: How the World's Best Compnies Simulte to Innovte, Hrvrd Business Press, Boston, MA. [7] Clrk, K., Fujimoto, T., 1991, Product Development Performnce: Strtegy, Orgniztion, nd Mngement in the World Auto Industry, Hrvrd Business Press, Boston, MA. [8] Ulrich, K. T., Eppinger S. D., 2000, Product Design nd Development, Mc Grw-Hill, New York. [9] Cooper, R. G., 1993, Winning t New Products: Accelerting the Process from Ide to Lunch, Perseus, Cmridge, MA. [10] Viswnthn, K. 2012, Cognitive effects of physicl models in engineering dt genertion, Texs A&M University, College Sttion, TX. [11] Drezner, J., 2009, On Prototyping: Lessons From RAND Reserch, RAND Corportion, Snt Monic, CA. [12] Dhn, E., Mendelson, H., 1998, Optiml prllel nd sequentil prototyping in product design, unpulished mnuscript, Stnford University Grdute School of Business, Stnford, CA. [13] Thomke, S.H., 1998, Mnging Experimenttion in the Design of New Products, Mgmt Sci. 44(June) [14] Thomke, S.H., Bell, D., 2001, Sequentil testing in product mngement science, Mgmt Sci. 47(Ferury) [15] Hggmn, A., Hond, T., Yng, M., 2013, The Influence of Timing in Explortory Prototyping nd Other Activities in Design Projects, ASME IDETC/CIE Design Theory nd Methodology Conference, Portlnd, Oregon. Pper Numer: DETC [16] Kershw, T. C., Holtt-Otto, K., Lee, Y. S., 2011, The Effect of Prototyping nd Criticl Feedck on Fixtion in Engineering Design, CogSci, Boston, MA. [17] Yng, M. C., 2005, "A study of prototypes, design ctivity, nd design outcome", Design Studies, 26(6), [18] Jng, J., Schunn, C. D., 2012, Physicl Design Tools Support nd Hinder Innovtive Engineering Design, Journl of Mechnicl Design, 134 (April). [19] Dow, S. P., Klemmer, S. R., 2011, The Efficcy of Prototyping Under Time Constrints, Design Thinking, Springer Berlin Heidelerg, [20] Cmurn, B. A., Dunlp, B. U., Kuhr, R., Viswnthn, V. K., Linsey, J. S., Jensen, D. D., Crwford, R. H., Otto, K. N., Wood, K. L., 2013, Methods for Prototyping Strtegies in Conceptul Phses of Design: Frmework nd Experimentl Assessment, ASME IDETC/CIE Design Theory nd Methodology Conference, Portlnd, Oregon. Pper Numer: DETC [21] Moe, R., Jensen, D., Wood, K., 2004, "Prototype prtitioning sed on requirement flexiility, ASME

9 IDETC/CIE Design Theory nd Methodology Conference, Slt Lke City, UT. Pper Numer: DETC [22] Christie, E., Buckley, R. Ziegler, K., Jensen, D., Wood, K., 2012, Prototyping strtegies: Literture review nd criticl vriles, ASME IDETC/CIE Design Theory nd Methodology Conference, Chicgo, IL. [23] Hmon, C., Green, M. Dunlp, B., Cmurn, B, Crwford, R., Jensen, D., 2014 Virtul or Physicl Prototypes? Development nd Testing of Prototyping Plnning Tool, ASEE Annul Conference & Exposition, Design in Engineering Eduction Division, Indinpolis IN, ccepted [24] Cmurn, B. A., Dunlp, B. U.,, Viswnthn, V. K., Linsey, J. S., Jensen, D. D., Crwford, R. H., Otto, K. N., Wood, K. L., 2013, Connecting Design Prolem Chrcteristics to Prototyping Choices to Form Prototyping Strtegy, ASEE Annul Conference nd Exposition, Atlnt, GA