GenderMag: A Method for Evaluating Software s Gender Inclusiveness

Transcription

1 c The Author Publshed by Oxford Unversty Press on behalf of The Brtsh Computer Socety. All rghts reserved. For Permssons, please emal: Advance Access publcaton on 27 January 2016 do: /wc/wv046 GenderMag: A Method for Evaluatng Software s Gender Inclusveness Margaret Burnett 1, *, Smone Stumpf 2, Jame Macbeth 3, Stephann Makr 2, Laura Beckwth 4, Irwn Kwan 5, Anca Peters 1 and Wllam Jerngan 1 1 Oregon State Unversty, Corvalls, OR, USA 2 Cty Unversty London, London, UK 3 Clemson Unversty, Clemson, SC, USA 4 HCI Researcher, Copenhagen, Denmark 5 The Mathworks, Boston, MA, USA Correspondng author: burnett@eecs.oregonstate.edu In recent years, research nto gender dfferences has establshed that ndvdual dfferences n how people problem-solve often cluster by gender. Research also shows that these dfferences have drect mplcatons for software that ams to support users problem-solvng actvtes, and that much of ths software s more supportve of problem-solvng processes favored (statstcally) more by males than by females. However, there s almost no work consderng how software practtoners such as User Experence (UX) professonals or software developers can fnd gendernclusveness ssues lke these n ther software. To address ths gap, we devsed the GenderMag method for evaluatng problem-solvng software from a gender-nclusveness perspectve. The method ncludes a set of faceted personas that brng fve facets of gender dfference research to lfe, and embeds use of the personas nto a concrete process through a gender-specalzed Cogntve Walkthrough. Our emprcal results show that a varety of practtoners who desgn software wthout needng any background n gender research were able to use the GenderMag method to fnd gender-nclusveness ssues n problem-solvng software. Our results also show that the ssues the practtoners found were real and fxable. Ths work s the frst systematc method to fnd gender-nclusveness ssues n software, so that practtoners can desgn and produce problemsolvng software that s more usable by everyone. RESEARCH HIGHLIGHTS We dscuss fve facets of pror gender research wth tes to males and females usage of problem-solvng software. We present GenderMag, the frst systematc method to evaluate gender-nclusveness ssues n problemsolvng software. We show how GenderMag draws upon and encapsulates these fve facets. We present three qualtatve emprcal studes that were used to nform and to valdate varous aspects of GenderMag, and show the knds of ssues that partcpants found and how gender of the evaluator nteracted wth usage of the method. Keywords: gender; walkthrough evaluatons; software nclusveness Edtoral Board Member: Smone Barbosa Receved 29 July 2015; revsed 9 December 2015; accepted 14 December 2015 Downloaded from by guest on 24 November INTRODUCTION Recent research calls nto queston the nclusveness of software that ams to support dverse people n problem-solvng stuatons. The users who tend to be best supported by problem-solvng software tend to be those best represented n software development teams (e.g. relatvely young,

2 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 761 able-boded, males), wth other users perspectves often overlooked. Perhaps the best-studed of underrepresented groups use of software are those wth physcal dsabltes, but even that group remans underserved (Power et al., 2012), and many other underrepresented groups uses of software reman barely consdered (Arjona-Rena et al., 2014; Burnett et al., 2011; Davdson and Jensen, 2013; Joyce et al., 2007; Power et al., 2012). In the realm of one underrepresented group n software, namely females, Wllams recently coned the term gender lens (Wllams, 2014), whch refers to the need for software development practces that nclude a gender perspectve. In ths paper, we present such a lens, n the form of GenderMag (Gender Inclusveness Magnfer). GenderMag s an nspecton method for evaluatng problemsolvng software from a gender-nclusveness perspectve What s gender? In ths paper, we use the term gender as a socally constructed concept (Butler, 1999; West and Zmmerman, 1987) where gender dentfcaton, dsplay and performance mght or mght not algn wth bologcal sex. As West and Zmmerman defne t, someone s gender choce affects and s affected by how they manage ther stuated conduct n lght of normatve conceptons of atttudes and actvtes approprate to the category wth whch they most dentfy. We especally emphasze that when someone dentfes as a male or female, ths s not the same thng as a clam to algn wth a stereotypcal male or female gender role or expresson. To reflect ths socal concept of gender, n ths paper, we follow the lead of West and Zmmerman by usng the term males as a shorthand for people who dentfy as male, and females to denote people who dentfy as female Gender nclusveness Research over the past decade has emerged showng that the ndvdual dfferences n how people use software features amed at supportng problem-solvng tend to cluster by gender, and further, that many such features are nadvertently desgned around approaches favored more by males than by females. These dfferences have been found n a varety of problem-solvng software; for example, n spreadsheets Beckwth et al., 2005, 2006, 2007; Grgoreanu et al., 2012, n vsualzatons (Borkn et al., 2013; Tan et al., 2003), n onlne classwork platforms (Pazza Blog, 2015), n web and home applance development and scrptng (Cao et al., 2010a; Rode, 2008; Rosson et al., 2010) and n ntellgent systems (Kulesza et al., 2011; Szafr and Mutlu, 2012). Further, research shows that desgnng software to be more gender-nclusve can beneft everyone, regardless of gender. For example, Tan et al. (2003) showed that dsplayng optcal flow cues benefted both females and males n vrtual world navgaton; Grgoreanu et al. (2008, 2010) showed how changes to spreadsheet features relatng to confdence, feature support and strategy workflows reduced gender gaps whle mprovng everyone s atttudes and feature usage; and Jerngan et al. (2015) showed how a tool desgned for a dversty of ndvdual styles and stuatons enabled both female and male novce programmers who dd not receve very much n-person help to program as well as novces who receved extensve n-person help. These fndngs are consstent wth smlar fndngs n changng educatonal practces to mprove gender nclusveness. For example, n educaton, researchers found that par programmng, whch was expected to help female computer scence students, not only reduced the gender gap but also ncreased success and reduced attrton among male and female students (Berenson et al., 2004; McDowell et al., 2003). Successful nstances lke these are encouragng, but manly what they show s proof of concept, not a path toward nclusveness. One mechansm to promote nclusveness that several researchers have advocated s gender-nclusve practces at desgn tme (Bardzell, 2010; Brggs et al., 2014; Wllams, 2014). These are mportant, but they are not a panacea. What s also needed s a systematc method that can be used even f few members of the software team are mndful of gender dfferences, and even f the software s more mature than beng n the ntal desgn stages. Ths ponts to the followng gap: How can ordnary practtoners, wth no background n gender research, dentfy whch aspects of ther software have gender-nclusveness ssues, realze why those ssues are ssues and thereby know what they should change? To address ths gap, we devsed the GenderMag method (Gender Inclusveness Magnfer). GenderMag evaluates features n problem-solvng software from a gender-nclusveness perspectve. At the core of GenderMag are fve facets of gender dfferences that have been extensvely nvestgated n the lterature. GenderMag encapsulates the facets nto personas to brng them to lfe, and embeds the personas and the facets nto a process based on the Cogntve Walkthrough (CW). The method ams to provde a systematc and practcal way for software practtoners (UX professonals, software developers, etc.) wth no background n gender research to fnd gender-nclusveness ssues n the problem-solvng software 1 they are producng. Ths paper presents the GenderMag method, along wth our nvestgatons to nform and evaluate our approach 1 When we refer to problem-solvng software, we mean software features and platforms n whch the user s actvely tryng to work out a soluton to some knd of problem or task, such as wth the examples at the begnnng of ths secton. Downloaded from by guest on 24 November 2018

3 762 Margaret Burnett et al. emprcally a formatve case study at a company that produces software allowng medcal practtoners to customze programmable hearng ads; a formatve workshop event n whch researchers evaluated Lookng Glass (Gross et al., 2012), a tool that teaches mddle school students to program 3D anmatons; and a qualtatve laboratory study n whch UX practtoners used GenderMag to evaluate Gdget (Lee et al., 2014), a game-lke programmng envronment desgned to teach debuggng. 2. BACKGROUND AND RELATED WORK 2.1. Gender dfferences n problem-solvng and programmng We have just cted extensve emprcal evdence over the past decade showng ndvdual dfferences that cluster by gender n the ways people use problem-solvng software. We now consder these dfferences at the foundatonal level. Fve facets of gender dfferences relatng to problem solvng that have been repeatedly mplcated by research from other felds, such as psychology, educaton and communcatons, are: () Motvaton: Research spannng over a decade has found that females tend (statstcally) to be motvated to use technology for what t enables them to accomplsh, whereas males motvatons sometmes come from ther enjoyment of the technology for ts own sake (Burnett et al., 2010, 2011; Cassell, 2002; Hou et al., 2006; Margols and Fsher, 2003; Smon, 2001). Ths dfference can affect whch features of problem-solvng software females vs. males choose to use. () Informaton processng styles: To solve problems, people often need to process new nformaton, and there s extensve research reportng gender dfferences here too. In essence, when problem-solvng, females are more statstcally lkely to use comprehensve nformaton processng styles gatherng farly complete nformaton before proceedng whereas males are more statstcally lkely to use selectve styles followng the frst promsng nformaton, then potentally backtrackng, n depth frst order (Cafferata and Tybout, 1989; Coursars et al., 2008; Meyers-Levy and Loken, 2015; Meyers-Levy and Maheswaran, 1991; Redl et al., 2010). Each of these styles has partcular advantages, but ether s at a dsadvantage when not supported by the problem-solvng software envronment. Partcularly relevant here are studes tyng gender dfferences n nformaton processng style to software-based tasks, such as wth e-commerce web stes (Smon, 2001), software-based audtng (O Donnell and Johnson, 2001) and workng wth spreadsheets (Grgoreanu et al., 2012; Subrahmanyan et al., 2008). () Computer self-effcacy: One specfc form of confdence s self-effcacy: a person s confdence about succeedng gven a specfc task (Bandura, 1986). Selfeffcacy matters to problem solvng because a person s self-effcacy nfluences ther use of cogntve strateges, amount of effort put forth, level of persstence and strateges for copng wth obstacles (Bandura, 1986). Emprcal data have shown that females tend statstcally to have lower computer self-effcacy than males, as one would expect gven phenomena lke stereotype threat, and non-nclusve work envronments and educaton practces (Appel et al., 2011; Huffman et al., 2013; Luger, 2014). Self-effcacy levels, n turn, affect people s behavor wth technology, such as whch features they choose to use and how wllng they are to persst wth hard-to-use features (Burnett et al., 2010, 2011; Durndell and Haag, 2002; Hartzel, 2003; O Leary-Kelly et al., 2004; Pazza Blog, 2015; Sngh et al., 2013). Fortunately, features desgned explctly for dverse self-effcacy levels have been shown to be preferred by everyone (e.g. Grgoreanu et al., 2008). (v) Rsk averson: Studes have shown that females tend statstcally to be more rsk-averse than males (Dohmen et al., 2011), surveyed n Weber et al. (2002) and meta-analyzed n Charness and Gneezy (2012) n numerous decson-makng domans, such as n ethcal decsons, nvestment decsons, gamblng decsons, health/safety decsons, career decsons and others. In contrast, we have been unable to locate any study n any doman reportng males to be more rsk-averse than females. Applyng these fndngs on rsk averson to software usage suggests that rsk averson may mpact females decsons as to whch feature sets to use. (v) Tnkerng: Research across age groups and professons reports females beng statstcally less lkely to playfully experment ( tnker ) wth features new to them, compared wth males. However, studes also show that when females do tnker, they are more lkely to reflect more n the process and thereby sometmes proft from t more than males do, and further, that some males tnker excessvely (Beckwth et al., 2006; Burnett et al., 2010; Cao et al., 2010a; Chang et al., 2009; Hou et al., 2006; Rosner and Bean, 2009). One effect of these dfferences n tnkerng behavors s ther mpact on whch features of software females vs. males wll elect to use, especally when a desgn choce underlyng the software product s that users wll learn new features by explorng and tnkerng wth them. These facets play out n software-based problem-solvng stuatons n a varety of ways, ncludng whch features females and males choose to use, the ways they use them and the strateges they employ nvolvng such features. The followng examples help us to llustrate ths pont. Downloaded from by guest on 24 November 2018

4 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 763 Frst, consder spreadsheets, a common settng for problem solvng about numerc calculatons such as for budgets, grades and fnances. In a study of Seattle-area experenced Excel users workng wth Excel (Beckwth et al., 2007), females self-effcacy predcted ther level of success completng a task, but the same dd not hold true for the males; for males, selfeffcacy dd not matter to how successful they were. Ths translated to feature use for females: low self-effcacy females reled more than males dd on the famlar type of features, partcularly value edts. At frst glance, a possble reason mght seem to be that females were smply better judges of ther lack of comprehenson of the new features, but the evdence does not support that reason: females comprehenson of the software features was no dfferent than the males and was not predcted by self-effcacy. In fact, ths study re-confrmed other studes fndngs of self-effcacy playng out dfferently for females vs. males (e.g. Burnett et al., 2011). Our second example nvolves a dfferent knd of problemsolvng software: customzng ntellgent systems. Intellgent systems, such as emal spam flters and recommender systems, learn computatonal behavors customzed to one end user and these learned behavors sometmes requre adjustment ( debuggng ). Here, one facet that turned out to be very relevant was that of nformaton processng. In one study n whch end users attempted to gude an ntellgent system to better sort emals nto folders by pontng out keywords n the emal messages (Stumpf et al., 2008), females spent sgnfcantly more tme than males workng wth the system, and also produced more thorough results. Ths was because females used the provded features more comprehensvely (as per the nformaton processng facet above), provdng the system wth sgnfcantly more keywords than males dd even though they consdered the same number of emal messages. Another study n ths doman (Kulesza et al., 2011) found that females had sgnfcantly lower self-effcacy than males, had more dffcultes choosng whch keywords to select (a selecton barrer) and how to proceed wth gudng the ntellgent system (a desgn barrer). Females also more often than males encountered these selecton barrers n a sequence, repeatedly runnng nto the same barrers (Kulesza et al., 2011). Web development and scrptng provdes a thrd example doman. In a study of web development by end users (Cao et al., 2010a), as wth the above studes, females had lower self-effcacy and focused ther efforts on famlar webservce features (versus unfamlar webservce features) sgnfcantly more than the males dd. Rosson et al. s study of web developers also showed suggestve gender dfferences n the use of novel web-based database features that are consstent wth these fndngs (Rosson et al., 2007). Fourth, a mult-study (Burnett et al., 2010) looked at generalzable patterns across a wde range of problem solvers rangng from admnstrators to professonal programmers usng a varety of problem-solvng software. The mult-study nvolved a gender-based secondary analyss of almost 3000 partcpants from multple studes data at a large software company, ncludng, two studes of hobbyst programmers usng Vsual Studo Express, two studes of professonal software developers usng Vsual Studo, as well as techncal problem-solvng practces of multple populatons usng a varety of other platforms. The results showed sgnfcant gender dfferences across all programmng envronments and populatons as to whch features males and females elected to use, as to males and females tnkerng and explorng behavors and between males or females techncal problemsolvng confdence. Further, as wth the other studes reported n ths paper, the confdence dfferences were not the sole explanaton for the dfferences n feature usage and tnkerng. Table 1 summarzes the results of the mult-study. We also mentoned n the Introducton several examples n whch the above gender dfferences were accommodated through more nclusve feature desgn (Grgoreanu et al., 2008; Tan et al., 2003). Other examples of supportng these dfferences through more nclusve desgns are Storytellng Alce (Kelleher et al., 2007), n whch dfferences n female vs. male motvatons to use technology were leveraged to ncrease mddle-school grls learnng of computer programmng, and Gdget (Lee et al., 2014), a game desgned to teach Table 1. Results of a fve-study analyss of gender dfferences n feature usage (RQ1), tnkerng (RQ2) and computer-related confdence/ self-effcacy (RQ3). RQ1: Whch features Interestng example: Wzards RQ2: Tnkerng, explorng One aspect: Atttude re: new technology RQ3: Confdence dfferences One aspect: Evdence of tes wth only some dfferences IT-support users: Hobbysts: (#2) survey, Professonal developers: (#1) survey (#3) feld ntervew (#4) needs survey, (#5) beta-testers survey (#1) (#2) (#4) (#1) (#2) (#1) (#3) (#5) (#1) (#3) (#5) (#1)? (#2), (#3) (#5) (#1) (#2) (#5) Note: (#study) denotes whch study produced each result. denotes statstcally sgnfcant dfferences, except where (#study) refers to a qualtatve analyss. The? refers to nuances n the hobbysts confdence results (Burnett et al., 2010). Downloaded from by guest on 24 November 2018

5 764 Margaret Burnett et al. programmng n a gender-nclusve way. Gdget s gender nclusveness comes from nnovatng certan programmng envronment characterstcs. For example, t portrays the computer as fallble and personfes error messages (Lee and Ko, 2011; Lee et al., 2014). A contrbutng technology to Gdget s the Idea Garden. The Idea Garden supports dversty n a varety of ways, one of whch s presentng explanatory help n ways that are compatble wth both females tendences toward comprehensve nformaton processng and males tendences toward depth-frst nformaton processng (Cao et al., 2013; Jerngan et al., 2015). (We wll return to Gdget later n ths paper). Gven how sgnfcantly such gender dfferences are ted wth software usage, how should developers proceed? The GenderMag method ams to enable software developers to answer that queston for themselves n the context of the software products they are producng Analytcal evaluaton and personas The GenderMag method s an analytcal method for evaluatng usablty. Analytcal methods rely on expert analyss, supported by gudelnes, prncples or prompts. They can be less laborntensve than user testng and can reveal problems early n the desgn process, when they are less expensve to fx (Blandford et al., 2008). The Cogntve Walkthrough (CW) s one such analytcal evaluaton method. The CW s a partcularly good ft to GenderMag s scope of problem-solvng software, because the CW was orgnally developed from theores of problem solvng (Anderson, 1987; Greeno and Smon, 1988) and learnng by exploraton (Polson and Lews, 1990; Polson et al., 1992). Because the Gender- Mag method s based n part on the CW, we descrbe CWs n detal here. The CW focuses specfcally on ease of learnng (Blandford et al., 2008; Lews et al., 1990; Mahatody et al., 2010; Wharton et al., 1994) and supports systematc evaluaton of how a frsttme user would carry out a task by usng nterface features. In a CW, a team of evaluators walks through the nterface step by step, evaluatng the nterface s usablty and learnablty at each step, n the sequence a user would do when completng some partcular task for the frst tme. The orgnal method conssted of a page wth bref questons, and also assumed a background n Cogntve Scence (Wharton et al., 1994). Snce then, the method has evolved over several teratons (Wharton et al., 1994). The frst teraton made t more formal and complex (Lews et al., 1991), but problems wth the usablty of the method and the need for Cogntve Scence knowledge as a prerequste stll left t dffcult to use. A smplfed verson then emerged (Wharton et al., 1994). Ths verson, whch s often cted and appled today, dd not requre the evaluator to place as much emphass on understandng the user s explct and mplct goal structures for partcular walkthrough steps. Several extensons to the Wharton et al. method have snce been developed, wth dfferent foc and for dfferent contexts. In 2010, Mahatody et al. dentfed 11 CW varatons: Heurstc Walkthrough, Norman Cogntve Walkthrough Method, Streamlned Cogntve Walkthrough, Cogntve Walkthrough for the Web, Groupware Walkthrough, Actvty Walkthrough, Interacton Walkthrough, Cogntve Walkthrough wth Users, Extended Cogntve Walkthrough, Dstrbuted Cogntve Walkthrough and Enhanced Cogntve Walkthrough (Mahatody et al., 2010). There s also a Programmng Walkthrough varant especally for evaluatng programmng envronments (Bell et al., 1991). In developng GenderMag, we drew from the Wharton et al. verson (Wharton et al., 1994), and from a more recent streamlned verson of the CW (Spencer, 2000), whch suggests provdng preparatory materals to the team n advance and a strong facltator wthn the team to keep the team on track and to avod lengthy desgn fxes and dscusson. In the Wharton et al. CW, evaluators perform a CW n two phases (Wharton et al., 1994). In the Preparatory Phase, they descrbe the target user, the task for evaluaton and an deal (or at least correct) sequence of goals, subgoals and respectve actons to acheve the task. Then, n the Analyss Phase, they use a prototype of the system to systematcally work through the deal subgoal sequence as f they were the target user, usng a set of questons (actng as prompts) to structure ther evaluaton and uncover possble usablty or learnablty ssues. For each subgoal step, evaluators ask whether users wll have formed ths subgoal as a step to achevng ther overall goal. Not dong so may mean that users mght not reach ther overall goal, or get stuck. For each of the acton steps, the evaluators ask three questons: (1) whether users wll note that the acton s avalable to them, (2) whether they wll assocate the ntended effect wth the acton and (3) whether they wll understand that they have made progress toward completng the task. Negatve answers to these questons ndcate the presence of potental ssues that mght affect usablty and learnablty. The CW method has several strengths. Lews et al. (1990) found that the CW method s more robust than Heurstc Evaluaton or tradtonal thnk-aloud user studes n terms of varablty n evaluator performance. It has been suggested ths mght be due to ts structured nature (Hertzum and Ebbe Jacobsen, 1999). Another strength of the CW method s that t can uncover desgn errors that may mpede novces learnng by dong, but t can also uncover usablty ssues that extend beyond ease of learnng (Mahatody et al., 2010; Wharton et al., 1994). Ths strength has been attrbuted to ts unconstraned nature (Hertzum and Ebbe Jacobsen, 1999) and correlaton of ease of learnng wth ease of use and functonalty (Mahatody et al., 2010; Wharton et al., 1994). It can be used early, n the desgn phase wth early stage prototypes, to uncover errors (Spencer, 2000), and can also be used later, throughout desgn and development phases (Wharton et al., 1994). Another strength s that the method can llumnate what background knowledge the user should possess to complete tasks (Wharton et al., 1994). A CW strength Downloaded from by guest on 24 November 2018

6 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 765 partcularly pertnent for uncoverng gender ssues s that the CW can reveal assumptons and msconceptons about the user that the desgner mght have unwttngly bult nto the system (Mahatody et al., 2010; Wharton et al., 1994). The CW also has weaknesses (Hertzum and Ebbe Jacobsen, 1999; Mahatody et al., 2010; Wharton et al., 1992, 1994). For example, choces made n task selecton and ther decomposton nto subgoals and actons durng the Preparatory Phase have mportant consequences on fndng ssues durng the Analyss Phase. Tedum can also be an ssue: the same questons are asked multple tmes and ths can become repettve for an evaluator. Perhaps the most mportant weakness from a dversty/ nclusveness perspectve s the danger of descrbng users n very hgh-level terms (e.g. people who use exstng ATM machnes (Wharton et al., 1994)), whch may encourage anchorng or stereotypng (Hertzum and Ebbe Jacobsen, 1999). Ths weakness could be partcularly detrmental to the GenderMag goals of helpng desgners make nformed decsons about gender dfferences relevant to software usage. To head off ths weakness, the GenderMag method ncludes a set of faceted personas to descrbe a target set of female and male users of the software beng evaluated, embeddng the facets mplcated n problem-solvng dfferences descrbed n Secton 2.1. A persona s a vvd descrpton of an archetype of some subset of a system s ntended users, ncludng ther goals, motvatons and atttudes (Adln and Prutt, 2010; Cooper, 2004), and personas are becomng ncreasngly popular n UX practce. Research on the usage of personas shows that desgners often use personas to communcate about user needs durng desgn phases of software development, such as va deaton and role-playng durng nformal tests of prototypes (Fress, 2012; Matthews et al., 2012; Nelsen and Storgaard Hansen, 2014) although a few researchers also suggest ther use wth analytcal evaluaton methods lke the CW (Adln and Prutt, 2010; Fress, 2012). The creaton of personas requres care. For valdty and credblty, personas need to be based on qualtatve and/or quanttatve emprcal data about target users (Adln and Prutt, 2010; Faly and Flechas, 2011; McGnn and Kotamraju, 2008; Prutt and Grudn, 2003). For applcablty and buy-n, they also need to be customzable to some extent (Adln and Prutt, 2010), but only n aspects that do not nteract wth the persona s valdty. In keepng wth these recommendatons, we derved our personas from prevous qualtatve and quanttatve gender studes, and explctly defned whch parts are customzable, as we explan further n Secton THE GENDERMAG METHOD 3.1. The method GenderMag s an evaluaton method wth whch software practtoners can evaluate the problem-solvng software they desgn and produce. The method focuses on the fve facets of gender dfferences that we descrbed n Secton 2.1, encapsulates them nto personas to brng them to lfe and embeds use of the facets nto a systematc process va a gender specalzaton of the CW (Wharton et al., 1994). More formally: Defnton: The GenderMag method s an analytcal method for evaluatng software () accordng to the followng fve facets of gender dfferences: motvaton, nformaton processng style, computer self-effcacy, rsk averson and tnkerng; () whch are encapsulated nto a set of faceted personas, each of whch has a gender and has research-based facet values for all fve facets; () usng a gender-specalzed CW process that ntegrates references to the facets and to the selected persona throughout. To nstantate the GenderMag method to evaluate a partcular software product, the evaluaton team selects one or more personas from the GenderMag persona set, optonally customzes the selected personas n the customzable portons of the persona and performs the set-up requred for CWs (.e. defnng an deal sequence of each task to be evaluated) n the Preparatory Phase. The evaluaton team then uses ths nstance of GenderMag n the Analyss Phase to evaluate ther own software/prototype by followng the gender-specalzed CW wth each persona they have selected. We explan each of these aspects n the next subsectons. To facltate GenderMag s nstantaton, we have created a GenderMag kt, whch contans practcal nstructons on how to prepare for and conduct the GenderMag CW process, the set of personas and examples and forms. The kt s avalable at The facets and ther ntegraton nto personas There are more than fve facets that could be obtaned from gender theory and emprcal lterature, but t seems unreasonable to expect GenderMag users (evaluators) to keep a large number of facets n mnd throughout an evaluaton. Thus, we settled upon fve facets as the maxmum we would nclude. Includng only fve facets requred us to accept the lmtaton that there are mportant gender-nclusveness aspects that nfluence problem solvng but would have to be omtted; however, we accepted ths trade-off to support the method s usablty. As to whch fve facets we should nclude, we terated over ths choce through our formatve studes. Our crtera were that the facets (1) needed to be extensvely researched n the lterature, (2) needed to be usable by ordnary software developers or user experence (UX) practtoners who had no pror background n gender research and (3) needed to have mplcatons for software usage. Ths process ultmately resulted n the lst of facets whose provenance we dscussed Downloaded from by guest on 24 November 2018

7 766 Margaret Burnett et al. n Secton 2.1: motvaton, nformaton processng, computer self-effcacy, rsk averson and tnkerng. Usng emprcal data for these fve facets, we ncrementally began to create four personas as follows: For each facet, we consdered ts range of possble values, and how ndvduals dentfyng wth each gender cluster across those values. To llustrate, Fg. 1 shows one facet by gender, usng data from (Burnett et al., 2010). Each of the four personas n the GenderMag persona set Tm, Abby, Pat(rca) and Pat(rck) has a value for each of the fve facets, and background consstent wth those facet values. Together, the four personas cover a wde sweep across these facet values: () We assgned to Tm the facet values most frequently seen n males, choosng as a tebreaker those most dfferent from those seen frequently n females. Thus, Tm represents a large fracton of males (as well as a few females), as n Value A from Fg. 1. () We assgned to Abby the facet values frequently seen n females that are most dfferent from those seen n males. Thus, Abby represents a large fracton of females (as well as a few males), as n Value C from Fg. 1. Intutvely, Abby s meant to represent the opposte of Tm n terms of the fve facets. () We assgned (dentcally) to the two Pats facet values that combned (1) facet values often occurrng for females wth (2) facet values somewhat less often occurrng for females wth (3) facet values often occurrng wth both groups, resultng n a composte along the lnes of Value B from Fg. 1. The two Pats are dentcal except for ther genders. One am of Pat(rca) s to combat napproprate stereotypng of females by showng nuanced dfferences (and lkewse Fgure 1. Values for one of the facets. Note that, although females values (lght yellow) are farly unformly dstrbuted among Values A, B and C for ths facet, the males values (dark blue) fall much more nto Value A than nto the other two values. Thus, f Value A s the only one supported at ths tme n the software, addng support for Value B and Value C would mprove nclusveness for both females and males. for Patrck and males). The dentcal Pats together also am to rase awareness that the mportant dfferences relevant to nclusveness le n the facets themselves, and not n a person s gender dentty. That s, they demonstrate that, although ndvdual dfferences often cluster by gender, the gender label tself s not the pont the road to nclusveness les n the facets. By communcatng ths through Patrca s and Patrck s commonaltes, we am to encourage evaluators to thnk n terms of the facets ( s ths feature effectve for people who have a comprehensve nformaton processng style? ) as the road toward nclusveness across genders. Thus, these four personas are charged wth rasng awareness of the ndvdual dfferences that often cluster by gender, and to cover a wde range of the facet values from the lterature. For example, Abby s, Patrca s and Patrck s motvaton to use technology comes from what t can accomplsh, whereas Tm enjoys technology for ts own sake. As a more nuanced example, Abby prefers ways of learnng new features other than tnkerng (e.g. va tutorals); Tm, Patrca and Patrck all tend to prefer tnkerng, but Patrca and Patrck go about tnkerng dfferently than the way Tm does. Table 2 enumerates all of the personas smlartes and dfferences for each facet, and all four personas are shown n Fgs. A1 A4 n Appendx Personas and stereotypng Personas, by defnton, represent a group of users (Marsden, 2014; Turner and Turner, 2011) wth the facet value the persona ncludes; personas are archetypes of user groups. In our context, ths rases a rsk of napproprate gender stereotypng. We consdered several ways to amelorate ths rsk. At frst glance, t mght seem that the answer could le n somehow removng gender from the personas. However, ths s not a promsng soluton because, wth supposedly gender-neutral terms lke user, most people envson males (Bradley et al., 2015), whch would be at odds wth our goal of encouragng them to deeply consder males and females. Ths phenomenon s n keepng wth Luger s argument that gnorng/removng gender merely hdes mplct stereotypcal assumptons about gender, makng them harder to address (Luger, 2014). Thus, our approach nstead goes for explctness, puttng faceted females and males squarely n the center of the evaluaton effort, thereby encouragng the feelngs of empathy that personas person-lke presentatons can generate (Grudn, 2006). Gven use of gendered personas, we have taken three measures to amelorate the rsk of napproprate stereotypng. We have already alluded to the frst two frst, that the four personas show nuanced wthn-gender dfferences; and second, that the two Pats dentcal facet values but dfferent genders am to partcularly emphasze that nclusveness ssues le not Downloaded from by guest on 24 November 2018

8 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 767 Tm Abby Pat(rca) Pat(rck) Table 2. The personas facet values. Motvaton Informaton Computer Rsk Purpose for usng processng style self-effcacy averson Tnkerng Tm represents users wth facet values most common n males, as n Fg. 1 s Value A Abby represents users whose facet values statstcally assocated wth females are most dssmlar to Tm s, as n Fg. 1 s Value C. As such, t provdes the strongest lens for dentfyng ssues that male development teams may not have notced before The Pats (Patrca and Patrck) represent users wth values lke Value B,.e. statstcally assocated wth many females and some males. Identcal female and male personas am to guard aganst overgeneralzng Technology s a source of fun Wants what the software/ feature can accomplsh Wants what the software/ feature can accomplsh Selectve Hgh (Fg. 1 s Value A ) Comprehensve Low. (Fg. 1 s Value C ) Comprehensve Medum. (Fg. 1 s Value B ) Rsk-tolerant Rsk-averse Rsk-averse Prefers; sometmes tnkers to excess Avods Prefers; tnkers reflectvely All facet values are based on extensve emprcal data. For example, the computer self-effcacy column reflects computer self-effcacy data lke Fg. 1, whch shows self-effcacy data reported n Burnett et al. (2010). n broad between-gender groupngs, but n each facet s range of possble values. Thrd, the personas explctly counteract a number of common assumptons not supported by data (Churchll, 2010). One example of such an assumpton s wth gender and mathematcs, an area closely assocated wth computng. Recent research has shown that when stereotype threat s controlled for, there are no dfferences n male and female mathematcal performance (Else-Quest et al., 2010). Therefore, all four personas are equally profcent wth accountng-level mathematcs. In fact, all four have equvalent background, job ttle and responsbltes, math sklls, doman knowledge and skll wth the technology that they use regularly. All of them even lke to play computer games as per research showng that about the same number of males and females play games (ESA, 2015) although the partcular games they lke sometmes dffer (ESA, 2015). In keepng wth these measures, GenderMag constrans personas that have the same job ttle and responsbltes to be entrely dentcal n everythng else too, except emprcally establshed dfferences. All dfferences beyond those of the fve facets must fulfll these three constrants: (1) they must be emprcally supported, (2) they must not suggest a dfference n ntellgence or educaton and (3) they must algn wth that persona s facet values or skll level Persona customzablty Wthn these bounds, personas must be customzable, so that the software team ultmately usng GenderMag can relate to the personas. For example, a product amed at professonal chefs may need a professonal chef nstead of an accountant representng the user, the Sudoku game may become passé, and a software team n Brazl may not empathze much wth a user from Wales. Thus, we have made explct the parts that can be customzed wthout losng the essence of the four personas (Fg. 2). Of course, f an evaluaton team changes one persona s unshaded sectons, they must also change all the other personas accordngly Tyng the faceted personas to a systematc process GenderMag connects these personas to a gender-specalzed CW at a fne granularty. Our prmary specalzaton s that the gender-specalzed CW explctly encourages reflecton on the facets n the personas at every step of the evaluaton, to help Downloaded from by guest on 24 November 2018

9 768 Margaret Burnett et al. Downloaded from by guest on 24 November 2018 Fgure 2. The parts of the personas that an evaluaton team s allowed to change, llustrated wth Patrca. We have shaded the parts that are not changeable, but the evaluaton team can talor unshaded parts to reflect the target user populaton. (Appendx shows the unshaded Patrca.) Fgure 3. The analyss phase of GenderMag CW. The persona name and Why questons are GenderMag s addtons over a standard CW. evaluators reman cognzant of the pertnent evdence-based gender dfferences throughout the Analyss Phase. As wth the tradtonal CW, the method has a Preparatory Phase and an Analyss Phase. In the Preparatory Phase, tasks and deal sequence of actons are defned based on sample forms, just as n the tradtonal CW. Durng the Analyss Phase, evaluators walk systematcally through pre-defned tasks usng a prototype of the system and evaluate whether

10 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 769 the GenderMag persona they are usng would have formed the goals, subgoals and deal acton sequences as specfed by the developers/desgners of the system. The GenderMag CW ncludes why questons and explct references to the current persona s facet values at each goal and acton step. In Fg. 3, we show how ths changes the tradtonal, full CW, whch s the verson we have evaluated so far. (We are now consderng movng to a specalzaton of the more modern, Spencer verson nstead.) 4. TWO EARLY FORMATIVE STUDIES OF GENDERMAG We have been teratvely developng the GenderMag method snce As part of ts teratve evoluton, we descrbe here two formatve studes, whch helped to shape the verson we evaluated n more detal n a thrd study (Secton 5) The method s hstory and an early case study at company X GenderMag was frst conceved when we receved an unexpected emal from John, a product manager at Company X (Fg. 4). In essence, the emal was a cry for help. It made us realze that despte all the research nto gender dfferences n software usage there was nothng for practcal use by ndustry practtoners wthout gender research backgrounds, to help them dentfy ther products gender-nclusveness ssues. In John s case, the gender nclusveness of hs company s software product was crtcal to ts compettveness. Toward provdng a method useful for such ndustry practtoners, we began by specalzng the CW method. In the nstructons, we created to accompany ths specalzed CW, we provded bref overvews of fve facets of gender dfferences n software use (a slghtly dfferent lst from that of Secton 2.1): Motvaton, Rsk Averseness, Self-effcacy, Tnkerng, Strategy. We also added nstructons on how to specfy values for each of these facets. For example, our nstructons requred a score from 1 to 10 to specfy the user s level of rsk averseness n the user descrpton. In the summer of 2012, Company X tred out the ntal verson of the method on ther software, and we came and observed them. Our goals were to fnd out what knds of dffcultes would arse, and whether the method would brng benefts to an evaluaton team of non-researchers. The frst dffculty John and hs colleagues experenced was n tryng to descrbe the user n the way our nstructons requred. In wrestlng wth ths problem, John decded to turn to the personas he had seen ther Marketng Department use. He knew that personas were often used as a way to descrbe users, so he proposed to adapt one from the set Marketng had developed. However, he ran nto dffcultes ntegratng the fve facets we had provded nto Marketng s personas, so upon our arrval at Company X, we worked together wth John to modfy one of the Marketng personas, producng the faceted persona shown n Fg. 5. To complete the Preparatory Phase, John and hs colleagues worked out three task descrptons, whch we ll refer to here as Tasks 1, 2 and 3. Task 1 was very basc, ntended to help everyone fgure out how the specalzed CW process worked. Then, for the Analyss Phase, John gathered four other employees of Company X an HCI researcher, two software Fgure 5. Excerpts from the persona used at Company X. The fve facets and ther values used n the gender-specalzed CW are n the lower part of ths persona. Downloaded from by guest on 24 November 2018 Fgure 4. An unexpected emal.

11 770 Margaret Burnett et al. developers and a senor applcaton desgner to help carry out the gender-specalzed CW. None had ever done a CW before, and none had carred out any research nto gender dfferences. The group commenced wth the CW on Task 1, usng the gender-specalzed user descrptons, to get up to speed. Even n the basc Task 1, the group found three ssues that could affect the product s gender nclusveness. Ths surprsed everyone, because Task 1 was so smple, we had thought of t only as a warm-up task. Gven these results, after a lunch break, four more company employees (software developers) decded to jon n, and together ths larger group carred out the evaluaton on Tasks 2 and 3. The after-lunch group revealed a number of addtonal ssues n ther software, based on the female persona n Fg. 5. Specfcally, n Task 2, a less smplstc task than Task 1, the group found 6 ssues, and n Task 3, a farly complex task, the group found 12 ssues. The ssues they found tended to be classc usablty ssues arsng from the Gulf of Evaluaton and Gulf of Executon (Norman, 2002), such as controls that were not obvous or oddly postoned on the screen, an end user (here, a medcal practtoner) havng to understand that certan functonaltes were avalable only on certan screens, and lack of feedback as to the effects of an acton. Many of the ssues they found revealed an underlyng expectaton by the software desgn and development team that these thngs would be clear after a user tnkered and expermented. Indeed, n dscussng the ssues they found, the team sometmes explctly gave reasons relatng to the two facets of Wllngness to Tnker and Self-Effcacy, but ths only happened occasonally. Overall, the team thought the method was easy enough to apply durng the Analyss Phase wthout pror experence; ths led two of the software developers to ask for more evaluatons on other parts of the software. Company X s n-the-feld experence revealed the need for two mprovements to the method n order to enable the practtoners to not lose ther focus on the facets. Frst, the team seemed to forget about the facet values farly often. Ths emphaszed the need to embed remnders to the facets n the CW process tself, so that an evaluaton team does not lose sght of them. Ths led to changes we later made to each queston n the gender-specalzed CW to fnally produce the verson presented n Secton 3.5. Second, the study revealed the mportance of the Preparatory Phase, wth faceted personas carefully prepared n advance of the Analyss Phase to concretely capture the facets for evaluators who are not famlar wth gender research. We descrbe next how we ncorporated these lessons nto the method, and how they played out n our second formatve nvestgaton. academc conference that emphaszes human aspects of programmng. Pror to ths event, we had made changes to the method, as per the results of the case study at Company X, as follows. Our frst change to the method was to add remnders to the CW of the approprate facets at each step of the way. Our second change was to elmnate the Strategy facet, whch had caused dffculty n the case study; ths left the followng four: Motvaton, Self-effcacy, Rsk and Tnkerng. The thrd change was that, nstead of usng Marketng personas talored at the last moment to our facets, two HCI researchers drew upon two users from prevous nclusveness research, Louanne and F4, and brought them to lfe by role-playng them (Fg. 6). Louanne, a retred busnesswoman, represented an experenced, self-taught computer user (not a programmer) who s somewhat acceptng of rsk and mght be wllng to tnker (Davdson and Jensen, 2013). F4 was a college freshman who was ntroduced n (Cao et al., 2010b); she was a computer user whose rsk averson, lack of wllngness to explore and low self-effcacy fgured extensvely n her partcpaton n a study of end-user mashup actvtes (Cao et al., 2010b). Our task envronment was Lookng Glass (Gross et al., 2012), a programmng envronment that bulds upon Storytellng Alce (Kelleher et al., 2007). Storytellng Alce s earler work led by the same researcher as Lookng Glass (Kelleher), whch teaches mddle school students to program Downloaded from by guest on 24 November A formatve workshop event Our second formatve nvestgaton took place va a workshop event n Fall 2013 at the IEEE Symposum on Vsual Languages and Human-Centrc Computng (VL/HCC), an Fgure 6. An HCI researcher role-playng F4, a college freshman descrbed n (Cao et al., 2010b).

12 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 771 3D anmatons. In the Preparatory Phase, the Lookng Glass researcher provded three tasks; she then served as the expert when questons about the software arose durng the Analyss Phase. For the Analyss Phase, another experenced HCI researcher served as the CW facltator, and three volunteers at a tme from the workshop audence cycled through 10-mn stnts as the team of evaluators. The CW lasted about 45 mn, and went as follows. Frst, the Lookng Glass researcher (Kelleher) projected an nstance of Storytellng Alce on the screen and performed the steps of the task. At each step, the facltator asked the current team of evaluators the CW questons and remnders. After the team gave answers to the questons, the role-players of F4 and Louanne each gave answers to the questons from the perspectve of ther persona. After a few steps, new audence members were rotated nto the team to nclude as many vews as possble. Most of the members of ths audence-based evaluaton team had no pror CW experence. Applyng ths verson of the method n ths way revealed 10 nclusveness ssues 2 that relate to the facets, despte Alce or Lookng Glass beng a relatvely mature platform. The ssues dentfed were manly assocated wth three of the personas facets Self-effcacy, Rsk and Tnkerng such as not explorng system features, stckng to establshed routnes or beng surprsed by unexpected system actons. In the workshop event unlke n the prevous case study the evaluators dd not seem to lose sght of the facets, and n fact seemed to become more attuned to nuances of the facets as the CW progressed. Ths suggests the value of the genderspecalzed CW s remnders at keepng the evaluators focused on the facets. We beleve ths was also partly because the roleplayers brought the personas to lfe at every step of the CW. Of course, HCI researchers who are attuned to gender dfferences and can role-play the personas are not readly avalable to UX professonals and software developers, so we decded t was necessary to ntegrate nto the method a set of faceted personas, carefully derved from research data, to make the method more accessble to practtoners. Ths decson led to the personas we presented n Secton A LABORATORY THINK-ALOUD STUDY OF GENDERMAG Iteratng agan on the method to ncorporate feedback we had gathered from the case study at Company X and the 2 Note that these ssues were found for personas who do not match the Alce system s ntended users. Alce s users are mddle-schoolers n an educatonal settng, whereas our personas represented a much older populaton wth very dfferent motvatons, and n entrely dfferent settngs. Thus, the ssues found should not be vewed as ssues about Alce or Lookng Glass n ts ntended usage, but smply as a demonstraton of the method s ablty to hghlght ssues for the personas that are beng used durng the evaluaton. Ths ponts to the crtcalty of the personas component of the method the personas affect the knds of ssues that evaluators wll fnd. workshop event at VLHCC, we ncorporated our frst two faceted personas, Abby and Tm (Secton 3). To evaluate and nform the method n much more detal than our prevous nvestgatons, we conducted a thnk-aloud study under the controls possble n a lab settng, usng these frst two personas. Our am at ths stage of GenderMag s development was to evaluate how professonal UX practtoners would apply the GenderMag method. Thus, ths study consdered the followng research questons: RQ1 (Research Transfer): How do UX practtoners use the facets n the GenderMag method? RQ2 (Gender): How does a UX practtoner s gender affect method use? RQ3 (Value): How does the method nfluence the usablty ssues UX practtoners fnd? Are the ssues dentfed of real, practcal value? 5.1. Partcpants and procedures Ten experenced UX practtoners (4 female, 6 male, 5.25 years mean work experence) took part n the study and, nstead of workng n a team, evaluated the software ndvdually. All were famlar wth CWs, but none had any background n gender research. Because we were not nterested n nvestgatng the genderspecalzed CW component alone or the personas component alone, we dd not solate them nto separate treatments; rather, we wanted to evaluate the entre method, and to compare ts gender-nclusveness results wth an establshed UX practce. Thus, we randomly assgned partcpants to one of two condtons: half the partcpants (referred to as PGM) appled the GenderMag method usng the faceted Abby persona and the gender-specalzed CW; the other half (referred to as PS) evaluated the software usng a standard CW and the Tm persona. We realzed that ncludng Tm wth the standard CW could muddy the waters, n that t would remove a clean separaton between the GenderMag method (GenderMag group) versus a tradtonal method (Standard group). Further, usng a standard CW does not often nclude personas so why nclude Tm? The answer was farness: the vague the user the standard CW allows s known to be problematc, sometmes leadng to anchorng (Hertzum and Ebbe Jacobsen, 1999). Further, there are proposals to use personas to rectfy t (e.g. (Fress, 2012; Adln and Prutt, 2010)), so t seemed unfar to arbtrarly choose to gnore ths known soluton to a known problem. Thus, we decded that the farest comparson was to provde Tm as a persona of a male the user to the Standard group. The partcpants task was to evaluate a porton of Gdget (Lee et al., 2014), a game-lke envronment n whch novce users program and debug code. Pror to ths task, partcpants flled out a bref background questonnare, took Gdget s standard short tutoral that explans basc nterface elements Downloaded from by guest on 24 November 2018

13 772 Margaret Burnett et al. to Gdget users and looked over the persona and CW forms we gave them (GenderMag CW wth Abby or Standard CW wth Tm, dependng on the group). Partcpants verbalzed ther thoughts as they worked, and we recorded the sesson audo and screen actvty. Partcpants then analyzed three Gdget tasks Gdget Levels 1, 5 and 20 step-by-step as to the persona s ablty to accomplsh these tasks, wth a maxmum of 45 mn per level. (One partcpant (P7GM) went over ths tme lmt and was stopped before completng Level 5.) We selected these levels to represent a range of task dffculty that all Gdget users must overcome to fnsh the game. Level 1 teaches smple programmng constructs. Level 5 has bugs that Gdget users must fx nvolvng arguments and object manpulaton. Level 20 ntroduces functons and s often very challengng to users (Lee et al., 2014)). (Updated versons of these levels can be experenced at The facltator encouraged all partcpants to refer to the personas, and stepped n only f a partcpant fell slent or devated from the method they had been gven. Fnally, partcpants completed an ext questonnare about ther experence. The questonnare focused n partcular on how useful and usable partcpants regarded the method they had just used, and how lkely they were to use t n the future The Gdget envronment Gdget (Lee et al., 2014) (Fg. 7) s an onlne game desgned to teach programmng concepts to non-programmers. Gdget s a robot character that tres to save anmals endangered by a chemcal spll, but ts code s faulty. Each level n the game ntroduces a new anmal-savng msson, and n the process ntroduces new programmng concepts n ts faulty (Pythonlke) code. To progress to each new level, users have to solve the problem of how to save the anmals n the current level by debuggng the provded code. For ths study, we used the Gdget verson from 28 May In ths verson, the basc UI elements are as follows: Gdget users have three executon buttons ( one step, one lne and to end ) that run Gdget s code and ncrementally show the effects of the code on the world. Users can edt the code, can reset all edts made by pressng the restore to orgnal code button, and can nspect objects propertes by clckng on the object n the world. Note that the partcpants were evaluatng the entre envronment s ablty to enable users to succeed at ther problem-solvng tasks namely learnng the aspect of programmng targeted n each of Levels 1, 5 and 20 not just the UI wdgets. For example, for the Level 5 task, an excerpt from the deal task sequence we provded was as follows: subgoal g6: Identfy problem: Gdget can t grab the goop because t s too far away... acton g6-a: User runs code to end. [Gdget stops wth error]. subgoal g7: Fx problem: Move Gdget to goop locaton. acton g7-a: User stops code. acton g7-b: User edts code from down to down 3. acton g7-c: User nserts code left 3 at next lne. Downloaded from by guest on 24 November 2018 Fgure 7. A porton of the Gdget game envronment at Level 5, expanded to show Gdget code (left) and the Gdget character n the world (mddle). In Level 5, Gdget users debug the code to manpulate several objects.

14 GenderMag: A Method for Evaluatng Software s Gender Inclusveness Analyss procedures We transcrbed the recordngs of the sessons and combned these transcrpts wth the notes partcpants made on the CW forms as the bass for our analyss Facets referenced For RQ1 (Research Transfer), a measure of how much of the applcable gender research the GenderMag method transferred to practtoners s n how they used the facets. Thus, for each subgoal and acton, we coded the CW forms and the transcrpts for each facet that partcpants referred to (.e. Motvaton, Informaton Processng, Self-Effcacy, Rsk and Tnkerng). Two coders ndependently coded 23% of the data usng these codes to check for relablty, wth 85% agreement (Jaccard measure of agreement) ndcatng hgh relablty of code applcaton. Gven ths consstency between coders, one of the coders then fnshed up the codng Issue types found RQ3 consders how the method mpacted the usablty ssues partcpants found. To nvestgate ths queston, we coded the types of ssues the partcpants dentfed. For ths, we analyzed each nstance n whch partcpants ether sad that the persona would struggle or explctly ndcated a problem on the CW forms. We coded the nstances nto types that reuse and extend those used n prevous Gdget studes wth real users (Lee et al., 2014). Table 3 shows the provenance of each code. Table 3. The ssue types code set. The rghtmost column of Table 3 shows two broad categores of types from the pror Gdget studes: programmng concepts and problem-solvng ant-patterns (Lee et al., 2014). In the pror studes, users had dffcultes understandng certan programmng concepts, such as strng equalty, functon calls vs. functon defntons, so we looked for all those ssues n ths study s results as well. We also looked for all the problemsolvng ant-patterns reported n those studes (problemsolvng strateges that do not lead n a productve drecton). We also looked for the algorthm barrers reported n those studes, but dd not fnd them n our data. We then added to that lst ssue types n our data that had not been reported for the prevous Gdget studes. We evaluated the relablty of ths codng scheme on 21% of the data, reachng agreement of 85% (Jaccard measure of agreement) between two coders dfferent than the coders of Secton Then, gven ths level of consstency, one of these two coders fnshed up the codng. 6. LAB STUDY RESULTS 6.1. RQ1 (research transfer): from gender research to practcal facet usage Our frst research queston (RQ1) consders the research transfer queston enablng practtoners to apply fndngs of past gender research to ther own stuaton. Because these fndngs are encapsulated n the GenderMag facets, we Issue type Descrpton Provenance Renvent the Wheel Persona deletes code wthout even readng t Ant-patterns (Lee et al., 2014) I don t want to try t Persona s afrad to try an dea Ant-patterns (Lee et al., 2014) When all you have s a hammer Persona stcks to a programmng construct that no Ant-patterns (Lee et al., 2014) longer works All-knowng computer Persona assumes provded code must be ok. Does not Ant-patterns (Lee et al., 2014) read code Functons Persona has problems usng functons (e.g. functon Programmng concepts (Lee et al., 2014) defntons and calls) Assertons Persona has problems usng assertons (goal statements) Programmng concepts (Lee et al., 2014) Strng equalty Persona has problems usng strng equalty (e.g. Programmng concepts (Lee et al., 2014) case-senstvty and exact strng matchng) Objects Persona has problems usng objects (e.g. objects can be Programmng concepts (Lee et al., 2014) nspected or have behavor) Dve In Persona omts testng, just modfes code Derved from ths study s data Fault localzaton Persona cannot fnd where error s or where a fx needs Derved from ths study s data to be made What to select Persona does not know whch feature to use (but does know what to do) Derved from ths study s data; smlar to selecton learnng barrer n Ko et al. (2004) Restore Persona starts over (dscard all work) prematurely Derved from ths study s data Fxed? Persona does not know f fx s correct Derved from ths study s data Tracng Persona has trouble fndng code effects n the world Derved from ths study s data Downloaded from by guest on 24 November 2018

15 774 Margaret Burnett et al. nvestgate ths queston by consderng whether and how the UX practtoners made use of the facets. Although we expected GenderMag partcpants facet usage to be hgher than those of the Standard partcpants, the dfferences between the two groups exceeded our expectatons. Four of the fve GenderMag partcpants talked about the facets more than any of the Standard partcpants dd, and overall they referred to facets nearly twce as often as Standard partcpants (Fg. 8, left). Further, ths pattern held across every facet: four of the fve GenderMag partcpants referred to every ndvdual facet more than any Standard partcpant dd (Fg. 8, rght). Snce the Standard partcpants had a faceted persona (but not the GenderMag CW), ths suggests that the combnaton of the GenderMag CW process and the faceted persona mattered to partcpants applcaton of the facets. Specfcally, nether provdng personas wthout provdng remnders throughout the process (as n the case study at Company X n Secton 4.1), nor provdng faceted personas wthout the GenderMag CW as n the Standard group, seemed as effectve at encouragng evaluators usage of the facets compared wth usng the entrety of the GenderMag method as a tghtly coupled whole. As to how partcpants used the facets, t was usually by talkng about how that facet appeared n ther specfc persona. In fact, ther verbalzatons mrrored the gender dfference lterature very well. Ths was true of both groups, although as Fg. 8 just showed, more often by the GenderMag group than the Standard group. For example, gender dfferences n nformaton processng styles were almost paraphrased n these partcpants uses of that facet: P3GM (Abby s Informaton Processng): She s gatherng everythng to understand the problem before tryng to solve t. versus P4S (Tm s Informaton Processng): He just sorta pcks one and tres t out. Tnkerng was the facet mentoned most frequently, and also llustrates ths phenomenon well: P2GM (Abby s Tnkerng): Probably not, because... she s not someone who would try... P3GM (Abby s Tnkerng):... and she does brng herself to tnker... versus P5S (Tm s Tnkerng): But, I thnk Tm s someone that s qute confdent to clck around, so he would fnd t... P4S (Tm s Tnkerng):... He s more of an explorer and a tnkerer. However, the groups how-ness dverged n nterestng ways wth the Rsk facet. Frst, the Standard partcpants rarely brought up Rsk at all and second, even when they dd, they mentoned only stuatonal rsks. For example, P1S (Tm s Rsk): If I started typng here... I clck on to end, I assume... would be takng me to the end of the lne, because that are the terms that I use on my general keyboard. So there s a danger. In contrast, GenderMag partcpants frequently brought up Rsk, and, n most of these cases, they mentoned personal feelngs of rsk that Abby herself would experence: P2GM (Abby s Rsk):... but t does let her worry a lttle bt because ths tme s dfferent from last tme... P3GM (Abby s Rsk): She s a bt rsk-averse, so maybe she mght not go straght to the end. Ths may suggest that ether the Abby persona nspred more empathy among the partcpants than Tm dd, and/or that the entrety of the GenderMag method helped promote empathy better. We speculate that both factors contrbute, because of the attenton the GenderMag partcpants gave not only to Abby, but also to Abby s facets ndvdually. Relevant to empathy but not qute the same are these two quotes from male GenderMag partcpants, who explctly expressed Abby s value for them n takng on someone else s perspectve: P7GM (questonnare):... t was really useful. I mean t s partcularly n terms of evaluatng from someone else s perspectve because t actually forced me to be more objectve. Downloaded from by guest on 24 November 2018 Fgure 8. Left: Each partcpant s total mentons of facets, for GenderMag (lght orange) vs. Standard (dark blue). Partcpants dscussed gender facets for GenderMag almost twce as often as they dd for Standard. Rght: Each partcpant s menton for each facet, for GenderMag vs. Standard. Except for Informaton Processng, partcpants n GenderMag dscussed each facet more than n Standard.

16 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 775 P2GM (questonnare): Wth a persona, I was able to take user s vew further, as n puttng myself n the user s shoes be more aware the walkthrough sn t about me. In some ways, ther comments could be about empathy, but the comments also sound mpersonal, suggestng that at least these two males were keepng Abby at arm s length. Ths brngs up the possblty that the evaluators gender may factor nto the process, whch we consder next RQ2 (gender): the mpact of evaluators genders Several hypotheses are possble regardng the genders of the evaluators. One such hypothess s that f a team s mostly male, they may not beneft very much from GenderMag because they do not relate well to the female personas and/or the female personas facet values, and are smlar enough to the male personas that they already had the ntutons they needed for consderng male users needs. A contrastng hypothess s that male practtoners and female practtoners mght not dffer much n ther use of GenderMag snce the facet connectons t provdes are ntegrated nto the process n a fne-graned way. Stll other possbltes nclude females beng especally engaged wth GenderMag due to hgher empathy, or conversely, dstancng themselves due to a resstance to seeng themselves as beng characterzed by one of the female personas. To nvestgate such possbltes, we analyzed facet usage by gender dentty of the evaluator. Fgure 9 shows the ndvdual partcpants usage of the facets by gender wthn groups (GenderMag on the left, Standard on the rght). The female GenderMag partcpants mentoned Tnkerng a great deal more than males dd, but otherwse, gender dfferences were not apparent, suggestng that the partcpants n general, regardless of gender, were more facet-focused usng GenderMag than usng Standard. Consderng facet mentons by group wthn gender (Fg. 10: females on the left, males on the rght) lkewse reveals that both female and male UX professonals tended to consder the facets more usng GenderMag than ther counterparts who used the Standard CW. But t also shows that the dfference was especally notable for the female partcpants for four of the fve facets, GenderMag female partcpants referred to Abby s facets at least twce as much as Standard females referred to Tm s facets. The males and females comments regardng ther empathy toward the personas may help to explan the trends n Fgs. 9 and 10. For example, one female GenderMag partcpant, P2GM, dentfed strongly wth Abby and stated that she and Abby would explore the nterface n a smlar way: P2GM (female): Although, f she knows thesameasidd,soshe got ntroduced to the same thng I dd... she probably wll have worked out what ths restore orgnal code means. Male GenderMag partcpants also occasonally dentfed wth Abby, but, n general, provded far less evdence of dentfyng wth the Abby persona. At one pont, partcpant P7GM (male) even stated: P7GM (male): I thnk the persona s almost mmateral at ths pont. Although the same partcpant dstanced hmself from Abby, he nonetheless reported that GenderMag made hm objectve and unbased: P7GM (male, questonnare): I thnk t actually clarfed for me... a consstent lesson of evaluatng sort of anythng, whch s beng objectve and not applyng your personal bas. Thus, overall the results pant a nuanced pcture of the mpact of the evaluator s gender on use of the method. GenderMag seemed to nspre more empathy wth Abby n female partcpants, and seemed to partcularly encourage them to consder Tnkerng as they carred out the process. However, although males dd not seem to relate as strongly to the Abby persona as the females dd, GenderMag stll appeared to help male partcpants consder the persona s facets more Downloaded from by guest on 24 November 2018 Fgure 9. Each partcpant s mentons of facets, for GenderMag (left) and Standard (rght). Crcles are female partcpants and crosses are male partcpants. GenderMag females had a great deal more to say than males dd about Tnkerng. Otherwse, lttle dfference was apparent n male vs. female partcpants use of facets.

17 776 Margaret Burnett et al. Fgure 10. Partcpants mentons of facets, for GenderMag (lght orange) and Standard (dark blue). Left: Female GenderMag partcpants mentoned four of the fve facets more than Standard females. Rght: Male GenderMag partcpants tended to refer to facets more than Standard males. frequently than ther counterparts who used the Standard CW. These results suggest that usng GenderMag as a gender lens had utlty regardless of the gender makeup of the evaluaton team, but that females mght have experenced a greater magnfcaton of gender nclusveness ssues than males dd RQ3 (value): the usablty ssues found What knds of ssues dd partcpants fnd, and how? A key crteron for the effectveness of any analytcal method s ts ablty to dentfy usablty ssues. Numercally, GenderMag partcpants dentfed about the same number of ssues as Standard partcpants. We nterpret ths result as evdence of both the GenderMag method and of the Standard CW (wth persona) beng effectve usablty methods from a fnd usablty ssues perspectve. However, the ways partcpants went about dentfyng ssues seemed to dffer by group. As Fg. 11 shows, the GenderMag group was more lkely to report ssues as they related to one of the facet values. Specfcally, the rghtmost bars of Fg. 11 show that, n contrast to GenderMag partcpants ssues, almost half of the ssue dentfcatons by the Standard partcpants dd not take the facets nto account at all. For example, partcpants P2GM,P3GM and P7GM all used Rsk to fnd usablty ssues at acton #g2-a, where the user s supposed to stop Gdget at the locaton of a bug n order to edt the code and fx the bug. In ther dscusson of ths acton, partcpants stated that Abby would press restore orgnal code nstead, n part because of her averson to rsk. P7GM put t as follows: P7GM (Rsk): That seems a lttle bt doubtful agan, gven her persona, to just jump n and start edtng. I d knd of magne she would be a bt more cautous about dong that... She mght be nclned to actually clck on restore orgnal code The ssues the GenderMag group dentfed for Abby were also often dfferent from those the Standard group dentfed for Fgure 11. Medan number of ssues that partcpants dentfed usng each facet; by GenderMag partcpants (lght orange) and by Standard partcpants (dark blue). GenderMag partcpants found far more ssues usng the Rsk and Tnkerng facets than Standard partcpants dd; n fact, many ssues found by the Standard group dd not consder any gender facet at all (rghtmost bars). Tm. To nvestgate what knds of ssues arose, we categorzed the ssues as per the code set presented earler (Table 3). Table 4 enumerates the results. For example, partcpants dentfed several ssue types (I don t want to try t, When all you have s a hammer, Assertons) as problematc for Abby more often than for Tm, whereas they dentfed other ssue types (Tracng, Renvent, Dve In) to be problematc more often for Tm than for Abby. These dfferences n ssue types rase two more questons: whether the ssues are real, and f so, whether they really algn by gender n the same way as they algned by Abby and Tm. We consder these questons next The dentfed ssues valdty To nvestgate whether and to what extent the ssues the lab study partcpants found correspond to real ssues for real Gdget users, we valdated them n three ways. Frst, we compared the ssues our partcpants dentfed wth a prevously publshed analyss by the development team of Downloaded from by guest on 24 November 2018

18 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 777 Table 4. The 14 ssue types dentfed by the lab study s partcpants n order of frequency for Abby, wth the percentage of all ssues found for the Abby or Tm persona. Abby (%) Tm (%) Issue valdated? Gender dstrbuton Issue type valdated? I don t want to try t Fault localzaton What to select 14 9 When all you have s a hammer 11 5 Restore 8 0 Functons 8 11 Assertons 5 0 Strng equalty 3 5 All-knowng computer 3 2 Objects 3 2 How valdated? Follow-up to fx the ssue? Development team Hoped for (Outsde the development team s responsblty area.) Development team Yes, dealt wth pror to our contact wth the development team. Fxes appled: (1) Syntax hghlghter had a bug, fxed (2) Whle typng, syntax and undefned varables are now underlned n red (3) Idea Garden help system: mproved gudance to help fnd the error Development team Yes, pror to our contact wth the development team. Fxes appled: Improved the bult-n tutoral Development team Development team Hoped for (Outsde the development team s responsblty area.) Development team and Lee et al. (2014) Development team and Lee et al. (2014) Yes, pror to our contact wth the development team. Fxes appled: Idea Garden (Cao et al., 2013) provded help about the concept Yes, pror to our contact wth the development team. Fxes appled: Idea Garden help system provded help about the concept Development team Yes, pror to our contact wth the development team. Development team and Lee et al. (2014) Development team and Lee et al. (2014) Fxes appled: mproved Gdget s nstructons at the begnnng of the level Yes, pror to our contact wth the development team. Fxes appled: Idea Garden help system dentfed some errors n startng code Yes, pror to our contact wth the development team. Fxes: Idea Garden help system provded help about the concept Fxed? 3 4 N/A N/A Tracng 3 7 Development team Renvent 0 4 Development team Yes, pror to our contact wth the development team. Fxes appled: added a button that says clear everythng. The frst tme they clck t, t asks f they are sure because they wll lose the hnts that the code contans Dve In 0 25 Development team Gdget s development team valdated 13/14 of these ssue types. 49% of the valdated ssues (n the sx shaded rows) had gender dfferences. Downloaded from by guest on 24 November 2018

19 778 Margaret Burnett et al. problems wth usng Gdget (Lee et al., 2014). Second, we verfed each ssue wth the Gdget development team by showng the Gdget team the transcrpt so that they could see for themselves the stuatons n whch our partcpants reported the ssues, and then asked the team whether they had observed these ssues n ther own users. Fnally, we asked the team whether they had notced gender dfferences for these ssues. Table 4 s Issue valdated? and How valdated? columns show the results. The development team s observatons over the prevous 2 years of seeng how males and females used Gdget n camps and lab experments served as the ground truth for all 14 of the ssue types. In total, the Gdget team verfed that they had ndependently observed 13 of the 14 of the ssue types. Four of the ssue types had formally been publshed the year before (Lee et al., 2014) but note that the study partcpants who found these ssues had not seen Gdget or the publcaton. The verfed ssue types accounted for 97% of the ssue nstances that the study partcpants dentfed for Abby and 96% of those dentfed for Tm. Regardng gender, 10 of the 13 valdated ssues matched the Gdget team s experence of gender dstrbuton (Table 4, gender dstrbuton valdated? column). Of these 10, four were reported and valdated about equally across gender, and sx (shaded n the table) were reported and valdated to dffer across gender. In total, these verfcatons of the gender dstrbuton of ssue types covered 81% of the ssues the study partcpants had dentfed. Fnally, the Gdget team verfed that (at least) sx of the ssue types were mportant so mportant that they need to be fxed. We know ths because, at the tme of the ntervew, they had already proceeded to fx four of them. (They could not fx the other two as they were outsde that team s responsblty.) Perhaps due n part to the ssues the Gdget team had fxed n tme for the Gdget publc release, ther software s qute popular wth females. At the tme of wrtng, 47% of the users of Gdget s regstered users dentfy as female. 7. DISCUSSION: THE ROAD AHEAD The studes descrbed n ths paper provde proof-of-concept evdence that the GenderMag method can (and dd) alert ordnary software practtoners such as the product manager and software developers of Company X and the UX practtoners of the lab study to nclusveness ssues n problem-solvng software that can mpact dfferent genders dfferently. We have also very recently completed a feld study nvolvng ndustral uses of GenderMag (Burnett et al., 2016). In that study, four teams of software practtoners (manly software managers and software developers) at a government agency and at two large hardware/software companes conducted GenderMag evaluatons on ther own software. All four teams found gender-nclusveness ssues n ther own software usng the method. These results are encouragng evdence of GenderMag s effectveness. Our next research goals revolve around conductng longterm studes n real-world settngs to nvestgate possble obstacles to adoptng GenderMag. In fact, n some organzatons, there may be barrers to even tryng, let alone adoptng, the GenderMag method. One of our next research goals s to catalog these barrers and to understand the potental changes we mght make to the method to address them. Our prevous research has suggested several possble obstacles to adopton that GenderMag could face. Frst, we have seen nstances of phlosophcal obstacles to nvestgatng gender dfferences. Some people call nto queston whether females and males behave dfferently wth software at all; our work takes the opposte stance, restng upon the evdence presented n Secton 2. As to people who do acknowledge the exstence of gender dfferences, they hold numerous vews of those dfferences. One spectrum of these vews runs from essentalst perspectves, whch hold that cogntve and behavoral dfferences between males and females are nnate, to socal-construct perspectves, whch see gender dfferences as arsng through socety s atttudes toward gender roles. Although nether end of ths spectrum questons whether gender dfferences exst, these perspectves suggest dfferent drectons as to how to address gender dfferences. That s, f gender s vewed as a socal construct, ths suggests that a way to address gender dfferences s by breakng down barrers that may have come about through learned gender roles (e.g. as wth stereotype threat (Appel et al., 2011; Zhang et al., 2013)). Ths s the drecton the GenderMag method takes. In addton to the above phlosophcal obstacles, certan organzatonal and practcal obstacles to adopton can arse. For example, some organzatons may beleve that females are not an mportant customer group to target, and thus ther software does not need to be gender-nclusve. We have also encountered some software teams who were uncomfortable wth or unable to nvestgate gender ssues because of ther organzaton s prvacy or equal opportuntes polces. In other nstances, some teams may beleve that they do not need GenderMag f the team ncludes a number of females on t. Indeed, one of our ndustral contacts expressed a lack of nterest for precsely ths reason. Ths queston remans open: data from our lab study suggest that the female partcpants got abtmore out of GenderMag than the males (recall Secton 2). Stll, regardless of whether t helps to have females on the team, t makes sense for any team to use GenderMag f t feels the need; the all-male Company X software development team offers a case n pont (recall Fg. 4). Fnally, obstacles can be methodologcal. For example, methodologcal objectons could stem from a team s resstance to one of the GenderMag components, such as use of personas. Indeed, Adln and Prutt stress the need for a number of steps to encourage adopton of persona-based methods nto organzatons, wthout whch persona adopton can fal (Adln Downloaded from by guest on 24 November 2018

20 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 779 and Prutt, 2010). As another example, teams not accustomed to usng analytcal methods mght be pessmstc about the expected cost/beneft of usng analytcal methods (such as GenderMag) vs. emprcal methods. In our lab study, when we asked partcpants f they would use our method n the future, one partcpant sad precsely ths, explanng that t was not vable because t was too tme consumng (Partcpant P5S). However, wthn the GenderMag group, all fve partcpants stated they would be lkely to use GenderMag n the future; for example, for breakng down dffcult or complex evaluatons nto component parts (P7GM), to provde nterestng nsghts (P2GM) and ts effcency (P6GM). Especally apprecated was GenderMag s potental to spot problems and nform desgn, for example: I could see ssues I wanted to fx by redesgnng rght away (P6GM). Stll, analytcal methods can be as tme-ntensve as some knds of quck user studes n ndustry, and the perceved benefts of user studes mght be hgher. In fact, one of our ndustral contacts, an ndustry-based UX professonal, commented that quck studes nvolvng users seem more persuasve to ther software teams and managers than analytcal results. Some of these obstacles to adopton could suggest mprovements to the method that can address the obstacles. For example, the phlosophcal category has already nspred the mechansms explaned n Secton 3.3 to guard aganst napproprate stereotypng. Others, such as a methodologcal dstrust of persona-based methods, may smply suggest ntroducton and presentaton processes to allay concerns about the method that are unwarranted. Stll others, such as organzatonal clmate, may dentfy organzatons and stuatons that are not rght for use of the GenderMag method. 8. CONCLUSION In ths paper, we have ntroduced GenderMag the frst systematc evaluaton method for practtoners to fnd gendernclusveness ssues n problem-solvng software. At the GenderMag method s core are fve facets drawn from an extensve body of research lterature on gender dfferences that can mpact use and usablty of problem-solvng software. The fve facets are the central pont of the method. That s, by promotng support for the fve facets, GenderMag s not ultmately about labelng people by gender, t s about desgnng for a dversty of ndvduals problem-solvng facets that happen to cluster statstcally by gender. We have teratvely evolved and emprcally nformed GenderMag across a range of settngs and wth a varety of evaluator types. In these nvestgatons, evaluators were software developers, software managers, HCI researchers and UX practtoners; personas represented a medcal practtoner, a college student, a retred busnesswoman and an accountant; and software products evaluated were a system for customzng medcal devces, an end-user programmng envronment for storytellng and a debuggng game. In addton, HCI students and local software developers have used t nformally (betatested t) to evaluate a programmng tool for bocomputng researchers, a support system for travelers, a moble-based document system and a decson support system to help chemsts or envronmental engneers choose whch materals to use n ther manufacturng processes. Emergng results suggest that the scope of GenderMag mght be slghtly larger than for software that drectly targets problem-solvng; t seems to be useful n evaluatng any nterface that s tself complex enough to nvolve problem-solvng (e.g. how do I make the system do what I want? ), even f the task beng supported by that complex nterface s not a problem-solvng task. In all of these uses of GenderMag, the evaluators have always found ssues. Further, most of the ssues they have found were real ssues, as wth the evaluaton of the lab study results wth the Gdget team. Fnally, as the lab study n ths paper also llustrated, GenderMag enabled partcpants to dentfy gender-nclusveness ssues even though none of them had a background n gender research. The method s avalable n kt form at consortum.org/gender/, and s beng beta-tested n several HCI educaton and producton software settngs n Denmark, Germany, Sngapore, Sweden, the UK and the USA. We am for ts usage to not only contnue to nform the method tself, but also to nform the expanson of ts personas corpus and our understandng of the boundares of the method s scope. Ultmately, ths research ams to help software teams avod unntentonally producng software that s not gendernclusve. Past research shows that ssues of gendernclusveness are pervasve n problem-solvng software, and untl now, software teams lke John s at Company X have had no mechansm to fnd out f ther products suffer from such ssues, and f so, exactly where the ssues are or why they are ssues. Wth GenderMag, we hope that John and others lke hm wll have a tool that helps them head off stuatons lke the one experenced by F4, the female end-user programmer n (Cao et al., 2010b): F4: Ths s so hard for me. Why s t so dffcult? ACKNOWLEDGEMENTS We thank the partcpants of our studes for ther tme; Catln Kelleher, Mary Beth Rosson and Rachel Whte for ther help durng formatve stages of ths work; and NSF for ther support under CNS , IIS , and IIS REFERENCES Adln, T. and Prutt, J. (2010) The Essental Persona Lfecycle: Your Gude to Buldng and Usng Personas. Morgan Kaufmann/Elsever, Burlngton, MA, USA. Anderson, J.R. (1987) Skll acquston: complaton of weakmethod solutons. Psychol. Rev., 94, Downloaded from by guest on 24 November 2018

21 780 Margaret Burnett et al. Appel, M., Kronberger, N. and Aronson, J. (2011) Stereotype threat mpars ablty buldng: effects on test preparaton among women n scence and technology. Eur. J. Soc. Psychol., 41, Arjona-Rena, L., Robles, G. and Dueñas, S. (2014) The FLOSS2013 Free/Lbre/Open Source Survey. Bandura, A. (1986) Socal Foundatons of Thought and Acton. Prentce Hall. NJ, USA: Englewood Clffs. Bardzell, S. (2010) Femnst HCI: Takng Stock and Outlnng an Agenda for Desgn. In Proc. CHI, pp ACM, New York, NY, USA. Beckwth, L., Burnett, M., Wedenbeck, S., Cook, C., Sorte, S. and Hastng, M. (2005) Effectveness of End-user Debuggng Software Features: Are there Gender Issues? In Proc. CHI, pp ACM, New York, NY, USA. Beckwth, L., Kssnger, C., Burnett, M., Wedenbeck, S., Lawrance, J., Blackwell, A. and Cook, C. (2006) Tnkerng and Gender n End-user Programmers Debuggng. In Proc. CHI, pp ACM, New York, NY, USA. Beckwth, L., Inman, D., Rector, K. and Burnett, M. (2007) On to the Real World: Gender and Self-effcacy n Excel. In Proc. Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Bell, B., Reman, J. and Lews, C. (1991) Usablty Testng of a Graphcal Programmng System: Thngs We Mssed n a Programmng Walkthrough. In Proc. CHI, pp ACM, New York, NY, USA. Berenson, S.B., Slaten, K.M., Wllams, L. and Ho, C.-W. (2004) Voces of women n a software engneerng course: reflectons on collaboraton. J. Educ. Resour. Comput., 4, 1. Blandford, A., Hyde, J., Green, T.R.G. and Connell, I. (2008) Scopng usablty evaluaton methods: a case study. Hum. Comput. Interact. J., 23, Borkn, M., Yeh, C., Boyd, M., Macko, P., Gajos, K., Seltzer, M. and Pfster, H. (2013) Evaluaton of Flesystem Provenance Vsualzaton Tools. In Trans. Vs. Comput. Graphcs 19, 12, pp IEEE, Los Alamtos, CA, USA. Bradley, A., MacArthur, C., Hancock, M. and Carpendale, S. (2015) Gendered or Neutral? Consderng the Language of HCI. In Proc. Graphcs Interface Conference, pp ACM, New York, NY, USA. Brggs, P., Thomas, L. and Mavn, S. (2014) On Famly and Fear: A Gendered Perspectve on the Desgn of Identty Technologes. In CHI 2014 Workshop: Perspectves on Gender and Product Desgn. tves/papers (accessed February 23, 2015). Burnett, M., Flemng, S., Iqbal, S., Venola, G., Rajaram, V., Farooq, U., Grgoreanu, V. and Czerwnsk, M. (2010) Gender Dfferences and Programmng Envronments: Across Programmng Populatons. In Proc. ACM Emprcal Software Engneerng and Measurement (ESEM). ACM, New York, NY, USA. Burnett, M., Beckwth, L., Wedenbeck, S., Flemng, S., Cao, J., Park, T., Grgoreanu, V. and Rector, K. (2011) Gender pluralsm n problem-solvng software. Interact. Comput., 23, Burnett, M., Peters, A., Hll, C. and Elaref, N. (2016) Fndng Gender-Inclusveness Software Issues wth GenderMag: A Feld Investgaton. In Proc. CHI. ACM, New York, NY, USA. Butler, J. (1999) Gender Trouble: Femnsm and the Subverson of Identty. Routledge, New York, NY, USA. Cafferata, P. and Tybout, A. (1989) Gender Dfferences n Informaton Processng: A Selectvty Interpretaton, Cogntve and Affectve Responses to Advertsng. Lexngton Books, MA, USA. Cao, J., Rector, K., Park, T., Flemng, S., Burnett, M. and Wedenbeck, S. (2010a) A Debuggng Perspectve on End-User Mashup Programmng. In Proc. IEEE Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Cao, J., Rche, Y., Wedenbeck, S., Burnett, M. and Grgoreanu, V. (2010b) End-User Mashup Programmng: Through the Desgn Lens. In Proc. CHI, pp ACM, New York, NY, USA. Cao, J., Kwan, I., Bahman, F., Burnett, M., Flemng, S., Jordahl, J., Horvath, A. and Yang, S. (2013) End-User Programmers n Trouble: Can the Idea Garden Help them to Help Themselves? In Proc. Symposum on Vsual Languages and Human-Centrc Computng. IEEE, Los Alamtos, CA, USA. Cassell, J. (2002) Genderzng HCI. In Jacko, J. and Sears, A. (eds) The Handbook of Human Computer Interacton, pp Lawrence Erlbaum, Mahwah, NJ, USA. Chang, S., Kumar, V., Glbert, E. and Terveen, L. (2009) Specalzaton, Homophly, and Gender n a Socal Curaton Ste: Fndngs from Pnterest. In Proc. ACM Computer Supported Cooperatve Work & Socal Computng, pp ACM, New York, NY, USA. Charness, G. and Gneezy, U. (2012) Strong evdence for gender dfferences n rsk takng. J. Econ. Behav. Organ., 83, Churchll, E. (2010) Sugared puppy-dog tals: gender and desgn. Interactons, 17, Cooper, A. (2004) The Inmates Are Runnng the Asylum: Why Hgh Tech Products Drve Us Crazy and How to Restore the Santy. Sams Publshng, USA. Downloaded from by guest on 24 November 2018

22 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 781 Coursars, C., Swerenga, S. and Watrall, E. (2008) An emprcal nvestgaton of color temperature and gender effects on web aesthetcs. J. Usablty Stud., 3, Davdson, J. and Jensen, C. (2013) Partcpatory Desgn wth Older Adults: An Analyss of Creatvty n the Desgn of Moble Healthcare Applcatons. In Proc. Creatvty & Cognton, pp ACM, New York, NY, USA. Dohmen, T., Falk, A., Huffman, D., Sunde, U., Schupp, J. and Wagner, G.G. (2011) Indvdual rsk atttudes: measurement, determnants, and behavoral consequences. J. Eur. Econ. Assoc., 9, Durndell, A. and Haag, Z. (2002) Computer self-effcacy, computer anxety, atttudes towards the Internet and reported experence wth the Internet, by gender, n an East European sample. Comput. Hum. Behav., 18, Else-Quest, N.M., Hyde, J.S. and Lnn, M.C. (2010) Crossnatonal patterns of gender dfferences n mathematcs: a meta-analyss. Psychol. Bull., 136, ESA (Entertanment Software Assocaton) (2015) Essental Facts About the Computer and Vdeo Game Industry: 2015 Sales, Demographc, and Usage Data. Essental-Facts-2015.pdf. Faly, S. and Flechas, I. (2011) Persona Cases: A Technque for Groundng Personas. In Proc. CHI, pp ACM, New York, NY, USA. Fress, E. (2012) Personas and Decson-Makng n the Desgn Process: An Ethnographc Case Study. In Proc. CHI, pp ACM, New York, NY, USA. Greeno, J.G. and Smon, H.A. (1988) Problem Solvng and Reasonng. In Atknson, R.C., Hernsten, R., Lndzey, G. and Luce, R.D. (eds) Stevens Handbook of Expermental Psychology. John Wley and Sons, NY. Grgoreanu, V., Cao, J., Kulesza, T., Bogart, C., Rector, K., Burnett, M. and Wedenbeck, S. (2008) Can Feature Desgn Reduce the Gender Gap n End-user Software Development Envronments? In Proc. Symposum on Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Grgoreanu, V., Burnett, M. and Robertson, G. (2010) A Strategy-centrc Approach to the Desgn of End-user Debuggng Tools. In Proc. CHI, pp ACM, New York, NY, USA. Grgoreanu, V., Burnett, M., Wedenbeck, S., Cao, J., Rector, K. and Kwan, I. (2012) End-user debuggng strateges: a sensemakng perspectve. ACM Trans. Comput.-Hum. Interact., 19, do: / Gross, P., Herstand, M., Hodges, J. and Kelleher, C. (2012) A Code Reuse Interface for Non-programmer Mddle School Students. In Proceedngs Intellgent User Interfaces, pp ACM, New York, NY, USA. Grudn, J. (2006) Why Personas Work: The Psychologcal Evdence. In Prutt, J. and Adln, T. (eds) The Persona LfeCycle: Keepng People n Mnd Throughout Product Desgn. Morgan Kaufmann Publshers, Burlngton, MA, USA. Hartzel, K. (2003) How Self-effcacy and Gender Issues Affect Software Adopton and Use. In Communcatons ACM 46, pp ACM, New York, NY, USA. Hertzum, M., Ebbe, Jacobsen, N. (1999) The Evaluator Effect Durng Frst-tme Use of the Cogntve Walkthrough Technque. In Proc. HCI Internatonal, Vol. I, pp Sprnger, Hedelberg, Germany. Hou, W., Kaur, M., Komlod, A., Lutters, W.G., Boot, L., Cotten, S.R., Morrell, C., Ant, Ozok, A. and Tufekc, Z. (2006) Grls Don t Waste Tme : Pre-adolescent Atttudes Toward ICT. In Proc. CHI Extended Abstracts, pp ACM. Huffman, A.H., Whetten, J. and Huffman, W.H. (2013) Usng technology n hgher educaton: the nfluence of gender roles on technology self-effcacy. Comput. Hum. Behav., 29, Jerngan, W., Horvath, A., Lee, M., Burnett, M., Culty, T., Kuttal, S., Peters, A., Kwan, I., Bahman, F. and Ko, A. (2015) A Prncpled Evaluaton for a Prncpled Idea Garden. In Proc. IEEE Vsual Languages and Human- Centrc Computng pp IEEE, Los Alamtos, CA, USA. Joyce, K., Wllamson, J. and Mamo, L. (2007) Technology, scence, and agesm: an examnaton of three patterns of dscrmnaton. Indan J. Gerontol., 21, Kelleher, C., Pausch, R. and Kesler, S. (2007) Storytellng Alce Motvates Mddle School Grls to Learn Computer Programmng. In Proc. CHI, pp ACM, New York, NY, USA. Ko, A., Myers, B. and Aung, H. (2004) Sx Learnng Barrers n End-user Programmng Systems. In Proc. Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Kulesza, T., Stumpf, S., Wong, W.-K., Burnett, M., Perona, S., Ko, A. and Oberst, I. (2011) Why-orented end-user debuggng of naïve bayes text classfcaton. ACM Trans. Interact. Intell. Syst., 1, do: / Lee, M. and Ko, A. (2011) Personfyng Programmng Tool Feedback Improves Novce Programmers Learnng. In Proc. ICER, pp ACM, New York, NY, USA. Lee, M., Bahman, F., Kwan, I., Laferte, J., Charters, P., Horvath, A., Luor, F., Cao, J., Law, C., Bethwetherck, M., Long, S., Burnett, M. and Ko, A. (2014) Prncples of Downloaded from by guest on 24 November 2018

23 782 Margaret Burnett et al. a Debuggng-frst Puzzle Game for Computng Educaton. In Proc. Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Lews, C., Polson, P., Wharton, C. and Reman, J. (1990) Testng a Walkthrough Methodology for Theory-based Desgn of Walk-up-and-Use Interfaces. In Proc. CHI, pp ACM, New York, NY, USA. Lews, C., Polson, P. and Reman, J. (1991) Cogntve walkthrough forms and nstructons. Insttute of Cogntve Scence Techncal Report #ICS Unversty of Colorado, Boulder, CO, Luger, E. (2014) A Desgn for Lfe: Recognzng the Gendered Poltcs Affectng Product Desgn. In CHI Workshop: Perspectves on Gender and Product Desgn. tves/papers. Mahatody, T., Sagar, M. and Kolsk, C. (2010) State of the art on the Cogntve Walkthrough method, ts varants and evolutons. Int. J. HCI, 26, Margols, J. and Fsher, A. (2003) Unlockng the Clubhouse: Women n Computng. MIT Press, Cambrdge, MA, USA. Marsden, N. (2014) CHI 2014 Workshop on Perspectves on Gender and Product Desgn. google.com/ste/technologydesgnperspectves/papers. Matthews, T., Judge, T. and Whttaker, S. (2012) How Do Desgners and User Experence Professonals Actually Perceve and Use Personas? In Proc. CHI, pp ACM, New York, NY, USA. McDowell, C., Werner, L., Bullock, H.E. and Fernald, J. (2003) The Impact of Par Programmng on Student Performance, Percepton and Persstence. In Proc. ICSE. ACM, New York, NY, USA. McGnn, J. and Kotamraju, N. (2008) Data-drven Persona Development. In Proc. CHI, pp ACM, New York, NY, USA. Meyers-Levy, J. and Loken, B. (2015) Revstng gender dfferences: what we know and what les ahead. J. Consum. Psychol. 25, Meyers-Levy, J. and Maheswaran, D. (1991) Explorng dfferences n males and females processng strateges. J. Consum. Res., 18, Nelsen, L., Storgaard, Hansen, K. (2014) Personas s Applcable: A Study on the use of Personas n Denmark. In Proc. CHI, pp ACM, New York, NY, USA. Norman, D.A. (2002) The Desgn of Everyday Thngs. Basc Books, New York. O Donnell, E. and Johnson, E.N. (2001) Gender effects on processng effort durng analytcal procedures. Int. J. Audt., 5, O Leary-Kelly, A., Hardgrave, B., McKnney, V. and Wlson, D. (2004) The Influence of Professonal Identfcaton on the Retenton of Women and Racal Mnortes n the IT Workforce. NSF ITWF & ITR/EWF Prncpal Investgator Conference, Phladelpha, PA, pp Pazza, Blog. (2015) STEM Confdence Gap. pazza.com/stem-confdence-gap/. Polson, P. and Lews, C. (1990) Theory-based desgn for easly learned nterfaces. Hum. Comput. Interact., 5, Polson, P., Lews, C., Reman, J. and Wharton, C. (1992) Cogntve walkthroughs: a method for theory-based evaluaton of user nterfaces. Int. J. Man-Machne Stud., 36, Power, C., Frere, A., Petre, H. and Swallow, D. (2012) Gudelnes are Only Half of the Story: Accessblty Problems Encountered by Blnd users on the Web. In Proc. CHI, pp ACM, New York, NY, USA. Prutt, J. and Grudn, J. (2003) Personas: Practce and Theory. In Proc. DUX, pp ACM, New York, NY, USA. Redl, R., Hubert, M. and Kennng, P. (2010) Are there neural gender dfferences n onlne trust? An fmri study on the perceved trustworthness of ebay offers. MIS Q., 34, Rode, J. (2008) An ethnographc examnaton of the relatonshp of gender & end-user programmng. PhD Thess, Unversty of Calforna Irvne. Rosner, D. and Bean, J. (2009) Learnng from IKEA Hackng: I m Not One to Decoupage a Tabletop and Call t a Day. In Proc. CHI, pp ACM, New York, NY, USA. Rosson, M., Snha, H., Bhattacharya, M. and Zhao, D. (2007) Desgn Plannng n End-user Web Development. In Proc. Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Rosson, M., Snha, H. and Edor, T. (2010) Desgn Plannng n End-user Web Development: Gender, Feature Exploraton, and Feelngs of Success. In Proc. Vsual Languages and Human-Centrc Computng, pp IEEE, Los Alamtos, CA, USA. Smon, S.J. (2001) The mpact of culture and gender on web stes: an emprcal study. Data Base Adv. Inf. Syst., 32, Sngh, A., Bhadaura, V., Jan, A. and Gurung, A. (2013) Role of gender, self-effcacy, anxety and testng formats n learnng spreadsheets. Comput. Hum. Behav., 29, Spencer, R. (2000) The Streamlned Cogntve Walkthrough Method, Workng Around Socal Constrants Encountered n a Software Development Company. In Proc. CHI, pp ACM, New York, NY, USA. Downloaded from by guest on 24 November 2018

24 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 783 Stumpf, S., Sullvan, E., Ftzhenry, E., Oberst, I., Wong, W.-K. and Burnett, M. (2008) Integratng Rch User Feedback nto Intellgent User Interfaces. In Proc. Intellgent User Interfaces, pp ACM, New York, NY, USA. Subrahmanyan, N., Beckwth, L., Grgoreanu, V., Burnett, M., Wedenbeck, S., Narayanan, V., Bucht, K., Drummond, R. and Fern, X. (2008) Testng vs. Code Inspecton vs.... What Else? Male and Female end Users Debuggng Strateges. In Proc. CHI, pp ACM, New York, NY, USA. Szafr, D. and Mutlu, B. (2012) Pay Attenton! Desgnng Adaptve Agents that Montor and Improve user Engagement. In Proc. CHI, pp ACM, New York, NY, USA. Tan, D., Czerwnsk, M. and Robertson, G. (2003) Women Go wth the (Optcal) Flow. In Proc. CHI, pp ACM, New York, NY, USA. Turner, P. and Turner, S. (2011) Is stereotypng nevtable when desgnng wth personas? Des. Stud., 32, Weber, E., Blas, A. and Betz, N. (2002) A doman-specfc rsk-atttude scale: measurng rsk perceptons and rsk behavors. J. Behav. Decs. Mak., 15, West, C. and Zmmerman, D.H. (1987) Dong gender. Gender Soc., 1, Wharton, C., Bradford, J., Jeffres, R. and Franzke, M. (1992) Applyng Cogntve Walkthroughs to More Complex user Interfaces: Experences, Issues, and Recommendatons. In Proc. CHI, pp ACM, New York, NY, USA. Wharton, C., Reman, J., Lews, C. and Polson, P. (1994) The Cogntve Walkthrough Method: A Practtoner s Gude. In Nelsen. J. and Mack R. (eds) Usablty Inspecton Methods, pp John Wley, NY. Wllams, G. (2014) Are you sure your software s genderneutral? Interactons, 21, Zhang, S., Schmader, T. and Hall, W.M. (2013) L eggo my ego: Reducng the gender gap n math by unlnkng the self from performance. Self Identty, 12, Downloaded from by guest on 24 November 2018

25 784 Margaret Burnett et al. APPENDIX THE FOUR PERSONAS Downloaded from by guest on 24 November 2018 Fgure A1. The Tm persona, representng users wth facet values most common n males, as n Fg. 1 s Value A. The fve facets are bulleted n the bottom two rounded rectangles. The red, underlned parts are to enable the evaluaton team to quckly remnd themselves of the man ponts.

26 GenderMag: A Method for Evaluatng Software s Gender Inclusveness Downloaded from by guest on 24 November 2018 Fgure A2. The Abby persona, representng female users wth facet values most dssmlar to Tm s, as n Value C. 785

27 786 Margaret Burnett et al. Downloaded from by guest on 24 November 2018 Fgure A3. The Patrca persona, representng female users wth most values along the lnes of Value B. Patrca s dentcal to Patrck (Fg. A4) except for her gender.

28 GenderMag: A Method for Evaluatng Software s Gender Inclusveness 787 Downloaded from by guest on 24 November 2018 Fgure A4. The Patrck persona, representng male users wth most values along the lnes of Value B. Patrck s dentcal to Patrca except for hs gender.