Interaction Technology

Transcription

1 Interaction Technology Quantifying User Research Remco Veltkamp

2 Users

3 User Research Quantify user behavior Systematically study goals, needs, capabilities of users De anatomische les van Dr. Deijman, Rembrandt van Rhijn, 1656

4 Usability Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context

5 User experience Perceptions, emotions, beliefs, preferences, behaviors, physical and psychological responses Before, during, of after use of a system

6 Emperical Methods Lab Experiment: artificial situation, highly controlled, typically comparing alternative user interfaces or measures how usability varies with some design parameter Field Study: real situation in the actual environment where people use the system, using real tasks Initial field studies just observe without intervening, final field studies see usage in practice Survey: questionnaire, conducted by paper, phone, web, or in person Good statistical sampling techniques exist to make the results generalizable

7 Emperical Methods Bill White, Southern Illinois University Edwardsville

8 Considerations Sample size Representativeness Randomness Data collection

9 Sample Size Rule of thumb: at least sample size of 30 However when using t distribution instead of z distribution assumption about probability density function of variable values, less than 30 can work out well See Scalable Learning course on statistical evaluation

10 Weak Arguments for Large Samples If the population is large, then the sample needs to be large. The sample size determines variance in statistical sampling, the size of the overall population does not. The evaluation of quality is independent of how many people are going to use it. The more features in the interface, the larger the sample size. More features means more tests, not more users in each test. However, test subjects will be overwhelmed by too many features. The interface is being designed to accommodate many target audiences. This only requires larger sample sizes if the different target audiences will use the interface in very different ways (e.g., buyers vs. sellers, teachers vs. students, doctors vs. patients).

11 Representativeness Sample should be representative of the parent population Not the same as large sample size Large sample of men says little about women Sample of voters more important than large sample of population

12 Gender Much research only based on male subjects Sex refers to biological characteristics of women and men, boys and girls, in terms of reproductive organs and functions based on chromosomal complement and physiology. As such, sex is globally understood as the classification of living beings as male and female, and intersexed. Sex differences relevant to the research and innovation should be investigated and addressed. Gender is a key analytical and explanatory variable in research. Gender as a socio cultural process refers to cultural values and social attitudes that together shape and sanction "feminine" and "masculine" behaviors, and also affect products, technologies, environments, and knowledge. Gender assumptions often go unquestioned and can unconsciously influence scientific priorities, research questions, and choice of methods.

13 Gender factors Household primary earner Personal income Number of hours per week spent doing housework Status of primary person responsible for doing housework Level of stress at home [Bem Sex Role Inventory masculinity/femininity score] Subjects with a higher femininity score can experience differently, regardless whether they were a man or a woman

14 Randomness Random means: no pattern, not taken from a probability distribution function In particular not the uniform distribution! Often difficult to select random subjects Better less than perfectly random sample from right population, than a perfectly random sample from the wrong population

15 Testing, Data collection Formative testing (monitoring to provide ongoing feedback that can be used to improve performance) Task level satisfaction Help access Usability problems Learnability Efficiency Recoverability Note: even open ended comments and problem descriptions can be categorized and quantified Mental model

16 Mental model What the user believes about the system Basis for interaction, and expectations about reaction of the system There can be a gap between designer s and user s mental model Example: gesture control of Unmanned Arial Vehicle Imitative My hand represents UAV Intelligent Expect interpretation of command Instrumented Control via joystick

17 Natural Interaction What is natural? Different mental models: which plant needs the least amount of water? The Artificiality of Natural User Interfaces Communications of the ACM, 2012

18 Testing, Data collection Summative (evaluating at the end of test by comparing it against some standard or benchmark): Task completion time Completion rate Error rate Conversion (persuasion) rate Acquiring data: A/B testing Eye tracking Interviews Surveys Online surveys Wizard of Oz experiment

19 A/B Testing Test small differences, like the titles or button colors, so you know what caused the effect Tools:

20 Eye Tracking

21 Eye Tracking Goals Track the attention of the user Identify locations of interest, distractions Compare designs, lay outs, functionalities, etc. Support other methods, such A B tests or surveys

22

23

24 Virtual supermarket Supermarket testing UMCU: brainstroke patients

25 Virtual supermarket Test cognitive tasks, e.g. remembering, navigation

26 Eye Tracking Tools

27 Usability Interview

28 Usability Interview Some guidelines: Invite the correct target group (your users) Don t ask suggestive or ambiguous questions (open questions) Do not be intrusive (position yourself correctly) Be helpful, start with chit chat, make your user feel comfortable Stay alert, observe important behavior and comments Stay within the time limits, cut off ongoing opinions if necessary

29 Survey Data Open questions Yes/No questions Likert rating scales Original Likert scale format: I think that I would like to use this system frequently: Strongly Disagree Disagree Neither agree not disagree Agree Strongly Agree Likert like scales: Rensis Likert

30 Online Surveys Access to all visitors, only a small % of respondents is enough Collect qualitative as well as quantitative data On the user s initiative, thus watch out for angry users Keep it short and simple (KISS), attention span is short online

31 Standardized Usability Questionnaires Obtain info from respondents Advantage of standardized questionnaires: Objectivity Replicability Quantification Cost effective Communication Scientific generalization All birds can fly, find the ostrich

32 Questionaires Post study After a complete session For example SUS System Usability Scale Post task After each separate task For example ASQ After scenario Questionnaire Website questionnaires Game questionnaires There are many other questionnaires Chapter 8 Sauro & Lewis Tell me what you think about this low fat diet shampoo Do you think it's Crunchy, Half Cruncy, or Not Crunchy at all? Put a tick in the appropriate box there's nothing to it... The Rutles, Archeology, 1996

33 Poststudy: SUS Software/System Usability Scale No cost cite Brooke, 1996 Alternating pos and neg questions

34 SUS Question i, answer position x i Positive questions: add (x i 1) Negative quesions: add (5 x i ) SUS score: sum x 2.5, range [0, 100] Questions 1 3, 5 9 address Usability, 4, 10 address Learnability

35 Post task: ASQ After scenario Questionnaire Probing overall ease of satisfaction, task completion, and information No cost Cite Sauro & Lewis ASQ score: average of response values

36 Websites Different fromother systems: Importance of effective browsing Shift to self service Elements of trust, trust scale Dynamic mash up Navigation help for the impaired

37 Websites: SUPR Q Standardized Universal Percentile Rank Questionnaire Commercial license Usability: 1 3,5 Credibility: 4,6,8,10,11 Appearance: 7,9 Loyalty: 12,13 Score: sum 1 12, plus half of 13, compare to ref database, convert to percentile rank Score of 50 is average

38 Game Experience Questionnaire IJsselsteijn, de Kort, Poels, 2007 Post study questionnaire Core questionnaire: Immersion, Flow, Competence, Positive and Negative Affect, Tension, and Challenge In game version is concise version Social presence module: Psychological and behavioral involvement of player with other social entities Virtual, mediated, or co located Post game module Assess how players feel after they had stopped playing

39 Game Experience Questionnaire Focus on game characteristics, rather than player characteristics However, both player and game characteristics affect usability and experience No published results on reliability and validity

40 In game GEQ Please indicate how you felt while playing the game for each of the items, on the following scale: not at all slightly moderately fairly extremely I was interested in the game's story 2 I felt successful 3 I felt bored 4 I found it impressive 5 I forgot everything around me 6 I felt frustrated 7 I found it tiresome 8 I felt irritable 9 I felt skilful 10 I felt completely absorbed 11 I felt content 12 I felt challenged 13 I had to put a lot of effort into it 14 I felt good

41 In game GEQ scoring Score is the average values of ratings Competence: Items 2 and 9 Sensory and Imaginative Immersion: Items 1 and 4 Flow: Items 5 and 10 Tension: Items 6 and 8 Challenge: Items 12 and 13 Negative affect: Items 3 and 7 Positive affect: Items 11 and 14

42 Game Engagement Questionnaire Brockmyer, Fox, Curtiss, McBroom, Burkhart, Pidruzny, 2009 Focus on player characteristics, rather than game characteristics Immersion, flow, presence, absorption and dissociation taken as increasing levels of a one dimensional scale of engagement

43 Wizard of Oz experiment Experimenter ( wizard ) simulates behavior of a computer application Intercepting all communications between participant and system Originally for learning natural language processing Making believe! cried Dorothy. Are you not a Great Wizard?

44 Wizard of Oz experiment Example: Gesture control of Unmanned Arial Vehicle Which gestures are natural? Gestures are recorded, but also interpreted by wizard For safety, wizard operates drone Ng & Sharlin 2011

45 Controlled Experiment 1. Formulate a testable hypothesis For example: Sticking up number of fingers works better than number of arm waves to indicate a number one or a number two 2. Choose the independent variables to manipulate to test the hypothesis In this case: the mode of operation 3. Measure the dependent variables to test the hypothesis, e.g. time, error rate, non error event count, user satisfaction 4. Use statistical tests to accept or reject the hypothesis Analyze how changes in the independent variables affected the dependent variables, and whether those effects were significant (i.e., indicating a definite cause and effect) Previous lecture

46 Further Reading Chapter 2 of J. Sauro and J.R. Lewis, Quantifying the User experience: Practical statistics for user research (2012)