Evaluation: Scientific Studies. Title Text

Transcription

1 Evaluation: Scientific Studies Title Text 1

2 Evaluation Beyond Usability Tests 2

3 Usability Evaluation (last week) Expert tests / walkthroughs Usability Tests with users Main goal: formative identify usability problems improve the tool 3

4 Summative Evaluation (focus today) How good is it? Useful? Better than other tools? 4

5 Formative and Summative: Usually combined formative summative Evaluation over time 5

6 Evaluation goals (summative) Generalizability Results can be applied to other people Precision We measured what we wanted to measure (controlling factors that were not intended to study) Realism Study context is realistic... usually trade-off between them! 6

7 The selection of a research method depends on the research question and the object under study! 7 McGrath / Carpendale

8 Controlled Experiments 8

9 Controlled experiment Or: Laboratory Experiment Lab study User Study A/B Testing (used in marketing) 9

10 Focus Precision Generalizability (?) Overall goal Reveal cause-effect relationships e.g. smoking causes cancer 10

11 Scenario A B Which is better? 11

12 Test it with users! Carpendale 12

13 Hypothesis A precise problem statement Example: H1 = Participants will buy more beer when using variant B than variant A Null-Hypothesis H0 = no difference in beer purchase A B 13

14 Independent Variables Factors to be studied A Typical independent variables (in HCI) Different types of design Task type: e.g., searching/browsing B Participant demographics: e.g., male/female Different technologies: touch pad vs. keyboard Control of Independent Variable Levels: The number of variables in each factor Limited by the length of the study and the number of participants How different? Entire interfaces vs. very specific parts How different? entire interfaces: I can t tell what actually causes the difference (not so good for research?) very specific parts: expensive (not so good for industry?) 14

15 Control Environment Make sure nothing else could cause your effect Control confounding variables Randomization! A e.g. naturally observe and see that girls are always using A, and guys B is it that they buys more beer because of interface A or because they are girls can t say anything about causality B 15

16 Different Designs: Between-Subjects Divide the participants into groups, each group does one condition Randomize: Group Assignment Potential problem? Group 1 A Group 2 B 16

17 Different Designs: Within-Subjects Everybody does all the conditions Can account for individual differences and reduce noise (that s why it may be more powerful and requires less participants) A Severely limits the number of conditions, and even types of tasks tested (may be B able to workaround by having multiple sessions) Can lead to ordering effects > Randomize Order Common: Ordering Effects: * Learning effect: Did everybody use the interface in a certain order? If so, are people faster because they are more practiced, or because of the effect of the interface? * Fatigue / Boredom 17

18 Dependent Variable The things that you measure Performance indicators: task completion time, error rates, mouse movement (numbers of beers bought) Subjective participant feedback: satisfaction ratings, closed-ended questions, interviews questionnaires (HCI lecture last week) Observations: behaviors, signs of frustrations 18

19 Tasks Specifying good tasks for controlled experiments is tricky Specifically, if you are measuring performance criteria Task criteria comparability for different interfaces clear end point Example usability test: >>buy a book for a 4 year old<< controlled experiment: >>find and buy the book Doctor Faustus by Thomas Mann<< 19

20 Results: Application of Statistics Descriptive Statistics Describes the data you gathered (e.g. visually) Inferential Statistics Make predictions/inferences from your study to the larger population 20

21 Descriptive statistics Central tendency mean {1, 2, 4, 5} median {15, 19, 22, 29, 33, 45, 50} mode {12, 15, 22, 22, 22, 34, 34} 21

22 Descriptive statistics Central tendency mean {1, 2, 4, 5} 3 median {15, 19, 22, 29, 33, 45, 50} 29 mode {12, 15, 22, 22, 22, 34, 34} 22 22

23 Descriptive statistics Central tendency mean {1, 2, 4, 5} 3 median {15, 19, 22, 29, 33, 45, 50} 29 mode {12, 15, 22, 22, 22, 34, 34} 22 Measures of spread range variance standard deviation note: for inferential standard deviation N becomes (N-1) > estimate for sampled population 23

24 Visualization of descriptive statistics e.g., Boxplot Mean 25/75% Quartiles Min / Max (alternative: with outliers) Boxplots are not completely standardized - different possible interpretations of Whiskers/ Outliers/.. what you are showing > Important: Describe what you are visually encoding 24

25 Validity Errors: Type I: False positives Type II: False negatives External Validity Can we generalize the study? E.g. generalizable to the larger population of undergrad students Internal Validity Is there a causal relationship? Are there alternate causes? 25

26 Inferential statistics Goal: Generalize findings to the larger population 26

27 Excursus: Tragedy of the error bars CI = Confidence intervals SE = Standard Error (SD of the sampling distribution of the sample mean) SD = Standard Deviation 27

28 Excursus: 95% Confidence intervals USE THEM! Interpretation: We can be 95% confident that the real mean lies within our confidence interval! More intuition about stats: Seeing theory: 28

29 Null Hypothesis Testing Statistically significant results p <.05 The probability that we incorrectly reject the Null-Hypothesis (Type I error) Many different tests t-test, ANOVA, A B CI 29

30 Internal Validity: Storks deliver babies!? R. Matthews, Storks Deliver Babies. Journal of Teaching Statistics, vol. 22, issue 2, pages 36-38, 2001; There is a correlation coefficient of r=0.62 (reasonably high) A statistical test can be employed that shows that this correlation is in fact significant (p = 0.008) What are the flaws? (Reason/Solution on the next slide.) 30

31 (Last slide:) Correlation does not imply causation. Relevant for M4! :-) Pragmatically A step-by-step how-to 31

32 Experimental Procedure: Typical example Identify research hypothesis Specify the design of the study Think about statistics *before* you run the study Run a pilot study Recruit participants Run the actual data collection sessions Analyze the data Report the results 32

33 Experimental Procedure: Typical example Identify research hypothesis Specify the design of the study Think about statistics *before* you run the study Run a pilot study Recruit participants Run the actual data collection sessions Analyze the data Report the results 33

34 Run a pilot study to test the study design to test the system to test the study instruments 34

35 Recruit participants Reflecting the larger population? in the best case yes pragmatic decision though How many? Depends on effect size and study design--power of experiment Usually 15+ (per group) Note: much higher than for usability test (~5) 35

36 Run the actual data collection process System and instruments ready? Greet participants Introduce purpose of study and procedure or deliberately don t Don t bias: compare my interface vs. this other interface, Get consent of the participants ethics! 36

37 Run the actual data collection process Assign participants to specific experiment condition according to pre-defined randomization method Introduction to system(s) and/or training tasks Participants complete the actual tasks take measures of dependent variables Participants answer questionnaire (if any) Debriefing session Payment (if any). monetary, coupons, chocolate 37

38 Report the results Introduction / motivation Study design Results Discussion Conclusions References / Appendix See, for instance, Saul Greenberg s recommendation: assignments/controlled_expt/ass1_reports.html 38

39 Other, more qualitative Evaluation Methods 39

40 Quantitative? Qualitative? Qualitative Focus: Meaning & experience from participants perspectives Key issue: Relevance why & how? In-depth, rich description Approach: more exploratory, open-ended, interpretive; eg interviews, observations, case studies, focus groups etc 40

41 Qualitative Methods as Add-on : Mixed Methods Approach Often controlled experiment + Experimenter Observations Collecting Participants Opinions Core methods: Observation, Semi-structured Interviewing Helpful for... Usability Improvement (cf. HCI last weeks) New insights, explanation of unforeseen results, new questions Can help to confirm results 41

42 Qualitative Methods as Primary Method Pre-design studies Rich understanding of a complex domain Problems, challenges, domain language During-, Post-design studies Case studies/ Field studies Helpful for... holistic understanding 42

43 Qualitative Methods as Primary Method In Situ Observations Participatory Observations Laboratory Observational Studies Contextual Interviews Focus Groups 43

44 Qualitative Challenges Sample Sizes Doing intensive studies with a lot of participants? Time? Data produced? Subjectivity Social relationship? Analyzing the data Grounded theory Open and axial coding 44

45 Further Reading Material ) :! d e d n e m m o Rec 45