Experimental Design. What is it? When to use it? Types of Variables Setting up an Experiment Case Study Analyzing the data

Transcription

1 Experimental Design What is it? When to use it? Types of Variables Setting up an Experiment Case Study Analyzing the data

2 Types of evaluation Users not involved Supported by practice/theory Occurs in realistic setting External validity: degree to which research results applies to real situations Large Sampling Subjective/qualitative

3 Done this someway In one form or another we have resorted to experimenting Also an important tool for survival! experimented with various types of ear plugs experimented with different types of pacifiers experimented with various types of snow tires etc But somewhat different, i.e. less formal

4

5 Naturalistic: Approaches: Naturalistic describes an ongoing process as it evolves over time observation occurs in realistic setting ecologically valid real life External validity degree to which research results applies to real situations

6 Advantage Approaches: Naturalistic Can state something about the user s behavior in an actual environment Disadvantage Cannot control (or even know) all the contributing factors to user s performance i.e. do they use menus more frequently than toolbar buttons because the icons are not comprehensible OR because the buttons are too small OR simply because they do not know that they exist OR. [can go on]

7 Approaches: Experimental In certain cases you want to make a statement about a particular UI design choice i.e. I really want to know whether the size of buttons contribute to how quickly users click on them or i.e. I want to know whether a menu designed in a circular shape (pie menu) is more effective than a regular menu or i.e. I want to find out whether technique 1 (or system 1) is better than technique 2 (system 2) You want to make some generic statements that can be widely applicable (not only restrained to your app)

8 Approaches: Experimental Experimental study relations by manipulating one or more independent variables experimenter controls all environmental factors observe effect on one or more dependent variables Internal validity confidence that we have in our explanation of experimental results Trade-off: Natural vs. Experimental precision and direct control over experimental design versus desire for maximum generalizability in real life situations

9 Quantitative Evaluation What task to evaluate? Depends on application Attempt to find canonical task(s) i.e. what would be a set of tasks that can be used to test whether larger icons contribute to faster selection? Common measures Task completion time Error rate Learning rate (novice -> expert transition) Fatigue, comfort? etc.

10 What task to evaluate? Example: Pointing Device Evaluation Real task: interacting with GUI s pointing is fundamental Experimental task: target acquisition abstract, elementary, essential W D

11 Example Is it easier to read with CAPS or without Caps? Want to make a conclusive and general statement whether CAPS are more efficient than non-caps Conclusion would look like: for text, CAPS are 20% less efficient than non-caps or for text, CAPS are 25% more efficient than non-caps

12 Example How do we test this question? Need to come up with a hypothesis a set of variables we are going to manipulate a set of variables we are going to measure reduce the number of confounding variables a task a set of randomized trials

13 Example THE BROWN FOX JUMPED OVER THE MOON. OR, SHOULD IT SAY THE BROWN FOX JUMPED OVER THE CAT.

14 Example The brown fox jumped over the moon. Or, should it say the brown fox jumped over the cat.

15 Example Would it be sufficient to simply show those two slides and do some measurements? What are some problems with this kind of setup? What would we measure? Lets first look at some definitions

16 Hypothesis Statement or claim that the experimenter wants to test Defines the nature of the relationship between two types of variables

17 Hypothesis H 0 : there is no difference in the number of cavities in children and teenagers using crest and no-teeth toothpaste H 1 : children and teenagers using crest toothpaste have fewer cavities than those who use no-teeth toothpaste

18 Hypothesis H 0 : there is no difference in user performance (time and error rate) when selecting a single item from a pop-up or a pull down menu, regardless of the subject s previous expertise in using a mouse or using the different menu types File Edit View Insert New Open Close Save File Edit View Insert New Open Close Save

19 Hypothesis Hypothesis can be softer and uncertain: Will color affect recognition speed? Will proximity affect perceptual organization? Etc

20 Independent Variables At least one circumstance is of major interest in an experiment i.e. menu type in selection time experiment OR text type Referred to as an independent variable Independent of the subject s behavior or performance Want to choose two or more levels of this circumstance to present (manipulate) Nothing the subject does can change the levels of the independent variable CAPS vs. non-caps What are the independent variables in the toothpaste experiment? What are the different levels?

21 Dependent Variables Want to measure a subject s behavior in response to manipulations of the independent variable Dependent variable, depends on what the subject does Statement about the expected nature of the relationship between the independent and dependent variables is referred to as hypothesis (as seen previously)

22 Control Variables Only want to manipulate one circumstance the independent variable All other circumstances need to be controlled These become control variables control font of two different types of menus control color coding on two different types of visualizations Have to be controlled across all levels of the IV confirm that change in dependent variable due to change in independent variable However impossible to control everything More control leads to less generalization

23 Confounding Variables A confounding variable is any factor that varies with the independent variable Suppose we want to use 5 different levels for text type subjects respond more quickly to the last 2 subjects respond more quickly after practice Practice confounded with speed Coke vs. Pepsi

24 Random Variables Want to avoid confounded effects; allow variables to randomly vary: random variables Selecting subjects is usually done randomly For testing effect of color on visibility of an object choose subjects randomly from a large population choose colors to be tested on randomly as well Age factors, eye deficiencies, and other elements would randomly enter into the equation (can eliminate some of these) Can flip a coin, throw dice, allow a random number generator to select for us

25 Example In the previous example what may be a hypothesis H1: Users are slower reading CAPS H2: There is no difference in reading rates H3: CAPS are less memorable What variables do we manipulate, i.e. what are the independent variables? Text type, i.e. CAPS or no Caps (Two levels) What variables do we measure, i.e. what are the dependent variables? Lets look first at the hypothesis H1 or H2: reading speed H3: recall after 2 hours

26 Example What variables do we control? What may be some confounding variables and how do we counter these? More on this next

27 Experimental Design Manipulating and Measuring Variables Within vs. Between Subjects Design Single vs. Multiple Variable Experiment

28 Choosing an Independent Variable Should be what the experimenter wants to manipulate: Font 10 vs. 12 vs. 14 (IV=font size) Bar graph vs. line graph (IV=type of graph) Are children more violent after being exposed to games with violence. What is the IV? In the last question need to define violence, i.e. what is the operational definition of violence in games? Is there shooting/hurting/physical contact? Are the actions moral/immoral (stealing, deceiving, etc.)? Language abuse? Would it be considered violent if outside the game?

29 Single Variable Experiment Only one independent variable Two-level experiment: the IV has two levels (simplest case, where one is the experimental group and the other control group), i.e. existence vs. non-existence Advantages: Way of finding out if IV is worth studying Results easy to interpret and analyze Some cases do not need more than two levels investigating two interaction techniques two educational methods etc.

30 Reading Time Reading Time Reading Time Single Variable Experiment Disadvantages: Sometimes does not say much about the relationship between the IV and the DV Print Size Print Size Print Size

31 Average Test Score Average Test Score Single Variable Experiment Multilevel Experiments: single variable experiments where IV has > 2 levels Advantages: Have better handle over IV-DV relationship Low High Anxiety Level Low Neutral High Anxiety Level The more levels added the less critical is the range of IV (balance between realistic and large enough) Disadvantages: Requires more time and effort than 2-level (within-subjects increases time for each subject, between-subjects requires additional subjects) Statistical tests more complex Need to know when to limit the number of levels

32 Caps Multiple Variable Experiment Most frequent design combines several variables in a factorial combination that pairs each level of IV with the others referred to as a factorial design 2 levels for Caps/no-caps and 3 levels for font size (small/medium/large) Gives 2 x 3 design Font Size Small Medium Large Yes No

33 Advantages Multiple Variable Experiment Interactions between IVs can be studied (interaction occurs when the relationship between one IV and subject s behavior depends on the level of a second IV) Can add additional circumstances by making them IVs When circumstance that could add variability to the data is made into a factor, the amount of variability decreases Disadvantages Time-consuming and costly Analysis more complicated, need to typically do an ANOVA Assumption that variability in data approximates a normal distribution (don t know until completed experiment) Interpretation of results is more complex

34 Range of the Independent Variable Range is the difference between the highest and lowest level of a variable; no specific guidelines, need to fit it in the experiment Realistic range: do not choose levels that are so wide that effects will definitely be found without carrying out the experiment Range that shows effect: should be large enough to have an effect If interested in effect of font size on reading speed choosing between font 14 vs. font 15 will could lead to false conclusions Pilot experiment: similar to real experiment but data thrown out; can test design before proceeding

35 Choosing a Dependent Variable Measure of the subject s behavior Need operational definition; i.e. do violent games result in children s aggression? How do we measure aggressiveness? Panel of judges observing playing behavior + rating Give a selection of toys and observe how they play Narrate frustrating stories and count number of directattacks In HCI it can be a bit more straightforward fortunately But need to also define validity and reliability of the measurements

36 Directly Observable Dependent Variables Directly observable DVs can be measured directly; indirect DVs use secondary measures i.e. physiological measures with a lie detector response time to measure how much info. is processed Single dependent variable: measuring only accuracy or speed; usually not sufficiently indicative of performance i.e. could be very fast but also very inaccurate Multiple dependent variable: speed-accuracy tradeoffs for example gives an overall better indication of performance i.e. more valid Composite dependent variable: multiple dependent variables combined to form one variable

37 Experimental Design Individual differences Need more than one subject Usually multiple subjects (n=at least 10, ideally much more) how to distribute tasks amongst subjects?

38 Within vs. Between Subjects Design Within subject design: Pros: All subjects do all conditions Fewer subjects, less individual differences Easier stats analysis Cons: Transfer effects Doing 1 condition affects following condition Often you want subjects to learn extensively Condition 1 Subject 1 Subject 2. Subject 10 Condition 2 Subject 1 Subject 2. Subject 10 Between subjects design: Pros: Subjects only do one condition No transfer effects Train to high skill Cons: More subjects, individual differences Harder stats analysis Condition 1 Subject 1 Subject 2. Subject 10 Condition 2 Subject 11 Subject 12. Subject 20

39 Experimental Design Order of presentation in within-subjects designs ABBA counterbalancing: Every subject does trials in the order: A, B, B, A Any confounding effect (e.g., learning curve) is counterbalanced Trial# Condition A B B A Linear Confounding effect Resulting Confound: A: = 50 B: = 50 Nonlinear confounding effect Resulting Confound: A: 5+60 =65 B: = 80

40 Experimental Design Order of presentation in within-subjects designs Make order a between-subjects variable Fully counterbalanced: A B B A A B C A C B B A C B C A C A B C B A Combinatorial explosion when n>4 Needs lots of subjects

41 Experimental Design Order of presentation in within-subjects designs Partial counterbalancing. e.g., Latin square: Ensures each level appears in every position in order equally often n rows x n columns and each treatment occurs once in each row and in each column A B C B C A C A B Balanced Latin Square: Each condition precedes and follows each of the other conditions equally often: A B C D B D A C D C B A C A D B

42 Why counterbalance? Reduce transfer effects Experimental Design Assumes symmetric transfer A-B transfer == B-A transfer If asymmetric transfer i.e., A-B transfer > or < B-A transfer then use a betweensubjects design Range effects People tend to perform best in middle of range of trials does between-subjects design solve this? Context effect when one level of IV is used subjects establish a context

43 Activity How would you carry out the experiment for comparing CAPS to non-caps, i.e. what would be your design?

44 Activity Design an experiment to compare a pop-up linear menu vs. a pie menu Subjects? Hypothesis? IV? DV? Design? Task (s)? Day Shift Evening Shift Night Shift Split Shift Day Evening Night Split

45 Activity

46 Activity Design an experiment to test whether adding color coding to a menu interface improves accuracy? Subjects? Hypothesis? IV? DV? Design? Task (s)?

47 Activity Only one form of solution, many others exist Subjects: Taken from user population Hypothesis: Color coding will make selection more accurate IV: Color coding DV: Accuracy measured as number of errors Design: between groups to ensure no transfer of learning (or within groups with appropriate safeguards if subjects are scarce) Task: the interfaces are identical in each of the conditions, except that, in the second color is added to indicate related menu items. Subjects are presented with a screen of menu choices (ordered randomly) and verbally told what they have to select. Selection must be done within a strict time limit when the screen clears. Failure to select the correct item is deemed an error. Each presentation places items in new positions. Subjects perform in one of two conditions.

48 Example The Effect of Shading in Extracting Structure from Space-Filling Visualizations July 14-16, 2004

49 Motivation Hierarchies are abundant and interacted with on a regular basis For adequate navigation, the structure has to be explicit Hierarchies are generally represented as trees Structure is explicit, but space-inefficient & navigation complexity increases with size

50 Space-Filling Visualization Developed to make more efficient use of display space i.e.: Treemap [Shneiderman, 1990] Characterized by compactness and effectiveness of showing node size However, the structure is no longer explicit Can shading facilitate the extraction of structure information?

51 CushionMap: Shaded Treemap CushionMap (SequoiaView ) uses shading to give a 2½-D impression, to make structure more explicit [van Wijk, 1999]

52 Structure-from-Shading (1) Evidence that our visual system extracts shading information early on Simple shading information processed preattentively [Enns & Rensink, 1990]

53 Structure-from-Shading (2) Shading and contour combine to strongly influence the shape of an object [Sun and Perona, 1996] We innately make assumptions about shading information [Ramachandran, 1988]

54 Structure-from-Shading (3) Shading useful in extracting structure information in node-link diagrams [Irani and Ware, 2001]

55 Structure-from-Shading (4) Some evidence that shading impairs size judgments 2D bar/pie charts better than 3D counterpart [Carswell et al, 1991] Similarly 2D line graphs lower accuracy than 3D counterpart [Zacks et al, 1998]

56 Hypotheses Participants Apparatus and task Experimental factors Study Design Study Methodology

57 Experiment - Hypotheses Hypothesis 1: shading (CM) will result in higher performance on structure related tasks than the noshading condition (TM) Hypothesis 2: shading (CM) will result in lower performance on tasks related to file and directory size comparisons than the no-shading condition (TM)

58 Participants 20 undergraduate students (paid) participated Random assignment to one of two condition CM or TM first All familiar with concept of file and directory management tasks/routines None had experience with SequoiaView

59 Experiment Method Half started on TreeMap (TM) the other half on CushionMap (CM) Used 2 different hierarchies H1 and H2 {CM-H1, TM-H2}, {CM-H2, TM-H1}, {TM-H1, CM-H2}, and {TM-H2, CM-H1}.

60 Experiment Tasks Tasks divided into two major categories: Structure-based Count the number of directories in the hierarchy Find the directory with the most number of files Count the number of subdirectories in a given directory Count the number of files in a given subdirectory Find the directory with the most number of bit map files (.bmp) Count the number of sub-directories that contain bitmap (.bmp) files Size-based Find the smallest directory in the hierarchy Find the largest file in the hierarchy Find the largest file in a given directory Find the largest mp3 file in the hierarchy

61 Experiment Measurements Measure: subjects performance on each task with respect to two variables: time until completion (0 to 45 seconds) successful/unsuccessful completion (0/1) Timeouts classified as failures Unsuccessful and timeouts not included in average completion time calculations

62 Experiment Results (2) Structure Size Average Completion Time (seconds) Average # of tasks successfully completed TM = 21.5 (6.1) CM = 16.2 (3.7) TM = 2.7 (1.5) CM = 4.9 (0.8) TM = 17.9 (5.4) CM = 20.2 (5.4) TM = 3.4 (0.7) CM = 3.1 (0.9) TM CM 4 3 TM CM St ruct ure Size 0 Str uctur e Size Completion Time # of Tasks Successfully Completed

63 Experiment Results (3) Structure Size Completion Time CM significantly faster that TM (p=0.0021) No significant difference between CM and TM Completion Success Subjects significantly more accurate on CM over TM (p<0.001) No significant difference between CM and TM

64 Experiment Subjective Evaluation Statement TM CM 1. I was able to count the number of directories using toolname I was able to find the bitmap (.bmp) files using toolname I was able to detect the type of files using toolname I was able to find subdirectories using toolname I was able to find the files inside a sub-directory using toolname I was able to find the largest file using toolname I was able to compare the sizes of files using toolname I was able to find the largest directory using toolname After the training session I knew how to use toolname I found toolname confusing to use = strongly agree, 1 = strongly disagree

65 Level of Support for Tasks Based on Size Low Medium High Very High Discussion? n9 n1 n5 n10 Sunburst? n7 n8 n0 n1 n2 n3 n4 n5 n6 n7 n8 n9 n10 Low Medium High Very High Level of Support for Tasks Based on Structure

66 Discussion Tested the effect of shading on non-explicit structures (CM vs. TM) Confirmed the first hypothesis Users were faster and more accurate in completing directory management tasks with the shaded hierarchies Did not obtain any conclusive results on the unfavorable effect of shading for size-based tasks Need to investigate the ability of users to extract structure from space-filling techniques

67 Choosing IVs and DVs Recap Range of IVs Determining reliability and validity Within-subjects & between-subjects design Single variable vs. multi-variable designs

68 Interpreting Experimental Results Plotting Frequency Distributions Statistics for Describing Distributions Plotting Relationships Between Variables Describing the Strength of a Relationship Interpreting Results from Factorial Experiments Inferential Statistics

69 Calculations that tell us Statistical analysis mathematical attributes about our data sets mean, amount of variance,... how data sets relate to each other whether we are sampling from the same or different distributions the probability that our claims are correct statistical significance

70 Questions one might ask Is there a difference? Is one system better than another? Techniques addressing this are called hypothesis testing The answers are not simply yes/no, but of the form: we are 99% certain that selection on 5 item menus is faster than 7 item menus How big is the difference? i.e. selection from 5 items is 270 ms faster than from 7 items Called point estimation, often obtained by averages How accurate is the estimate? i.e. selection is faster by 270 +/- 30 ms Answers to this are in the form of standard deviations or confidence intervals we are 95% certain that the difference in response time is between 240 and 310 ms

71 First two rules: Look at the data Interpreting Results a graph, histogram or table of results could be more instructive Exposes outliers, which need to be removed to avoid biases Save the data May want to try different analyses on the data Trace back the analysis to the raw data collected Choice of statistical analysis depends on type of data and questions to be answered

72 Plotting Frequency Distributions Plot a frequency distribution telling us how frequently each score appears in the data Frequency is the number of raw data points that fall into each score category Useful first step in finding out whether there is a difference between conditions Example: two groups Want to determine whether video game player who plays racing games is more comfortable (less anxious) with fast drivers

73 Plotting Frequency Distributions Game Player Non-Player Game Player By looking at distributions we can notice that there are no differences Non-Player

74 Plotting Frequency Distributions Normal distribution, fits a complex mathematical formula. For our purposes, dist is normal if fits a bellshaped curve Important to know whether distribution is normal so that you can apply appropriate statistical tests Could also have bimodal, truncated or skewed distributions Although nice to see frequency distribution, nice to have a single number representing how subjects performed

75 Statistics for Describing Distributions Use typically two types of statistics: descriptive and inferential Descriptive statistic is simply a number that allows the experimenter to describe some characteristics Inferential will be discussed later

76 Statistics for Describing Distributions One important descriptor is the location of the middle of a distribution (central tendency) Mode, the most frequently occurring score Median, it s the middle score, equal number of scores above it and below it Mean, weighted average of the scores Which to use depends on the distribution, what purpose the average plays, and your judgment outliers vs. no outliers

77 Statistics for Describing Distributions Another important statistic is the measure of dispersion, or how spread out the scores are Range, difference between largest and smallest value Variance, calculated by computing deviation of each score from the mean, squaring these, adding them up, and dividing by number of scores Std deviation, simply the square root of the variance The smaller the std dev, indicates that mean is with fewer errors

78 Plotting Relationships Between Variables Reason for experiment is to determine if there is a relationship between IV and DV Find it useful to draw a graph to represent the experimental relationship Plot DV on y-axis and IV on x-axis What types of graphs to use: If IV levels cannot be represented by numbers use bar graphs If IV is continuous use histogram or line graph

79 Plotting Relationships Between Variables P NP Bar Graph showing mean comfort scores for players (P) and non-players (NP) Line graph showing mean comfort scores for players after several months of gaming

80 Strength of a Relationship The previous graphs were functions of a descriptive statistic rather than that of individual points Rarely will every data point fall on a smooth function If you use raw data will very likely find some variability or spread a scatter plot

81 Scatterplots

82 Correlation: Strength of a Relationship Measures the extent to which two concepts are related How? e.g. years of university training vs. computer ownership per capita obtain the two sets of measurements calculate correlation coefficient +1: positively correlated 0: no correlation (no relation) 1: negatively correlated

83 Salary per year (*10,000) Strength of a Relationship 10 condition 1 condition r 2 = Pickles eaten per month

84 Salary per year (*10,000) Correlation Pickles eaten per month Salary per year (*10,000) r 2 =.668 Which conclusion could be correct? - Eating pickles causes your salary to increase - Making more money causes you to eat more pickles - Pickle consumption predicts higher salaries because older people tend to like pickles better than younger people, and older people tend to make more money than younger people Pickles eaten per month

85 Dangers attributing causality Correlation a correlation does not imply cause and effect cause may be due to a third hidden variable related to both other variables drawing strong conclusion from small numbers unreliable with small groups be weary of accepting anything more than the direction of correlation unless you have at least 40 subjects

86 Correlation Cigarette Consumption Crude Male death rate for lung cancer in 1950 per capita consumption of cigarettes in 1930 in various countries. While strong correlation (.73), can you prove that cigarette smoking causes death from this data? Possible hidden variables: age poverty

87 Condition 2 Regression Calculates a line of best fit Use the value of one variable to predict the value of the other e.g., 60% of people with 3 years of university own a computer 10 y =.988x , r2 = condition 1 condition Condition 1

88 Interpreting Results from Factorial Experiments Example: time it takes subjects to read paragraphs typed in 12-point or 10-point print 8-year olds in one group, 12-year olds in another group Cannot simply ask whether the independent variable has had an effect on the dependent variable Must ask more specifically: Is there an effect of print size? (main effect) Is there an effect of age? (main effect) Does the effect of one variable depend on the level of the other? (interaction)

89 Interpreting Results from Factorial Experiments Main Effects To evaluate main effects of an IV must average across levels of the other variable To determine effect of print size we need to find a point halfway between the two levels of age at each level of print size We observe a change in print size (10-point to 12-point) causes a change in DV (time) yes, there is main effect of print size To determine effect of age we need to find a point halfway between the two levels of print size at each level of age We observe that a change in age (increase) causes a change in DV (time decreases) yes, there is a main effect of age

90 Time Reading Time Time Interpreting Results from Factorial Experiments 40 Main effect of print size? yes Print Size Print Size Age 8 years 12 years Main effect of age? yes Print Size

91 Interpreting Results from Factorial Experiments Interactions To determine whether the IVs interact we must ask: is the effect of print size different for each age? (or) is the effect of age different for each print size? 1 st question: we see that going from 10-point to 12-point causes a decrease in reading time for 8-year old but no diff for 12- year old 2 nd question: we see that the difference between reading times for the two ages is larger for 10-point than for 12-point

92 Time Time Interpreting Results from Factorial Experiments Interaction? Age 8 years 12 years Print Size Print Size yes

93 Time Time Activity Print size? Age? Interaction? Age 8 years 12 years Print Size No Yes No Print Size Print size? Age? Interaction? Yes Yes No

94 Inferential Statistics In many experiments testing one design against another i.e. the independent variable is usually discrete Can have discrete variables or continuous variables Discrete take on finite number of values (screen color) Continuous take on any value (person s height, time to complete task) Special case when continuous variable is positive (response time cannot be < 0)

95 Choosing a Statistical Technique Independent Variable Parametric Two-valued Discrete Continuous Non-parametric Dependent Variable Normal Normal Normal Student s t-test on difference of means ANOVA (ANalysis Of VAriance) Linear (non-linear) regression factor analysis Two-valued Discrete Continuous Continuous Continuous Continuous Wilcoxon (Mann-Whitney) rank-sum test Rank-sum versions of ANOVA Spearman s rank correlation