From Analytics to Action

Transcription

1 From Analytics to Action

2 Overview So, what are we talking about here? How to make incremental changes to your digital content that will result in positive user actions

3 Overview Rooted in the hard sciences and hypothesis testing

4 Overview By the end, you should understand: 1. What a content experiment is 2. Benefits of content experiments 3. Why they work 4. How to start conducting them

5 Overview Warning: Numbers

6 Overview Beyond the basics of web analytics

7 Overview Tools Abound!

8 Overview Questions!?

9 About Josh Who is this guy?

10 Why Focus on Experiments? They work!

11 Why Focus on Experiments? Nonprofits are missing out

12 Why Focus on Experiments? It s within reach for just about any org

13 The Brochure Problem

14 The Brochure Problem

15 Instead: Make small changes and learn from your users

16 Web Analytics What we talk about when we talk about analytics

17 Web Analytics Starts With Data

18 Data + Brains = Analysis

19 Aggregations

20 Visualizations

21 Induction, My Dear Watson 80% of new site visitors leave without viewing another page. Therefore, the content on my homepage is not very engaging.

22 Insight: Explaining User Behavior Start with an educated guess based on observations, then test it experimentally to draw conclusions

23 Insight: Explaining User Behavior Learn More

24 Insight: Explaining User Behavior Learn More

25 Insight: Explaining User Behavior Learn More

26 Insight: Explaining User Behavior Click for Inspiration

27 Insight: Explaining User Behavior The button was too small The color was not attention-grabbing The language was not engaging enough

28 Optimization Changing your digital content based on experimental evidence, in order to increase positive user actions

29 Digital Content Experiments Two main types: A/B and Multivariate

30 A/B Tests

31 Multivariate Tests

32 Improvement Over Time Static vs 2% monthly donations improvement Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Annual 4-Year $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $12,000 $48,000 $1,000 $1,020 $1,040 $1,061 $1,082 $1,104 $1,126 $1,149 $1,172 $1,195 $1,219 $1,243 $14,920 $59,682

33 Critical Concept: Optimization & Confidence A B Donate Now Donate Now 5% CTR 10% CTR

34 Critical Concept: Optimization & Confidence A B 40 Users Total Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks

35 Critical Concept: Optimization & Confidence A B 4000 Users Total Donate Now 5% CTR 2000 users 100 clicks Donate Now 10% CTR 2000 users 200 clicks

36 Critical Concept: Optimization & Confidence When you have few data points to compare, it is harder to know whether a change is due to chance

37 Critical Concept: Optimization & Confidence Statistical Confidence: How likely is it that the result we see is due to the change we made?

38 Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks BUT only 73% confidence

39 Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 2000 users 100 click Donate Now 10% CTR 2000 users 200 clicks

40 Critical Concept: Optimization & Confidence A normal distribution of z

41 Critical Concept: Optimization & Confidence Statistical Significance: The bar you set for how confident you must be before trusting results of a test Typically: 95% confidence

42 Critical Concept: Optimization & Confidence A General Rule: Changes that make small impacts need more data to detect. Changes that Changes that make a large impact will be apparent without much data much data needed at all.

43 Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks 7 clicks 97% confidence

44 Setting Up Good Experiments What is a good experiment?

45 Setting Up Good Experiments Submit Donation

46 Setting Up Good Experiments

47 Setting Up Good Experiments

48 Setting Up Good Experiments Align Tests with Organizational Goals What are you Key Performance Indicators?

49 Setting Up Good Experiments KPI

50 Setting Up Good Experiments: KPIs How K is your PI?

51 Setting Up Good Experiments: KPIs

52 Setting Up Good Experiments: User Path Understand the User s Path

53 Setting Up Good Experiments: User Path Get Newsletter me@ .com Click to confirm!

54 Setting Up Good Experiments: User Path Where Are Users Falling Out?

55 Setting Up Good Experiments: User Path

56 Setting Up Good Experiments Formulate a Hypothesis Why are people dropping out?

57 Setting Up Good Experiments A: The newsletter button CTA is not obvious enough

58 Setting Up Good Experiments B: The value of the newsletter is not communicated

59 Setting Up Good Experiments C: There are too many steps in the signup process

60 Setting Up Good Experiments D: People don t see the confirmation link

61 Setting Up Good Experiments Choose a single hypothesis to test Aim for a hypothesis that seems both easily testable and highly impactful if highly impactful if true

62 Setting Up Good Experiments People don t see the confirmation link

63 Setting Up Good Experiments There are too many steps in the signup process

64 Setting Up Good Experiments The value of the newsletter is not communicated

65 Setting Up Good Experiments The newsletter button CTA is not obvious enough

66 Setting Up Good Experiments Write It Down! We suspect that more people will sign up for the newsletter if the button on the button on the homepage stands out more

67 Setting Up Good Experiments Design your experiment 50% of users will see the existing newsletter button. The other 50% will see a red will see a red version of the button.

68 Setting Up Good Experiments Run Your Experiment!

69 Getting Started Tools at Your Disposal

70 Getting Started: Web Tools

71 Getting Started: Tools

72 Getting Started: What To Test Donate Now Donate Now

73 Getting Started: What To Test Donate Now Donate Now

74 Getting Started: What To Test Donate Now Donate, Or Else

75 Getting Started: What To Test Donate Now Donate Now

76 Getting Started: What To Test Donate Now Donate Now Donate Now

77 Getting Started: What To Test Donate Now Donate Now

78 Simply the Best

79 Google Optimize

80 Google Optimize

81 Google Optimize

82 Google Optimize

83 Google Optimize

84 Wrapping Up

85 Wrapping Up

86 Wrapping Up

87 Wrapping Up

88 Get In Touch!