From Analytics to Action

Overview So, what are we talking about here? How to make incremental changes to your digital content that will result in positive user actions

Overview Rooted in the hard sciences and hypothesis testing

Overview By the end, you should understand: 1. What a content experiment is 2. Benefits of content experiments 3. Why they work 4. How to start conducting them

Overview Warning: Numbers

Overview Beyond the basics of web analytics

Overview Tools Abound!

Overview Questions!?

About Josh Who is this guy?

Why Focus on Experiments? They work!

Why Focus on Experiments? Nonprofits are missing out

Why Focus on Experiments? It s within reach for just about any org

The Brochure Problem

Instead: Make small changes and learn from your users

Web Analytics What we talk about when we talk about analytics

Web Analytics Starts With Data

Data + Brains = Analysis

Aggregations

Visualizations

Induction, My Dear Watson 80% of new site visitors leave without viewing another page. Therefore, the content on my homepage is not very engaging.

Insight: Explaining User Behavior Start with an educated guess based on observations, then test it experimentally to draw conclusions

Insight: Explaining User Behavior Learn More

Insight: Explaining User Behavior Click for Inspiration

Insight: Explaining User Behavior The button was too small The color was not attention-grabbing The language was not engaging enough

Optimization Changing your digital content based on experimental evidence, in order to increase positive user actions

Digital Content Experiments Two main types: A/B and Multivariate

A/B Tests

Multivariate Tests

Improvement Over Time Static vs 2% monthly donations improvement Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Annual 4-Year $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $1,000 $12,000 $48,000 $1,000 $1,020 $1,040 $1,061 $1,082 $1,104 $1,126 $1,149 $1,172 $1,195 $1,219 $1,243 $14,920 $59,682

Critical Concept: Optimization & Confidence A B Donate Now Donate Now 5% CTR 10% CTR

Critical Concept: Optimization & Confidence A B 40 Users Total Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks

Critical Concept: Optimization & Confidence A B 4000 Users Total Donate Now 5% CTR 2000 users 100 clicks Donate Now 10% CTR 2000 users 200 clicks

Critical Concept: Optimization & Confidence When you have few data points to compare, it is harder to know whether a change is due to chance

Critical Concept: Optimization & Confidence Statistical Confidence: How likely is it that the result we see is due to the change we made?

Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks BUT only 73% confidence

Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 2000 users 100 click Donate Now 10% CTR 2000 users 200 clicks

Critical Concept: Optimization & Confidence A normal distribution of z

Critical Concept: Optimization & Confidence Statistical Significance: The bar you set for how confident you must be before trusting results of a test Typically: 95% confidence

Critical Concept: Optimization & Confidence A General Rule: Changes that make small impacts need more data to detect. Changes that Changes that make a large impact will be apparent without much data much data needed at all.

Critical Concept: Optimization & Confidence A B Donate Now 5% CTR 20 users 1 click Donate Now 10% CTR 20 users 2 clicks 7 clicks 97% confidence

Setting Up Good Experiments What is a good experiment?

Setting Up Good Experiments Submit Donation

Setting Up Good Experiments

Setting Up Good Experiments Align Tests with Organizational Goals What are you Key Performance Indicators?

Setting Up Good Experiments KPI

Setting Up Good Experiments: KPIs How K is your PI?

Setting Up Good Experiments: KPIs

Setting Up Good Experiments: User Path Understand the User s Path

Setting Up Good Experiments: User Path 1 2 3 Get Newsletter me@email.com Click to confirm!

Setting Up Good Experiments: User Path Where Are Users Falling Out?

Setting Up Good Experiments: User Path

Setting Up Good Experiments Formulate a Hypothesis Why are people dropping out?

Setting Up Good Experiments A: The newsletter button CTA is not obvious enough

Setting Up Good Experiments B: The value of the newsletter is not communicated

Setting Up Good Experiments C: There are too many steps in the signup process

Setting Up Good Experiments D: People don t see the confirmation email link

Setting Up Good Experiments Choose a single hypothesis to test Aim for a hypothesis that seems both easily testable and highly impactful if highly impactful if true

Setting Up Good Experiments People don t see the confirmation email link

Setting Up Good Experiments There are too many steps in the signup process

Setting Up Good Experiments The value of the newsletter is not communicated

Setting Up Good Experiments The newsletter button CTA is not obvious enough

Setting Up Good Experiments Write It Down! We suspect that more people will sign up for the newsletter if the button on the button on the homepage stands out more

Setting Up Good Experiments Design your experiment 50% of users will see the existing newsletter button. The other 50% will see a red will see a red version of the button.

Setting Up Good Experiments Run Your Experiment!

Getting Started Tools at Your Disposal

Getting Started: Web Tools

Getting Started: Email Tools

Getting Started: What To Test Donate Now Donate Now

Getting Started: What To Test Donate Now Donate, Or Else

Getting Started: What To Test Donate Now Donate Now

Getting Started: What To Test Donate Now Donate Now Donate Now

Getting Started: What To Test Donate Now Donate Now

Simply the Best

Google Optimize

Wrapping Up

Get In Touch! josh@getdigits.io